“Real life is not like that”
The beginning and the end of the ascent were busier than usual, but not when compared to some of the simulations that the flight crew and flight control team are put through pre-launch.
When asked if STS-93 seemed like a busy ascent or a quiet simulation run, Shannon said: “When we would do integrated simulations with the crew, they’re just filled with failures. They try to exercise each of the positions — you practice your communication, your work with your backrooms, how you work with each team member to come up with good solutions and go work problems and things. that’s expected and the team gets very good at doing those kind of sims — you may have upwards of twenty failures in a single ascent.”
“But it’s different,” he noted compared to a real launch. “Despite all the efforts to make it as realistic as possible with comm loops and the crew and the simulators and all that, the simulation supervisor and the simulation team did not give you a set of problems that you could not work your way out of. They didn’t take the vehicle away from you unless you made some kind of a mistake and you kind of knew that, that there’s a way out of this.”
“Real life is not like that and whenever you have an issue with the orbiter, there is no ‘sim sup’ there to make sure you have a case that you can work your way out of, and you’re very aware of that, at least I was for a launch.”
Credit: Philip Sloss for NSF.
(Photo Caption: The Flight Director console in the Shuttle Flight Control Room, as seen from the observation gallery above and behind the room during an ascent simulation for the last Shuttle mission in 2011. In this picture, Tony Ceccacci is the Weather Flight Director on the left and Richard Jones is the Ascent Flight Director on the right. For the STS-93 launch, Leroy Cain was Weather Flight and John Shannon was Ascent Flight Director.)
With the two issues that the team was working on, Shannon also could focus on those in contrast to ground simulations with the high volume of problems.
“[During simulations when] you’re having twenty failures going on, you can’t be listening to everyone’s backroom loop, you’ve got to focus on the air-to-ground loops and your flight director loop,” Shannon said. “But again it’s a real ascent where not a lot is happening, so I have all the air-to-grounds and the flight loops but I also have Booster, FDO (Flight Dynamics Officer), EGIL, EECOM (Electrical, Environmental and Consumables), DPS (Data Processing System), GNC (Guidance Navigation and Control), PROP (Propulsion Systems), all their backroom loops I have available to me and I could also listen to loops where they’re talking to each other and generally as a rule of thumb I would not ever go to it because it’s kind of a bad practice.”
“It was very common [on a real ascent] with nothing else going on or with a very severe problem to go and listen to their backroom loop to get a head start on what they were talking about or what their recommendation was going to be. Plus you could listen in to the logic that they’re using to make sure that you agreed with it and it would cut down on the time wasted or the chatter on the flight loop that you really didn’t need.”
“You’d never talk on it, you’d never say a word on it, it’s their loop to go talk,” Shannon emphasized. “It just gave you a step up when you finally had a resolution or plan of action and they could talk to you on the flight loop, so on flights it was pretty typical if there was something going on or I suspected something going on that I would dial up that backroom loop and just listen to it as they’re talking.”
“If things got busy with air-to-grounds or things got busy in some other areas I’d punch it off pretty quick.”
Credit: NASA.
(Photo Caption: A slide from the STS-106 Flight Readiness Review in 2000 showing the two cases where a Shuttle missed a “guided” MECO by fractions of a second.)
The last surprise of the ascent came at MECO, when the Main Propulsion System backroom engineer called out that it was a liquid oxygen (LOX) low-level cutoff, meaning Columbia had run out of fuel. “It was the second time in the program that we had flashed the low-level cutoff sensors and had a non-guided cutoff, had a cutoff on the propellant,” Shannon noted. “The other was STS-78 when we had a loading error.”
“So right then when Booster called me and said ‘MECO, Flight, low-level cut’, I realized ‘oh’. We’d had more going on with the engines than we could see and the FDO, who was Lisa Shore, she piped up real quick and said we were just fifteen feet per second short and no OMS-1 was required, that we could make up the shortfall with a little bit longer OMS-2 burn and we did that, it was not a big deal.”
During the ascent just after the “Negative Return” call to the crew that they’d passed the point where an RTLS abort was possible, Lisa Shore reported a thrust update to Shannon on the Flight Director loop and Shannon then called Reding to check on the engines.
“I was listening to his backroom loop, he was talking potential nozzle leak but not of the size that we eventually saw that it was,” he said. “We were a little worried about the tags. I was wrong, I was sitting there thinking there’s something going on in the center engine because I can’t see it and was not realizing that the problem was really with the right engine.”
As it turned out, a coincidence and a consequence of the other problem ended up compensating for the nozzle leak on the right engine. In another one of the series of posts he made on the STS-93 flight, Wayne Hale noted that one of the transducers that measured the pressure in the main combustion chamber (MCC) of the center engine was not perfectly calibrated; it was reading a slightly higher pressure than the other. The electrical short actually disconnected the more accurate transducer, which ended up slightly reducing propellant consumption.
“If the right engine would have just kept performing the way it was, we would have had a significant underspeed at MECO,” Shannon pointed out. “Luckily the center engine had a sensor bias in it that when the digital controller unit A went down, it used just that B sensor of main combustion chamber pressure and that biased us to use a little more hydrogen, a little less oxygen and so the additional oxygen we were using to keep the performance on the right engine was being made up by the center engine performance.”
“So they kind of canceled each other out and we only came up about fifteen feet per second short of a guided MECO.” The engines shut down early, but only 0.15 seconds early.
Post-insertion
There was still one more surprise a few minutes after MECO. “We had just gone through this unusual launch where we were really on our toes, watching for the next failure, being prepared to react and had a very short shortfall on the guided MECO and were relieved that OMS-1 was not required, we could make it up with the OMS-2 burn and had just told the team that and everybody started to kind of relax and the FIDO was building targets for that OMS-2 and all of sudden you heard this ‘pop’ and a shower of sparks came down — right on the GNC and PROP console,” Shannon remembered.
Credit: Philip Sloss for NSF.
(Photo Caption: A wide shot of the Shuttle Flight Control Room in 2011, showing the projectors for the three large displays in the front of the room. The left-most one chose to overheat and go out shortly after Columbia reached orbit on STS-93.)
“I don’t even actually know if it was a short or the bulb burst or whatever, [but] our projector on the left-hand side went out and the GC (Ground Control), Bill Foster, at the time shut it down and PROP and GNC jumped up and got away from their console and I don’t even remember if they went to their backroom to go monitor from that point on, but I just remember turning around. First I looked at Leroy and just thought ‘what in the world’ and turned around and looked at Wayne and thought ‘wow, this is quite a day.'”
“They got it all shut down and everything was fine and I think they came in the next shift and fixed that projector but it was just one of those days,” he added. “The team I felt like was very professional, really understood their systems and how to monitor them, and even through some adversity they were extremely professional and did a fantastic job, the crew did a great job as well.”
In-flight anomaly investigations
A few hours after launch, while Collins and her flight crew worked with Austin and his flight control team to deploy Chandra, long-range video and film tracking camera replays of the launch provided the first view of the right engine nozzle leak. “The flight director sent us an email with a picture of the leak,” Collins noted. “We saw it before we went to sleep on Flight Day 1.”
Credit: NASA.
(Photo Caption: A frame from the Playalinda Beach ET-207 video tracking camera shortly after liftoff of STS-93. The leaking nozzle cooling tubes on the right engine can be seen as a vertical white line in the middle bottom of the image.)
“I did not press the issue when I first got to orbit, as everything was normal to me, I was focused on the deployment of Chandra,” she added. “We discussed the leak after the deployment.”
The engines were only used during ascent and the short was isolated; the impact of the anomalies was to the big picture. “It was not a big impact to us, we had sufficient workarounds for that AC bus, we just left it isolated for the rest of the flight,” Shannon said. “It did kick off a very significant investigation into the wiring of all the other orbiters and Columbia as well.”
“After the mission, inspection revealed a single 14 ga. polyimide wire had arced to a burred screw head. The wire was located in the aft left-hand mid-body bay #11 lower wire tray,” the Shuttle Independent Assessment Team (SIAT) reported. The SIAT was formed in September 1999, to do an independent assessment of the Shuttle Program after the close-calls during the launch.
Credits: NASA.
(Photo Caption: A composite of three images related to the electrical short that occurred seconds after liftoff on STS-93. The leftmost image shows the general location of the wire tray in the orbiter mid-body. The center image is a close-up of the screw head with the burr and the electrical arcing damage after re-positioning the wiring. The labeled image on the right is from the Columbia Accident Investigation Board report, showing the damaged wiring and the screw from the STS-93 incident.)
“The issue we had with the short, which caused the AC 1 Phase A failure, that issue had been probably on the vehicle since it was initially built, it was a little burr on the top of a screw head,” Shannon recalled. “Somebody probably stepped on the wiring harness and rubbed off some of the Kapton insulation and you got a significant short to that screw head from the vibration of the launch.”
“We really went through and we changed our processes for how the technicians would get in and inspect the vehicle and do any kind of a reconfiguration of the vehicle and we went through and we opened up a lot of wire harnesses in heavily traveled areas to look for any kind of issues that you could potentially find,” he added. “We were duly worried about having shorts where you would lose a bus or something critical, but also a Kapton wire fire.”
“If you had a significant, sustained short you could catch the insulation on fire and end up losing some significant amount of wiring. So we were very nervous about that, it kicked off quite an investigation on the orbiters to go understand the state of the wiring in the other vehicles.”
Credit: NASA.
(Photo Caption: A chart from the SIAT report showing occurrences of LOX post pin ejections along with flight use cases in which pins were not ejected. As noted in the report, in cases where the post pins were ejected, it tended to occur on the first start after they were inserted.)
The SIAT noted that the LOX injector post pin that was ejected on STS-93 during engine startup and ruptured three nozzle coolant tubes was never “green run” or acceptance tested. “The practice was to insert the pin and perform a vacuum leak check,” they wrote in their report. “If there was no leak, an engine firing and subsequent successful vacuum leak check was required to ascertain that the pin would not be ejected.”
“It is significant to note that 19 of 20 pins were ejected on the first engine firing,” the report added. In spite of this, beginning with STS-38 in November 1990, nine LOX post pins were flown on five Shuttle launches without a “green run” engine firing before STS-93. “The SIAT considers that a serious lapse in judgment and/or inattention to the engine database occurred, which allowed two pins to be used in STS-93 without ground test verification firing. The second pin was not ejected.”
STS-93 was the last flight of Phase II Shuttle main engines which had flown since STS-6 in 1983. “During the recent engine block changes (I & IIA), Main Injector manufacturing processes were improved to preclude liquid oxygen post damage,” the SIAT report noted. “Currently, there are no pinned posts in the fleet. All future STS flights, starting with STS-103 are planned as either Block II or Block II-A Space Shuttle Main Engines.”