After another launch scrub for weather, Space Shuttle mission STS-93 lifted off from the Kennedy Space Center (KSC) in Florida twenty years ago in 1999 to begin an eventful ascent into low Earth orbit. Shuttle Columbia, with NASA’s first female spaceflight commander Eileen Collins leading a crew of five, successfully reached orbit and went on to deploy the Chandra X-Ray Observatory hours later, but two incidents occurred one on top of the other around liftoff to make for one of the busier launches in Shuttle history.
An electrical short reduced critical elements of the ascending vehicle to zero-fault tolerance before the Shuttle cleared the launch tower, and a fuel leak had started even before liftoff that caused Columbia to run out of fuel early. Although the incidents weren’t directly related, a lucky coincidence of one of the failures meant that instead of running out of fuel significantly early the Shuttle’s main engines shut down only a fraction of a second before they needed to.
Weather scrubs second attempt
Following a last-second cutoff of the countdown on the first launch attempt after midnight local time on July 20, 1999, a 48-hour scrub turnaround was required in order to reservice the Radial Outward Firing Initiators (ROFIs) that burn off free hydrogen gas under the Shuttle main engines during their start sequence. Liftoff was rescheduled for the opening of an approximately 45-minute long launch window at 12:28 am Eastern time on July 22, 1999.
Following another smooth countdown, the launch was scrubbed again due to a complex of thunderstorms that developed and stayed close to the launch area. “I’m going to give you a little more confusing background here, but because we had decided we were just going to do a Day 1 deploy as the plan, that gave us a shorter launch window as I recall it,” STS-93 Lead Flight Director Bryan Austin explained.
“Had we planned nominally to go to a Day 2 deploy and not deploy on Day 1 that would have actually given us a longer launch window but the IUS (Inertial Upper Stage) doesn’t like that because that changes their whole scheme there. And then AXAF (the Advanced X-Ray Astronomical Facility that subsequently was renamed Chandra) also had some thermal challenges, but waiting to Day 2 allowed us to have a more casual checkout and work through some problems we had.”
“We actually debated until the very end of the [period] getting ready to launch, should we plan for a Day 2 deploy and finally we called it off, I mean knocked off that discussion a couple of weeks before launch,” Austin added. “So now here we are on the second launch attempt, the weather is getting in our way and we’re at the end of the launch window and the payload program manager for AXAF comes up and we started discussing it for a minute.”
“Ultimately and thank goodness the window closed. I think we were still no-go because of weather at the time but that became a distraction in there kind of out of the blue.”
Wayne Hale, who was one of the Shuttle Ascent/Entry Flight Directors in 1999 but was filling the role of Mission Operations Director for the STS-93 ascent and entry, went into more detail on the second launch attempt in a post a few years ago. Following the scrub of the second attempt, the launch was rescheduled for just under 24 hours later at 12:24 am Eastern time on July 23, 1999, from Launch Pad 39B.
A disguise works
The next night, there was a seven-minute extension of the hold at T-20 minutes to fix an issue at the local Merritt Island Launch Annex (MILA) tracking station, but otherwise the countdown once again smoothly ticked down to a liftoff now scheduled for 12:31 am Eastern time.
“The count was very smooth,” John Shannon said in a recent interview. Twenty years ago he was the STS-93 Ascent/Entry Flight Director in the Shuttle Flight Control Room at the Mission Control Center in Houston, Texas. “Loved this crew, a great crew — we’d had really good integrated sims.”
“I had probably the most experienced team I had ever had as an ascent flight director. We had Jon Reding as the Booster, Tim North was the EGIL (Electrical General Instrumentation and Lighting), who did all the electrical systems. The count was great.”
Now getting into Columbia for a third time in four days, Flight Engineer and Mission Specialist 2 Steve Hawley was perhaps reminded by either the unusual manual cutoff that stopped the first launch attempt or the unusual overnight weather that scrubbed the previous night’s attempt that he’d better wear a disguise. Hawley had a lot of experience ingressing and egressing Discovery and Columbia before STS-93; across five Shuttle launch campaigns, he’d seen a wide variety of last-minute or last-second issues with Space Shuttle Main Engine and launch pad valves, orbiter and Solid Rocket Booster hydraulic power units, and bad weather.
(Photo Caption: A couple of disguises Steve Hawley used to try trick Columbia, which apparently worked. The disguise on left for the last of several STS-61C launch attempt in January 1986, included blocking his name tag. The one on the right was for the third and final launch attempt for STS-93 in July 1999.)
Although somehow one of the missions launched on the first attempt, Hawley entered the White Room anonymously for the third STS-93 launch attempt. “And we have Mission Specialist 2 Steve Hawley with his traditional bag on the third launch attempt hoping that he’s not the one causing the delays, he’s hoping the Shuttle won’t recognize him,” NASA Public Affairs Officer and Launch Commentator Lisa Malone said during the live NASA TV broadcast at the time.
“It was a fun bunch, you know Steve Hawley’s personality from his bags and his commentary from some other scrubs that he’s had,” Austin said. “There’s a spectrum of personalities you can get from that type of people, pilots and astronauts, and this was just a great mix of people, I really enjoyed them.”
Redundancy, insight lost “right off the pad”
Thinking back a couple of days to the issue that stopped the first launch attempt, Shannon noted: “We were all watching the haz gas system as we counted down late and I remember Leroy Cain was sitting next to me, he was watching the weather for me, it was a pretty good night to go launch and we lifted off and the first thing that came to us was Eileen calling down the fuel cell Ph, which meant that we got an alarm.”
“[That] was not unusual, sometimes the way that system worked is it would determine that there was too much of the base, the potassium hydroxide, going into flow and it would give you a spurious Ph [message] right when you lifted off because it would mix up that system a little bit and so she called ‘fuel cell Ph.'”
Following liftoff, the Shuttle Commander would make a voice call to Mission Control as the vehicle cleared the tower and began its roll and pitch program. For this launch, her nominal call to acknowledge the roll program was underway included the fuel cell Ph message.
In an email, Collins noted when asked what she saw and heard as the vehicle lifted off, “There was no Master alarm. We had a ‘tone’ with the pH message. The water pump light flashed but only for a second.”
“I felt for a second as if I was in a simulator,” she added. “An interesting note is that this failure was in our last simulation run, less than a week before launch.”
(Photo Caption: Left, Columbia lifts off to start STS-93 early on July 23, 1999. Right, the Flight Director, CapCom, and MOD consoles in the Shuttle Flight Control Room shortly after Columbia reached orbit.)
“I looked up at the board that had all of the alarms and messages on it and we had a water pump, too, which hadn’t been there right before liftoff along with the fuel cell Ph so I knew something was going on,” Shannon remembered. “I listened to the EGIL’s backroom loop right then and it was clear that we’d had a real short on one of our AC (alternating current/electrical) buses, I think it was AC 1 Phase Alpha and that had slowed down the water pump enough to give us the message and also gave us the fuel cell Ph message as well.”
For the Booster position on the Flight Control Team, Reding was at the “front room” console in the Flight Control Room behind Shannon, with the Main Engines and Main Propulsion System “back room” positions working from the Multi-Purpose Support Room nearby. Immediately after Collins called the pH message from Columbia, the Main Engines position reported the loss of two engine controllers (also known as digital controller units or DCUs) — one on the center engine and one on the right.
“Now the impact [of the Ph message], and this was right off the pad — we were just getting clear of the tower when this was going on,” Shannon emphasized. “The bigger thing was that AC bus provided power to the digital controller units.”
“There’s two of them on the main engines and they’re fully redundant and [with the electrical short] we lost the digital control unit A on the center engine and (DCU) B on the right engine. The B [controller] on the right engine wasn’t a big deal because you’re normally running off the A side but losing the [A-side] digital controller unit on the center engine was a big deal because it took almost all of the data away from us.”
In response to the loss of functionality of those controllers when their electricity was interrupted, a request for a crew action was relayed from the Main Engines backroom position to Booster, to the Flight Director loop. “They did something called taking AC bus sensors to off,” Shannon explained. “You may have recalled ‘Scooter’, Scott Altman was my CapCom (Capsule Communicator position).”
“He called to have [Pilot] Jeff Ashby take AC bus sensors off, that just took off automatic monitoring of the AC bus sensors so we wouldn’t get a false failure of another AC bus system which would take one or two engines out at that point.”
(Photo Caption: A diagram from the Shuttle Crew Operations Manual showing how the Main Engines were wired for redundancy. The three AC buses were connected to the six main engine controllers for redundancy and so that one failure wouldn’t shut down an engine. The transient electrical short on the AC1 bus highlighted in red here disconnected DCU A on the center engine and DCU B on the right. That still left a working controller for all three engines, with the AC2 and AC3 buses wired to controllers highlighted in yellow and blue, respectively. A second AC bus issue would cause one engine to shut down, but the vehicle was certified to make an abort landing for a single-engine failure any time during ascent.)
The DCUs for each of the three Space Shuttle Main Engines (SSME) were fully redundant to begin the launch, but the electrical short now meant that two of the engines had lost a significant amount of their fault tolerance. The flight control team was trained to stay aware of the consequences of “the next failure”; for the engines, a next failure could shut down one of them.
“The last thing we wanted on this flight was any engine issues because this was a really challenging flight from a trajectory standpoint,” Shannon recalled. “Chandra with the IUS was a very heavy flight and we’d had a lot of discussions with Eileen and Jeff Ashby and Steve Hawley over the abort modes and whether we had full coverage for two-engine out or three-engine out abort modes and we had a discussion about black zones which are areas where you don’t have an abort capability for two or three engines out.”
“So getting right off the pad with all of that fresh in my mind and having a redundancy loss of computers on two engines was pretty nerve-racking. What you’re aware of is that you had this issue and now you’ve lost a significant leg of redundancy in two engines.”
“That gets you because you’ve lost some insurance, your margin has just decreased to not very much and what you’re relying on is the design of the vehicle and the testing of the vehicle and the workmanship of the vehicle to make sure that last leg of redundancy hangs in there,” he noted. “It was not lost on me that we lost that first set of redundancy about five seconds into the mission, and it was a very long eight and a half minutes until we got to Main Engine Cut Off (MECO) before I was really breathing again.”
A single engine shutdown during the first few minutes of powered flight would have required Collins and crew to do a Return To Launch Site (RTLS) abort, which was the trickiest of the intact abort modes the Shuttle was certified to fly. As Shannon noted, an abort would have made it much more likely that Columbia would have to return and land with Chandra and the IUS still in the payload bay.
Columbia was flying due-east away from Pad 39B, instead of up the East Coast of the United States, which meant there were no “East Coast Abort Landing” (ECAL) contingency options available. The heavyweight launch also extended the duration of Columbia’s RTLS exposure; typically, the Shuttle would gain enough speed to be able to do a Transoceanic Abort Landing (TAL) abort if necessary within the first two and a half to three minutes of liftoff but for STS-93, the “two-engine TAL” capability wasn’t available for almost three and a half minutes.
“The tags are off” on the right engine
Communications back and forth from CapCom Scott Altman to Collins and Ashby were much busier than usual in the first minute-plus of flight as Mission Control gave the crew additional information for their situational awareness going forward in the ascent. A few seconds later, as the thrust from the Solid Rocket Boosters began to rapidly tail off before burn out and separation, the Main Engines backroom controller called Reding to note that operating temperatures on the right SSME (engine #3) were high.
(Photo Caption: An image in the Shuttle Independent Assessment Team (SIAT) report showing the impact damage from the LOX injector post pin on the main combustion chamber of the right engine. The pin struck the closer MCC first and eventually hit the inside of the engine’s nozzle, puncturing three of the 1080 cooling tubes there shortly after ignition.)
Although the temperatures weren’t high enough to cause the engine to shutdown, they were significantly higher than normal. What wasn’t seen clearly until video and film launch replays hours later was that the nozzle of the right engine had sprung a leak while still on the launch pad.
“The LOX injector post in the main combustion chamber, there’s quite a few of them and sometimes they fail,” Shannon explained. “Instead of trying to take them out or go dissect the engine to replace it, they would just hammer a gold pin into the outlet port and one of those pins shot out right at startup and hit the side of the main combustion chamber and put a pretty good ding in it and then went down and ripped through three of the hydrogen cooling tubes that make up the nozzle — three of one-thousand eighty of them and it gave us a fairly significant leak on the hot wall side of the hydrogen.”
“We got really lucky there because the analysis kind of showed that if you lost five or six of those tubes you’d get a localized hot spot on that nozzle and potentially burn through it and then you could end up losing that whole engine. So we were blissfully unaware of all that going on, but we could see the performance on the right engine was a little bit off, we were burning a little more oxygen than we thought we should based on what called the tags or how the engine had run in its preflight testing and so we knew something was going on on it.”