The STS-129 ascent/entry flight control team and flight crew concluded the fourth and final integrated ascent simulation at the Johnson Space Center (JSC) on Tuesday. NASASpaceflight.com’s Philip Sloss was given a unique insight by observing the sim runs conducted by Flight Director Bryan Lunney and his team.
STS-129 Sim Runs:
In preparation for the upcoming launch, the ascent/entry flight control team – led by Flight Director Lunney, and Mission Commander Charles Hobaugh – went through four ascent simulation runs, each of which presented different scenarios.
The flight control team was assembled in the Shuttle Flight Control Room, with their “backroom” support personnel working from the Multi-Purpose Support Room. Along with Commander Hobaugh, Pilot Barry Wilmore, Mission Specialists Randy Bresnik and Leland Melvin also took part in the simulations from the motion-based Shuttle Mission Simulator.
The scenarios were choreographed by a simulation team that monitors the actions taken by the flight control team and flight crew during the different sim “runs.” For this last simulation prior to flight, four ascent “runs” were conducted.
Run #1:
In order to fine-tune the reactions of the flight controllers and crew, several malfunctions are scripted into each run. The teams aren’t aware when such issues may arise, and anomalies can even occur simultaneously.
The high pace of ascent events became apparent while listening to the flight director loop. For example, just about the time that CapCom Chris Ferguson relayed the “two-engine TAL” call from the Flight Dynamics Officer to the crew, the Booster officer reported a small helium leak on the left main engine. The crew were subsequently asked to make a few switch throws on the flight deck, in order to close related valves in an attempt to try to isolate the leak.
“The main engines consume helium, it goes into a seal and keeps the hydrogen and oxygen separated by just pushing helium through there and then it just exhausts out to the ambient area,” explained Lunney in an interview with NASASpaceflight.com after the simulation.
“So it’s using helium from its tanks and it has a certain flow rate; the Boosters [Booster officers] are staring at their displays and in this case, the flow rate was a little bit higher than it should have been. I think it was about twice (as much, if I remember. So to him that’s considered a “small” helium leak.
“Since it was so ‘small’ – and I use quotes, because in my opinion, anything I can see is not small – it was small enough to not be a concern to the overall mission. The big helium leaks mean I’m not going to be able to run the engine until MECO, for eight and a half minutes – it’s going to shutdown sooner.
“The procedures onboard would be difficult for the crew to do by themselves, so Booster just walked them through what few isolation steps we can do to see if the leak was in a particular piece of the plumbing. So we did that manually, ‘OK, close this valve, no joy, close that valve, no joy, open back up’. And the in-open, out-open stuff, that just refers to a particular valve that allows that engine to take helium from, or provide helium to.”
There were several other simulated malfunctions during, and shortly after, powered flight, including a simulated fuel leak on APU number three, a slow helium leak on the left OMS engine, and a partial freon loop blockage. Despite all of the malfunctions, the teams were still able to fly the orbiter to a nominal MECO and get set up for an OMS-2 burn. The run was ended about 25 minutes after simulated liftoff.
Run #2:
Shortly after liftoff on the second run, the gaseous hydrogen Flow Control Valves (FCVs) for the center and right main engines failed closed. The Booster Officer asked to call up to the crew to take the LH2 ullage pressure switch to the open position. Less than a minute after liftoff, the Booster officer reported “no joy” on the switch throw, requesting the crew take the main engine limit switch to “inhibit / hard enable.” The crew was also told they would need to begin to manually throttle the engines later in the ascent.
“Each engine provides GH2 back into the hydrogen tank on the ET to keep it pressurized as we’re consuming propellant out of it,” Mr Lunney explained. “So we want to regulate how we flow the gas back into the tank, so the pressure stays where it should, and the FCV is what does that for us.
“We have three Flow Control Valves – one on each engine – and they respond opened/closed to what the ullage pressure in the tank is telling them. If it gets too low it says open, if it gets too high it says closed – and when I say ‘open’ and ‘closed,’ it’s not fully open or fully closed, it’s about 30 percent open when it’s ‘closed’ and then we take it to open I think it’s 60 or 70 percent open, I can’t remember the exact number. Also, the valve is shimmed, so it’s partially open/partially closed when we say ‘open’ and ‘closed.’
“In this case, two of them failed closed – and we have analysis that shows us that with one full open, we can throttle back and maintain the inlet conditions at the main engines at sufficient head pressure. If you think of the turbopumps that are spinning, you have pressure sitting on top of them; if you don’t have pressure sitting on top of them, those pumps will overspeed, come apart, which is bad, so we don’t do that,” he added, dryly.
“So you keep pressure sitting on top of them, and we have to throttle back a little bit at a time. We have analysis that shows that by playing this game of slowing it down, slowing it down, slowing it down, we can keep engines running long enough and keep enough pressure on the pumps long enough to get to a good MECO.”
During this event, at approximately three minutes, 45 seconds mission elapsed time (MET) after the simulated liftoff, the crew throttled the engines back to 95 percent. Then around 6 minutes, 45 seconds, they throttled the engines back to 80 percent. When the propellant remaining in the External Tank was down to 2 percent, the crew throttled down the minimum power level and performed a manual MECO.
On the engine limit switch throw, Lunney noted: “We have redlines on the engines that are running to automatic shutdown of an engine that is violating a redline. If things are going bad, you want the computer to shutdown the engine.
“There’s also a timer going and the timer is saying ‘hey at some point you get too far and we should be shutting down the engines’, so we want to disable that timer because we know we’re going to go long, because we’re having to throttle back, less thrust, it takes longer to get there. So we do this ‘inhibit, hard enable’ which resets that timer for me, that’s all that’s doing.”
In the meantime, still during first stage, the Propulsion Officer reported one of the other malfunctions – a large oxidizer leak on the left-side Reaction Control System (RCS). A few minutes later, he reported a slower leak of the right side oxidizer. Because of the leaks on both sides, the crew was asked to abort TAL a little more than six minutes after the simulated liftoff.
Lunney explained later in our interview: “The RCS is essential for attitude control during re-entry and from EI (Entry Interface) down to about a Q-bar [dynamic pressure] of 20 [pounds per square foot]. After about a Q-bar of 20, my aerosurfaces can do enough to get by to control the vehicle without the RCS. However, for that window when the aerosurfaces aren’t doing any good because there’s not enough air I need my RCS for attitude control.
“The left, in this case, was a big leak – so it was mostly going to be gone. The right was a little bit slower, but if I go and rendezvous with the station, well I’m going to leak out before I get there, so I won’t be able to fly to the station.
“If I wait for a rev 3 (next PLS), all that propellant may be leaked out, and I may not be able to make it around, so we go ahead and abort TAL – assuming the leak rate tells us for at least one of the systems I can get down to a Q-bar (of) 20, which is what PROP did for us. (The controller) checked his rates, recommended TAL, because he knew that even though both were leaking, the right side would support down to a Q-bar of about 20.”
In addition to these simulated problems, there were a few others issues during powered flight, including problems with one of the Inertial Measurement Units and the Flash Evaporator System. Main engine cutoff occurred at around 8 minutes and forty seconds MET, and the teams managed to get the orbiter configured for a simulated landing at Zaragoza. The run was ended at about fourteen minutes MET.
Run #3:
Another series of simulated malfunctions occurred one on top of the other around one minute after liftoff on the third ascent run. Watching on screens at the Public Affairs Officer console, we saw the data values “freeze” temporarily. This appeared to be an issue with one of the Pulse Code Modulation Master Units.
“PCMMU is the little box onboard that formats the data for downlink, so when it breaks, we don’t get data,” Lunney said. “So we asked them to swap to the other one so we could get data.”
A few things were happening in the same timeframe; the Booster Officer asked for AC bus sensors off around the same time as the data loss and subsequent call for the PCMMU swap, which recovered the telemetry data. Shortly after that, Booster reported that the B-side controller for the center engine was lost.
“So in that case we’re critical to AC1, which powers electronics A on that engine,” Lunney said. “…he said “AC bus sensors off” – we take the action – and then he explains to us why we did that, because we lost this controller box.”
After SRB separation, the Propulsion Officer reported a slow left OMS gaseous nitrogen leak and began troubleshooting. A couple of minutes later, he reported it was an accumulator leak, just a few moments before the Booster Officer called again to report a high mixture fuel flow meter shift on the right main engine.
“The accumulator leak is on the OMS engine, and it’s independent of what’s going on in the main engines, and in that case, you got the GN2 tank and you got a little bitty accumulator, and those pressurize the ball valves, to open them up and flow propellant and make the engine work,” Lunney added. “So PROP was troubleshooting that leak.
“In the other case, it’s a fuel flow meter shift, it’s one of many things that we can simulate that can go wrong with the main engines. The main engines are closed loop systems, they can modify the propellant flow into the engine, both the ox and the fuel, so that they get the right conditions.
“For an OMS engine, an RCS thruster, you open the valves, you flow propellant, and what happens is going to happen – those are open loop systems. The main engines are closed loop systems, the engine monitors itself, and if there’s a blockage – if there’s something going on where it needs a little extra flow of ox or a little extra flow of fuel, or a little less of each of those – it can tweak that mixture ratio in real-time.
“What happened in this case is, the fuel flow meter is what is used to monitor how much fuel is flowing in. The instrumentation became biased, and you know that because there’s two of them, and it takes an average of the two. So (for the high mixture case) it’s going to flow in too much ox [liquid oxygen], not enough fuel [liquid hydrogen], and then we’re going to consume our ox that much quicker.
“Also, I think in that case we had a LOX low-level cutoff, also, because that fuel flow meter was lying to us, we pulled more LOX than we should have into the engine.” [There was a 30 foot-per-second underspeed at MECO.] “It was within operating conditions that the engine could operate in – none of the other redlines, temperatures, pressures were violated – but it was flowing more LOX than we wanted out of the tanks.
“Also, I think in that case, later we lost the controller associated with the bad fuel flow meter and we went back to a good engine, it rebalanced back to a good flow condition.”
The Flight Dynamics Officer was modeling the performance for the fuel flow meter shift when the loss of AC1 system one caused the remaining good controller on the center main engine to quit, which caused the engine to shut down a little after six minutes MET. Shortly after calls about the engine shutdown, the Electrical Generation and Illumination (EGIL) Officer reported a three-phase motor stop on AC1.
“When the AC system has a short – which is what that case was – there’s three phases, A, B, and C, and they’re all out of phase by 120 degrees,” Mr Lunney explained. “We had a phase-to-phase short between the phases and we didn’t know which one was causing the problem. But when that happens, the voltage drops on the AC system, and when the [main engine] controller sees the voltage drop, it dies, the controller just flat cuts off.
“It’s pretty sensitive to AC changes, so we lost the controller and that’s what happened there with the main engines. Going back to the AC, dealing with the problem there, we have the capability to drop one phase, and then EGIL can look and see if the short is still present. In this case, we went through all three phases and the short was still present in all cases. So this was the case where we had to just drop the entire AC bus.”
A little later, before main engine cutoff, the EGIL Officer noted steps in the Ascent Pocket Checklist that the crew were required to carry out within a few minutes after MECO.
“The fuel cells have coolant flowing through them, driven by a pump,” Lunney added. “With the AC system down, the pump stops, so fuel cell 1 is dependent on AC1, fuel cell 2 is dependent on AC2, fuel cell 3 on AC3. So with the pump stopped, it’s basically not getting any cooling, and when it’s not getting cooling, it’s going to heat up and eventually overheat.
“After about nine minutes, our analysis says it’s going to overheat, so we want to shut it down within nine minutes, and those were the numbers she (EGIL Officer) was quoting. That was probably three, four minutes prior to MECO, I can let it run through MECO, and I can get it post-MECO.”
The simulated MECO occurred at about nine minutes, twenty seconds after liftoff, at which time the low-level cutoff was reported by the Booster Officer. While the post-MECO work was going on, the Electrical, Environmental and Consumables Engineer (EECOM) reported a cryo oxygen leak at about fifteen minutes MET and asked to take the O2 Tank 2 manifold valve to close to try to isolate the leak. However, that didn’t work – as Lunney explains:
“Well, remember the sim people are very mean, they don’t like us – no, I’m kidding – they want to make it hard on us. We had an O2 tank leak and you want to close the valve to it. They failed that valve, which we’ve actually seen from time to time on these cryogenic valves. It’s a momentary switch, it’s a switch that you push open and when you let go it flops back to the middle state, if you push close and vice versa.
“So, what we’ve learned is that if you continuously energize – hold in the open position or hold in the close position – then you can sometimes get lucky and the valve will defrost or start working – however you want to look at that. They put a retention device, which is just a little widget we created, to shove in there and hold the position.”
Due to an impending loss of cryo, the team began evaluating whether they would need to do an Abort Once Around (AOA) or whether it was a “next PLS” (Primary Landing Site) situation that would buy some additional evaluation time. At around twenty-two minutes MET, fuel cell 3 was shutdown to try to isolate the cryo oxygen leak, which was successful. By the time that the sim run was ended about five minutes later, the flight control team had worked out a way to bring back fuel cell 1 and possibly not just get out of the AOA, but get out of the next PLS case.
“The way that one could have played out depends on where the failures are and how things work, if we’re able to basically use jumper cables, literally like a car, jumping a car,” Mr Lunney explained. “Fuel cell 1 was the one we were trying to get back, essentially the AC system is what we’re trying do. AC transfer cables is what we’re using [to get power from AC system 3 to AC system 1] – so I’m able to keep a couple of fuel cells running and if I can look at that and see it’s stable and healthy, I can get out of the PLS case.
“We want to get to OMS-2 safely, get on-orbit, and then we can get time to talk about it. If you can re-route the power and get that second fuel cell running, then you have two viable, independent fuel cells. So at that point, you’re good to go for probably a nominal end-of-mission. I think, in that case though, I’d have to check flight rules.
“We’d have probably one or two meetings – or ten – to discuss it, and make sure we all understood what we had, why it happened, and that we were comfortable continuing. We might still cut the mission short in that case, it just depends on what all the data showed in that particular case.”
Run #4:
In comparison to the first three runs, the fourth and final run was relatively free of simulated malfunctions.
“That was a nominal run, which is typical for our last ascent or last run,” Lunney noted. “We try and have it fairly nominal, so we can kind of look at the timeline, do the nominal calls as a matter of practice, so we all remember what the nominal stuff is supposed to look like – because that’s how we expect to operate on the real day.”