January-April 2010: Safety and Understanding Triumph over Schedule

by Chris Gebhardt

If 2009 demonstrated an amazing execution of both unprecedented missions of science, cooperation, and exploration for NASA, 2010 undoubtedly demonstrated the agency’s unwavering commitment to investigation and understanding over launch and mission schedules. During the first four months of 2010 alone, NASA’s commitment to safety over schedule was demonstrated in large ways during the STS-130 (Endeavour) and STS-131 (Discovery) Space Shuttle mission launch campaigns.

Starting off the year: the rollout, launch campaign, and flight of STS-130/Endeavour:

After spending the holiday season of 2009 inside of the VAB safely mated to her External Tank/Solid Rocket Booster (ET/SRB) stack, the Space Shuttle orbiter Endeavour and the STS-130 vehicle began the slow crawl to Launch Complex 39A at 0413 EST on January 6 – a rollout that garnered attention not only as the first rollout of the year, but also as one of the coldest rollouts undertaken by the Space Shuttle Program.

So cold were the overnight/morning temperatures at the Kennedy Space Center, that the normal, pre-rollout ops at the Kennedy Space Center were spread out over a longer period of time than the started 4hrs from rollout Call to Stations to rollout first motion.

In fact, Call to Stations occurred nearly 11 hours prior to rollout, starting at 1700 EST January 5th. Much of this extra time was dedicated to the activation of special heating and purging equipment to ensure that certain Endeavour systems could be maintained at adequate temperatures for the duration of the rollout since the temperature at the start of rollout was 32 degrees F and dipped as low at 30 degrees F roughly three hours into the rollout.

Once at Pad-A, ground crews completed all pad integration activities and began a planned 32-day flow toward a targeted February 7 launch date. Less than one day into the pad flow came word that Endeavour’s primary payload, the Node-3/Tranquility module had a rather significant engineering issue relating to its exterior ammonia (NH3) cooling hoses.

As noted by ISS Flight Director Robert Dempsey, “While proof testing one of the new flight unit ammonia lines there was another rupture. A tiger team has been formed and dispatched to California to begin looking at options and schedule impacts.”

The lines, which were expected to be delivered to the Kennedy Space Center for launch at the end of January, were undergoing additional ground proof testing following two previous ruptures in December 2009.

NASA immediately began reviewing options which ranged from truncating the STS-130 mission from 3 EVAs (Spacewalks) to 1 EVA and still launching in the February timeframe without a resolution/modification to the ammonia lines, or delaying the mission until later in the year when a resolution would be available – a move that would have placed the following STS-131 mission on Discovery ahead of Endeavour/STS-130 in launch order.

“Due to the history, options could be a full redesign, a tweak, a new vendor – which would mean a slip to the nominal mission timeline. As you might expect, (ISS Program Manager Mike) Suffredini wants us to assess the impact to performing a modified mission where we just install the Node 3 and hook up the LTA (Launch To Activation) cables.

“We will also consider swapping the order of the missions.”

The issue itself arised primarily because of a last-minute change to Node-3’s berthing location on the ISS. This change in berthing location meant that the ammonia coolant lines would have to bend at a near 90-degree angle to properly interface with ISS systems.

Nonetheless, within a few days of the issue’s rise to prominence, NASA had demonstrated an amazing work effort to understand the jumper hose failure issue and design a potential workaround that would allow the nominally planned STS-130 mission to maintain its target launch date of Feb. 7.

According to another ISS Flight Director Robert Dempsey memo from early January, “There are two parallel paths being worked to get us NH3 lines in time for our launch date, or with a slight delay. We are by far not out of the woods yet, but this is promising that we can launch on or near 2/7 and have a full mission.”

These two options pertained to a redesign of the original ammonia lines via a double braid or the welding together of smaller, spare hoses identified at the Kennedy Space Center.

These two options were identified early in the process as being the preferable options since they preserved – at least initially – the ability to launch in the February window. However, NASA refused to allow these two paths to usurp other potential options – demonstrating a firm commitment to finding the best possible solution to the issue and simply letting the launch schedule fall where it would.

STS-130 Specific Articles: http://www.nasaspaceflight.com/tag/sts-130/

By January 18th, both hose options were being pursued, with the welding together of spare hoses at the Kennedy Space Center being the preferred option. Under the mid-January schedule, the newly welded hoses carried an estimated delivery date of February 2 – a delivery date that would allow for a February 7 launch attempt.

Over the next 10 days, work proceeded along those fronts and, as a result of the amazing and inspiring work across the agency from thousands of dedicated engineers and managers, NASA officially and unanimously cleared Endeavour for launch on February 7 at the January 27 SOMD FRR (Space Operations Mission Directorate Flight Readiness Review) – an amazing turnaround from an issue which initially threatened any chance of launching Endeavour on time.

Moreover, the Agency FRR for STS-130 also brought to fruition the year-long investigation into the Flow Control Valve (FCV) poppet cracks discovered after the November 2008 flight Endeavour/STS-126.

The year-long investigation yielded six recommendations to the Space Shuttle Program, the first of which was an official closure to the investigation with “ground flow induced damage as the most likely source of crack initiation.”

The second recommendation advised the continued use of flight rationale based on NDE (Non-Destructive Evaluation) inspections and hardware observations (eddy current, Materials Laboratory review, MPI, and SEM). The third recommendation was to maintain “evaluation of the vast acoustic emission data that has yet to be analyzed.”

The fourth recommendation, specifically targeted for STS-130, advised for the “emphasized attention” to Endeavour’s LV-57 position FCV. “This particular position is the only one of the 1301 family of poppets to exhibit considerable crack growth consecutively (STS-126 failure, and STS-127 zone 3 crack development to .22 inch flaw).”

The final two recommendations pertained to the ground processing side of the FCVs.

Recommendation #5 advocated the implementation of a redesign of the outlet tube housing for the GH2 and GN2 ground testing model in order to remove numerous acoustic frequencies to the noise level and demonstrate the effectiveness of Acoustic Environment sensors.

The final recommendation was to remove or reduce the full stroke shimmed flow balance test, and/or implement effective GH2 and GN2 ground testing controls to “improve repeatability and reduce runtimes.”

However, the final push toward the 24th and penultimate flight of Endeavour would not be as smooth sailing as some would have hoped, with a burst FIREX pipe causing Launch Control Center (LCC) water damage, the announcement of the cancellation of the Constellation Program by President Obama (and the creation of an even more uncertain plan for NASA’s future), and weather all playing key roles in the final days of the STS-130 launch campaign.

On Thursday, January 28th at 2025 EST “the LCC experienced a FIREX pipe burst that flooded multiple floors,” noted the January 29 NTD (NASA Test Director) morning update. The burst pipe resulted in the closure of the Firing Room to be used just days later for Endeavour’s launch.

Thankfully, the damage from the pipe burst was not serious and all wiring and console systems necessary for launch operations were not damaged thanks to thorough weather proofing already in place and a quick response from LCC personnel to cover all equipment to prevent any damage from water dripping onto the console from the floors above.

In the end, the LCC flooding did not impact STS-130 S0007 operations and Endeavour’s 3-day countdown began right on time.

As attention shifted more and more toward launch and the actual mission, Shuttle Flight Director Kwatsi Alibaruho rallied his teams with an Admiral Bill Adama (Battlestar Galactica) style address, stating that “I know many of us have watched with great interest as the President’s 2011 budget proposal was unveiled.

“The President’s plan and direction for the Agency calls for a dramatic restructuring of the Constellation Program, including indefinite suspension of plans for returning humans to the moon, and an expansion of support for development of commercial interests in space and Earth Sciences interests.

“There will be much dialogue between the President and the Congress before a final direction is codified.

“Such redirection can be incredibly challenging; however, it is especially difficult for many since we are at the beginning of what is to be the last year of the Space Shuttle Program. It is at these times that we are reminded that we are agents of the United States Government and servants of the Executive Branch. We are living in difficult times, and such times call for leadership.

“Leadership is not about making everyone happy, although it would be nice if one could. Real leadership is about setting a direction, developing a plan, leveraging resources, and making the difficult trades demanded by the difficult times.

“Please, do NOT be discouraged. This is not the end, by any means. All of this has happened before, and all of this will happen again. This is part of the natural process that is endemic to our special and enduring system of government. This is the nature of the process by which policy is made.

“But we are not policy makers. We are executors and stewards of a unique trust. History will barely remember the underlying fiscal and political factors that drove this policy and the subsequent direction for NASA, but what history will record in bold letters is how we perform.”

With everything in place, the stage was set for a spectacular night time launch of Endeavour. As the countdown proceeded, Endeavour behaved extraordinarily well on the pad (a testament to the hard work of her processing teams), allowing launch personnel more than enough time to analyze a few (very few) minor issues that cropped up during the count.

But in the end, it would come down to the weather. While official forecasts in the days leading up to the February 7 attempt called for an 80 percent chance of acceptable conditions, it would be the slight possibility of low clouds that would eventually force a scrub at the T-9mins and holding mark.

But this call proved slightly more dynamic than other recent weather scrubs, with a fascinating on-net discussion of the weather rules and whether or not the cloud cover hanging over the Kennedy Space Center was acceptable in the event of an RTLS (Return To Launch Site) abort or not.

As discussed in the post-launch press conference of February 8, Mike Moses explained that “Our Flight Rules cover what we need for RTLS in terms of ceiling and visibility. [The cloud ceiling] limit is 5,000ft for RTLS and the Range rules for visibility sit at about 4,000ft depending on thickness of clouds.

“Basically, we’ve always had this slight disconnect about what the Range would be go for and what RTLS landing would be go for. And they’re off by about a 1,000ft. Over the years, we’ve learned to constantly evaluate that if you have a condition that is acceptable for the Range but not RTLS, what does it take to be acceptable [across the board].

“Each of the pilots practice various visibility runs. They learn what they need to have to be comfortable: what would be good [and] what would be bad. Really, you’re looking for a [cloud] deck that allows you to gain the horizon as you’re rolling around the HAC (Heading Alignment Circle) so you understand clearly where you’re at.

“Last night we had that in the fact that there was nothing above that [cloud] deck. So clear skies up to that point (about 3,800ft).

“Then you look at the thickness of [the cloud deck] as you go through it. So you’ve got a good reference point, you know where you’re coming through, you punch through the ceiling really fast so you know you’re on your approach path. And once you get through it’s clear on down [the runway].

“The other big factor is what the wind’s doing from about 10,000ft on down and the work load that that then drives for the pilot. If all he has to do is pay attention to that glides slope as he comes in, that’s a little more acceptable. So we send the STA (Shuttle Training Aircraft) up there to fly that path and characterize that for us. So we send the STA up there with an astronaut to go fly that approach and go see what it really is compared to what our weather data is telling us.

“So what we were talking about yesterday was: could we get comfortable with the fact that, even though we were a little bit lower than our limits, we could still have good [conditions for RTLS]. And we were debating back and forth as to how comfortable we really were there. And we just didn’t get there on that yesterday.”

Despite this on-net discussion, weather was never “go” for launch during the T-9min hold and Launch Director Mike Leinbach made the correct call that even with potential weather waivers (of which none were being discussed) the weather would have been unacceptable from a flight safety standpoint.

Recycling for a 24hr turnaround, teams configured for a second launch attempt for Endeavour on the morning of February 8. This time, it would be the TAL (Trans-oceanic Abort Landing) sites in Europe that would pose an issue in terms of weather.

Thanks to real-time weather reconnaissance, however, one TAL site was declared “go” and at 04:14:08 EST, Endeavour lit up the partly-cloudy and cold night sky of Florida, thundering away from the Kennedy Space Center to begin STS-130 – the first of what would be three Space Shuttle missions in 2010.

In all, Endeavour reached orbit unscathed thanks to a superb job from the External Tank teams in preparing Endeavour’s tank for flight. TPS (Thermal Protection System) inspection data revealed NO (0 – ZERO) underbelly TPS scuffs or damage zones – a first and so far only occurrence in the Space Shuttle Program.

In fact, there were only 3 areas of TPS concern across the entire vehicle, a milestone that creates a highly visible accomplishment of all those across NASA who dedicated themselves to the reduction/elimination of damaging ET foam liberation from the tank during ascent.

The extremely clean nature of Endeavour’s TPS allowed all those involved in the execution of the mission to focus on the mission’s objectives – the delivery and installation of Node-3 and Cupola.

After successfully installing Node-3 and configuring Cupola, the STS-130 crew had the honor of opening up Cupola’s window covers for the first time and taking in the breath-taking 360 degree view of Earth from ~200miles above the planet’s surface.

After 13 days 18 hours 8 minutes and 3 seconds in space, Endeavour Commander George Zamka and Pilot Terry Virts fired Endeavour’s OMS (Orbital Maneuvering System) engines for 2minutes 38seconds, dropping Endeavour to suborbital speed.

Earth’s gravity pulled Endeavour into the atmosphere where friction between Endeavour and her contact with atmosphere created a stream of plasma around and behind the orbiter – an event imaged by crewmembers aboard the International Space Station from the newly installed Cupola.

Endeavour glided to a dark landing at the Kennedy Space Center at 22:22:10 EST on February 22 to cap off the first NASA flight of 2010.

STS-131: The Challenges of Launch Processing and the Rewards of Orbital Success:

With weather playing a significant role in the rollout, pad processing, launch, and landing of Space Shuttle Endeavour/STS-130, weather would again prove troublesome during early launch processing ops for STS-131/Discovery.

Initially targeting launch on March 18, NASA managers pushed back the mission’s targeted launch date due to Dual Docked Ops (DDO) constraints with the departing and arriving Russian Soyuz spacecraft for ISS crew rotation.

While Discovery would have initially been able to meet the March 18 launch date, the decision to delay launch due to DDO constraints proved fruitful when unacceptably cold weather at the Kennedy Space Center prevented workers from rolling Discovery from her OPF (Orbiter Processing Facility) to the VAB for 10 days.

Technicians were forced to delay Discovery’s rollover to the VAB from the OPF due to a requirement that temperatures in the VAB not drop below 45 degrees for more than 12 hours at the time of rollover and in the initial days of mating ops. This rule is in place to protect a Shuttle orbiter’s OMS and RCS (Reaction Control System) systems from potential cold weather damage since those systems cannot be hooked up to heating and purged air for rollover and mate.

STS-131 Specific Articles: http://www.nasaspaceflight.com/tag/sts-131/

The long delay to rollover ops would have eliminated March 18 as a launch opportunity for Discovery/STS-131, thus making the delay to April 5 from a DDO perspective a fortunate coincidence as it restored over 7 days of contingency time to the STS-131 pad flow.

After 8 days in the VAB, Discovery rolled out to launch pad 39A on March 2 at 23:58 EST – one day later than planned due to thunderstorms in the vicinity on March 1/2. Discovery arrived at Pad-A at 0649 EST March 3 and technicians began processing the vehicle for launch.

While processing initially went smoothly for Discovery, a visible RCS hydrazine leak occurred during the multi-day OMS/RCS fueling operations. As noted by the daily NTD reports, “Leak: RCS Helium Iso Valve on the right pod for the fuel system.” This meant that the Iso valve was stuck in the “open” configuration.

During RCS thruster operations, the correct flow of helium is essential for providing the proper flow path of oxidizer from the propellant tanks into the combustion chamber of the thrusters, allowing them to carry out their function of aiding the orbiter’s maneuvering on orbit.

While replacing the valve would have solved the issue, replacement could not have been performed at the launch pad. Instead, Discovery would have had to have been rolled back to the VAB, destacked, returned to the OPF, her right OMS pod removed, and the Right OMS Pod from Endeavour donated to Discovery for STS-131.

Since that would have induced a multi-month delay to the mission, NASA looked for other, less intrusive solutions for the issue.

The two less intrusive options developed were a “blowdown” procedure to blow helium through the system for a period of time and then cycle the helium valves to purge any contaminants that could have been causing the valve in question to remain in the open position.

The second option would be to simply accept the condition “as is” via flight rationale and fly the mission as planned thanks to built-in redundancy and on-orbit flight experience with a stuck open and stuck closed Helium Iso valve.

In the end, even though root cause of the issue could not be identified, flight rationale was accepted and the decision made to fly “as is.”

But this would not be the only technical issue to plague pad workers during the month of March. Shortly after Discovery’s arrival at Pad-A, Pulse Code Modulator Master Unit (PCMMU) 1 on Discovery experienced a “momentary failure” on March 5, kicking off a round of troubleshooting and engineering evaluations.

The PCMMU “routes the orbiter data to the OPS recorders and the communications systems for downlink to the MCC (Mission Control Center),” notes a Data Processing System (DPS) Familiarization Workbook (L2).

According to the STS-131 Space Shuttle Program (SSP) FRR summary (available on L2), “On March 5th, the four GPCs (General Purpose Computers) running in the redundant set all logged an I/O (input/output) error. Reviewed the data and saw that the PCMMU and several OI MDMs (Multiplexer Demultiplexers) had logged data validity errors.”

Following the March 5 failure, a second failure occurred the following day with a third failure occurring on March 9. “The GPCs responded nominally so the focus is now on the instrumentation buses,” notes the 131 SSP FRR summary.

In the end, no anomalies could be identified on PCMMU 1 and the engineering community recommended switching to and launching on PCMMU 2 without the replacement of PCMMU 1 – which operated nominally after the three previous issues on March 5, 6 and 9.

On March 26, the SOMD FRR unanimously approved launch of Discovery for April 5, and the three day countdown began in the early morning hours of April 2 and proceeded without major issue.

On the morning of April 5, 15-minutes before the targeted liftoff, the International Space Station flew almost directly over the Kennedy Space Center and Space Shuttle Discovery/STS-131, bisecting the moon as viewed from the KSC press site.

Fifteen minutes later, at 06:21:22 EDT (49 minutes before sunrise and with a light glow on the horizon), Space Shuttle Discovery lifted off from Pad-A, a light layer of fog creating the presence of a halo-like glow around Discovery as she headed toward space.

With sunlight barely breaking across Earth’s horizon as seen from Discovery’s ET camera, Discovery slipped into orbit after what was arguably the most visibly impressive launch of the Shuttle Program.

Following SRB separation, high altitude sunlight refracted from the water vapor contrail created by Discovery’s three main engines, creating a visible second stage contrail.

Furthermore, the SRB contrail, drifting into the upper atmosphere, formed a shape remarkably similar to that of a dragon. When this contrail caught the pre-sunrise light, it turned an impressive shade of orange and red – a natural tribute to Japanese astronaut Naoko Yamazaki who launched onboard Discovery.

Once in orbit, Discovery’s crew quickly realized that the orbiter’s Ku-Band antenna – the primary communication’s antenna on the vehicle – was experiencing an issue following several failed self-tests. These tests are conducted automatically by the Ku-Band antenna upon activation and deployment on orbit.

Commands to power cycle the two black boxes on the hardware failed to correct the problem. Thus, the Ku-band antenna was inoperative in both the forward and return link configurations, which prevented high bandwidth download and upload between the ground the Discovery.

Nonetheless, all TPS inspections were performed nominally with some information downlinked via the S-Band antenna during passage over the continental US.

The remaining data was downlinked following docking to the ISS when Discovery’s crew could make use of the Station’s Ku-Band antenna for the duration of the docked mission.

Like Endeavour and the previous mission, Discovery’s TPS was found to be in excellent condition with only a Rudder Speed Brake tile partial liberation gaining the bulk of the Debris Assessment Team’s attention.

However, the failure of Discovery’s Ku-Band antenna led NASA to extend the mission and move the customary TPS late-inspection into the docked portion of the flight – a procedure which had been accomplished before on STS-123 in March 2008.

This was done to ensure that all scan imagery could be transmitted to the ground and analyzed in a timely manner prior to reentry.

With the docked late-inspection in place, Discovery undocked from the ISS for a targeted April 19th landing. However, poor weather at the Kennedy Space Center forced a wave-off of the landing to the following day.

After 6.23 million miles, Discovery reentered Earth’s atmosphere over the North American continent on April 20, performing the second – and to date last – descending node reentry over the heartland of America in the post-Columbia era.

After 15days 2hours 47minutes and 11seconds in flight, Discovery came to a stop on Runway 33 at the Kennedy Space Center. With this mission duration, STS-131 became Discovery’s longest flight, beating out STS-120’s mission duration by a mere 23 minutes.

The STS-131 flight of Discovery also marked the final Space Shuttle mission with a seven member crew, the final Space Shuttle mission to carry a rookie astronaut to orbit, and the first time that four women worked together in space (three on Discovery and one on the Space Station), and the first time that two Japanese astronauts worked together in space (one on Discovery and one on the ISS).

The flight was the 18th post-Columbia flight, the 33rd flight of the Space Shuttle to the ISS, the 35th American manned night launch (of which 34 have been by the Space Shuttle and 22 of which have occurred from Pad-A), the 38th and penultimate voyage of Discovery, the 106th post-Challenger flight, and the 162nd American manned spaceflight.

(Part 2 of NASASpaceflight.com’s 3-part 2010 Year in Review series will be published December 31.)
(Images via L2, Larry Sullivan: MaxQ Entertainment, NASA.gov)

Related Articles