Shuttle Endeavour is back in space for what is scheduled to be her final mission. A nominal countdown, which included an issue-free tanking, resulted in an on time launch at 8:58am Eastern. The countdown also provided a full test of the mitigation effort on the Auxiliary Power Unit (APU-1) Heater system issue which caused the April 29 scrub, with no issues reported.
Endeavour continued to enjoy a trouble-free countdown on Monday, with the retraction of the Rotating Service Structure (RSS) completed by 12:24pm local time the previous day, revealing the youngest orbiter of the fleet for what was her final scheduled starring role on a launch pad.
The first major task for Endeavour’s launch day was the meeting of the Mission Management Team (MMT), which reviewed the upcoming weather for morning, prior to making the decision to proceed with tanking the Shuttle’s ET-122.
The only real item of interest was a loose tyvek cover on two of the orbiter’s Forward Reaction Control System (FRCS) trusters. However, with no rain observed on the pad, there are no concerns of rain intrusion into the exposed thruster.
With a go for tanking given by the mangers, engineers pressed on with the starting of the pumps to chilldown the transfer lines between the storage tanks and Endeavour’s Main Propulsion System (MPS), which is the path taken by the LOX and LH2. This was followed by the main stage of tanking, known as fast fill.
As with all recent tankings, several areas of interest will closely monitored, such as the low level and Engine Cut Off (ECO) sensors, which have performed as advertised since a successful resolution to what was a problem in part of the LH2 Feedthrough connector system.
STS-134 Specific Articles: http://www.nasaspaceflight.com/tag/sts-134/
As the three hour tanking process continued, sensors sniffed around the top of the tank for any indicators of a leak from the Ground Umbilical Carrier Plate (GUCP), which has been problematic during a number of countdowns, such as STS-119, STS-127 and the previous mission, STS-133.
With tanking complete, cameras around the pad – including infra-red systems – kept a close eye on the flange area between the Intertank and LO2 tank, ensuring no observations of cracks are found in the Thermal Protection System (TPS) foam, which would be an indication of a cracked stringer.
This was unlikely to occur, following the STS-133 event, with mitigation in place via the installation of radius blocks around the circumference of the flange, which worked well on Discovery’s ET-137.
Atlantis’ ET-138 tank also sports the stringer modifications, given its close relation to Discovery’s tank, although ET-122′s modifications are only installed as a contingency, as this tank is much older than most of the recent tanks to launch. ET-122 wasn’t fabricated during the period where suspect “mottled” materials were used on the production of the stringers.
Notes of interest during the count:
An Interim Problem Report (IPR) was charged to Endeavour as the tanking process began, per L2 updates, relating to a pressure delta – outside of 15psi. Noted as IPR-65, an OMS05 waiver is in work and was reviewed by the MMT at 6am Eastern, with approval.
While a minor issue with the Merrit Island Launch Annex (MILA) was cleared after a switch to a land line, IPR-66 was noted, relating to a failed autostart of ET-122’s bellow heaters, located on the LOX feedline. A manual start was initiated, resolving the problem, which is believed to have been caused by a console configuration error.
APU-1 B string heaters 116 and 117 are currently working as planned, along with heaters 114 and 115 which took a little longer to heat up, as designed. Tanking completed, into stable replenish.
Numerous key events follow after tanking, including the checks on the tank by the Final Inspection Team (FIT), who checked for ice build up on the tank, prior to the crew making their way to the pad to ingress Endeavour. The crew are currently on board Endeavour.
With the remainder of the count issue-free, a go was given to come out of the T-9 minute hold, prior to a successful launch into space. Flight Day 1 coverage is continuing on the live updates, with another article to follow later on Monday.
Refer to the live update coverage – linked above – for live coverage. Key notes will be added to this article.
APU-1 Heater System Mitigation:
Endeavour’s extended stay at Pad 39A was the result of “APU-1’s Fuel Test Line and Fuel Service Line ‘B’ heaters failing to activate during STS-134 pre-launch operations”. The failure was deemed to be persistent with both ground command and panel switch activation, which is an off-nominal condition for an orbiter, as much as the redundant ‘A’ heaters performed nominally.
Each APU system – which is used to pump hydraulic power to the aerosurfaces and engine gimbals – contains multiple heater circuits to maintain the system within nominal design operating temperature range. Temperature control is via thermostats located along the fuel line.
The heaters are organized into redundant strings (‘A’ and ‘B’) and can be activated independently. Activation is carried out via toggle switch panel control on-orbit and via panel control or ground command during pre-launch operations.
Electrical power distribution and control is via Aft Load Control Assemblies (ALCA) 1, 2 and 3, with Hybrid Drivers contained within the ALCAs, providing the logic and power switching components to control the electrical power to the heater loads.
As a breach of Launch Commit Criteria (LCC) rules – which requires both heater strings to work as advertised, the concern of launching in such a configuration relates to the loss of APU 1 heater redundancy, otherwise documented as the loss of 1 of 2 active thermal control mechanisms for APU 1 line heaters.
STS-134 APU Troubleshooting Articles: http://www.nasaspaceflight.com/tag/APU
The threat to the orbiter is based along the next failure scenario, where the lack of heater operation could lead to line rupture with potential for a fuel leakage leading to a combustion risk when the hydrazine reacts with the oxygen as the vehicle re-enters the atmosphere.
“Launch Day Observations: APU (1-3) heaters were initially configured to operate on the ‘B’ strings via launch data bus issued ground command. As aft compartment cooled temperatures on the APU 1 fuel test line and fuel service line continued to decrease beyond thermostat activation temperature and past LCC lower limit,” noted one of numerous program level presentations on the issue, available on L2.
“Heater (HR116B / 117B / 118B) anomaly was suspected. 14B/15B functionality was not confirmed on launch day. The APU 1 ‘A’ heaters were activated via ground command. Nominal heater performance was observed. The ‘A’ heaters were activated via panel switch (ground command deactivated). Nominal heater performance was observed.
“The ‘B’ heaters were activated (‘A’ deactivated) via panel switch and temperatures began to decrease. The ‘B’ heaters were confirmed non-operational via both panel and ground command. Launch was scrubbed and ‘A’ heaters used during drain.”
Engineers and technicians waited for ET-122 to be detanked and inerted prior to gaining access to Endeavour’s aft compartment to begin troubleshooting the cause of the heater system failure, focusing initially on the thermostats, prior to checking the electrical paths.
“Vehicle Level Troubleshooting: Panel A12 switch fuses were verified intact via control bus drops. Freeze mist to activate thermostats verified a failure in the HR116 / HR117 / HR118 and HR14 / HR15 heater circuit(s). Confirmed launch day observation with addition of 14B/15B,” added the presentation.
“Subsequent steps verified a good ground path through the LCA and exonerated the common Type IV hybrid driver. A Break-Through configuration determined that ALCA 2’s outputs associated with the heater circuits were failed – no output.”
With the problem observed in the ALCA-2, technicians carefully removed the box, prior to it being sent to the nearby NASA Shuttle Logistics Depot (NSLD) for a forensic examination of its multitude of wires and circuit cards.
“ALCA 2 Test and Analysis: The removed ALCA 2 was routed to the NSLD where an incoming functional was attempted. A failure in the special test equipment (STE) necessitated a manual test which verified the vehicle failure. The driver module A49 (2 hybrid drivers) was removed and the failed driver routed to the F/A Lab. The ALCA tested nominally with a spare driver module installed,” the presentation continued.
“The failed driver was delidded where the fusible element was observed to be open due to a high current event. Power was provided to the driver downstream of the fuse and it performed nominally. Two Type III downstream drivers were removed from the ALCA (Module A51) and routed for destructive physical analysis to determine whether high current propagated to / damaged the downstream components.”
While the ALCA-2 was replaced on Endeavour with a spare, the findings from the NSLD concluded that a short circuit occurred downstream of the failed driver and warranted additional on-vehicle testing in order to seek out a short circuit via Hi Pot tests.
Notably, vehicle heater connections were established and the heaters functionally tested via ground power supply with nominal results. Heater vendor leads were flexed with no negative effect observed. Technicians then removed the overtemp thermostats S12B and S118B, prior to their replacement, as they became suspects of a potential root cause.
This was a key moment for the investigation, as teams in the NSLD concluded their tests on the removed ALCA-2, noting it was not at fault for the heater system failure, rather a casualty of the overall problem.
“ALCA 2 Test to Screen for Internal Short: Since a short circuit internal to the ALCA (downstream of the failed driver) could have caused the failure signature, additional testing of the LRU (Line Replacement Unit) was warranted,” the presentation noted.
“The ALCA was repopulated and readied for vibration testing. Functional vibe testing was performed in 3 independent axes with outputs monitored. Test results were nominal relative to the APU heater circuitry. Two unrelated anomalies were noted: A small current exceedance on a Type IV driver in a Hydraulics circuit and a intermittent failed output on a spare Type III driver.
“Isolated to a crimp specific to the failed Type III driver. Test and analysis indicates that the failed ALCA driver performed nominally in the presence of an external high current demand. There is no indication of an ALCA-based root cause.”
Focus was then placed on one of the overtemp thermostats, namely the S12B overtemp thermostat, after an engineering review of tests conducted when Endeavour was in her Orbiter Processing Facility (OPF) back in June of last year showed a spike on this particular thermostat when it was being heated up to show it would perform its role nominally in the event of a problem on orbit.
“Previous APU 1 Heater Test Data Review: Data was collected and analyzed for the previous testing of the APU 1 heater circuits during the STS-134 processing flow. Bus Monitor Unit (BMU) high speed current data identified an unexpectedly high current (18 Amps minimum) during testing of the circuit on June 22, 2010,” noted the presentation.
“The current spike was specifically time correlated with testing of the S12B thermostat using a metallic heat gun in close proximity. The test was opening S12B with heat; the macroscopic effect of the device opening was observed, however the high current event occurred simultaneously. The Main Bus B current decreased by 2 Amps in the wake of the high current event.
“The ALCA 2 circuit passed 18 Amps (minimum) during the event. The thermostat had already been removed at the time of the data discovery and was then expedited to the NSLD F/A Lab.”
Closer examinations of the thermostat showed an exposed conductor, which – while apparently not affecting the thermostats function during testing – became the main focus of the root cause.
“S12B Overtemp Thermostat: Following the review of high speed current data from the June 22nd test, S12B was expedited to the lab for analysis. Visual assessment identified exposed conductor on the vendor lead adjacent to the thermostat’s head.
“Damage to conductor did not breach the nickel plating. Nickel’s low oxidization rate precludes dating the wire damage. Optical and electron microscopy did not identify indications of a short at the exposed conductor location. Indications could be obfuscated by the presence of the remaining insulation / insulation debris.
“The thermostat was tested and functioned nominally. S12B cannot be definitively implicated or exonerated as root cause.”
Indeed the presentation’s findings fails to fully conclude the root cause effort as specific to the thermostat, although the working theory is the spike – which then propagated through the heater system – was simply down to the contact between the metal head on the heating gun – used by engineers to test the overtemp thermostats – and the exposed conductor.
The reason there’s high confidence the issue won’t reoccur during the countdown relates to the replacement of all major elements on the heater system path, such as the wiring, the ALCA-2 and the thermostats.
“Summary: The APU heater circuit was non-functional in ALCA 2. The LRU was replaced with a spare. LRU and component analysis indicates that the driver was exposed to a short circuit and responded nominally,” noted the findings.
“The driver was replaced and the LRU tested nominally including during vibration testing. Vehicle wiring was nominal during Hi-Pot testing. New wire and splices were installed from ALCA 2 to the heater strings to mitigate an undetected transient short. ‘B’ heaters tested nominally during continuity and 450 Vdc isolation testing. Heaters were functionally tested with GSE power and via the replacement ALCA 2 and performed nominally.
“Data review from the June 2010 heater testing revealed a 2 mSec 18 Amp current spike during test of thermostat S12B. The thermostat was found to have exposed conductor on the vendor lead.”
“Flight Rationale: Most likely cause(s) of heater circuit short circuit have been removed / replaced. ALCA 2 was replaced with spare unit. New vehicle wiring from ALCA 2 to heater splices installed. Thermostats S12B, S16B and S118B were replaced. S12B thermostat implicated via June 2010 test data and identified as UA Most Probable Cause.
“‘B’ heater elements Hi-Pot tested at 450 Vdc. Redundant APU 1 ‘A’ string heaters are nominal providing full heater redundancy. ‘A’ and ‘B’ functionally tested at ambient with nominal results. Pre-launch testing will verify affected ‘B’ and ‘A’ heater strings with the aft compartment cryo chilled.”
So, despite the issue being classed as an “Unexplained Anomaly (UA)” confidence is high the system will perform without issue when tested not long after tanking has entered stable replenish. If, in the unlikely event, the issue is seen again, a scrub will be called, per LCC rules.
“Pre-Launch Planned Functional Verification: As an additional verification of the functionality of APU 1’s heater strings ‘A’ and ‘B’, pre-launch operation on both strings will be performed. Activation of each heater string, via ground command will be performed with the aft in a cryo chilled state. Each string will be verified to respond within the expected thermostat band to ensure functionality. On-orbit operation of the heaters will be discussed by MOD.
“Launch Commit Criteria: In the unexpected event of a recurrence of the original or comparable anomaly the OPO (Orbiter Project Office) recommends adherence to the applicable APU LCCs as written.”
(All images via L2. Extensive coverage is being provided on the news site, forum and L2 special sections – the latter of which is the world’s best front row seat to Shuttle launches and missions, breaking every major event over the last 10 missions before any other site, including NASA sites/TV).