Root cause found on Atlantis’ STS-125 ASA issue – SSMEs removed

by Chris Bergin

Engineers have found the root cause of the Aerosurface Servoamplifer (ASA) channel 1 failure on Atlantis, after an expansive failure analysis effort found a short in the wiring. Atlantis is being processed for STS-129, following the highly successful STS-125 mission to Hubble. As part of her turnaround, the Space Shuttle Main Engines (SSMEs) that helped power the vehicle into orbit have now been removed.

STS-129 Processing Latest:

Atlantis – housed on OPF-1 (Orbiter Processing Facility) – has a slightly longer flow than normal, as she is reconfigured for International Space Station (ISS) missions.

The removal of her Hubble payload equipment has all but been completed, along with the removal of her three SSMEs, and the replacement of two slightly damaged windows (1 and 7).

“Windows1 R&R is complete less 24 hr re-torque. Window #7 R&R complete,” noted the latest processing information (L2). “Fuel Cell 2 voltage reduction test complete. OMS Pod Functional is in work. S-Band antenna troubleshooting in work.

“Payload download is in work. FRCS (Forward Reaction Control System) functional is complete. Wheel and tire installation continues to be in work.

“Weekend Work: Remove Bay 11 & 12 longerons. Waterproofing Sunday (Bay Clear).”

Atlantis has at least two more missions ahead of her prior to the retirement of the fleet – pending a final decision on shuttle extension – along with the assignment as the STS-135 LON (Launch On Need) support role of the final mission on the current manifest.

Currently targeting launch No Earlier Than November 12, the STS-129 mission will see Atlantis deliver the Express Logistics Carriers 1 & 2 (ELC-1 and ELC -2), along with SASA and MISSE (Materials on International Space Station Experiment) 7A and 7B, to the Space Station.

ASA Root Cause:

a61Work started on the ASA channel 1 failure – a system required for carrying commands from the flight computers to the aerosurfaces such as the elevons and tail – shortly after Atlantis landed in California at the conclusion to STS-125.

The issue was first registered during SSME ignition, which – it has since been revealed via Flight Readiness Review (FRR) documentation – had the potential to cause a RSLS (Redundant Set Launch Sequencer) hold with just seconds remaining on the countdown.

“During SSME ignition an electrical short occurred on DC buses MN-A and MN-B which resulted in the loss of ASA-1 and caused several MDM (Multiplexer Demultiplexer) FA1 and OA1 measurements to become erratic momentarily,” noted a 260 page Agency FRR presentation from the Orbiter Project, available to download on L2.

“If this failure were to occur prior to T-10s, the result would be a violation of GNC Launch Commit Criteria. If this failure were to occur between T-31s and T-0s, the potential for an RSLS hold or abort exists (however, “Instrumentation shift only – not true cap voltages,” allowed Atlantis to launch and continue to orbit without concern).

“At ~T-3s FC currents indicated a current-limited short of ~12.5A on FC1 (MN A) and ~7.5A on FC2 (MN B). At ~T-0s the short circuit was isolated from the main busses (total short circuit time of ~3s). At ~T+1s the event indicator for APCA4 RPC 10 (10A) transitions from ON to OFF followed by the event indicator for APCA5 RPC 12 (5A) at ~T+2s.

“FC (Fuel Cell) current short circuit signatures were consistent with RPC trip characteristic signatures (150 percent for 2-3s). APCA4 RPC 10 and APCA5 RPC 12 provide redundant power to Aerosurface Servoamplifer-1 (ASA-1). Follow on flight control signatures support loss of ASA-1.”

a28The short itself was caused by the noise of the SSMEs powering up injecting into Atlantis’ aft, and causing some disruption of Atlantis’ avionics.

“Review of Bus Monitoring Unit (high rate) data showed that during the time of the short circuit a distorted, 400Hz sinusoidal wave was injected onto the MDM FA1 and OA1 signal return buses,” added documentation.

“This resulted in measurements which use the MDM signal return bus as a reference to become erratic (single ended). Differential measurements were not affected. Once the short circuit was isolated all measurements on MDMs FA1 and OA1 returned to expected values.

“Review of historical data uncovered a very similar event during the STS-37 (104 Flight Eight) ground flow (1991). Short circuit was determined to be caused by an exposed conductor contacting vehicle structure on the 26Vac excitation circuit between ASA-1 and a left wing elevon actuator

“Wire was pinched in the non-insulated portion of a wire harness clamp that shorted while personnel were in the area. MDM interference was determined to be due to the grounding scheme of the ASA box.”

In-flight short circuits associated with orbiter wiring are rare, with the previous incident occurring during Columbia’s highly eventful STS-93 launch

a73*Click image for a clippedd video sample (full video on L2) showing flight controllers reacting to the STS-93 event.*

“There have been seven previous in-flight short circuits which were later isolated to Orbiter wiring damage.

“The last in-flight short circuit isolated to Orbiter wire was the AC-1 phase A short during ascent of STS-93 (1999). This resulted in a stand down period during which all vehicles were inspected for previously undetected wiring discrepancies.”

Following the STS-93 short, corrective actions were implemented that both reduced the potential for wire damage and heightened detection capability should it occur.

“Inspection Processes Enhanced Following STS-93. Wire inspections are now performed by two independent quality inspectors anytime wire bundles or connectors are opened for any type of work (category 1),” outlined documentation.

“Harnesses are flared and a detailed inspection of accessible wire performed with 10X magnification, lighting and mirrors as required. A baseline wire inspection was scheduled to be performed on each vehicle at its next OMM (category 2).

“A hands-on outer-periphery inspection of ALL accessible wire bundles with criteria and methods similar to category 1 intended to capture and repair latent wire damage (i.e. baseline the vehicle).

“Closeout inspection criteria was updated to require two independent inspectors and inspections are now performed prior to installation of hardware that would cover wire. Arms-length inspections of wire bundles are performed as areas are backed out and closed for flight.”

During flight, the failure of the single channel ASA – part of a quad channel system – holds no mission impact. However, due to flight rules, a second failure – depending on the cause – forces a discussion on whether to bring the orbiter home at the next landing opportunity.

Expansive plans were drawn up in the event of a second channel failure, resulting in over 20 presentations to the Mission Management Team (MMT) during STS-125’s mission (all available on L2).

a35The need to find the root cause of the ASA failure was deemed a priority for engineering teams, although the task was daunting to say the least, due to the amount of wiring that required inspection.

“(The ASA wire network) represents approximately 300 feet of bundled cables or approximately 800 feet of cable end to end,” noted documentation, with initial resistance checks during Atlantis’ stay at the Dryden Flight Research Facility (DFRC) finding no obvious fault candidate.

Working through wiring from the nose of Atlantis, all the way through to connections with the Solid Rocket Boosters (SRBs) and External Tank, sections were cleared one at a time during the investigation, prior to a breakthrough last week.

“ASA 1 Update: Troubleshooting found a Hi-Pot failure in Right Hand inboard Elevon,” noted processing information, relating to a test that applies a higher than normal voltage to the signal wires, with the aim of applying a greater stress to the insulating coverings in order to uncover any weakness.

The voltage is usually applied with respect to system ‘ground’ reference and any excessive current indicates a breakdown of the insulation (due to cracks, damage, etc.)

Root cause for the ASA failure is thought to be a short from one of the power signal wires to ground, and further testing on the associated harness were undertaken to reveal the specific fault.

a43Updating status, the Orbiter Project Office (OPO) noted observations of worn-through insulation specific to the area in question, adding confidence they have found the cause of the short associated with the ASA-1 failure.

“On the ASA, did find the short and the wire harness was removed. Guys did a good failure analysis on it, and at NSLD (NASA Shuttle Logistics Depot) they could see the worn-through over-braid and the worn-through insulation of the wire below the over-braid,” OPO noted on the latest Shuttle Standup/Integration report (L2).

“They could see the damage from the HPOTP (High Pressure Oxygen Turbopumps), but were able to find a small pit that appears to be the original short location. Have been working with Ground Ops to do the inspections on the other vehicles in similar locations and making sure we have everything protected.”

This associated harness will be replaced with a new version that is being built in time for installation into Atlantis next month. Endeavour was already cleared due to added in-built braiding protection on her wiring.

“At the NSLD, we’ve started the planning and fabrication of a new ASA wire harness to replace the one removed from OV-104 (Atlantis),” added processing notes. “Looking at July 10 as a delivery date.”

L2 members: Documentation – from which the above article has quoted snippets – is available in full in the related L2 sections, now over 4000 gbs in size.

Related Articles