Engineers have removed the external LH2 Feed-through connector from ET-125, beginning the next stage of the major troubleshooting that has caused the delay to STS-122’s launch.
Moving forward with repair Option A, engineers hope that subsequent testing will confirm the ECO (Engine Cut Off) system anomaly is specific to the external elements of the connector, allowing work to be completed at the pad, thus avoiding rollback.
**The most comprehensive collection of Shuttle, Ares, Orion and ISS related presentations and mission documentation, plus expansive daily processing documentation and updates are available to download on L2 **
L2: STS-122 to STS-127 documentation already available.
STS-122 L2 Special NOW LIVE. **Click here for sample of L2 menu and content**
**STS-122 Special: 12 MOD Flight Readiness Review (FRR) Presentations, Baselines and Mission Overviews. 17 Shuttle FRR Presentations. Payload Presentations and vast amounts of live, uploaded images and presentations. Installation Movies (several) – already 1900 megabytes strong**
**COMPLETE Section on ECO/Feedthrough Connector issues and troubleshooting, graphical and data based – expansive.**
**LIVE updates on Atlantis STS-122 Troubleshooting Latest**
ET-125 Troubleshooting Latest:
Thursday’s PRCB (Program Requirements Control Board) meeting approved the go-forward plan of ‘Option A’ – which involves the removal of the external connector elements for testing, to be replaced by a ‘cherry picked’ replacement.
A tanking test may follow, with timelines showing the completion of work is targeted for the second half of January.
The stack will then have to be placed back into a launch posture, with estimates of a re-aligned launch date sometime in late January/early February.
Although four shuttle departments recommended ‘Option B’ – which calls for rollback and the replacing of ET-125 with ET-126, Shuttle manager Wayne Hale decided to continue with Option A, in the hope that testing will confirm the problem relates only to the external connector elements.
All options remain on the table, driven by the subsequent data that will be gained via the repair process, ranging from the successful implementation of Option A, to rollback and full feed-through (external and internal) hardware R&R, to potentially re-visiting the viability of flying without the full use of the ECO system during the ride uphill – if all else fails.
Characterization of the problem:
Originally deemed to be a problem with the ECO (Engine Cut Off) sensors, the reoccurrence of the anomalies during the two scrubs of STS-122 came as a surprise, following the mitigation effort that involved ‘cherry picking’ the best possible sensors for tanks such as ET-125.
Hand picking the ‘best’ sensors was called after the root cause was found via the manufacturing process, when slightly off spec tooling (factory machine) was found at the vendor, related to the swages in the sensors.
No issues with the ECO sensors followed with the subsequent launch countdowns, until STS-122, when a proverbal spanner was thrown into the works after the circuit anomalies – characterized as ‘open or lack of continuity during tanking sensor integrity evaluations’ – returned.
Even before the tanking test – called to pinpoint exactly what was causing the anomalies – engineers pointed the finger at two possible areas, with ET engineers originally claiming it had to be related to Atlantis’ Point Sensor Box (PSB) – in wiring downstream of the tank, while Orbiter engineers refuted that claim, blaming the ET as the source of the problem.
However, the general engineering consensus was already focusing on the LH2 Feed-through connector, due to all sensor circuits – LH2 ECOs 1-4 and 5 percent – sharing commonality to that element of hardware.
‘Open circuit conditions occurred at times consistent with rapid cryogenic conditioning of connector hardware,’ noted documentation. ‘Failure signatures consistent with previous experience since RTF (Return To Flight).
‘Opens occur during fast-fill after LH2 tank liquid level is above feed through location. All circuits return to normal state during / following tanking drain back.
‘Onboard TDR (Time-Domain Reflectometry) instrumentation during third tanking identified open circuit condition at feed through location. Open circuit considered external to feed through connector with 60 percent confidence.’
It is still unclear if NASA will actually find the root cause of the issue with the connector, given the time such problems occur. Recreating the exact environment that would allow for testing of future connectors can only be achieved at the point when the connector and ECO system is subjected to the exact same cryo conditions of tanking.
The closest engineers can get is at the test lab at the Marshall Space Flight Center (MSFC), which was used to test ET-120’s connector – though no issues were found via initial test results. This leaves engineers with only a hypothesis as to what is happening to cause the anomalies.
‘Root cause of ET-125 anomalies undetermined but most likely attributed to separation of plug socket / feed-through pin contacts,’ added documentation. ‘Contact separation caused by thermally-induced effects (CTE (Coefficient of Thermal Expansion), delta pressure, contamination, relative motion between parts) result in open circuit at external plug / feed through connection.
‘Bench testing used to demonstrate sustained open circuit due to pin/socket relative motion and ice contamination. Open circuit requires break in pin – recovery of circuit performance not possible.’
Interestingly, as of the latest PRCB (Program Requirements Control Board) meeting on Thursday, most of the connector elements being evaluated as potentially responsible for the root cause ranged from the classification of ‘improbable’ to ‘remote’.
However, the least likely element – according to documentation – is the internal elements of the connector hardware.
‘Internal Plug / Feed Through Receptacle Pin Contacts – Improbable. Minimal relative motion between socket and pin contacts. Sockets fully locked to retaining ring. Thermally-induced delta pressure loading within connector cavity not possible (GHe purge / LH2). Teflon grommet material thermally stable at cryogenic temperatures (no cracking)
‘Less chance of contamination as compared to external plug. Contamination due to solid air / moisture not possible (GHe purge). Other contamination possible but unlikely to remain following 3 LH2 loadings. ET-120 internal plug flown with no issue.’
That raises hopes that the STS-122 stack won’t be required to rollback to the Vehicle Assembly Building (VAB) – which would only be called if work is required on the internal connector, due to the need for engineers to enter the tank, something that cannot be done at the pad.
As to why only the ECO team came to the same conclusion (no rollback), contrary to JSC Engineering, KSC Engineering, NESC and SE&I’s recommendation (rollback) is a mystery, though sources note that the aforementioned bodies are historically ‘conservative’ in their evaluations – backed up by their stance during previous major issues, such as the tile ‘gouge’ on Endeavour during STS-118, and the RCC panels ahead of STS-120.
The ECO team were also backed up by related documentation from the PRCB meeting, which updated the root cause status for the external connector as the ‘probable’ cause.
‘External Plug / Feed Through Receptacle Pin Contacts – Probable.
‘Significant relative motion possible between socket and pin contacts CTE-induced effects increase contact pressure between sockets and pins. Sockets allowed to move towards receptacles,’ noted documentation, with additional information that the ET-120 connector test – originally documented as seeing no issues with the connector – has since been updated to show evidence of a potential culprit.
‘Review of ET-120 plug showed evidence of grommet material / socket retention degradation. Thermally-induced delta pressure loading possible at cryogenic temperatures (vacuum). Greater chance for contamination as compared to internal plug.
‘Contamination due to solid air / moisture probable (cryopumping through harness). Other contamination possible. Bench testing demonstrated open circuit condition with ice contamination and relative motion between pin / sockets.’
‘Review of available data supports External Plug Contacts as most likely cause of STS-122 Anomaly.’
The forward plan is staged, with the immediate process involving the removal of the external connector from the tank. While the connector will be shipped to MSFC for testing, an array of methods will be implemented to help aid data collection.
‘Current plan requires partial removal of TPS to provide unobstructed access for in-situ inspection of feed through connector hardware and surrounding TPS (Thermal Protection System) foam. Detailed dissection of TPS within J-box (surrounding connector) planned to identify any potential contributors (i.e. excessive cracking or intrusion into connector body) – not expected based on results from ET-120.
‘Visual and NDE (x-ray) inspections of connector hardware (internal / external) with moisture proofing and TPS material in place. Inspections provide baseline of hardware configuration prior to removal. Continuous circuit monitoring during hardware manipulation. Additional TDR can be performed following internal plug disconnect for comparison with tanking test data.’
Work at the pad will involve a more focus effort on inspecting the remaining internal connector elements, while a replacement for the external plug is ‘cherry picked’ for installation. The replacement hardware has already been identified, though the real test will begin once ET-125 is filled with cryos – either during a new tanking test, or the STS-122 countdown.
‘Inspection and testing of ET-125 internal plug (in-situ) and feed through replacement being used to mitigate potential causes associated with internal plug and feed through (contamination, bent pins, damaged sockets, etc.)’
However, there are some doubts surrounding the status of the internal connector, which could be part of the supporting rationale for rollback, as started by the four departments at the PRCB. The concern relates to the ‘backshell adaptor’ – which is classed as a ‘medium’ risk of causing the open circuit and highlighted by engineers.
‘Partial disassembly available to provide additional risk mitigation for faulty backshell adapter,’ added another presentation. Should that element of hardware ultimately be responsible, rollback would have to be called for replacement.
**Ride home through the fire, sparks and plasma of re-entry with Atlantis, Discovery and Endeavour. FIVE Stunning high quality 2hr, 355-400mb Camcorder and HUD videos – from payload bay closure – through re-entry with an astronaut held camcorder video – to post landing – several more videos showing landing from 90,000 ft also available and HUD videos from STA landings. Now includes HALO II Re-entry video, and re-entry videos from Gemini and Apollo (converted from 8mm)**
For the meantime, the PRCB’s forward plan is clear, only subject to change if data gained over the coming week points to an additional problem with the internal connector. The plan involves soldering the external connector pins/sockets, following a process last seen on the Atlas program, which proved to be successful.
‘(This) eliminates known failure mode at external Feed-through Connector,’ outlined the forward plan. ‘Does not eliminate potential unknown root cause at internal plug (ATLAS heritage). Process development and new technician cert required. Soldering and Swage-Lok Options Provide ‘Solid State’ Connection Between Sockets and Pins.’
The above plan grew from the multiple options that were drawn up ahead of the PRCB, at the time noted as ‘Option 4’. As noted at the time, this process is likely – though not confirmed at present – another tanking test, to be carried out near the end of the month – which is outlined as the R&R schedule for this option (completion January 22-25).
‘ET Project Recommendation – ET-125 / ET-126. Pursue design solution that addresses most likely root cause by implementing solid state connection between external harness and feed through connector (Option 4). Pin soldering provides proven technology and can be rigorously screened prior to implementation (NDE, acceptance test loading),’ noted documentation.
‘Technical Basis for Recommendation – Eliminates significant contributors to most likely root cause of STS-122 ECO circuit anomalies. Contamination / relative motion effects between external plug socket and feed through pins eliminated. Internal plug improbable contributor to anomaly – In-situ inspections / tests will provide additional risk mitigation against potential causes identified.
‘Pin soldering used on Atlas program to mitigate intermittent circuit anomalies isolated to connection between external plug and feed through receptacles. Similar design as used on ET.
‘Same pin / socket material and pin diameter. Testing performed on Atlas program to verify design performance. Pin / socket tensile strength and qualification testing with cryogenic cycling (LN2) with vibration environments. Design successfully flown with no issues.
‘Solder contacts with harness ‘pig tail’ to feed through receptacle at component level. Provides optimum access and opportunity for acceptance test / inspection.’
Should engineers be forced to take Option B, rollback would led to a destack and tank swap with ET-126 inside the VAB. This option will not be taken unless test data shows the requirement for internal tank access for the repair effort.
‘Repair options requiring internal tank access make ET-126 most likely tank for implementation. Tank currently in checkout cell. Horizontal access required for egress through aft manhole cover and replacement of SLA TPS,’ noted R&R options from Thursday’s PRCB meeting.
‘Around 4 week additional schedule time to implement on ET-126 as compared to ET-125. ET-125 roll back and de-stack required to implement any option requiring internal tank access. Horizontal access required for egress through aft manhole cover and replacement of SLA TPS. High risk of collateral damage during operation.’
Overview and Timeline:
Engineers from the Michoud Assembly Facility (MAF) are supervising the whole R&R process, both at the pad, and with the other tanks they have in production. They, along with MSFC, outlined the overall thinking and schedules for returning to the point that will allow USA (United Space Alliance) engineers to return to a launch posture.
‘Recommending near term design solution that addresses most likely root cause by implementing solid state connection between external harness and feed through connector. Pin soldering provides proven technology (Atlas) and can be rigorously screened prior to implementation (NDE, acceptance test loading).
‘Soldered design eliminates significant contributors to most likely root cause of STS-122 ECO circuit anomalies. Contamination / relative motion effects between external plug socket and feed through pins eliminated. Qualification testing required but not a constraint to hardware installation (flight constraint). High confidence in design based on Atlas experience and development test results.
‘Technical reviews planned prior to implementation on ET-125. Production Readiness Review: 1/03/08. Certification Review: 01/23/08. Current plan shows ET-125 work complete prior to 2/01.
‘Analysis and data review used to show Internal plug as an improbable contributor to anomaly. In-situ inspections / tests operation will provide additional risk mitigation against potential causes identified. Decision to replace internal plug would change program plan. Requires internal tank access – Better to implement on ET-126. ET-125 would require roll back and destack operations to gain internal access.’
Overall, hopes remain high – especially due to the commonality with the success of the plan on the Atlas program – that the repair will solve the key suspect that is causing the issues.
If that proves to be the case, the fix will be implemented into at least four tanks, ET-125 with Atlantis, ET-126 in the VAB, ET-128 and ‘one more’, as per documentation from Lockheed Martin.
Option A’s plan will also see ET-126’s R&R process completed by February 2, providing plenty of time for mating with Endeavour in time for LON (Launch on Need) requirements. Ultimately impacting the launch schedule by only a matter of around a month in total, should everything go to plan.
L2 members: All documentation – from which the above article has quoted snippets – is available in full in the related L2 sections, updated live.
**STARLIGHT: NEW STS-117 MISSION REVIEW MUSIC VIDEO** – STS-118 and STS-120 Mission review video also available.
**ROCK YOU LIKE A HURRICANE: STS-117 MISSION REVIEW MUSIC VIDEO – AS SEEN BY THE STS-117 CREW!!** -.
(Video section is FREE, but you need to sign up as a member of the forum to enter the video section of the site. We only use your e-mail to send you your password. It will not be used for spam etc.)