Discovery’s return to the Vehicle Assembly Building (VAB) – in order to gain extra data on the health of the stringers on areas of her tank (ET-137) which can’t be scanned out at the Pad – was completed early on Wednesday. Efforts to return Discovery to the VAB comes just days after a successful Tanking Test, which gained solid data on on the stringers, whilst also confirming the successful troubleshooting on the Ground Umbilical Carrier Plate (GUCP).
STS-133 Latest:
Now returned to the huge building, Discovery is likely to spend two to three weeks inside the VAB, allowing access to the 360 degree circumference of the two flanges – the upper LO2 area and the lower LH2 area – on the intertank.
At present, the data gained from the Tanking Test may have provided the required confidence that both the two doubler stringers – which replaced the S6 and S7 stringers, both of which suffered from two cracks each – acted in the required manner during tanking and pressurization during the test.
An option to add additional strength via modifications is open to managers, should the Tanking Test and ground test data call for it.
At the Pad, engineers cut the instrumentation wiring from the tank, allowing for rollback. The array of sensors – located in two areas of the tank – will be fully removed once the stack inside the VAB. Foam repairs will then follow, with the ultimate goal of returning the STS-133 stack in a nominal condition for the early February launch attempt.
“OV-103/SRB BI-144/RSRM 112/ET-137 (Pad-A): ET Instrumentation post ops are scheduled. Work will be performed to prepare the instrumentation for rollback and later removal in the VAB,” noted the NASA Test Director (NTD) via processing latest (L2) ahead of rollback.
“S0038/A5217 Rollback to the VAB: The vehicle will be rolled back to the VAB to perform 360 degree NDE (Non Destructive Evaluations) of LO2 and LH2 flanges, remove instrumentation, and repair foam areas associated with the instrumentation.”
STS-133 Specific Articles (click for numerous background content relating to this article): http://www.nasaspaceflight.com/tag/sts-133/
Although the rollback was scheduled for 00:30 on Tuesday morning, first motion was pushed back by delays in removing the hydrogen vent system arm from the tank – a process which has to be completed carefully, in order to avoid any troublesome misalignments to the GUCP on the end of the arm.
However, a problem with the Crawler Transporter resulted in a decision to postpone the rollback for the night. Troubleshooting is focusing on the D-Side of the CT, with scanner conversations noting a potential contact between the D-Side and a cable tray when in transit to underneath the MLP – which may have affected the Jacking, Equalization and Leveling (JEL) hydraulic system.
The NTD noted the issue was pinpointed to a faulty transducer, which has been replaced in time for another rollback attempt to take place around 10pm Eastern. Following a smooth operation, the CT began first motion at 10:48 local, arriving back at the VAB at 7am, with the completion of rollback pending harddown.
Meanwhile, checking the GUCP was classed as a “free” test during Friday’s tanking, which was dedicated towards the gaining of data on ET-137’s stringers. Had the GUCP leak been the only problem with ET-137, a Tanking Test wouldn’t have been carried out. Regardless, the lack of leaks during the tanking has provided a level of confidence that the system will behave once Discovery undergoes another launch attempt.
“S0037 ET Tanking Test: The tanking test was completed successfully on Friday. Good data was collected from all instrumentation installed on the ET. The data is under engineering evaluation but a preliminary assessment shows that the tank performed nominally with actual data closely matching predictions,” added the NTD.
“GUCP repairs were also proven successful with no hydrogen leaks detected. De-tanking, boil-off, and securing from S0037 were completed over the weekend.”
Notes from the Mission Evaluation Room (MER) during the Tanking Test confirmed the GUCP provided no issues to controllers on Friday, with the main focus on the key transitional periods of the loading process.
During tanking, and once the chilldown has been completed, both tanks enter a slow fill, avoiding the cyro-shocking of the feedlines. Loading then transitions into Fast Fill, from which the tank volume is partially filled with LH2, with ullage gas consisting of mixture of GHe and GH2 concentration.
The vent valve begins cycling – the point at which STS-133’s first launch attempt in November was scrubbed – with a tank pressure target of between 32.9 and 43.2 psia.
During the transition to topping, the valve opens and vents high concentration of GH2 and some LH2 into the vent line to be burned off at the Flare Stack. The vent line/umbilical interface is exposed to rapid cryo cool-down and pressure. Once topping has been completed, stable replenish is used to compensate for the natural boil off of the propellant.
“We have started the LO2 and LH2 loading portions of the Tanking Test, on time. All ET Stringer Instrumentation is working properly. Also, now that we’ve started tanking, we will not scrub for any future ET Stringer Instrumentation failures,” added MER notes, confirming the status of tanking (L2), with only one minor IPR (Interim Problem Report) noted during the process, relating only to instrumentation.
“Both LO2 and LH2 are in the Fast Fill portion of ET Loading. Loading is proceeding nominally except for a minor IPR (80) taken for the LO2 replenish valve located on the MLP. Both open indications did not turn on after the valve was opened. A secondary cue (a potentiometer indicating 99 percent open) has verified the valve did, indeed, open when commanded.”
Confirming no leaks from the GUCP at either the time of the previous incident, or closer to topping – the usual time for previous leaks from the GUCP – controllers classed the loading portion of the Tanking Test as nominal.
“LH2 has transitioned to the topping phase of loading, which will bring the LH2 tank load to 100 percent. The ET GH2 Vent Valve has been opened with no leakage occurring at the GUCP. The ET Instrumentation verbal report says all data looks “nominal” with no failed data,” added MER notes.
“We have reached Stable Replenish for both the LO2 and LH2 systems. Other than the Replenish valve indication issue (which, due to a thermal increase from the Sun now indicates OPEN), this has been a nominal ET load.”
The Final Inspection Team (FIT), plus one member with a telescope, then inspected the tank, as is usual post-tanking. Later, managers gained a “quick look” of the test results, all of which showed expected data on the stringers.
GUCP Investigation Results:
Although the stringers were the primary focus of the test, the amount of work that has gone into ensuring the GUCP is no longer an issue for ET-137 is staggering, and typically NASA.
All had been proceeding to plan during the November 5 launch attempt, with the tank ‘fast filled’ during tanking, with no issues recorded with either the loading process, or the Low Level/Engine Cut Off (ECO) sensors via their customary SIM checks – until the first leak indication was revealed.
Firstly, a 33,000 ppm leak – below the 40-44,000 ppm (HAZ-09 limit in the Launch Commit Criteria – LCC) – was recorded, before reducing to a level below 20,000 ppm. The leak was only being observed during the cycling of the vent valve to “open” – to release the gaseous hydrogen from the tank and through the vent arm plumbing to the flare stack, as designed.
With controllers deciding to stop the cycling of the valve – in order to increase the pressure and attempt to force a seal – before attempting to complete the fast fill process and transition into “topping”, the leak spiked and pegged at the highest 60,000 ppm level, indicating a serious problem with the GUCP’s seal.
With cycling of the valve resumed (six times) – as part of the troubleshooting efforts to clear any potential obstructions such as ice from the hardware – and no resolution forthcoming, a scrub was the only outcome.
With a vast amount of experience from previous GUCP leaks, previously observed during STS-127 – engineers went to work on methodically evaluating the root cause of STS-133’s leak via a fault tree.
“Evaluation: STS-133 troubleshooting plan as compared to the STS-127 Fault Tree. STS-133 GUCP Leak Investigation Team has developed a troubleshooting plan to address the STS-127 GUCP Leak Fault Tree. Residual risk still exists for dynamic characteristics of the system that are not fully understood,” outlined one of a host of expansive – and impressive – presentations on the STS-133 GUCP troubleshooting, available on L2.
“Mitigated through bench leak checks, disassembly and inspection, and analysis. Leak Through QD (Quick Disconnect) Assembly. Sealing surface defects and material properties of the Flight Seal, Bellows Probe and ET Carrier Assembly (ETCA). Materials testing of Bellows Probe and Flight Seal still in work, however considered unlikely contributor…
“…Eccentricity Mismatch Between Seal and Probe. Probe dimensions included for STS-133 Concentricity Assessment. Sealing Surface Contamination including ice and FOD (Foreign Object Debris). Flight Seal Deformation. Flight Seal Retaining Ring Loose. Improper Probe, Flight Seal, and ETCA Build Dimensions ETCA Movement (Intertank movement /vibration including Shuttle).”
With the likely causes pinpointed to a deformation of the flight seal, or a misalignment of the hardware (concentricity), an investigation team found that a misalignment was a leading candidate, following observations of movement in the GUCP ahead of tanking.
“Ground Umbilical Carrier Plate (GUCP) Investigation Team was reconvened. Detailed disassembly of the Ground Umbilical Carrier Assembly (GUCA) and Vent Line Assembly was performed. Observed no anomalies in the vent line assembly,” added another presentation.
“Observed that the GUCP had shifted since the final measurements taken prior to S0007 (Launch Countdown). The plate was originally offset (0.049”) to the 8 O’clock position. Shifted an additional amount toward the 7 to 8 O’clock position. First post disassembly concentricity value was 0.061” to the 8 o’clock position.”
The other leading candidate for the leak was the flight seal, which contributed to the previous GUCP leaks, resulting in a change of design to a more capable two-part flight seal. Initially, engineers checked the seal out at the pad.
For all NASASpaceflight.com articles related to the GUCP, click here: http://www.nasaspaceflight.com/tag/gucp/
“Flight Seal observations. Tactile and visual inspections performed (3 independent inspections). Some circumferential deformation at the 7 to 9 O’clock position. Gap of the seal edge to retainer ring appeared smaller at the 7 to 9 O’clock position,” the presentation continued.
The Flight Seal was removed from the hardware and sent to the KSC lab – and later to the Michoud Assembly Facility (MAF) – for analysis, which included Chemical and Nondestructive examinations. Those evaluations revealed some observations of interest, but no anomalies which could have contributed to the leak.
“Chemical analysis was perform on wipes taken during tactile inspection and on debris captured prior to seal removal and on debris captured. Wipes analysis revealed unidentifiable materials that were organic in nature. Debris analysis revealed variety of small metallic and organic particles. Magnified visual inspection of both pieces of the flight seal revealed no anomalies.
“Dimensional analysis of the seal found one minor dimension to be slightly undersized. Dimensional analysis of the metallic outer spring found its diameter to be slightly undersized (is 0.116 s/b 0.120 +/-.002). This dimension was consistent around the diameter indicating the spring was not out-of-round.”
A misalignment soon became the root cause, or major contributor, to the leak, confirmed by concentricity measurements on the SN2 plate used by ET-137, and backed up by previous investigation notes that loads on the vent line can cause the GUCP to shift slightly in one direction. This shift – by only 0.061 inches – provided a gap for the leak to manifest.
“GH2 vent system is composed of numerous components (flight and ground) that combined together on STS-133 to provide a tolerance stack-up towards the 7-8 o’clock position. GUCP to ETCA and to QD Concentricity,” noted the findings presentation.
“Previous GUCP investigation uncovered tendency for ground GH2 vent line loads to affect carrier plate position by pulling the plate towards the 7-8 o’clock. First opportunity for concentricity measurements during disassembly corroborates this tendency. SN2 was found offset 0.061” towards 7-8 o’clock direction.”
While the misalignment itself was an initial leading candidate for the leak, the shift in this particular direction wasn’t expected. However, it provided an immediate solution.
“Significant previously unaccounted for variable of QD concentricity discovered during STS-133 GUCP leak investigation. Clocking QD during installation uses this variable to our advantage. Controlling the variables we can optimize (GUCP & QD concentricity) in a manner that helps us mitigate loading (7-8 o’clock) will provide greater compliance to the system as a whole.”
Such was the confidence in the findings, the presentations also confirmed there was no need for a Tanking Test specific to the GUCP. However, with the stringer crack investigation providing a free opportunity to monitor the GUCP via the associated Tanking Test, managers decided to add reference marks – to be observed by two cameras – to the hardware to facilitate measuring movement of the GUCP/QD during tanking.
“Use imagery to detect movement at GUCP/ETCA Interface during Tanking. Expect to measure approximately 1/10” movement at plate with six Degrees of Freedom (translation and rotation). Reference Marks Requirements: 6 marks on GSE (Ground Support Equipment). 42 marks on ET TPS,” confirmed the recommendation of the Program Requirements Control Board (PRCB).
With the understanding of the issue, and the subsequent replacement of the hardware (carrier plate, flight seal etc) ahead of the Tanking Test, managers are confident the GUCP leak issue is finally behind them.
“The STS-133 Ground Umbilical Carrier Plate and QD concentricity offsets were considered the major contributors to the GUCP leak. The GUCP and QD have been replaced with a set that minimizes the concentricity offset,” added summary notes.
“Analysis of the fault tree identified residual risk due to the relative motion between the GUCP/QD and flight seal due to vent line loads and dynamics, bellows spring force, bellows binding/interference, and the pyro bolt assembly. However, the expectation is that the currently optimized GUCP and QD should eliminate the issue.”
The troubleshooting effort also earned praise from NASA manager LeRoy Cain, who thanked the NASA and contractor teams for their attitude, which he classed as exceptional.
“Thanks to all the folks who have been working the cracked stringer issue, as well as the issue to resolve the GUCP leak. Mr. Cain stated for the ET Project and Lockheed-Martin, the effort being put into this by the team is outstanding across the board,” added notes on the Shuttle Standup/Integration report.
“The attitude is exceptional; it is really a model for other folks to watch and pay attention on how to work through issues like this.”
(Further updates and articles will follow. Refer to live coverage threads linked above. L2 members refer to STS-133 live coverage sections for internal coverage, presentations, images and and updates from engineers and managers. Images used, Lead: Larry Sullivan, MaxQ Entertainment/NASASpaceflight.com. Within the article: via L2 acquired PRCB presentations).