STS-133 GUCP root cause to benefit from STS-119/127 investigation
As engineers prepare to take a close look at the Ground Umbilical Carrier Plate (GUCP) at the center of STS-133’s scrub on Friday, a pool of knowledge will be fully utilized in the drive towards finding a root cause. That knowledge comes via a large amount of documentation gained through testing and troubleshooting, after the GUCP caused scrubs on both STS-119 and STS-127 last year.
STS-133 Pad Flow:
Discovery has seen her final mission delayed until November 30 at the very earliest, following the recording of IPR-68 (Interim Problem Report) during Friday’s countdown, when leak detectors at the pad observed the gaseous hydrogen leak from the GUCP.
All had been proceeding to plan – with the tank “fast filled” during tanking, with no issues recorded with either the loading process, or the Low Level/Engine Cut Off (ECO) sensors via their customary SIM checks – until the first leak indication was revealed.
Firstly, a 33,000 ppm leak – below the 40-44,000 ppm (HAZ-09 limit in the Launch Commit Criteria – LCC) – was recorded, before reducing to a level below 20,000 ppm. The leak was only being observed during the cycling of the vent valve to “open” – to release the gaseous hydrogen from the tank and through the vent arm plumbing to the flare stack, as designed.
With controllers deciding to stop the cycling of the valve – in order to increase the pressure and attempt to force a seal – before attempting to complete the fast fill process and transition into “topping”, the leak spiked and pegged at the highest 60,000 ppm level, indicating a serious problem with the GUCP’s seal.
With cycling of the valve resumed – as part of the troubleshooting efforts to clear any potential obstructions such as ice from the hardware – and no resolution forthcoming, a scrub was the only outcome.
“Pegged leak detectors at topping. Violation. Scrubbed for today. Configuring for drain,” flashed the confirmation (L2), as controllers moved into emptying the External Tank, leading to the ECO sensors registering “dry” at 13:53 local time.
It took around a day for the tank to become inert, allowing engineers to prepare towards disconnecting the vent arm and the large amount of lines and ordnance on the hardware, prior to taking their first look at the potentially suspect seal and any potential alignment issues – the two leading candidates for the leak. A gantry walkway will be constructed over the coming hours to allow such access.
With the Rotating Service Structure (RSS) now mated back around the STS-133 stack, weekend information noted the completion of scrub activities is anticipated for sometime during Sunday’s third shift (early Monday morning), with tasks including the offloading of Discovery’s Power Reactant Storage and Distributation (PRSD).
It had already been a relatively eventful final day of countdown, with the Rotating Service Structure (RSS) rotated back to mate overnight, in order to allow for the replacement of Tyvek covers over the Forward Reaction Control System (FRCS).
“IPR-65: During surveillance of camera 61 it was noticed that the Tyvek cover on thruster F1L had partially de-bonded on the leading edge of the thruster lip,” noted the NASA Test Director (NTD) report (L2). “Based on weather forecast there was a potential to allow water intrusion into the thruster chamber, thus engineering recommended cover replacement.
“Following RSS rotation to mate complete, engineering inspection revealed Tyvek rain cover F2D was also partially detached. Techs began work establishing access for further inspection. Further inspections revealed that F3F requires replacement. Inspections were completed and planning put in place to replace F1L, F2D and F3F.
“Thruster inspections complete, no evidence of water intrusion into the thrust chamber. F1L, F2D and F3F Tyvek cover installation are complete, final inspections are complete, all nominal.”
Also observed during the morning of the countdown, one of the ET/orbiter umbilical baggies – a kapton bag that is attached to enclose the area to prevent ice from forming, purged by GN2 flowing out of the aft compartment – became detached from around the propellant lines.
However, the observation of a crack – or a separated ‘chunk’ – in the ET’s foam will require a large amount of evaluation and likely a repair effort. The crack would have been spotted by the Final Inspection Team (FIT) who would have been due to document ice/frost buildup – and any other potential issues – after the tank had been transitioned into stable replenish.
Although that task was cancelled due to the scrub, the first teams who re-entered the pad noticed the seven inch long crack on the offset up on the flange area between the intertank and the LOX tank. Such a crack would have likely resulted in ice forming in the gap, and/or may have resulted in a foam liberation threat – which in turn may have failed the inspection criteria for launch.
The area is also facing the orbiter (+Z side), just above the bipod ramps, increasing the debris threat to Discovery’s Thermal Protection System (TPS), making it an area that is almost certain to require repair. However, access at the pad will be difficult, resulting in a forward plan being developed over the coming days.
“We’re going to make sure we fix the problems and worry about the launch dates later,” noted Mission Management Team (MMT) chairman Mike Moses, when referencing the preliminary launch date of November 30 – the opening of the next launch window after the beta angle cut off – a date which is currently heading to all related departments as a Change Request (CR) in case of objections.
STS-133 Specific Articles: https://www.nasaspaceflight.com/tag/sts-133/
GUCP History – STS-119:
The GUCP had behaved nominally since the issues of STS-119 and STS-127, pointing to what was hoped had been a resolution of its previous leaks.
The first leak put an end to Discovery’s opening launch attempt on STS-119 back in March 2009, using ET-127. The leak, which was less than that observed during STS-133’s tanking, also occurred at a later time during tanking, at the point of the actual transition into topping, as opposed to during continued fast fill tanking.
Some similarities were noted on the Booster console inside the Launch Control Center (LCC) during the STS-119 problem, with the leak rate decreasing when the vent valve was closed. Also, once the tanking had been stopped, the same troubleshooting procedure of cycling the valve to clear any potential ice in the hardware failed to stop the leak.
“STS-119 / ET-127: Pre-launch: 1st loading resulted in scrub/LCC violation due to GH2 leakage at Ground Umbilical Carrier Assembly (>40,000 ppm). Leakage occurred during transition from fast fill to topping. Vent valve opened when 98 percent level sensor indicated wet. Detected by leak detectors (LD 23 & 25) located in ground umbilical shroud,” documentation noted at the time (L2). “Isolates leak to either ground side quick disconnect (QD) or interface with flight seal.”
Once engineers accessed the GUCP, no obvious cause of the leak was found – as the order to changeout the seal was carried out, along with a focus on the retorquing of the hardware. Engineering notes at the time pointed to a potential problem with the “left and right pivot seat”, which wasn’t fully connecting to the ET’s pin receptacle sleeve at the bottom of the GUCP.
“There was some damage to the flight seal, but we’re not sure that’s the cause,” noted launch director Mike Leinbach in review of the STS-119 troubleshooting. “There was a bit of discoloration on the QD (Quick Disconnect), but that might have been to the hydrogen flowing where it shouldn’t have been.”
With a new seal and the “tightening” of the hardware completed, STS-119’s second tanking was conducted without issue and no leak detectors tripped. Managers could be forgiven for thinking the issue was a one-off, solved by the replacement of the flight seal and the re-alignment of the pivot seats.
As per typical for NASA, an investigation was still conducted into the STS-119 scrub, which noted that out of the previous 31 loadings only one leak was observed (and only at 13,500 ppm). The investigation also noted the potential for issues with the flight seal being part of the root cause, along with the misalignment on the pivot seats – resulting in the hardware being “pulled” down and to the left.
“Most probable cause identified as momentary breach in flexible flight-seal to bellows probe due to ‘thermal shock’ of GH2/LH2 with vent valve in open position. Significant Disassembly Observations: Lower left pad was hard against skin,” noted the findings (L2).
“Other locations were not touching (0.014 – 0.030 gap / 0.001 requirement) indicating a pull downward and to the left. Peripheral seal compressed more on left side and toward bottom of GUCP. Left side pivot assembly in hard contact with pivot pin (pin would not rotate). Stain observed on external surface of bellows guard and peripheral seal at 6 o’clock position. Flight-side seal asymmetrically compressed at 3, 7 and 8 o’clock positions.”
Changes were then implemented to ensure the alignment issue wouldn’t reoccur, with additional focus placed on both the installation of the GUCP hardware and observations of any movement once the stack was out at the pad. STS-119 launched without any further issues.
With the next mission, STS-125, avoiding any leaks during tanking, STS-119’s GUCP-related scrub continued to appear as a one-off issue, with additional confidence in future tankings gained by the aforementioned mitigation procedures. However, STS-127 would see the problem return.
GUCP History – STS-127:
Interestingly, STS-127’s GUCP issues began before Endeavour had even rolled out to the pad with ET-131, with documentation showing work to install the hardware on the tank – carried out inside the Vehicle Assembly Building (VAB) – has been problematic, ultimately requiring a changeout of the GUCP and a redesign to its installation hardware.
“Interference between GUCA (Ground Umbilical Carrier Assembly) and ET-131 right hand hinge support observed during mate GUCP (Ground Umbilical Carrier Plate) installation in VAB,” noted STS-127 SSP (Space Shuttle Program) FRR (Flight Readiness Review) documentation (L2).
Such an interference is not permitted at the hinge location due to the fact that there is potential to induce un-intended loading on the pyro-bolt assembly – which could affect that separation mechanism at T-0. In order to correct the interference, the GUCP was removed and a different unit was installed. However, after this was accomplished, the interference remained.
Visual and Laser inspections revealed the slight misalignment between the centerline of the plate and the hinge bracket, leading to a modification to the pivot assembly, which was successfully installed by “locally machining outboard surface (0.1” removed) to create the required gap (0.03” gap provided),” according to the FRR documentation.
How much relation those changes had to the subsequent leak during STS-127’s tanking in June of last year remains unknown. The first tanking of STS-127 registered a leak at the same time as STS-119’s detection, leading to the scrub and call to dismantle the GUCP hardware once the tank was inert.
Once the vent arm was removed and engineers removed the flight seal, observations pointed to a potential root cause via small gaps on the right hand side of the seal.
It was also thought the tank’s GUCP may have suffered from being mated and then unmated at Pad 39B, before being mated once more at Pad 39A, as Endeavour switched rolls from being STS-125’s Launch On Need (LON) vehicle (STS-400) to her primary role with STS-127 – requiring the pad switch.
This was a potential candidate for being part of the root cause because the seal is a hard teflon ring with no resiliency, and thus presents a sharp corner edged to a smooth tapered metal probe. Any bump, dent or mis-alignment of the probe during installation could result in a leak caused by damage to the teflon edge on the seal.
“GH2 vent seal inspection results: rolled edge around entire circumference with worst case from 4 to 10 o’clock position,” noted one log report on the status of the old seal at the time of troubleshooting (L2). “No inclusions and no scratched observed.”
With the seal replaced, it was hoped that STS-127 would enjoy a smooth tanking at the second attempt, similar to STS-119 once the GUCP seal was changed-out. Unfortunately, the June 16, 2009 tanking once again registering a leak.
Interestingly, the second STS-127 scrub appears to have most of the characteristics as STS-133’s incident, with the leak seen 25 minutes prior to topping – appeared to increase at the end of fast fill operations, with the leak detectors observing the peak leak rate of 60,000 ppm.
“This time the leak started during fast fill which is a signature we’ve never seen before (relating to the difference between the previous leaks, observed as the tank loading process moved from fast fill to topping/stable replenish of the LH2). During fast fill we leaked to approx. 15,000 ppm,” noted the STS-127 attempt 2 scrub outline on L2. “Once we reached replenish, we violated the LCC like we’ve typically seen in the past. Leak eventually trended upward to 60,000 ppm.”
Initial theories pointed to several candidates as the root cause, such as unique thermal conditions associated with the hardware, notably the dynamics of the cryo temperatures that may be interacting with the hardware’s hinge brackets, resulting in a misalignment during tanking.
Also under evaluation were potential software issues, and even possible issues with the leak detectors that registered the leak during tanking – as much as the latter was ruled out as a specific reason for the scrub, due to the “visible” observation of venting from the tank.
Another investigation path pointed to the External Tank hardware itself, as opposed to the Ground Support Equipment (GSE) of the GUCP QD, as the reason for the specific leak issues observed with STS-119 and STS-127′s tanks.
However, the key area of interest is related to the two mounts, or feet, located on the tank where the GUCP hinge points attach. One of these mounts (right) was deemed to be offset from their preferred location.
“Have many folks across Agency supporting GUCP investigation. Appears to be going well. Appreciate folks at KSC showing us the hardware there. It looks like the ETCA plate that mounts (manually installed) to the ET is not properly aligned with the ET,” noted an Engineering overview presented via a Shuttle Standup meeting (L2) at the time.
“There are a couple of feet, below where the GUCP rotates off during separation, which are not mounted exactly correctly relative to the ETCA. When the GUCP is put on, there are forces between the pyro bolt, the large QD and the seat. If the alignment is not correct on the ET, the seat may be shifted as everything is tightened.”
This problem was also found on six other tanks set to fly – though sources note that didn’t include STS-133’s tank – and the misalignment on ET-131 was classed as “the worst”.
“Two adjustments were made to get additional clearance to allow centering and alignment, but after both attempts, the feet and brackets were found way over to the right side and we were not able to align properly,” added notes, again pointing to a problem being suffered at the actual time the tank transitioned into a cryogenic state.
With the second scrub resulting in a several week standdown, NASA’s engineering teams were put into full investigation mode, resulting in a hugely impressive mitigation drive involving several centers.
Test articles were put to use – such as the GUCP rig at the Marshall Space Flight Center (MSFC) – working on the main candidate that a misalignment was causing the leaks, along with a drive to use a new two part flight seal, one which would be more forgiving to small misalignments, and hopefully mitigate unacceptable leak levels.
The two part seal also allows the tank to “burp” – without the need for vent valve cycling – which had previously cleared a minor leak on a previous loading earlier in the program.
The two part seal had only been installed in two previous tanks ahead of the problems with STS-119 and STS-127, one of which leaked, but was successfully mitigated via the “burp”, allowing the launch to proceed.
A tanking test in June 2009 was called for, testing out the changes and allowing for additional data to be gained – via strain gauges on the feet of the GUCP hardware – during the loading of the cryogenic propellants, with the ultimate aim of conducting a successful test and allowance to proceed with STS-127’s launch.
“The engineering teams, after much analysis of the measurement data between the 2nd scrub disassembly and the 1st scrub disassembly, have high confidence that misalignment is the issue,” noted documentation ahead of the tanking test (L2).
In order to mitigate misalignments, a redesign to the “fitted feet” on the GUCP was implemented on to STS-127’s tank. This design – along with the two part seal – was implemented into all future tanks that were under construction at the Michoud Assembly Facility (MAF).
The tanking test proved to be a success with no leaks detected, allowing for Endeavour to proceed towards another launch attempt, which also suffered from no leaks during tanking. Ironically, Endeavour was delayed by weather constraints and took a total of six attempts to finally launch on her mission to the International Space Station (ISS). No further leaks were observed on her tankings after the tanking test success.
The successes provided additional confidence that the engineering work on correcting and mitigating what was then confirmed to be an alignment issue of just 0.357 degrees in the counter-clockwise direction, has been successful.
The amount of work that went into fixing the issue was listed in a 47 page presentation to the all-powerful Program Requirements Control Board (PRCB), dated July 7, 2009 and acquired by L2 at the time.
The presentation provided what was claimed at the time to be the closure of the GUCP leak IPRs ahead of STS-127’s successful launch, a path that appeared to be confirmation the problem was behind them, as would be seen in all further tankings until STS-133’s incident.
“Present the GUCP GH2 leak fault tree status, IPR closure (STS-119 and STS-127), and results of root cause assessment including affected materials, process/procedure/technique changes, and other associated relevant data. Present results to the PRCB,” prefaced the presentation.
“Identified 21 scenarios using inputs from community, new fault tree, timelines. Collected evidence to support/refute each scenario. 11 scenarios are fully (4) or partially (7) mitigated by the actions taken. Evidence reviewed by team ruled out 10 scenarios.”
Following an extensive review, engineers confirmed the misalignment was to blame. However, the a flight seal issue – since replaced with the “more forgiving” two-part seal – may have also contributed.
“Root cause: plate misalignment resulted in gapping at flight seal/bellows probe interface. Contributors: As-built flight hardware misalignment ETCA & hinge pin brackets. Insufficient controls during assembly to account for off-nominal ET geometry,” the presentation noted.
“Measurements, Alignment pins, Flight/ground plate relative motion (lateral) during assembly. Reduced capability to accommodate motion at interface during operations due to stiffer Inconel bellows. Unexplained Anomaly, possible contributors include: Flight seal defects and/or damage during assembly. Potential plate misalignment.
“Leak mitigation: Tighter tolerance alignment pins (0.515″). Tailored GUCP feet (0.180″ & 0.230″ offset). analysis shows adequate strength. Hinge pin washers restrain GUCP lateral motion. 2-piece flight seal has greater resiliency and provides additional capability for misalignment. 2-piece seal tested to 0.050″. Concentricity and other measurements during assembly show minimal motion of GUCP. Successful tanking test. Tanking test observations show minimal motion of GUCP feet.”
The PRCB investigation into the STS-119 and STS-127 leaks admits that “A lack of root cause for STS-119 and partially mitigated failures scenarios demonstrate some residual leak risk still exists,” but “recommended MMT action closure”.
For all NASASpaceflight.com articles related to the GUCP, click here: https://www.nasaspaceflight.com/tag/gucp/
As to why STS-133’s tank suffered a leak, the hope would be for an obvious and one-off misalignment being suffered on the tank, or a problem with the flight seal – allowing for a replacement to likely solve the issue.
If a misalignment is confirmed, engineers should be able to work on torquing the hardware in-situ at the pad. They will also likely replace the flight seal, even if no issues are observed once it’s removed.
If an issue with either the alignment or the seal is obvious, it may be possible to avoid a tanking test ahead of Discovery’s next launch attempt. No decision will be made on a tanking test until the results of the full GUCP observations have been made.
(Further articles will follow. Refer to the STS-133 troubleshooting thread, linked at the top of the article. L2 members refer to the STS-133 and GUCP Special Sections for live internal updates on the mitigation efforts. All images via L2 content.)