Shuttle Scrubber: SLS working to avoid GUCP leaks
With a deep heritage between the Shuttle’s External Tank and the Space Launch System (SLS) Core Stage, a NASA team recently evaluated recommendations to mitigate against leaks of the Ground Umbilical Carrier Plate (GUCP) – the cause of launch day delays for three Space Shuttle missions – on the monster rocket.
The GUCP is a critical element of hardware, located at the end of the gaseous hydrogen vent arm. Attached to the External Tank, a plate holds a large-diameter pipe that collects excess hydrogen gas from the tank as it’s being filled with liquid hydrogen on launch day.
The venting system funnels it to a larger pipe that takes it down the fixed service structure and out to a flare stack that burns the excess hydrogen off safely. At liftoff, the GUCP retracts away from the tank, cutting off the connection.
Due to the importance of the GUCP’s connection to the tank ahead of lift-off, the hardware has sensors in place to watch for hydrogen leaking from the point of tanking operations, through to launch.
These readings are closely monitored by the team in the Launch Control Center (LCC) Firing Room, with any readings outside the limits resulting in the Launch Director, the NASA Test Directors (NTDs) and Mission Management Team (MMT) making a decision based around Launch Commit Criteria – often resulting in a scrub for the day and the detanking of the ET.
Click here for the list of GUCP-related articles: http://www.nasaspaceflight.com/tag/gucp/
This was the scenario that impacted three missions in the post-RTF era.
GUCP History – STS-119 (All “L2-tagged” materials can be found in the L2 GUCP Section):
The first leak put an end to Discovery’s opening launch attempt on STS-119 back in March 2009, using ET-127. The leak was observed at the point of the actual transition into topping, as the ET was almost full to its brim.
With readings alerted to the Booster console inside the LCC, the leak rate appeared to decrease when the vent valve was closed. This led to an initial effort to troubleshoot via the procedure of cycling the valve to clear any potential ice in the hardware, but this effort failed to stop the leak.
“STS-119 / ET-127: Pre-launch: 1st loading resulted in scrub/LCC violation due to GH2 leakage at Ground Umbilical Carrier Assembly (>40,000 ppm). Leakage occurred during transition from fast fill to topping. Vent valve opened when 98 percent level sensor indicated wet. Detected by leak detectors (LD 23 & 25) located in ground umbilical shroud,” documentation noted at the time (L2). “Isolates leak to either ground side quick disconnect (QD) or interface with flight seal.”
Once engineers accessed the GUCP, no obvious cause of the leak was found. The decision was made to changeout the seal, along with a focus on the retorquing of the hardware.
Engineering notes at the time pointed to a potential problem with the “left and right pivot seat”, which wasn’t fully connecting to the ET’s pin receptacle sleeve at the bottom of the GUCP.
“There was some damage to the flight seal, but we’re not sure that’s the cause,” noted launch director Mike Leinbach in review of the STS-119 troubleshooting. “There was a bit of discoloration on the QD (Quick Disconnect), but that might have been to the hydrogen flowing where it shouldn’t have been.”
With a new seal and the “tightening” of the hardware completed, STS-119′s second tanking was conducted without issue and no leak detectors tripped.
As a result, managers could be forgiven for thinking the issue was a one-off, solved by the replacement of the flight seal and the re-alignment of the pivot seats.
Click here for STS-119 news articles: http://www.nasaspaceflight.com/tag/sts-119/
As is typical for NASA, an investigation was still conducted into the STS-119 scrub, which noted that out of the previous 31 loadings only one leak was observed (and only at 13,500 ppm). The investigation also noted the potential for issues with the flight seal being part of the root cause, along with the misalignment on the pivot seats – resulting in the hardware being “pulled” down and to the left.
“Most probable cause identified as momentary breach in flexible flight-seal to bellows probe due to ‘thermal shock’ of GH2/LH2 with vent valve in open position. Significant Disassembly Observations: Lower left pad was hard against skin,” noted the findings (L2).
“Other locations were not touching (0.014 – 0.030 gap / 0.001 requirement) indicating a pull downward and to the left. Peripheral seal compressed more on left side and toward bottom of GUCP. Left side pivot assembly in hard contact with pivot pin (pin would not rotate). Stain observed on external surface of bellows guard and peripheral seal at 6 o’clock position. Flight-side seal asymmetrically compressed at 3, 7 and 8 o’clock positions.”
Changes were then implemented to ensure the alignment issue wouldn’t reoccur, with additional focus placed on both the installation of the GUCP hardware and observations of any movement once the stack was out at the pad. STS-119 launched without any further issues.
With the next mission, STS-125, avoiding any leaks during tanking, STS-119′s GUCP-related scrub continued to appear as a one-off issue, with additional confidence in future tankings gained by the aforementioned mitigation procedures. However, STS-127 would see the problem return.
GUCP History – STS-127:
Interestingly, STS-127′s GUCP issues began before Endeavour had even rolled out to the pad with ET-131, with documentation showing work to install the hardware on the tank – carried out inside the Vehicle Assembly Building (VAB) – had been problematic, ultimately requiring a changeout of the GUCP and a redesign to its installation hardware.
“Interference between GUCA (Ground Umbilical Carrier Assembly) and ET-131 right hand hinge support observed during mate GUCP (Ground Umbilical Carrier Plate) installation in VAB,” noted STS-127 SSP (Space Shuttle Program) FRR (Flight Readiness Review) documentation (L2).
Such an interference is not permitted at the hinge location due to the fact that there is potential to induce un-intended loading on the pyro-bolt assembly – which could affect that separation mechanism at T-0. In order to correct the interference, the GUCP was removed and a different unit was installed. However, after this was accomplished, the interference remained.
Visual and Laser inspections revealed the slight misalignment between the centerline of the plate and the hinge bracket, leading to a modification to the pivot assembly, which was successfully installed by “locally machining outboard surface (0.1” removed) to create the required gap (0.03” gap provided),” according to the FRR documentation.
How much relation those changes had to the subsequent leak during STS-127′s tanking remained unknown.
The first tanking of STS-127 registered a leak at the same time as STS-119′s detection, leading to the scrub and call to dismantle the GUCP hardware once the tank was inert.
Once the vent arm was removed and engineers removed the flight seal, observations pointed to a potential root cause via small gaps on the right hand side of the seal.
It was also thought the tank’s GUCP may have suffered from being mated and then unmated at Pad 39B, before being mated once more at Pad 39A, as Endeavour switched rolls from being STS-125′s Launch On Need (LON) vehicle (STS-400) to her primary role with STS-127 – requiring the pad switch.
This was a potential candidate for being part of the root cause because the seal is a hard teflon ring with no resiliency, and thus presents a sharp corner edged to a smooth tapered metal probe. Any bump, dent or mis-alignment of the probe during installation could result in a leak caused by damage to the teflon edge on the seal.
“GH2 vent seal inspection results: rolled edge around entire circumference with worst case from 4 to 10 o’clock position,” noted one log report on the status of the old seal at the time of troubleshooting (L2). “No inclusions and no scratches observed.”
Click here for STS-127 news articles: http://www.nasaspaceflight.com/tag/sts-127/
With the seal replaced, it was hoped that STS-127 would enjoy a smooth tanking at the second attempt, similar to STS-119 once the GUCP seal was changed-out. Unfortunately, the June 16, 2009 tanking once again registering a leak.
The leak was observed 25 minutes prior to topping and appeared to increase at the end of fast fill operations, with the leak detectors observing the peak leak rate of 60,000 ppm.
“This time the leak started during fast fill which is a signature we’ve never seen before (relating to the difference between the previous leaks, observed as the tank loading process moved from fast fill to topping/stable replenish of the LH2). During fast fill we leaked to approx. 15,000 ppm,” noted the STS-127 attempt 2 scrub outline on L2.
“Once we reached replenish, we violated the LCC like we’ve typically seen in the past. Leak eventually trended upward to 60,000 ppm.”
Initial theories pointed to several candidates as the root cause, such as unique thermal conditions associated with the hardware, notably the dynamics of the cryo temperatures that may be interacting with the hardware’s hinge brackets, resulting in a misalignment during tanking.
Also under evaluation were potential software issues, and even possible issues with the leak detectors that registered the leak during tanking – as much as the latter was ruled out as a specific reason for the scrub, due to the “visible” observation of venting from the tank.
Another investigation path pointed to the External Tank hardware itself, as opposed to the Ground Support Equipment (GSE) of the GUCP QD, as the reason for the specific leak issues observed with STS-119 and STS-127′s tanks.
However, the key area of interest is related to the two mounts, or feet, located on the tank where the GUCP hinge points attach. One of these mounts (right) was deemed to be offset from its preferred location.
“Have many folks across Agency supporting GUCP investigation. Appears to be going well. Appreciate folks at KSC showing us the hardware there. It looks like the ETCA plate that mounts (manually installed) to the ET is not properly aligned with the ET,” noted an Engineering overview presented via a Shuttle Standup meeting (L2) at the time.
“There are a couple of feet, below where the GUCP rotates off during separation, which are not mounted exactly correctly relative to the ETCA. When the GUCP is put on, there are forces between the pyro bolt, the large QD and the seat. If the alignment is not correct on the ET, the seat may be shifted as everything is tightened.”
This problem was also found on six other tanks set to fly, although the misalignment on ET-131 was classed as “the worst”.
“Two adjustments were made to get additional clearance to allow centering and alignment, but after both attempts, the feet and brackets were found way over to the right side and we were not able to align properly,” added notes, again pointing to a problem being suffered at the actual time the tank transitioned into a cryogenic state.
With the second scrub resulting in a several week standdown, NASA’s engineering teams were put into full investigation mode, resulting in a hugely impressive mitigation drive involving several centers.
Test articles were put to use – such as the GUCP rig at the Marshall Space Flight Center (MSFC) – working on the main candidate that a misalignment was causing the leaks, along with a drive to use a new two part flight seal, one which would be more forgiving to small misalignments, and hopefully mitigate unacceptable leak levels.
The two part seal also allows the tank to “burp” – without the need for vent valve cycling – which had previously cleared a minor leak on a previous loading earlier in the program.
The two part seal had only been installed in two previous tanks ahead of the problems with STS-119 and STS-127, one of which leaked, but was successfully mitigated via the “burp”, allowing the launch to proceed.
A tanking test in June 2009 was called for, testing out the changes and allowing for additional data to be gained – via strain gauges on the feet of the GUCP hardware – during the loading of the cryogenic propellants, with the ultimate aim of conducting a successful test and allowance to proceed with STS-127′s launch.
“The engineering teams, after much analysis of the measurement data between the 2nd scrub disassembly and the 1st scrub disassembly, have high confidence that misalignment is the issue,” noted documentation ahead of the tanking test (L2).
In order to mitigate misalignments, a redesign to the “fitted feet” on the GUCP was implemented on to STS-127′s tank. This design – along with the two part seal – was implemented into all future tanks that were under construction at the Michoud Assembly Facility (MAF).
The tanking test proved to be a success with no leaks detected, allowing for Endeavour to proceed towards another launch attempt, which also suffered from no leaks during tanking. Ironically, Endeavour was delayed by weather constraints and took a total of six attempts to finally launch on her mission to the International Space Station (ISS). No further leaks were observed on her tankings after the tanking test success.
The successes provided additional confidence that the engineering work on correcting and mitigating what was then confirmed to be an alignment issue of just 0.357 degrees in the counter-clockwise direction, has been successful.
The amount of work that went into fixing the issue was listed in a 47 page presentation to the all-powerful Program Requirements Control Board (PRCB), dated July 7, 2009 and acquired by L2 at the time.
The presentation provided what was claimed at the time to be the closure of the GUCP leak IPRs ahead of STS-127′s successful launch, a path that appeared to be confirmation the problem was behind them.
“Present the GUCP GH2 leak fault tree status, IPR closure (STS-119 and STS-127), and results of root cause assessment including affected materials, process/procedure/technique changes, and other associated relevant data. Present results to the PRCB,” prefaced the presentation.
“Identified 21 scenarios using inputs from community, new fault tree, timelines. Collected evidence to support/refute each scenario. 11 scenarios are fully (4) or partially (7) mitigated by the actions taken. Evidence reviewed by team ruled out 10 scenarios.”
Following an extensive review, engineers confirmed the misalignment was to blame. However, the a flight seal issue – since replaced with the “more forgiving” two-part seal – may have also contributed.
“Root cause: plate misalignment resulted in gapping at flight seal/bellows probe interface. Contributors: As-built flight hardware misalignment ETCA & hinge pin brackets. Insufficient controls during assembly to account for off-nominal ET geometry,” the presentation noted.
“Measurements, Alignment pins, Flight/ground plate relative motion (lateral) during assembly. Reduced capability to accommodate motion at interface during operations due to stiffer Inconel bellows. Unexplained Anomaly, possible contributors include: Flight seal defects and/or damage during assembly. Potential plate misalignment.
“Leak mitigation: Tighter tolerance alignment pins (0.515”). Tailored GUCP feet (0.180” & 0.230” offset). analysis shows adequate strength. Hinge pin washers restrain GUCP lateral motion. 2-piece flight seal has greater resiliency and provides additional capability for misalignment. 2-piece seal tested to 0.050”. Concentricity and other measurements during assembly show minimal motion of GUCP. Successful tanking test. Tanking test observations show minimal motion of GUCP feet.”
However, the PRCB investigation into the STS-119 and STS-127 leaks admit that “A lack of root cause for STS-119 and partially mitigated failures scenarios demonstrate some residual leak risk still exists,” but “recommended MMT action closure”.
GUCP History – STS-133:
STS-133′s ET-137 proved to be a rather troublesome tank, following a double issue during its loading on launch day.
Discovery saw her final mission delayed, following the recording of IPR-68 (Interim Problem Report) during the countdown, when leak detectors at the pad observed the gaseous hydrogen leak from the GUCP.
All had been proceeding to plan – with the tank “fast filled” during tanking, with no issues recorded with either the loading process, or the Low Level/Engine Cut Off (ECO) sensors via their customary SIM checks – until the first leak indication was revealed.
Firstly, a 33,000 ppm leak – below the 40-44,000 ppm (HAZ-09 limit in the Launch Commit Criteria – LCC) – was recorded, before reducing to a level below 20,000 ppm. The leak was only being observed during the cycling of the vent valve to “open” – to release the gaseous hydrogen from the tank and through the vent arm plumbing to the flare stack, as designed.
With controllers deciding to stop the cycling of the valve – in order to increase the pressure and attempt to force a seal – before attempting to complete the fast fill process and transition into “topping”, the leak spiked and pegged at the highest 60,000 ppm level, indicating a serious problem with the GUCP’s seal.
With cycling of the valve resumed – as part of the troubleshooting efforts to clear any potential obstructions such as ice from the hardware – and no resolution forthcoming, a scrub was the only outcome.
Click here for STS-133 news articles: http://www.nasaspaceflight.com/tag/sts-133/
“Pegged leak detectors at topping. Violation. Scrubbed for today. Configuring for drain,” flashed the confirmation (L2), as controllers moved into emptying the External Tank, leading to the ECO sensors registering “dry” at 13:53 local time.
However, after the scrub was called, cameras at the pad picked up another serious problem, with cracks observed on the foam surrounding ET-137′s LO2/Intertank flange stringers.
It took around a day for the tank to become inert, allowing engineers to prepare towards disconnecting the vent arm and the large amount of lines and ordnance on the hardware, prior to taking their first look at the potentially suspect seal and any potential alignment issues – the two leading candidates for the leak.
At the same time, meetings were conducted to assess the reason for the crack, later found to be caused by the stringers themselves becoming cracked underneath the foam.
Ultimately, a huge amount of work was carried out, both to investigate the root cause of the cracks – found to be via a “mottled” batch of stringers at MAF, leading to a rollback and the installation of radius blocks to strengthen the local structure.
While this work was completed, engineers were called to “clock” the GUCP’s placement on the tank – and a new two-part flight seal installed. The team were provided with “free” test of the GUCP via a Tanking Test, called for to aid the investigation into the stringer cracks. The test showed the GUCP did not leak at any point during the tanking, adding confidence to the mitigation procedure.
With STS-133 launching successfully, the last two missions of the Space Shuttle Program did not suffer from any issues, either with the ET’s stringers or GUCP.
GUCP – SLS:
NASA managers often speak of “lesson’s learned” – the ability to draw on their vast experience in the rocket business, aided by their database of mitigation procedures. A shining example of this process was highlighted in a presentation that reviewed the aforementioned incidents with the GUCP and its relation to SLS.
Relating to how SLS will have commonality with ET hardware – given the SLS core is Shuttle-like ET, bar obvious changes due to the in-line design of the HLV – the experiences of the Shuttle Program are notably apt.
However, there will be differences between the Shuttle and SLS hardware, specific to what was the GUCP on the ET. Firstly, the ongoing design process has moved further away from a direct match with the Shuttle ET GUCP, instead opting for a Core Stage Inter-Tank Umbilical (CSITU).
According to the Ground Systems Development and Operations presentation (available on L2), “Final cost and schedule impacts to … baseline for change from ET Vent Line reuse to new swing arm umbilical” are pending final approval.
The new design will provide commodity services to the SLS’s Core Stage Inter-tank region. The umbilical arm will be 45 feet in length, 8 feet in width, and 15 feet in height.
An added benefit will come via the umbilical and ground carrier plates being mated in the VAB – due to no pad access for umbilical mating – allowing them to remain connected to the vehicle until liftoff. This will help avoid any problems that can occur with mating the hardware at the pad.
For previous SLS Articles, click here: http://www.nasaspaceflight.com/tag/hlv/
The umbilical plate size is now expected to be 34 inches x 54 inches, although it will still consist of a seal at its heart as the protective element against scrub-causing leaks.
This is where a team effort – between NASA Engineering (NE) and Team QNA Engineering personnel – is working towards a goal of providing support for the Umbilical Systems Development project, which is funded by Advanced Exploration Systems (AES) and 21st Century Launch Complex.
Reviewing the Shuttle issues with the GUCP, and consulting with the SLS team for their recommendations, an associated presentation noted that several “lesson’s learned” will be implemented for the HLV.
These recommendations included the development of a concentricity tool, to help alignment during assembly of flight side components. The use of Parallelism Retainer Clips, Ground Support Equipment bolts and clips that will maintain parallelism at the seal face during assembly of QD to launch vehicle. And the use of a Self-Aligning Probe, to accurately guide and centre the QD and the tank vent as they are brought together during the integration flow.
They also reviewed the old two piece seal designs, proposing three new alternative seal designs – to be down-selected after more detailed tests/analysis.
The end result of these early evaluations will hopefully avoid launch fans having to head back home after a launch day scrub caused by the detection of unacceptable levels of leaking gaseous hydrogen from a small vent on the side of the vehicle.
(Images: Via L2 content from L2′s GUCP section and L2′s SLS specific L2 section, which includes, presentations, videos, graphics and internal – interactive with actual SLS engineers – updates on the SLS and HLV, available on no other site. Other image via NASA)
(L2 is – as it has been for the past several years – providing full exclusive SLS and Exploration Planning coverage. To join L2, click here: http://www.nasaspaceflight.com/l2/)