Bad weather scrubs fifth attempt for STS-127 – GUCP investigation results

by Chris Bergin

Shuttle orbiter Endeavour was again ready to launch to the International Space Station. However, poor weather again put pay to the fifth launch attempt for STS-127, moving launch to Wednesday. Meanwhile, the results of the Ground Umbilical Carrier Plate (GUCP) investigation appear to have been validated via Sunday’s leak-free “near-complete duration” tanking.

STS-127 Fifth Attempt Scrub:

With two delays caused by a leak of Endeavour’s External Tank (ET-131) at the GUCP, followed by two weather related delays Endeavour was running out of time to make her trip to the ISS  – prior to standing down for the Russian Progress launch.

“Launch countdown: Saturday the Mission Management Team (MMT) delayed launch for 24 hours due to Lightning Strikes. Sunday the MMT gave the go for launch but a scrub was called due to weather violation for Return to Landing Site (RTLS). 24 Hour Scrub Turnaround.

“Air to GN2 changeover complete at 0513 EDT. LO2/LH2 final load preps are in work. BDA (Blast Danger Area) clear will pick up at 0630 EDT. Ops/Eng Tanking Briefing is scheduled for 0800 EDT. ET load Weather Briefing 0845 EDT. LO2/LH2 tanking around 0926 EDT (confirmed as 09:36). T-0 is expected at 1851 EDT,” outlined Monday morning processing status.

The weather was once again an issue for Monday’s attempt, with forecasts noting a 60 percent constraint to launch by the time STS-127’s countdown reaches T-0. Also, rain during the launch countdown would be very problematic for an exposed tyvek cover – as noted in attempt 4’s article.

With the delay to Wednesday, the Rotating Service Structure (RSS) will be placed back around the vehicle for a replacement of the cover.

Water intrusion into the thruster holds the threat of freezing during ascent, which could cause problems for its operation on orbit. However, it was the weather in the local area for around T-0 that resulted in another scrub being made official.

Several “funnies” had also been listed at the Mission Evaluation Room (MER) at the Johnson Space Center (JSC), ranging from GSE (Ground Support Equipment).

T-0_Umbilical_Slush2aThe main notes on the MER list related to findings by the Ice Inspection Team – which had arrived at the pad to look at all areas of the shuttle after tanking had entered stable replenish.

“TPE 127V-0134, During ICE Team inspections, 3” x 3” x 1/2” ice buildup on the bottom of the LH2 T-0 umbilical,” noted the MER funny, acquired by L2. “ETM 127V-0135, During ICE Team inspections, frost with crack on the LH2 Pyro Cannister.

T-0 Umbilical Ice – Constraints both S0007.200 OP 16-1151 coming out T-9 minute hold.”

During the hold ahead of T-9 minutes, a waiver was issued to allow the vehicle to proceed to launch.

Another issue related to a Freon (coolant) loop in Pad 39A’s Ground Coolant Loop Resevoir Level. However, this is a known condition, as an existing GSE PR (Problem Report) on the freon leak was noted.

A problem with a chiller pump at the pad was also not going to be a cause any problems for the remainder of the count.

While Wednesday was originally unavailable due to the Progress launch, NASA had appeared to have used an option to negotiate the extension of STS-127’s window by a day with the Russians, and/or reduce mission content in order to have a final opportunity available on Thursday – if approved by the MMT.

GUCP Status (images to be added shortly):

One benefit of Sunday’s – and indeed Monday’s – scrub was the confirmation the GUCP troubleshooting plan appears to have worked, with no leaks detected throughout the countdown, up to the point of polling to come out of the T-9 minute hold.

For all 19 articles on the GUCP, click here:

T1This provided additional confidence – following the tanking test at the start of the month – that the engineering work on correcting what was an alignment issue of just 0.357 degrees in the counter-clockwise direction, has been successful.

The amount of work that went into fixing the issue can be seen in a 47 page presentation to the all-powerful Program Requirements Control Board (PRCB), dated July 7 and acquired by L2.

“Present the GUCP GH2 leak fault tree status, IPR (Interim Problem Report) closure (STS-119 and STS-127), and results of root cause assessment including affected materials, process/procedure/technique changes, and other associated relevant data. Present results to the PRCB,” opened the presentation.

“The objective of the overview – to identify the configuration that allows us to proceed to a launch attempt with very high confidence of success.”

Engineers were tasked with checking historical issues with the hardware, in order to attempt a root cause analysis of not only ET-131’s leak during STS-127’s opening two countdowns, but the leak on the opening launch attempt of STS-119 with ET-127.

T5They were also asked to present the worst case outcome of the potential scenarios surrounding a problem with the GUCP hardware.

“Identified 21 scenarios using inputs from community, new fault tree, timelines. Collected evidence to support/refute each scenario,” added the presentation. “11 scenarios are fully (4) or partially (7) mitigated by the actions taken. Evidence reviewed by team ruled out 10 scenarios.

“Worst case effect/risk vary with individual scenario: Could result in an improper function of the vent line (leakage of flammable hydrogen external to the ET), resulting in a launch scrub.

“Premature release of GUCA (Ground Umbilical Carrier Assembly) resulting in improper vent line operation or inability to perform functions (nominal or on-pad abort configuration). Improper release / binding after pyro bolt separation.”

Following an extensive review, engineers confirmed the misalignment was to blame. However, the a flight seal issue – since replaced with a “more forgiving” two-part seal – may have also contributed.

T2“Root cause: plate misalignment resulted in gapping at flight seal/bellows probe interface. Contributors: As-built flight hardware misalignment ETCA & hinge pin brackets. Insufficient controls during assembly to account for off-nominal ET geometry.

“Measurements, Alignment pins, Flight/ground plate relative motion (lateral) during assembly. Reduced capability to accommodate motion at interface during operations due to stiffer Inconel bellows.

“Unexplained Anomaly, possible contributors include: Flight seal defects and/or damage during assembly. Potential plate misalignment.”

The solution, which included replacement of the aforementioned seal, involved new feet being installed on the GUCP, to avoid misalignment potential out at the pad.

“Leak mitigation: Tighter tolerance alignment pins (0.515”). Tailored GUCP feet (0.180” & 0.230” offset). analysis shows adequate strength. Hinge pin washers restrain GUCP lateral motion,” the presentation noted, which was backed up by the data gained via ET-131’s tanking test.

“2-piece flight seal has greater resiliency and provides additional capability for misalignment. 2-piece seal tested to 0.050”. Concentricity and other measurements during assembly show minimal motion of GUCP. Successful tanking test (50 ppm GH2). Note -successful tanking has never been followed by a leak resulting in a scrub. Tanking test observations show minimal motion of GUCP feet.”

The modifications noted will now be implemented on ET-132 (for STS-128) and all downstream tanks that are yet to arrive at the Kennedy Space Center (KSC) from their manufacturing base – the Michoud Assembly Facility (MAF) in New Orleans.

“Processing plan for ET-132 & subs. Apply ET-131 lessons learned. Assembly using tighter tolerance guide pins. Install 2-piece flight seal. Install washers on hinge pins. Increased measurements during assembly.”

T4Although this forward plan included flight rationale to the Mission Management Team (MMT) for STS-127’s tanking and launch, the investigation will continue, given the findings do not provide a clear root cause for STS-119/ET-127’s leak.

This will include further work at the test facility at the Marshall Space Flight Center (MSFC).

“Working theory supported by measurements. Hydrogen leak occurred between flight seal and ground QD (Quick Disconnect). Unique ET-131 umbilical geometry was a contributor to the leak,” the investigation noted.

“Mitigations implemented for ET-131: Fabricated new GUCP feet “tailored” to ET-131 geometry. Utilized tighter tolerance guide pins during assembly. Installed hinge pin washers to limit lateral motion. Installed 2-piece flight seal.

“ET-131 mitigations do not invalidate certification or hazard controls. Continue investigation to determine the sensitivity of this interface to off-nominal geometry for future tanks, other contributing factors, and finalize long-term changes to preclude leaks.

“Lack of root cause for STS-119 and partially mitigated failures scenarios demonstrate some residual leak risk still exists. Recommend MMT action closure.”

T3However, images of ET-127’s flight seal – which was replaced with a similar spare for STS-119 – shows the leak was likely caused by a faulty seal, with several anomalies observed on its edges.

“ET-127 Flight Seal Findings: Multiple dents, gouges, scratches and flattened areas were noted around circumference of seal face and edge. Abnormal gouge patterns observed in between 6:00 -7:00,” the presentation outlined.

“Seal edge inspections showed that the compression band formed on the seal edge by bellows probe contact was sporadically inconsistent around the entire circumference of the seal. Areas of wide, rolled, jagged, thin & wavy contact were all observed.

“Dents and gouges were also evident around the entire circumference of the seal edge.”

Given ET-131’s first flight seal only suffered from a few rolled edges – likely caused by the misalignment – ET-127’s leak is probably related to a specific fault with its flight seal, which in turn adds confidence ET-131’s issue was mainly via the now-solved misalignment problem.

For live updates of launch attempt 5 – refer to the live update pages, linked above.

L2 members: Documentation – from which the above article has quoted snippets – is available in full in the related L2 sections, now over 4000 gbs in size.

Related Articles