Understanding the threats – FCV flight rationale effort outlined

by Chris Bergin

Discovery’s processing flow at Pad 39A is continuing in part, in the hope the realigned NET (No Earlier Than) launch date of February 19 for STS-119 will not be further delayed. Minutes from the Agency level Flight Readiness Review (FRR) point to a huge amount of work being undertaken in order to achieve the requested flight rationale to fly with the recently re-installed Flow Control Valves (FCV).


Pad Flow Latest:

While the flow had continued to hold at the point of completing final ordinance Installation, closeouts of the vehicle are progressing – in preparation for what is hoped to be a firm launch date from the Space Shuttle Program (SSP) next week.

“At the Agency FRR, the decision was made to delay the launch to no earlier than February 19, 2009 to allow for further development of flight rationale for the MPS GH2 flow control valve issue,” noted processing information on L2.

“Particle impact testing, model analysis and discussion of flight rationale will be reviewed at a Special PRCB (Program Requirements Control Board) meeting on February 10, with a delta FRR expected on February 12, 2009.”

Final ordinance installation was initially on hold due to the requirement to carry out a repair on a pyro wire in the aft compartment of Discovery. However, by the second shift on Wednesday, the repair had been completed.

Engineers are awaiting instruction on whether to continue to hold with ordinance installation, pending the realignment of the launch date.

“S5009, Final Ordnance Installation/Connection, remains ‘on hold’ until the damaged Pyro wire in the aft compartment is repaired,” noted processing information on L2, which was updated during the day. “Damaged Pyro wire, update: As technicians began repair of the damaged wire, additional damage was noted 3 feet aft of the connector. Engineering is evaluating and initial repairs are in work.

“Update (2) Continuity, isolation and hipot testing of repaired pyro cable complete. All results good. Total resistance measurement of R/H Umb Sys B pyro circuit from MEC 2 connector to 50J828 calculated out to be 0.20 ohms and within prescribed limits.”

One new issue was also noted on the processing information, relating to the discovery of a bent line in the aft compartment. An update on any potential need to repair will be forthcoming later in the week.

“During inspection of the aft compartment, MPS (Main Propulsion System) engineering found a bend in the LO2 sense line. Disposition calls for mold impressions to be taken of the area and have dye penetration performed on the sense line.”

Agency FRR:

Tuesday’s Agency level FRR had managed to clear all outstanding issues – pending the “completion of open work” – bar the need for a greatly understanding of risk acceptance on the FCVs.

Click here for NASASpaceflight.com articles on the FCV issue since STS-126.

Interestingly, it was well known before the FRR that flight rationale was still being worked on, which led to the expectation that the meeting would clear February 12 for the launch of STS-119, pending the further meetings on the status of the risk acceptance findings in the days leading up to launch.

The amount of work required to get to that stage of acceptance ended the chances of making February 12, which in turn has raised concerns that February 19 may also be too tight on the schedule.

A number of engineers close to the situation believe the process may take up to another month, based on the requirements that came out of the Agency FRR.

“As expected, the most significant topic was the MPS flow control valves. The leading theory for most probable cause is an acoustic environment frequency which is coupling with the structural mode of the poppet to produce crack initiation and fracture,” noted an expansive overview/summary/minutes of the STS-119 Agency FRR on L2.

“There are two hazards associated with poppet lip failure, which include H2 venting within window of flammability concern (up to about MET 2 minutes) and critical damage due to particle impact.”

Flight rationale will be built in two stages. Firstly, in order to protect STS-119’s February window, rationale for one flight will be built. The second stage will work on completing flight rationale for all three orbiters and their downstream missions.

“Significant progress on this issue has been made but there were still enough unknowns that the risk to be accepted was not well understood,” added the summary. “The short term goal is to develop one flight rationale to support launch of STS-119.”

Initial FCV Concern (STS-126):

The issue with the FCVs first came to light during STS-126’s launch – believed to be the first time in shuttle history that the valve was deemed to be responsible for an anomaly. Notes also relate to two events with qualification valves.

The FCV relates to the Main Propulsion System (MPS) and the SSMEs (Space Shuttle Main Engines) during thrusting periods. In each SSME, gaseous hydrogen from the low-pressure fuel turbopump is directed through two check valves to two orifices and a flow control valve for each engine.

During the main engine thrusting period, the liquid hydrogen tank’s pressure is maintained between 32 and 34 psia by the orifices and the action of the flow control valve from each SSME. The flow control valve is controlled by one of three liquid hydrogen pressure transducers. When tank pressure decreases below 32 psia, the valve opens; and when tank pressure increases to 33 psia, the valve closes.

If the tank pressure is greater than 35 psia, the pressure is relieved through the liquid hydrogen tank’s vent and relief valve. If the pressure falls below 32 psia, the LH2 ullage press switch on panel R2 is positioned from auto to open , which will cause all three flow control valves to go to full open and remain in the full-open position.

An issue with one of the FCVs was observed on Endeavour during STS-126 launch, leading to the examination of Endeavour’s three FCVs, once she returned into her Orbiter Processing Facility (OPF) to determine if they were the cause of the anomaly – or if the problem was with Endeavour’s wiring.

“During ascent and after the thrust bucket, the E2 GH2 outlet pressure had a 200 psi step down without a corresponding ullage pressure signal conditioner command change,” notes the Orbiter and GFE IFA review presentation available on L2. “The delta outlet pressure for this FCV position was 260 psi during previous command changes before the thrust bucket.”

Potential Threats:

While the anomaly did not have an impact on ascent performance, simply resulting in an improper MPS configuration during SSME operation, any failure of the GH2 FCV is a 1R/2 condition – which is why the FRR are taking the issue so seriously, due to the potential risk of a LOV/C (Loss Of Vehicle/Crew) event.

One such LOV/C risk relates to a flammability event of O2 venting and setting the TPS (Thermal Protection System) foam on the External Tank on fire during first stage ascent. Assessments are taking place into this threat relating to serious FCV failures.

However, the current key concern – that involves outstanding work for the flight rationale – relates to a tiny crack and a loss of part of the valve observed on one of the FCVs from STS-126/Endeavour. Examinations have been made at the vendor “Vacco”, along with several NASA centers brought in to help with the investigation.

“MPS Flow Control Valve: During STS-126 ascent phase, engine #2 flow control valve appeared to transition from low towards high flow position without being commanded to do so. Post-flight inspection of this valve showed around 87 degrees of the lip of the poppet had broken off. Evaluation concluded the failure was caused by high cycle fatigue. No evidence of material defect or impact event.

“Orbiter Project concluded that hydrogen embrittlement was not a significant contributor. However, there were multiple questions on hydrogen embrittlement. MSFC (Marshall Space Flight Center) is initiating a study on growth propagation in hydrogen environment utilizing test coupons to understand this effect better.

“In addition to the STS-126 failure, two poppet failures occurred in 1990 during ATP GH2 flow test. These failures occurred early in the test. Fractology of these two failures found indications of fatigue but failed primarily by overload.

“They don’t know if same part of the circumference failed on these two as for STS-126 since that information was not recorded. They had concerns with the vendor, so they went to a different vendor, changed heat treatment, and implemented inspection requirements.”

Should a valve crack – under high cycle pressures – the liberated piece of the valve could damage systems along the pressure line between the SSMEs and the External Tank, and it could also prove to be a flammability event – both are a LOV/C threat.

Building Flight Rationale:

The crack was found on the lip area of the valve called the poppet, which is the threat area for the liberation. The FRR want to know if there are potential “common causes” that created the crack, and exactly what they could expect, should a piece liberate into the propulsion systems.

“Two hazards associated with poppet lip failure include H2 venting within window of flammability concern (up to about MET 2 minutes) and critical damage due to particle impact,” added the Agency FRR notes.

“A single poppet failure can result in venting in the flammability region if the circumference of the fracture is at least around 150 degrees (fracture circumference that will result in venting depends on time of failure). Based on the 3 poppet failures experienced, Orbiter concluded that the size would be limited to the 60-90 degree circumference.

“Engineering stated they could only bound the worst case circumference somewhere between 90 and 180 degrees. To assess particle impact concerns, flow field and particle trajectory analysis has been performed. Since the analysis results are marginal assuming a STS-126 sized piece, a more refined analysis is required. In addition, impact testing is being conducted at various centers to verify the analysis.”

Dates are given for testing results, with one noted as February 9. However, the test “will be later” if “real velocities and real impacts in hydrogen required,” which may in turn be a threat to the NET February 19 target.

So far, tests have shown that the current design FCVs – a total of 33 – that have been examined show no sign of cracking. One old configuration valve – from a batch of 14 – did show signs of a crack, but had flown 16 times.

“SEM (Scanning Electron microscope) inspection provides extremely high resolution (.005 inches in size can be examined). SEM inspection of 11 flown poppets and 22 spare poppets showed no evidence of degradation or crack growth. However, don’t know critical flaw size,” added the notes.

“Also, inspected 14 old configuration poppets (same material as current poppets), one of which showed a very fine crack. This poppet flew sixteen times, two different vehicles, three different locations. At least one flight was flown with a crack A fracture was induced and resulted in surface similar to STS-126 failure.”

Findings so far have come up with a leading theory for most probable cause, deemed to be “acoustic environment frequency, coupling with structural mode of poppet to produce crack initiation and fracture.”

“GO2 flow control valves may also be susceptible to environmentally induced high cycle fatigue but there are key design and flight history differences. No previous failures of a GO2 poppet have occurred. The GO2 poppet material is Monel which is more ductile and tougher than GH2 poppet material,” the assessment for the FRR continued.

“Venting assessment is in work to determine windows of exposure where the concern is O2 venting resulting in ET TPS (Thermal Protection System) ignition. Particle impact and blockage assessments are in work and GO2 spare FCV inspections are in work.”

“Key open work for STS-119 includes bounding the size of the poppet lip that can be liberated, assessing GH2 poppet particle impact concerns, and looking at GO2 side to determine whether there is common cause. There was general agreement that completing this work to better understand the risk to be accepted would not be complete by 2/12.”

Based on the above findings, William “Gerst” Gerstenmaier, Associate Administrator for Space Operations for NASA, is quoted in the minutes as concurring with the decision to move away from setting February 12 as the launch date

“Gerst summary: Overall goal is to get one flight rationale. Should head toward flying OV103 (Discovery) poppets that have been installed. Scanning electron microscope (SEM) inspection shows no cracks great than .005 inches is key piece of rationale.

“Thinks partial crack in the poppet that didn’t propagate to failure is important data but not sure what it’s telling us (potential as flight rationale).

“Need to determine failure bounding condition (90 degrees? 180 degrees?), GOX rationale, ET venting risk (best engineering judgement), bound flammability concern.”

It still remains the case that flight rationale will be approved, but when that will be forthcoming remains fluid.

Further notes from the Agency FRR will be included in upcoming articles.

L2 members: All documentation – from which the above article has quoted snippets – is available in full in the related L2 sections, now over 4000 gbs in size.

Related Articles