Flow Control Valve Investigation reveals probable cause of failure
A one-year long investigation into the liberation of part of a Flow Control Valve (FCV) poppet during Endeavour’s STS-126 launch in November 2008 has concluded with a recommendation to close the official investigation after findings revealed the most probable cause of the STS-126 liberation and subsequent eddy current reading increases following some of the Shuttle missions last year.
Review of the Issue:
During Endeavour’s launch on November 14, 2008, “MPS (Main Propulsion System) engine #2 GH2 (Gaseous Hydrogen) flow control valve (FCV) LV57 appeared to transition from low flow towards high flow position without being commanded to do so,” notes the FCV Poppet Investigation Summary presentation to the Program Requirements Control Board (PRCB) on January 7.
Click here for NASASpaceflight.com articles on the FCV issue since STS-126.
Post-flight inspections of the valve revealed a liberation of ~87-degrees of the poppet head’s circumference.
This finding lead to the immediate inspection of the FCVs throughout the Shuttle fleet and the subsequent grounding of Discovery and the STS-119 mission until the cause of the poppet liberation could be determined or the integrity of the FCVs to set to fly with Discovery verified.
The GH2 FCVs (of which there is one per SSME [Space Shuttle Main Engine]), are designed to provide each engine with the proper amount of pressurized GH2 from the ET LH2 tank while maintaining the ET LH2 tank flight pressurization limits.
The FCVs are “Commanded ON/OFF by corresponding ullage pressure signal conditioner. ON (low flow) when ET ullage pressure exceeds 33.2 psia. OFF (hi flow) when pressure drops below 32.8 psia,” notes the failure summary presentation – available for download on L2.
When the FCVs are commanded “ON,” a Solenoid valve with a “spring return” is energized, causing the poppet on the FCV to close against the spring.
Conversely, when the FCVs are commanded “OFF,” the Solenoid valve is de-energized, causing the poppet to open.
Given the important nature of the GH2 FCVs in maintaining ET LH2 tank pressure during the powered phase of ascent, the primary concerns for a liberation and multiple FCV failures are the over-pressurization of the Liquid Hydrogen tank (resulting in “overboard venting” and the creation of a fire hazard) and Liquid Hydrogen tank under-pressurization (resulting in “External Tank structural failure or low SSME turbo pump Net Positive Suction Pressure”).
Further implications for a liberated FCV poppet head are “downstream component/line damage or blockage and/or line rupture” resulting in the under-pressurization of the External Tank and corresponding venting and fire hazards in the aft fuselage of the Orbiter.
Review of Accepted Flight Rationale for STS-119 and Subsequent Flights:
In particular, the review of all FCV information revealed that the overall likelihood of a FCV poppet failure in flight is low – especially since all FCVs post-STS-126 are inspected via Non Destructive Evaluation (NDE) eddy current inspections.
Furthermore, the extensive flight history behind the FCVs allowed engineers to conclude that any non-detected cracks (after eddy current inspections) in a FCV poppet were unlikely to grow to the point of failure (liberation) in a single ascent.
This flight history, combined with fractography and fracture analysis “suggests that small cracks would likely require some number of flights to grow to failure due to HCF (High Cycle Fatigue) and H2 (Hydrogen) assisted static loading environment,” notes the failure summary analysis.
Still, the possibility – though extremely low – exists that a resonance between the vehicle and the FCV poppet could exacerbate a non-detected crack and accelerate poppet failure in a single flight.
Should a non-detected crack in a FCV poppet lead to a single-flight failure, experts believe that – at most – only 125-degrees of the circumference of the poppet would liberate.
“Bounding analysis based on engineering judgment of fracture experts predicts a maximum particle release of 125 degrees.”
Nonetheless, part of the failure analysis looked at the various possible failure modes.
“ET venting analysis shows no ET venting for a single poppet failure (up to 170 degrees of the circumference) even if that failure occurs as early as engine start. Venting prior to 120 seconds requires two 125 degree poppet failures in the first 90 seconds” of flight.
Additional confidence was gained via liberation impact testing which showed that, even if the largest possible circumferential area of a GH2 FCV poppet should liberate during ascent, the damage inflicted to the connected GH2 lines would not be large or significant enough to cause an ET under-pressurization event.
“ET under-pressurization analysis shows that the hole size in MPS/ET GH2 line required to underpress the tank is 60 times larger than the biggest damage created during impact testing,” notes the failure analysis report.
Furthermore, the amount of damage necessary to cause the over-pressurization of the ET is eight times (8x) larger than the damage created during the largest possible circumferential area liberation tests.
Moreover, “Orbiter flammability analysis indicates that damage created during impact testing is comparable to that required to exceed concentration limits. (Combustion requires oxygen source and ignition source to also exist).”
Lastly, a Monte Carlo probabilistic risk assessment confirmed the low risk of damage due to a GH2 FCV poppet liberation.
Considerations and Testing:
The initial results of the FCV analysis lead to the use of eddy current inspections to verify the integrity of all FCVs for STS-119.
Since then, every flight has had its FCVs inspected both pre-flight and post-flight, with some FCVs being removed from service due to the signatures returned during their eddy current inspections.
In some cases, the NDEs of the FCVs revealed an increase in the signature denoting the presense of unseen, subsurface cracks – cracks that could propagate during liftoff.
However, in a December 2009 interview with NASASpaceFlight.com, KSC Launch Integration Manager Mike Moses stated that the ongoing investigation into the FCV issue revealed that these NDE inspections and ground testing methods could be contributing to the cracks and increases in post-flight eddy current signatures.
“One of the things the teams are looking at is that the Non Destructive Evaluations might actually be causing some of the problems we’re trying to avoid,” stated Moses. “The teams are off looking at that data and there’s still some work to do there but it appears that our ground testing might actually be causing some of the cracks that can propagate during flight.”
This issue also gained mention in the FCV failure summary.
During ground testing, six un-flown FCVs experienced cracking of their poppets. “Hardware evidence indicates that the ground flow environment induces considerable damage to the poppet radius as shown by the cracked six 0-flight poppets.”
Since a very high alternating stress (54 ksi) is required to initiate a crack in a GH2 FCV poppet (those same stresses would accelerate crack growth in a GH2 environment), the most likely cause of the cracks has been traced to either the ground GN2 (Gaseous Nitrogen) processing or the ground GH2 testing environments.
Testing “points to GN2 processing as the likely source of initiation or a ground GH2 testing environment with higher amplitude responses than those in flight,” notes the failure analysis presentation.
A combination of a Computational Fluid Dynamics (CFD) acoustic model and the “intersection of structural modes, and their presence in the acoustic emission flow test data provides two areas of interest for damage consideration even though the state-of-the-art analyses conducted to date do not predict initiation.”
The first of these areas of interest is the GH2 flight and test environment and the second is the GN2 “full stroke force balance testing at Vacco.
For the GN2 full stroke test environment, “The flared 0361 housing at VACCO increases CFD response by a factor of about 3 to that of a straight housing. The full stroke case traverses a structural mode between 56 and 68 Khz as it seats on the housing.
“Chatter: repeatedly seating and unseating the poppet could have an amplification characteristic/flutter that is not captured in the CFD and dynamic analysis. Inlet conditions and stroke variability may influence responses considerably,” notes the presentation.
Additionally, a GN2 85% full stroke case was found to hold the potential of amplifying stresses on the FCV poppet based on Acoustic Environment data that showed an unexpected “non-symmetric” response “in the pull back-in phase of the flow balance GN2 test.”
This signature could be a sign of a “lock in with a mode and or a flutter response.”
Furthermore, “damping” has a significant effect on the alternating stress amplitude at structural resonance; however, this factor is highly uncertain given the lack of measurements of structural damping in operating conditions.
Results and Recommendations:
In all, the year-long investigation has yielded six recommendations to the Space Shuttle Program, the first of which being an official closure to the investigation with “ground flow induced damage as the most likely source of crack initiation.”
The second recommendation advises the continued use of flight rationale based on NDE inspections and hardware observations (eddy current, Mat Lab review, MPI, and SEM). This was followed by the recommendation to maintain “evaluation of the vast acoustic emission data that has yet to be analyzed.”
The fourth recommendation pertained specifically to STS-130 and advised for the “emphasized attention” to Endeavour’s LV-57 position FCV. “This particular position is the only one of the 1301 family of poppets to exhibit considerable crack growth consecutively (STS-126 failure, and STS-127 zone 3 crack development to .22 inch flaw).”
If this trend continues with STS-130, it is recommended that all housing/valve hardware be replaced.
The final two recommendations pertain to the ground processing side of the FCVs.
Recommendation #5 advocates the implementation of a redesign of the outlet tube housing for the GH2 and GN2 ground testing modes. This would be accomplished in order to remove numerous acoustic frequencies to the noise level and demonstrate the effectiveness of Acoustic Environment sensors.
The final recommendation is to remove or reduce the full stroke shimmed flow balance test, and/or implement effective GH2 and GN2 ground testing controls to “improve repeatability and reduce runtimes.”
In addition, new procurements of poppets and piston seals is currently being worked to support additional 0-flight poppet testing.
Further, all FCVs will continue to be removed after every flight, inspected, cleaned, reassembled, and – if they pass inspection – reinstalled on an orbiter for a future flight.
Therefore, engineers have concluded that most likely cause for the FCV liberation on STS-126 was the result of a constant pressure crack growth with subsequent ground test data from flown and un-flown FCVs revealing that the ground testing procedures are the most likely cause of poppet crack formation.
Because of all the extensive testing, strong Flight Rationale exists for the remaining five Space Shuttle missions, with safety measures in place to ensure that only viable FCVs are flown on those remaining missions.
As Mike Moses stated, “When you go back and look at everything that the teams have done in response to this issue, you can see that the appropriate steps were taken to ensure the highest level of safety. This is a complex vehicle and you can never allow yourself to think that you understand exactly how it works.
“We’ve been flying for almost 30-years now and we learn something new every flight. We thought we had a pretty good handle on the Flow Control Valves. STS-126 proved to us that we needed to take some time, go look at our data, re-evaluate our knowledge of this system, and prove to ourselves that we were – and that we are – safe to fly,” stated Moses.
L2 members: Documentation – from which the above article has quoted snippets – is available in full in the related L2 sections, now over 4000 gbs in size