STS-131: Special PRCB on RRCS notes split opinion on flight rationale

by Chris Bergin

Friday’s Agency Flight Readiness Review (FRR) will have several dissenters pitching against flight rationale for STS-131’s launch, relating to the faulty helium isolation valve on the Right Reaction Control System (RRCS). While dissent is not uncommon – and more so welcomed – the Special Program Requirements Control Board (PRCB) meeting still has strong support for launch via the powerful Orbiter Project Office (OPO).

STS-131 Processing Latest:

Discovery continues to be processed for an April 5 launch – pending approval at the Agency FRR – with the milestone of final ordnance installation, and preparations for the opening of her Payload Bay Doors (PLBDs) for the installation of the payload being worked.

“Orbiter Mid Body Umbilical Unit mate and leak checks were completed yesterday. Final ordnance installation and connection for flight is complete. Vehicle power up ordnance testing is in work and expected to complete today,” noted NASA Test Director (NTD) processing information on Tuesday (L2).

“Payload bay doors will be opened today in preparation for payload installation into the Orbiter tomorrow. Aft compartment closeout for flight continues with completion planned Friday. Four waves of LOX tankers were offloaded at the LOX storage area yesterday.”

Engineers also noted that a cracked Thermal Protection System (TPS) tile was noticed during the inspection of some of Discovery’s Window Inserts – with a Material Review (MR) being conducted at time of publishing.

“Window ceramic inserts inspection/pull tests are complete and good. MR options are being explored for the cracked tile found during the inspection/pull tests,” added the NTD report.

Three new IPRs (Interim Problem Reports) have been added to Discovery’s flow, although none are deemed to be a problem at this time. Meanwhile, the closure of the PCMMU (Pulse Code Modulator Master Unit) issue on the orbiter is expected this week.

“IPR 0029 Update: Troubleshooting/monitoring continued yesterday with no anomalies noted; instrumentation was removed last night to support ordnance installation. Engineering will recommend launching on PCMMU#2 and not replacing PCMMU#1 to PRCB,” noted the NTD, with the latest Shuttle Standup/Integration report (L2) adding their status via the OPO.

“On the PCMMU work done over the weekend, it still has not repeated again. A bus sniffer has been added onto the bus. Will continue to look as long as there is access. Think there is adequate Flight Rationale even without repeats. Will bring that story forward in a kind of continuation of the PRCB.”

STS-131 Specific Articles:

Flight rationale is not being classed as a problem for the SRB (Solid Rocket Motor) Rate Gyro Assembles (RGA) – as presented to the Special PRCB today (article will summarize this week), leaving just the one major item of debate outstanding for the FRR.

This came via the Special PRCB’s latest on the RRCS helium isolation valve, which dominated the meeting via an additional five presentations (all 8 Special PRCB presentations are available on L2).

Special PRCB RRCS Debate:

Outlining the issue which was observed during hypergolic loading at the pad, the Special PRCB allowed for a specific debate ahead of the FRR, in order to have a good understanding of progress that has been made ahead of the flight rationale decision.

“There were more issues post-SSP (Space Shuttle Program) FRR, pre-Agency FRR than we have had in awhile,” wrote SSP Manager John Shannon on the Standup report, pre-empting the Special PRCB meeting. “Was a tremendous job by the team on the RCS Helium ISO troubleshooting. Doing all the right testing, and showed that the rest of the system is healthy.

“There are many reviews, but (want) to just get the team together all in one spot and talk about these new issues before going into the Agency FRR. The Special PRCB will be very helpful to us. Otherwise, things seem to be going okay.”

The positive note on the valve troubleshooting was in reference to the weekend test on the redundancy in the RRCS – specifically two regulators on each path. The weekend test put the system through its paces to the equivalent of seven thruster firings, and found no unacceptable decay/leaks from the regulators.

“He iso valve and regulator flow test at 150 SCFM (equiv to 7 thrusters firing) performed (3/20-21/10). System pressurized to ~1800 psi. Each leg flowed to less than regulator lock-up. Target flow rate achieved – no evidence of any flow obstruction,” noted a 36 page review and update presentation to the Special PRCB.

“System repressurized to ~1800 psi. Both legs flowed (together) to primary regulator lock-up. Primary regs monitored for any creep. No creep (leak) detected.”

Problem was known as far back as last year:

Notably, the main presentation also revealed engineers were aware of a problem with the RRCS helium iso valve back in October of last year. Unfortunately, the problem was incorrectly documented, as the issue failed to show a more permanent problem until the vehicle was put through it paces at the pad.

“Review of data collected during STS-131 He System Checkouts in 10/2009 revealed an RRCS fuel A leg He iso valve issue. During helium system blowdown (helium tank vent), after venting 1/2 of the helium tank through the A leg, the A leg helium iso valves were commanded closed,” noted documentation.

“The ox helium iso valve operated nominally, but the fuel helium iso valve did not. The Open VPI (Valve Position Indicator) remained ON and Closed VPI remained OFF. There was also gas flow through the valve. Helium iso valves were cycled three more times with the same results.

“On the fifth attempt to close the valves, the fuel helium iso indicated Closed (Open VPI OFF/Closed VPI ON), but lagged the ox helium iso valve by 2 seconds. The valves were cycled open/closed again with the VPI’s indicating nominal operation (no lag).

“During the subsequent operation the A leg helium iso valves were cycled Open/Closed again with the VPI’s indicating nominal operation. After the first indication of a failure where gas flow through the valve occurred, there was no situation where the valve position was verified other than by VPI indication until the pad (no pressure differential across the valve).

“Likely, the valve stuck in the open position during the tank vent. After a few cycles it moved enough to get nominal VPI operation, but was stuck partially open where it is now. The problem was detected, but mistakenly assumed to be a known and documented problem.”

“VPI problems have occasionally occurred when the iso valves are cycled during tank vents. FRCS Fuel A-Leg He Iso Valve (was also seen to be) slow to show closed during STS-119 post-flight valve test.”

Flight Rationale Debate:

Given it has been known since hypergolic loading that the valve is a problem, and that the mission could still receive the required flight rationale via the redundancy of the regulators, flight rationale should expect approval at the FRR.

“Regulators are healthy, unlikely for dual failure. Operational impacts and management of potential subsequent anomalies in the RRCS helium pressurization system issues to be manageable in flight. All other RCS Helium components (left & right pod) showing nominal, in-spec performance,” noted an outline of OPO’s take on approving flight rationale.

“Nominal regulator performance on RP03 (Endeavour’s hardware) demonstrated in OPF (Orbiter Processing Facility) and past weeks’ testing. Mechanism causing helium isolation valve failure will not likely; propagate into downstream regulator failure during flight, block orifice/regulator filter causing loss of redundancy (leg), result in failed-closed isolation valve.

“Orbiter Project Office recommends fly-as-is for STS-131.”

However, numerous departments are dissenting flight rationale, based on a lack of a root cause for the helium valve’s issues, and what is being classed as a new failure signature. All of the dissenters aren’t listed in the Special PRCB documentation, although both NASA Safety and Boeing Safety are currently heading into the FRR with a recommendation to changing out the valve.

“Some dissenting opinions recommending changing valve prior to flight. Reason for opinion: New failure signature. Root cause not identified. Dissenting opinions held by Boeing Safety, NASA Safety (SAIC) PRT representative and several PRT SSE’s.”

Should the FRR side with such a dissenting opinion, a rollback would be the outcome, as the valve cannot be worked on at the pad.

Moreover, Discovery would have to be demated from the External Tank, returned to her OPF, and the Right OMS Pod removed and sent to the Hypergolic Maintenance Facility (HMF) for around three weeks of work. At the same time, a donated Right OMS Pod from Endeavour would be installed on Discovery, resulting in a total delay to STS-131 of around three months.

However, it is very important to note that dissenting opinion is not unusual even during the FRR meeting itself, where it is debated and evaluated prior to a decision. More often than not, dissenting opinion is based on an overly conservative stance, and as such is welcomed, as those in favor of flight rationale are pushed harder to prove their acceptance – thus aiding the overall confidence in the safety of the vehicle.

This can already be seen via the documented “forward work”, which will provide the FRR with simulated results of the problem becoming more serious while on orbit.

“Helium tank vent model still in work. Simulates time to vent tank in the event both primary and secondary regulators fail-open and vent through relief-valve and overboard vent. Will also provide estimates of thrust forces this will create. Expect thrust to be approximately that of a vernier thruster. NSLD still in work with tests to validate potential failure modes of RCS Helium Isolation Valve,” added one of the presentation.

“Unless results are more severe or different than expected, neither are a constraint to STS-131.”

It should also be stated that the problem holds no risk to the crew – even in the unlikely event of further downstream failures in the system. The worst case scenario – a major leak throughout the valve systems early into the mission – would see the crew being informed to land Discovery at the next available opportunity (nominal landing site).

Further pre-FRR coverage will be published up to – and including – Friday’s decision.

L2 members : Documentation – from which the above article has quoted snippets – is available in full in the related L2 sections, now over 4500 gbs in size

Related Articles