STS-133: Discovery closing in on launch day – MEC investigation review

by Chris Bergin

Engineers have successfully cleared all of Discovery’s minor issues, allowing the veteran shuttle to head towards the upcoming key countdown milestones. While all eyes will be on her External Tank (ET-137) on Thursday, previous issues during November’s countdown – such as the Main Engine Controller (MEC) problem – have not reoccurred.

STS-133 Latest:

Discovery has been charged with a number of new Interim Problem Reports (IPRs) this week, although none of them provided a challenge to her talented team of engineers – who resolved all of the issues to allow S0007 (Launch Countdown) operations to proceed unhindered.

Flight history and engineering experience was also put to task on one of the issues, relating to a small leak on a regulator valve during pressurization of the Reaction Control System (RCS).

“IPR 0092. During press to flight mass the right RCS primary fuel regulator creep was 3300 SCCH and should have been less than 360 SCCH,” noted the NASA Test Director (NTD) report (L2). “Will discuss the anomaly and consider a waiver (based on historic precedent). The constraint was assigned to S0007.200. Currently the A Leg is closed and the B leg is configured open.”

A waiver was finally issued on Wednesday at the L-1 Mission Management Team (MMT) meeting, which also provided the expected news that all systems were go for Thursday’s launch attempt.

The three most recent IPRs listed on Wednesday mainly related to cosmetic observations on the vehicle, very minor in nature but a sign of just how seriously the engineering teams take even the slightest out of spec element on the stack.

STS-133 Specific – Including ET Stringer Issue – Articles:

“IPR 0096 Leak Detector 12 failing to lower limit. Troubleshooting found an intermittent connection and the entire junction box was replaced. The sensor was recalibrated, no further work is required,” added the NTD report.

“IPR 0097 documented a sheen seen on SSME 2 (Space Shuttle Main Engine) engine bell. It was suspected to be a few drops of hydraulic fluid from an earlier hose demate and was cleaned off. IPR to close.

“IPR 0098 documented a bowed rib on a bracket of secondary structure in Av Bay 4 and 5. The paper was picked up for documentation and rework is not required. The IPR will be MR’d (Material Review) to fly as is.”

With all of the issues quickly resolved, Discovery’s milestones picked up at a pace, as the veteran shuttle heads towards Thursday morning’s tanking.

“OV-103/SRB BI-144/RSRM 112/ET-137 (Pad A): The S0007 countdown continue to progress well with no significant issues. The weather is forecast to be favorable for flight. A summary of the major events are; SRB (Solid Rocket Booster) PIC Test and S&A rotation tests were completed successfully,” listed the NTD report.

“PRSD (Power Reactant Storage and Distributation) system was loaded with O2 and H2 for the mission. OMBUU (Orbiter Mid Body Umbilical Unit) was closed out for flight. LO2 and LH2 preps for vehicle loading are complete. FRCS (Forward Reaction Control System) TYVEK cover final inspection is complete. Work to prepare the pad for closeout continues.

“Through Thursday the remainder of the preps for vehicle flight will be completed. The RSS (Rotating Service Structure) will be rotated at 1930 EST (Potentially 8pm) and clear the pad of non essential personnel at 0225 Thursday morning.

“Wednesday, the ET Cryo Load is planned to begin at 0725 EST. Launch is planned for a window open time of 1645 EST Thursday.”

(STS-133’s launch day article will be published at the time of tanking on Thursday).

Main Engine Controller Investigation:

Before problems with the tank resulted in Discovery’s stay of execution into 2011, the original launch date in November was slipped slightly due to discussions relating to problems with the redundant Main Engine Controller (MEC) on SSME 3.

“IPR 58 to CCME (SSME Controller Engineer Console): While performing the bus redundancy test on the main engine controllers, CCME reported Engine 1 and 2 performed nominally, however, Engine #3 redundant controller did not activate,” the NTD outlined at the time.

“CCME was expecting approximately 0.5 amp increase on all three phases. Data retrievals showed a very small current spike on Phase A and C, but nothing on Phase B. The controller has a built in ‘safe’ mode that when a phase is missing, it will shut itself down.”

The SSME controllers provides complete and continuous monitoring and control of engine operation. In addition, it performs maintenance and start preparation checks, and collects data for historical and maintenance purposes. The controller is an electronic package that contains five major sections; power supply section, input electronics section, output electronics sections, computer interface section, and digital computer unit.

Pressure, temperature, pump speed, flowrate, and position sensors supply the input signals. Output signals operate spark igniters, solenoid valves, and hydraulic actuators. The controller is dual redundant, which gives it normal, fail-operate, and fail-safe operational mode capability. The problem was specific to the redundant controller on ME-3.

“The CCME decided to cycle the Phase B circuit breaker to determine if there was any contamination in the breaker that could cause this problem,” the NTD report continued. “Prior to starting the troubleshooting CCME noticed the redundant controller had been activated without any action by engineering or the SCO (Spacecraft Operator) in the crew module.

“Data retrievals showed the switch scan for the controller cycled on/off on its own, the same signature on Phase A and Phase C were noticed and then shortly after that Phase B came up which ultimately activated the redundant controller.”

Interim troubleshooting involved the cycling of the Panel L-4 circuit breakers five times, resulting in the MECs powering up with no anomalies documented. This was carried out to attempt to remove any potential contamination underneath the switch, which may have caused the inconsistent readings.

The Engineering Review Board (ERB) then presented their data to the MMT for evaluation. Launch was delayed by one day to allow the engineering teams to debate the forward plan without being placed under schedule pressure.

With that allowance for the engineering community to collate their data for presentation at Wednesday’s MMT, an additional overnight test was conducted on the suspect redundant controller – which provided encouraging results, as no repeat of the original issue.

This allowed for the MMT to gain the required flight rationale to proceed towards the launch attempt. However, given the subsequent problems with ET-137 scrubbed the launch until February, the redundant MEC’s circuit breakers (CBs) have since been replaced while Discovery waited for repairs to be carried out on the tank.

The subsequent investigation report – aided by removal of MEC hardware from Discovery – has since been represented to Space Shuttle Program (SSP) managers.

“L-1 Status Recap: During S0007 SSME Controller power up, controller 3/B (engine 3, B side) did not power up until 1 hour, 45 minutes after activation. Subsequently a 5 volt transient/dip was noted on controller composite voltage measurement,” opened the expansive presentation – available on L2.

“Nominal power up was achieved after circuit breaker (CB) cycling. Fly ‘As Is’ rationale was accepted (by MMT – after review of:) history of CB internal debris. CB continuity regained with cycling.

“Stable controller performance after cycling. Power up signature was consistent with high resistance contact in CB. Possible explanation for voltage dip. Launch scrub provided time for additional work to occur”

Actions taken during Discovery’s delay included the installation of a breakout box and the testing of three single phase circuit breakers for SSMEC 3B on Panel L4.

This panel was also removed for testing. Vehicle wiring that could be at cause for the anomaly was also tested, although this inspection is limited by access, while engineers pro-actively replaced all 18 SSMEC circuit breakers at the recommendation of management.

The problem soon became clear when CB 109 was inspected, with a clear observation of non-conductive debris on the hardware, a key candidate for the original problem seen with SSME 3’s redundant MEC.

“Circuit Breaker 109 Failure Analysis Prelim Results. Confirmed non-conductive debris present,” noted the presentation, showing images of the observation. “Observed a build up of the debris at the contact patch area of the contacts. Results are consistent with prior CB failure analyses.”

This allowed engineers to provide their conclusion that the presence of non-conductive material, an internally generated debris condition confirmed in CB 109, was the root cause of the SSMEC power up condition.

The investigation team also noted they believe that previously enunciated flight rationale is still valid, with no dissenting opinions on that conclusion, along with the bonus that the removed hardware has since increased confidence. Indeed, the MECs have performed flawlessly throughout the latest pad flow.

Concluding the investigation, the lead member of the inspection team elaborated on their findings, which in turn updated the resulting Delta Flight Readiness Review (FRR) documentation.

“Failure analysis of CB-109 is complete. The anomalous operation of the Phase B circuit breaker was due to non-conductive debris build-up on the ‘fixed’ ‘B’ contact.

“There is evidence of debris particles throughout the interior of this circuit breaker. This is a known and well understood issue with these circuit breakers whereby a fine glass and epoxy particulate is generated by repeated operation of the circuit breaker,” added the notes on the presentation.

“This particulate becomes compacted at the contact points between the fixed and movable breaker contacts. It is particularly prone to collect in the microscopic “craters” caused by contact arcing.

“Images show a large mound of this material at the single contact-point for the ‘B’ Contact set. This contact set was particularly vulnerable because the contact pair is not parallel (this is not an infrequent condition) and only made contact at one end of the contact pair allowing the non-conductive debris to easily disrupt current flow.

“Due to the ‘plastic’ nature of the glass and epoxy mixture, the compressive force at this contact point would eventually ‘extrude’ this material from between that contact point allowing electrical current to flow, but it would have likely been through a somewhat high resistance due to that current flowing through very small contact ‘points’ created by the arcing at this location. The arcing is normal.

“Images confirm the presence of a large quantity of non-conductive, organic (both of these characteristics were determined by Energy Dispersive X-ray Spectroscopy) debris compacted on the ‘fixed’ ‘B’ contact in the circuit breaker.”

The conclusions were signed off at the Delta FRR and lessons learned were applied to both Endeavour and Atlantis.

(Numerous articles will follow. L2 members refer to STS-133 live coverage sections for internal coverage, presentations, images and and updates from engineers and managers – which will ramp up into full Flight Day coverage during the mission. Images used, Lead: Larry Sullivan (MaxQ Ent, Rest via L2).

Related Articles