Expedition 24 Flight Engineers Doug Wheelock and Tracy Caldwell Dyson will undertake two EVAs – on Saturday and Wednesday – slipping twice on the initial schedule – to changeout an ammonia coolant Pump Module on the S1 truss, which is at the center of the Loop A failure on the External Thermal Control System (ETCS) on Sunday, resulting in the shutdown of several systems.
ETCS Loop A failure:
Issues with the ETCS were heard over the loop on Saturday evening, when alarms sounded – waking up the ISS crew on the orbital outpost – due to a circuit breaker trip. Within half an hour, mission controllers updated managers by memo.
“Loop A Shut down: Around 213/00:00 we lost the Loop A of the ETCS system. Currently the team is working balancing the thermal loads and analyzing the cause of the shut down,” noted a running order of memos acquired by L2.
“It is not clear at this time if it is a hybrid FET failure, RPCM (Remote Power Control Module) trip or pump problem. Specialists are either in or on way, MER (Mission Evaluation Room) is here and staffing up.”
The External Thermal Control System (ETCS) provides cooling to the entire United States On-Orbit Segment (USOS). The ETCS consists of two independent loops, designated Loop A and Loop B.
The system is designed to collect heat from the Internal Thermal Control Systems in the US Lab, Node-2, COL, JEM, and Node-3, via IFHXs (Interface Heat Exchanger). The Pump Module (PM) provides the motive force to transport the ammonia around the loop.
Heat is rejected to the space environment as each loop flows through a separate set of rotating radiator assemblies mounted on the S1 (Loop A) and P1 (Loop B) truss segments. The heat rejection capability is 35 kW per loop (70 kW total for the USOS).
The majority of ETCS components are located on the S1 (Loop A) and P1 (Loop B) truss segments. ETCS fluid lines are routed from the various USOS modules through the S0 Truss to the S1/P1 truss segments.
As part of the contingency plan, controllers ensured Loop B was providing the required assistance, as they evaluated options.
“Team is pressing into Loss of Loop A Powerdown. We’re going to get on a stable config on Loop B and then reassess w/ MER if/when to attempt restart of Loop A,” added the next memo. “From 50 Hz dump, the RPC trip on Loop A ETC pump was real. O1 (Orbit 1) team now running the show.”
Tracy Caldwell Dyson provided on-site assistance as controllers ran through the procedures for shutting down numerous systems, in order to reduce the amount of electronics that would require the cooling from the ETCS.
“The team is continuing work Procedure 2.674 Loss of Thermal Loop A Powerdown. The major impacts are 2 CMG’s (Control Moment Gyros 1 and 4), S-Band String 1, GPS String 1, Prime EXT MDM (Multiplexer Demultiplexer), Node 2-1 MDM, P1-1 MDM, P3-1 MDM, SO-1 MDM, S1-1 MDM, S3-1 MDM , STR MDM and several DDCU’s (DC to DC Converter Units) in Node 2. Momentum is currently stable at 31 percent.
“Tracy Caldwell is still awake assisting with the powerdown. The rest of the crew is back in bed.”
Caldwell’s clearly tired – but highly professional – assistance also located a jumper cable, after its original location proved to be a mystery. Her actions also protected the remaining string on the S-Band communication system, as she joked it felt like she was “in a sim” (simulation).
“The team has completed most of the critical actions in ‘2.674 Loss of Thermal Loop A Powerdown – Warn’, the crew was up for their entire sleep period and is just now being put to bed,” summarized a memo sent out in the early hours of Sunday.
“They installed two jumpers to maintain maximum available redundancy, 1 contingency truss jumper to bypass DDCU S01A and maintain significant redundancy to systems especially driven by Sband string 1 heater and ops power (to maintain dual Sband redundancy capability) – good job by PHALCON and OSO to recommend that jumper.”
The crew were allowed to get in a decent sleep period prior to Sunday efforts to ensure the ISS remained in a good configuration. Attempts to restart the motor were unsuccessful, although the pump did show signs of physically working, leading to theories that the fault is a short in the power system.
“The forward plan is to let the crew sleep 6-8 hrs, and we told them that after careful review on the ground we may elect to attempt to repower the loop A pump to see if it can be recovered while they are sleeping (teams are off assessing options at this time given the signature and dump data),” memos continued.
“Systems are in a stable configuration after about 8 hrs of reconfig and safing commanding by the team, including powering down 7 DDCUs (and then repowering S01A), deactivating or reconfiguring downstream loads, reconfiguring ITCS (Internal Thermal Control System), SARJ (Solar Alpha Rotary Joint), and TRRJ (Thermal Radiator Rotary Joint). Necessary loads are being supported by the 2/3 power domains.
“Truly a great job by all, the entire team is doing an outstanding job. A big thank you to the folks called in to support this activity. This is why we sim as we do, and it paid dividends.”
Attention then focused on the need to carry out two EVAs to replace the cause of the problem, the primary ETCS heat transportation component known as the Pump Module (PM), which circulates liquid ammonia at a constant flowrate to a network of coldplates and heat exchangers located on the external trusses and USOS modules, respectively.
The PM contains an accumulator, instrumentation, isolation and relief valves, various heaters, and a Pump and Control Valve Package (PCVP) – which is responsible for regulating the flowrate, filtering impurities to protect loop components, and for controlling temperature.
Wheelock and Caldwell Dyson originally were scheduled to perform a spacewalk to outfit the Russian Zarya module for future robotics work and prepare the station for the installation of a new US permanent multipurpose module.
However, because of the importance of restoring redundancy to the station’s cooling and power systems, Saturday’s spacewalk will be dedicated to the pump module replacement. Another spacewalk is currently scheduled for Wednesday to complete the repairs.
This will be the first time a “Big 14” – the term for a major system on the ISS – Stage EVA will be conducted.
“There will be a team 4 effort spun up to start looking at EVA options for R&R, given a US EVA was already planned next week. There is a spare pump package and it is baselined as a full 2 EVA ‘Big 14’ task which will take a significant amount of work to get everyone prepared to execute,” added another memo (L2).
“The US EVA scheduled for Thursday 8/5/2010 (now delayed) will now be dedicated to a pump module (PM) R&R due to the criticality of restoring redundancy to the ISS. The pump module R&R will be a two EVA task with the second EVA no earlier than 2 days after EVA 1.” This slipped on Monday to a schedule of Friday and Monday for the EVAs – prior to more time being required, slipping the EVAs to Saturday and Wednesday.
The EVA tasks will be complex, with numerous electrical and fluid lines associated with the 780-pound pump module, along with robotics being required to aid the translation of the old and new PMs.
“The SSRMS (Space Station Robotic Manipulator System) will be an integral part of the EVA to assist in pump module translation and details are being worked by the team. A development NBL (Neutral Buoyancy Laboratory) run will be scheduled for Monday to run through the procedures, determine what modifications need to be made, and the feasibility of egress on Thursday (delayed).
“A final NBL run will be scheduled for Tuesday (8/3). The EV Crew (Caldwell/Wheels) had NBL task training on a pump module R&R back in Sept 09 and the MCC team will be sending refresher material on the R&R later today. Clearly if the collective team is not ready to proceed on Thursday the EVA (delayed) will be moved to the right accordingly (as was the case).”
An executive summary then followed, as the plans matured at a pace. Considerations included further robotic assistance, hazards and contingencies for issues during the planned EVAs.
“The crew timeline this week is being scrubbed and will contain the necessary preparation including crew study time for the EVA. The Mobile Transporter (MT) will be moved to work site 2 (WS2) to allow the SSRMS to assist in the PM replacement,” added the summary (L2).
“Two DDCU power up tests with no cooling will be performed to provide data regarding heat up rates and how long they can remain powered for nominal Plasma Contactor Unit (PCU) use to preclude EMU shock hazards and SSRMS contingency support if required.
“Much of the discussion from the meetings involved the EVA timeline details and how to operate during the EVA with next worse failure as well as system configuration details (i.e. SSRMS redundancy). Good news is that even though cooling is not being provided for external DDCU’s on power channels 1 and 4, they can be operated for some amount of time to provide critical redundancy if required.
“The MER is providing an assessment as to how long these will last but the initial thought is around 8 hours or more based on anticipated loads. The failed PM venting plan is being worked and details will be discussed at (further) meetings since this activity could have a major impact to ISS system configurations (i.e. solar arrays) and EV NH3 exposure risk depending on when performed.
“Details regarding solar array positioning for longeron shadowing, PCU redundancy, SSRMS redundancy, and combined EVA/SSRMS operations were also discussed and will be finalized over the next several days.”
Thanks to the speedy and professional work carried out by ground controllers, engineers and the crew on orbit, the ISS is in a good configuration – which earned additional praise on the executive summary.
“The ISS systems are all nominal with the exception of the PM. The on console team continues to work on impacts and actions for next worse failures as a result of the current ISS configuration. Excellent work by the teams today working through the complexities associated with the EVA’s required to remedy this failure.”
Further updates may be added to this article. Refer to live update pages on the ISS section. L2 members refer to the documentation and memo update threads.