Teamwork, Expertise and Confidence – solving the ISS’ troubles

Following the success in rebooting the German made computers, after Expedition 15 commander Fyodor Yurchikhin and flight engineer Oleg Kotov installed bypass jumpers on the secondary power supply switches, a successful attitude test is one of the last remaining hurdles to be negotiated, in order to allow Atlantis a nominal undocking.

Atlantis’ undocking is likely to take place on Tuesday, now that it appears that she won’t be required to stay the extra day to assist ISS attitude control via her RCS (Reaction Control System) thrusters. Atlantis has also officially been cleared for returning to Earth, after the successful OMS Pod blanket repair during Friday’s EVA-3.

Troubleshooting:

The successful repair to the computers started to show promise by Friday evening, as Russian ground teams managed to restart two lanes (subsets) each of the SM Terminal Computer (TVM) and Central Computer (TsVM), after Yurchikhin and Kotov isolated a faulty secondary power supply switch behind SM panels as cause of the computer shutdowns, by bypassing the switch with a jumper cable. The computers later were shut down and then restarted in self-test mode as planned.

‘Telemetry data was collected for additional analysis Saturday morning during RGS (Russian Ground Station) passes,’ added a weekend On Orbit status report, which proved to be successful

‘TVM (Terminal) and TsVM (Central) computers (specifically on subsets TVM1 & TVM3 and TsVM2 & TsVM3), allowed restart of the computers on these subsets, this morning (Saturday) the two computers and the U.S. GN&C (Guidance, Navigation & Control) MDM computers established communication between each other and are back in Ops mode, after running overnight in Test mode,’ updated a memo on Saturday night. ‘Normal systems conditions are slowly coming back.’

Plans to return the two computers – that finally became active on Saturday – via Atlantis have now been cancelled, although NASA noted the Russians may still work towards an advanced launch date – in late July – of their next Progress resupply ship, which will carry spares and replacement computers.

These computers have identical hardware and service software, and each one is comprised of three identical FCRs (Fault Containment Regions), or ‘lanes’, that are separate ORUs (Orbital Replaceable Units). They were developed by the German company DASA (today Astrium) for ESA and RSC-Energia under contract.

‘The TsVM, acting as C&C (Command & Control) computer, performs integral control tasks for the RS (Russian segment), controlling data streams, communicating with U.S. C&DH (Command & Data Handling), doing autonomous control tasks, etc,’ added documentation surrounding the issues with the ISS computers.

‘The TVM, acting as navigation computer, performs GN&C (Guidance, Navigation & Control) functions, communicating with the U.S. GN&C MDM and the SUDN (Motion Control & Navigation System) of other RS vehicles, and controls a host of other onboard systems.

Finding causes, using contingencies:

The troubleshooting process started with the power feed from the new S4 solar arrays to the RS via SNT (voltage/current stabilizers) #23 & #24 being removed in the Node, but leaving the ground path through P-12 connected. The installation of the S3/S4 truss segment, following deliver by Atlantis, was initially thought to be the cause of the problem.

Friday night saw the crew used a current scopemeter (oscilloscope) in the Node endcone to check on the quality of the current on three wire harnesses that feed power to the RS, in order to determine if EMI (electromagnetic interference) was travelling down one of the lines as noise which could upset a noise-sensitive power supply card.

‘Results were evaluated on the ground, and the S4 power was found clean of any noise and was exonerated,’ it was noted, clearing the truss from the fault tree. However, the installation of the new addition to the station may ultimately have started the chain of events with the Russian computers.

‘Root cause analysis is underway. One speculation under consideration is that changes in the ground state, i.e. the difference between the vehicle’s electrical potential and the surrounding plasma ‘vacuum’, caused sensitive protective secondary power circuitry to turn off the primary power supply on sensing an overvoltage,’ added NASA information. The jumpers bypass either the sensitive logic or the entire overvoltage protection.’

‘A grounding change could have been caused by the addition of the 36,000-lbs S3/S4 truss and a change in the plasma environment or a change in ground path caused by the rewiring and circuit changes. More analyses are required to confirm or change this hypothesis. The ISS is the largest structure ever erected in space, and this situation cannot be tested on the ground. But the experience will help teach us how to build the large space structures of the future.’

Contingency procedures were worked through, from the moment the computers initially went down, although it’s still uncertain as to exactly when this happened – only that they were up at the time Atlantis docked last week.

A short return to operation on two of the computers allowed Russian controllers to take a level of action, which bought them some time during the initial phase of troubleshooting. At no time were the crew in any danger.

‘When the SM computers came up on one lane each for a short time early this morning, it sufficed for recycling the FGB MDM (Multiplexer/ Demultiplexer) computer to connect it to the SM and to activate one ARCU (American-to-Russian Converter Unit) to feed power to the FGB systems.’ information on Friday stated. ‘As a result, U.S. power flow to the FGB and Soyuz TMA-10/14S was restored.

‘The Soyuz spacecraft has its main battery recharged (the entry battery was never touched). Supplying U.S. power via dragthrough cable is being studied as a potential future option.

‘The brief activation of two SNTs and one ARCU allowed Russian specialists to operate one of the SM external thermal loops and improve maintenance of internal temperatures in the SM (stabilizing at ~4 degC above nominal).

‘Elektron (O2 generation) and Vozdukh (CO2 removal) were down, but Vozdukh can operate without the TVM, and there is plenty of O2 on board, to be supplied from Progress tanks as required. Also, the new U.S. OGS (Oxygen Generation System) is scheduled to become operational later during the 13A Stage.

‘The SKV air conditioners remained off during the day, but humidity control is adequately served by the U.S. CCAA (Common Cabin Air Assembly), and specialists are looking at new procedures for using SKV without TVM. The BMP Harmful Impurities Removal System remained active, as did all other RS systems.’

**Ride home through the fire and plasma of re-entry with Atlantis on STS-115 – And now also with Discovery on STS-116 – TWO Stunning high quality 2hr, 355mb videos – from deorbit burn to post landing**

The main issue – which also related to Atlantis – was attitude control for the ISS, required for nominal undocking from the station. This also – during the time of troubleshooting – had contingencies in place.

‘The ISS has been in gravity-gradient stabilized attitude to minimize use of the CMG MM (Control Moment Gyroscope/Momentum Management) system,’ noted On Orbit status report information on Friday evening. ‘Russian reaction control jets (thrusters) are not required until the ISS needs to maneuver, e.g., for Shuttle undocking attitude, to desaturate the CMGs, or for debris avoidance.

‘Russian thrusters are available for desaturation as needed as attitude control mode is back on CMG TA (Control Moment Gyroscope with Thruster Assist).’

Teamwork:

Another impressive joint US/Russian effort involved ‘study teams’ from both countries carrying out an on-going effort to add ‘lessons learned’ from the recent events, should they occur again.

‘Joint US/Russian Study Teams: Alternative approaches to station attitude control, including off-nominal Shuttle undocking, are being studied by a special team of U.S. and Russian experts,’ one report stated.

‘Other joint teams are investigating the root cause of the computer failures, approaches to optimizing ISS configuration without Russian computer control, including critical systems and resources, and the updating – where necessary – previously developed and filed de-crewing criteria, including timeline, Soyuz safe departure, and optimum station systems configuration to facilitate subsequent re-crewing.’

During the height of the computer failures, NASA called on their experts to be on standby to support efforts for off nominal attitude problems.

‘Steve Poulos (Orbiter Project manager) has asked all MER (Mission Evaluation Room) subsystem teams to be ready to support any request coming from organizations working the ISS attitude control problems,’ noted one memo, which was soon followed by a call for multiple teams to be set up, with the ability of 24/7 support.

‘Multiple teams formed that encompasses a lot of different personnel: 1. Root Cause / System Recovery Team. 2. Alternate Attitude Control Team. 3. Crewed Operations Team. ‘How can the crew stay long as possible without Russian assets?’ 4. De-Crewing Team. ‘How to set up the ISS if it were evacuated.’ 5. Alternate Orbiter Entry Team.’

Other NASA departments – not normally directly related to Shuttle or ISS – even offered their support, should it have been required. That offer came from Constellation Program Manager Jeff Hanley.

‘Constellation program support to ISS – Explorers, I fully support Jeff’s offer of folks working Cx (Constellation) to support ISS as they recover attitude control,’ noted an e-mail sent by Mission Operations Project Office and former ISS manager Dennis Webb, which concluded with a rallying call – quoting from the ‘Foundations of Mission Control’ – to the troops, one that epitomized how NASA work, despite rarely being portrayed as such in the media.

‘These things go without saying it but I will say them anyway: Foundations of Mission Operations: To instil within ourselves these qualities essential to professional excellence. Discipline…Being able to follow as well as to lead, knowing that we must master ourselves before we can master our task. Competence…There being no substitute for total preparation and complete dedication, for space will not tolerate the careless or indifferent.

‘Confidence…Believing in ourselves as well as others, knowing that we must master fear and hesitation before we can succeed. Responsibility…Realizing that it cannot be shifted to others, for it belongs to each of us; we must answer for what we do, or fail to do. Toughness…Taking a stand when we must; to try again, and again, even if it means following a more difficult path.

‘Teamwork…Respecting and utilizing the abilities of others, realizing that we work toward a common goal, for success depends upon the efforts of all. Vigilance… Always attentive to the dangers of spaceflight; Never accepting success as a substitute for rigor in everything we do.

‘To always be aware that suddenly and unexpectedly we may find ourselves in a role where our performance has ultimate consequences. To recognize that the greatest error is not to have tried and failed, but that in the trying we do not give it our best effort. Ad astra per aspera, Dennis.’

EVA-4:

Flight Day 9 was mainly uneventful, as the crews conducted media events and preparations for the fourth and final spacewalk of STS-117, which will be conducted by Pat Forrester and Steven Swanson.

Objectives of the spacewalk, scheduled to begin just after midday EDT, include verification of DLA-2 installation, suspected of being mislabelled for DLA-1, removing the remaining SARJ (Solar Alpha Rotary Joint) launch restraints.

Also involved will be the stowing of the S3 drag link/keel pin, relocating an APFR (Articulated Portable Foot Restraint) and installing MMOD (Micrometeoroid/Orbital Debris) shields bolts, preparing the MT Mobile Transporter) translation path, removing GPS-4 antenna, removing SASA (S-band Antenna Support Assembly) gimbal locks, and more.

The EVA was completed on Sunday in just over six hours.

Follow Atlantis’ mission with live updates on our Flight Day specific interactive live event pages – open to all.

For L2 members, follow NASA’s evaluations on the L2 Flight Day specific pages (which contain 100s and same day documentation and presentations – uploaded live during each Flight Day – 60 presentations on Flight Day 7 and 8 alone).