Teams on the ground are working through possible root causes into the temporary failure of one of Atlantis’ General Purpose Computers (GPC-4), with a Coronal Mass Ejection (CME) listed as one of three potential contributing factors. The computer now appears to be functioning normally, as Atlantis and her crew head into the final part of their docked STS-135 mission.
With the crew ahead of the timeline on their huge list of transfers between the Multi-purpose Logistics Module (MPLM) and the International Space Station (ISS), Atlantis will be hauling a packed Raffaello back to Earth next week, at the same time as leaving the orbital outpost with a partial logistics bufferzone of supplies.
The crew went to sleep on Flight Day 9 with 90 percent of overall transfers completed, ahead of what will be their final full day on Station.
All orbiter systems are performing as required, with the Emergency, Environmental and Consumables Operation Manager (EECOM) reporting to the Mission Evaluation Room (MER): “All Life Support Systems are normal: Total GN2 Qty: 214.9 lbm. Total Waste Qty: 42.7 percent 73.7 lbm. Supply Water Total Qty: 460.9 lbm. CWC #11 Fill complete and filled with ~93.7 lbm of supply water.”
Atlantis’ propulsion systems are in good shape, with 25 of the 38 primary thrusters having already been fired during the mission, ahead of a full set of thruster firings during the EOM-1 (End Of Mission) checkout.
“Since vernier activation on FD1 (Flight Day 1), thruster L5L’s chamber pressure has been consistently lower than previous flights: Nominal peak Pc > 100 psia. For short pulses or elevated prop temps Pc as low as ~75 psia (this is) not unusual,” noted a 43 page Orbiter Project Office (OPO) presentation (L2).
“STS-135/L5L peak Pc (chamber pressure) 103 psia; was as low as 76 psia on FD3 (Flight Day 3). For comparison, STS-125/R5R peak Pc ~ 96 psia; was as low as 48 psia on FD2.”
The depth of evaluation is not unusual for orbiter engineering teams, but it’s also a sign of a lack of any other issue of note to evaluate on Atlantis’ propulsion systems.
Based on previous flight history, engineers do not expect the thruster to fail, as much as such an incident would not prove to be challenging for Atlantis.
“Current Status: Pc signature through docking did not change; average Pc is still ~100 psia for extended firings. Appears it may be performance problem. Based on past low Pc cases, L5L fail-off extremely unlikely,” added the presentation, which noted the thruster will still remain in use.
“Criticality/consequences of loss of thruster. Loss of verns while docked (all verniers required for docked attitude control). Could affect reboost (not currently planned), maneuver to undock, undock attitude hold. Has no impact for Orbiter alone – vernier control still possible with remaining five jets. Manual flying would require Alt DAP (Digital Auto Pilot) (e.g., flyaround).
“Recommendations: Continue to use jet unless fails off. Remove thruster valves post-flight.”
The only notable issue with Atlantis has been with her GPCs, with GPC-3 failing before rendezvous with the ISS – prior to being recovered, and GPC-4 failing during the crew sleep period after Flight Day 7 had been completed. This computer has also now recovered.
STS-135 Specific Articles: http://www.nasaspaceflight.com/tag/sts-135/
There are five identical GPCs aboard the orbiter. Each GPC is composed of two separate units, a central processor unit and an input/output processor. All five GPCs are IBM AP-101 computers. Each CPU and IOP contains a memory area for storing software and data. These memory areas are collectively referred to as the GPC’s main memory.
The hardware controls for the GPCs are located on panel O6. Each computer reads the position of its corresponding output, initial program load (IPL) and mode switches from discrete input lines that go directly to the GPC. Each GPC also has an output and mode talkback indicator on panel O6 that are driven from GPC output discretes.
Each GPC receives run, stby, or halt discrete inputs from its mode switch on panel O6, which determines whether that GPC can process software.
It was a “switch tease” on that panel when GPC-3 was physically turned on during the set expansion task for rendezvous, which caused the computer to temporarily join the common set prior to then failing out. GPC-3 was recovered following a dump and an IPL – effectively rebooting the computer, which has shown no signs of issues since.
A problem was first noticed with GPC-4 when Atlantis annunciated the problem with via a master alarm, resulting in Commander Chris Ferguson being awoken, prior to heading over to the orbiter to evaluate the issue. The GPC in question was running system management software.
“To attempt recovery of GPC 4, work GPC FRP-1. This procedure will dump GPC 1 as well as dump and IPL GPC 4. If GPC 4 is recovered, this will freeze-dry GPC 4,” opened a highly technical set of instructions in one presentation (L2).
Commander Ferguson ably followed the instructions from the ground, starting GPC-2 to take over the System Management (SM) task GPC-4 was running, allowing him to turn GPC-4 off. He then returned to his sleep period.
In the morning, the crew performed what is known as “hardware-initiated, standalone memory” (HISAM) tasks on GPC-1, prior to carrying out the same procedure on GPC-4. This was followed by an IPL task, prior to the ‘freeze dried’ procedure being carried out on the computer. During these tasks, a data dump from the troublesome GPC was sent to the ground for evaluation.
“GPC 4 (SM) failed during crew sleep. GPC 2 was brought online and configured for SM. GPC 4 was powered OFF,” explained a second presentation (L2). “GPC 1 HISAM was performed following crew wakeup. GPC 4 HISAM was performed. Data Dump first look shows: CPU Store Protect Error, PC 1 Timer Interrupt.
“GPC 4 was IPL’ed and added to Redundant Set in G2. GPC 4 was freeze dried in G2 and put in STBY.”
No specific root cause has been revealed at this stage, although teams are confident that the incident was a “single event upset” meaning GPC-4 is effectively still a healthy machine.
As to what the “single event upset” could have been, documentation shows analysis is still in work, although in the list of contributing factors is a “Coronal Mass Ejection (CME)” from the Sun.
“Analysis of Memory Dump continues. MOD investigating any contributing factors: South Atlantic Anomaly (SAA), Coronal Mass Ejection (CME), Stored Program Command (SPC). history review for similar failure signatures. Discussions on any Confidence Runs prior to deorbit prep. MOD discussing Entry configurations.”
Interestingly, docked photos from NASA – prior to GPC-4’s issues – show Atlantis attached to the ISS as aurora borealis (Northern Lights) and/or aurora australis (Southern Lights) lit up the backdrop of Earth – an effect of CMEs/Solar activity.
It is understood these photos were taken a matter of hours prior to GPC-4 failing.
While orbiter’s have five GPCs, one of the computers runs software designed by a different vendor than the other four – in order to protect against a commonality failure of all five computers. Technically, an orbiter can function and re-enter on just one GPC, although the preference is too have two GPCs running, with two more in standby mode.
Currently, all five GPCs are deemed to be in good working condition, following the troubleshooting. Future updates on the root cause of GPC-4’s incident will be provided when information is available.
To read about Atlantis and her sisters – from birth, processing, every single mission, through to retirement, click here for the links:
(Images: Via L2 content and NASA.gov. Further articles on STS-135 will be produced during and after her mission, driven by L2′s STS-135 Special Section which is following the mission at MMT/MER level, surrounded by a wealth of FRR/PRCB/MER/MMT and SSP documentation/pressentations, videos, images and more.
(As with all recent missions, L2 is providing full exclusive level mission coverage, available no where else on the internet. To join L2, click here: http://www.nasaspaceflight.com/l2/)