The Aerospace Safety Advisory Panel (ASAP) expressed its concern about the integrated computer testing capabilities across the three separate programs under the Exploration Systems Development (ESD) division of NASA’s Human Exploration and Operations Mission Directorate. The Exploration Ground Systems (EGS), Orion, and Space Launch System (SLS) programs under ESD are working together to accomplish upcoming Artemis lunar missions, but have been independently funded and managed throughout their post-Constellation program formulations.
During the ASAP’s fourth quarterly meeting of 2020, the panel highlighted the absence of the ability to test a complete set of the flight computer avionics and software that will fly on upcoming Artemis launches, which includes not just those ESD program components, but also United Launch Alliance’s (ULA) Common Avionics computer system. In the aftermath of the computer software in-flight anomalies on Boeing’s Orbital Flight Test-1 (OFT-1) mission last December, the panel agreed with points made in an internal NASA Engineering and Safety Center (NESC) report and recommended establishing the resources for end-to-end, integrated testing of all the flight computer systems.
ASAP observations on applying OFT-1 lessons
The ASAP meets on a quarterly basis; following private briefings by NASA representatives from different agency directorates, the Federal Advisory Committee Act (FACA) panel holds a public hearing to outline the private proceedings and their observations. At the conclusion of the fourth 2020 quarterly meeting on October 1, Paul Hill presented the panel’s observations about ESD, which included an emphasis of their concerns about lessons that ESD could apply to their own software development and computer testing efforts in the wake of the investigation into the Starliner OFT-1 in-flight anomalies.
“As presented by ESD, there is no end-to-end integrated avionics and software test capability,” ASAP member Paul Hill said in the October 1 public hearing. “Instead multiple and separate labs, emulators, and simulators are being used to test subsets of the software. The panel referred the agency to the NESC report from September 8, the details of which are proprietary, but the findings, observations, and recommendations are very well aligned with our concerns.”
The Boeing Starliner OFT-1 uncrewed Commercial Crew test flight last December was classified by NASA as a high-visibility close call after the mission failed to rendezvous and dock with the International Space Station.
Full results of the post-flight investigation were not fully published, but software development processes and independent oversight were cited as inadequate. Findings from the investigation prompted reviews from elsewhere around the agency to make sure lessons learned from OFT-1 were applied to software development and computer verification and validation testing efforts in other programs.
(Photo Caption: One of the test rings in the Avionics and Software System Integration Lab (SIL) at Marshall Space Flight Center (MSFC) set up with SLS Core Stage avionics. The SLS Program uses the lab to test the computer systems it is responsible for in the Core Stage and Boosters. The SLS upper stage for the first Artemis launches is a commercial service, owned and operated by United Launch Alliance, which is responsible for testing its own avionics and software.)
ESD management briefed results of those reviews to the ASAP during this most recent quarterly meeting. “Although the detailed lessons that we saw are proprietary, we can say that ESD has added more integrated software tests for SLS and is evaluating more Orion test cases,” Hill noted. “That being said, there are still no integrated avionics and software test capability for ESD missions, including for the upcoming Artemis 1, 2, and 3 missions.”
In a statement from the Public Affairs Office at NASA Headquarters in Washington, D.C., the agency said “NASA is conducting integrated end-to-end testing for the software, hardware, avionics, and integrated systems needed to fly Artemis missions.”
“Using the agency’s sophisticated software development laboratories, teams from SLS, Orion, and Exploration Ground Systems use actual flight hardware and software, as well as emulators — versions of the software that each team employs to test their code and how it works with the whole integrated system — to support both system-level interface testing and integrated mission testing to ensure the software and avionics systems work together.”
“Additionally, the agency completed hardware-in-the-loop testing for each element before delivery to Kennedy [Space Center] to verify end-to-end functionality, including full mission runs with Orion and thousands of nominal and off-nominal simulated SLS flights,” the agency noted. “At Kennedy, NASA will perform an integrated test and checkout with the flight hardware that includes a countdown demonstration and wet dress rehearsal with the rocket, spacecraft, and ground systems prior to the Artemis I launch.”
“All software, hardware, and combinations for every phase of the Artemis I mission is thoroughly tested and evaluated to ensure that it meets NASA’s strict safety requirements and is fully qualified for human spaceflight.”
Each Level II program under ESD at Level I has their own computer integration test facility to bring together computers and software from different program contractors and subcontractors. Orion Prime Contractor Lockheed Martin’s Integrated Testing Lab (ITL) in the Denver area supports integrated testing of Orion software and avionics.
The Avionics and Software System Integration Lab (SIL) at NASA’s Marshall Space Flight Center in Huntsville integrates the SLS flight software developed and managed by NASA with contractor avionics for the Boosters, Core Stage, and Engines. At the Kennedy Space Center in Firing Room 3 of the Launch Complex 39 Launch Control Center, EGS tests ground systems software and avionics.
As the ASAP emphasized, though, each program’s computer lab is using “outside” emulators to test its flight computers and software. So the Orion ITL lab tests Orion flight computer equipment with emulators from EGS and SLS that simulate their equipment. Likewise, the SIL has emulators from EGS and Orion to test the SLS computer system, and Firing Room 3 at KSC has emulators from Orion and SLS to test EGS avionics and software.
(Photo Caption: A 2012 graphic of the ESD verification and validation approach which also shows one way in which the program elements are organized and where qualification and acceptance testing is applied. ESD is at Level I, and the EGS, Orion, and SLS Programs are Level II and below.)
The capability that ASAP is recommending would be across the separately funded and independently managed programs. Citing one observation from the internal NESC report, Mr. Hill said “The hardware and software [is] developed and validated by each program in separate labs using numerous emulators and simulators, but it is required to operate in flight as a single, integrated system.”
A test lab at ESD Level I, integrating all the avionics and software for the Artemis 1 countdown, launch, and ascent would need to bring together flight computer equipment, flight software, personnel, support equipment, and other resources from across all three programs and commercial services, since ULA’s Common Avionics computer system is used by the Interim Cryogenic Propulsion Stage in this configuration of SLS.
Even with all the computer systems with flight-like wire-harness connections and electrical systems put together, emulation and simulation computers and software would still be necessary to mimic the rest of the rocket stages and spacecraft equipment, from mechanical systems to propulsion systems and more.
ESD integration relies on program resources
Although ESD is accountable for the critical integration of the programs, it has limited resources of its own to work with. “NASA’s current approach is organized with ESD, rather than a contractor, as the overarching integrator for the three separate human spaceflight programs — SLS, Orion, and EGS,” the Government Accounting Office stated in the GAO-18-28 October 2017 report to the U.S. Congress.
The funding goes directly to the three programs, primarily for development and operations, and ESD draws resources from them. “ESD manages both the programmatic and technical cross-program integration, and primarily relies on personnel within each program to implement its integration efforts,” 2017 GAO report stated.
(Photo Caption: A chart of the President’s budget request for Fiscal Year 2021 showing the money the White House asked for everything under Exploration Systems Development. While ESD is responsible for integration of the three programs, the money and almost all the people are funded at the program level. The chart also shows the magnitude of funding requested for “Program Integration and Support,” which is typically in the tens of millions of dollars and is under two percent of the total money requested.)
“The Cross-Program Systems Integration sub-office is responsible for technical integration and the Programmatic and Strategic Integration sub-office is responsible for integrating the financial, schedule, risk management, and other programmatic activities of the three programs. The three programs themselves perform the hardware and software integration activities.”
Within the large budgets for the three programs, integration allocations within them tend to be a couple of orders of magnitude smaller. In the President’s Fiscal Year 2021 budget request, a total of about $75 million was allocated for integration resources out of about $4 billion for EGS, Orion, and SLS altogether.
NASA’s statement cited the agency’s efforts to apply lessons learned from the Starliner close-call. “The NASA Chief Engineer established an independent review team to assess all Artemis I critical flight and ground software activities,” it said. “The team included members of the NASA Engineering and Safety Center and from the Starliner incident team.”
“The software development teams for each [ESD] program are now implementing key recommendations, including adding additional tests and end-to-end testing in multiple software testing labs to compare results across programs. The software teams are also planning for additional end-to-end testing for the more complex missions of Artemis II and Artemis III, including similar ground testing with Artemis II hardware-in-the-loop following the uncrewed Artemis I flight test.”
But ASAP still has concerns. “The panel has grave concern about the end-to-end integrated test capability plans, especially for flight software,” Mr. Hill said at the ASAP public hearing. “Although the agency expressed confidence that accountability for successful system development is clear, it is not evident to the panel that the current plan and processes take advantage of their lessons learned.”
(Photo Caption: EGS (black cabinet) and Orion (blue cabinet) emulator racks in the SIL at MSFC. Each of the three ESD programs develops and maintains simulation/emulation software and avionics that allows the others to test the cross-program computer interfaces and specifications. The programs use the simulation software not just to mimic computer interfaces, but also to test their own computers by simulating the behavior of the rest of the vehicle, particularly combinations of off-nominal failures and contingencies.)
“NASA and the ESD understand both of these realities and have clear accountability; the ASAP recommends, though, that in addition to accountability, ESD ensures that their management and test processes and practices are aligned to ‘test like you fly,’ to quote the NESC report,” Hill noted. “One element of that is more flight-like, end-to-end, integrated test capability, especially for new hardware and software and major upgrades, both of which will dominate NASA’s experience in the decades-long Moon and Mars campaign.”
“The NESC report makes the excellent point that, as much as possible, flight systems should be developed for success with the goal to test-like-you-fly in the same way that NASA’s operations teams train the way you fly and fly the way you train,” he added. “As Kathy Lueders told the panel in the Spring, from a certain perspective the Boeing OFT in-flight anomalies were a gift, and that gift was awareness and the opportunity to apply the OFT lessons learned across the Commercial Crew Program. It’s also an opportunity to apply them deliberately across the ESD programs.”
Lead image credit: Mack Crawford for NSF/L2.