Just over two months after the truncated Orbital Flight Test of their Starliner crew capsule, Boeing officials have provided an in-depth overview of the issues faced on the flight.
The update, held on Friday, 28 February, also provided information on the path forward for Starliner, including the on-going software audit, closing process escapes and software testing gaps, as well as work with NASA to determine the future flight schedule.
NASA will ultimately be the one to decide what kind of flight Starliner will fly next and when that next flight will occur, but John Mulholland, Vice President and Program Manager for Boeing’s Commercial Crew Program, confirmed there is room on United Launch Alliance’s busy Atlas V schedule this year for another Starliner flight.
However, Mr. Mulholland advised cautioned on pinning down exactly when Starliner’s next flight will happen, saying that until the final recommendations from the Independent Review Team are received (next week), until Boeing’s internal audit of the software and processes that lead to the in-flight issues not being discovered prior to flight are completed, and until Boeing can implement new requirements and rules for software verification and validation, there cannot be a realistic discussion of when Starliner will return to flight.
Of note, Mr. Mulholland reiterated — as he has throughout the last two months — that the Atlas V rocket performed flawlessly on the Orbital Flight Test launch on 20 December 2019 from the Cape Canaveral Air Force Station, Florida.
The Atlas V dropped Starliner off in the planned sub-orbital trajectory and is not in any way to blame for the issues faced after Starliner separated from the top of the Centaur upper stage.
What went right:
While much attention has been rightly paid to the issues encountered on Starliner’s debut, less focus has been given to the 80%+ of the mission that went perfectly.
Of note, numerous Starliner systems performed better than pre-flight predictions, including the craft’s critical Environmental Control and Life Support Systems, its thruster systems, and its avionics cooling systems (used when the radiators on the Service Module are unavailable during pre-launch, launch, and reentry/landing operations).
The craft’s Thermal Protection System — more commonly called a heat shield — also performed above predictions.
The same is true for the solar panels on the base of the Service Module and Starliner’s associated power systems — which actually provided more power than predicted.
In fact, some of these systems performed so well together that the conservative operating guidelines in place on the Orbital Flight Test can be relaxed going forward, providing additional efficiency to the craft.
Additionally, while not used to dock to Station, Starliner’s NASA Docking System also underwent a near-complete checkout during the two-day free-flight, with Mr. Mulholland noting “we know we have good control of the docking system.”
In specific regard to Starliner’s thruster system, Boeing talked in-depth about the stress put on the thrusters to operate outside their normal envelope.
After Starliner didn’t perform the Orbital Insertion burn (because of the Mission Elapsed Timer software anomaly) when intended, the spacecraft had to perform a “very inefficient three-axis prolonged thruster burn to get Starliner into a stable orbit,” related Mr. Mulholland.
This necessitated burning the thrusters far longer than the nominal mission plan, which introduced “false positive” overheat signatures.
These “false positives” were not unexpected once the system was put through this inefficient three-axis burn.
The thruster systems were shut down out of caution after the three-axis burn got Starliner into a power-positive and stable orbit, and teams with thruster experience from the Shuttle-era then investigated the “false positives” and worked to bring each thruster back online.
All but one thruster was successfully returned to service, and all that were brought back functioned perfectly for the rest of the mission — save one very minor issue with an attitude control thruster’s nitrogen pressurization valve that failed to close during reentry.
This valve closure issue was not a flight or safety concern, but Boeing has pulled the affected thruster to determine why the valve didn’t close.
The landing itself was within 182 meters (200 yards) of the pinpoint center of the landing zone, and landing loads were “benign, below prediction, and three times lower than NASA requirements,” noted Mr. Mulholland.
Parachute performance was also baseline normal.
Moreover, all post-flight data reviews — a separate set of reviews from the on-going software audit and Independent Review Team action/review items — are now complete.
Speaking of the data review, Mr. Mulholland noted that the Orbital Flight Test capsule had been cleared to continue post-mission processing and flight turn-around efforts.
“By and large, the vehicle hardware is in outstanding shape after the mission,” said Mr. Mulholland, who praised the technicians and engineers who built Starliner and are now in charge of turning this capsule around for its next flight.
The next capsule (Spacecraft #2) to fly is the one originally earmarked for the Crew Flight Test. It is currently in the hazardous production area of the Commercial Crew and Cargo Processing Facility at the Kennedy Space Center undergoing proof and leak tests.
What kind of mission this capsule will fly — a repeat of the uncrewed Orbital Flight Test or the Crew Flight Test — is not yet known.
NASA will brief the public on the results of the Independent Review Team report on Friday, 6 March at 11:00 EST (16:00 UTC). Boeing officials cautioned that NASA might not be ready to make an announcement at that briefing as to whether a reflight of the Orbital Flight Test will be required.
The path forward:
In all, Mr. Mulholland was appreciative of the interim recommendations provided to Boeing by the Independent Review Team, noting that those detailed, preliminary recs — including investigative and review steps — have allowed the team to start moving out and strengthening the robustness of the team, the systems, and the processes to ensure safe flights of Starliner.
A key step already outlined by the Independent Review Team is the need to improve System Engineering and Integration disciplines.
Some of those improvements have already been implemented — such as Boeing conducting on-site visits, reviews, and relationship-strengthening discussions with its various suppliers.
But there is still a great deal of work ahead on the software side of things.
“We need to go look at the software testing process for formal qualification,” noted Mr. Mulholland. “We’ve already completed an audit on the high and medium logic areas to compare to software testing to identify gaps, and we’ve identified those areas where we did and did not test the software.
“We’re looking for other anomalies, too, that might need to be fixed. So far, we haven’t found any, but the overall software audit continues.”
Mr. Mulholland pledged that if any more software issues are found in testing, Boeing will reveal those issues and their fixes; moreover, he expects to be able to provide an answer to that within the next month or so.
Mulholland also addressed tough and repeated questions as to why a complete end-to-end test of Starliner’s software, a test that would have caught the two issues that plagued the craft on the Orbital Flight Test, was not performed.
In short, he said that teams used a common industry practice of breaking up prolonged software qualification tests into chunks and tested each of those chunks independently.
For Starliner, this resulted in one chunk being the countdown and launch, stopping just after spacecraft separation from the Atlas V’s Centaur upper stage.
The second chunk picked up thereafter and ran through the critical Orbit Insertion burn.
The issue here is that because the countdown/launch software test chunk stopped before the Orbital Insertion burn, the test was unable to verify that the correct Mission Elapsed Time was pulled by Starliner from the Atlas V rocket.
Had the test actually proceeded through the Orbit Insertion burn, it would have caught the Mission Elapsed Timer issue and permitted teams to correct it on the ground before flight.
(The Mission Elapsed Timer issue was traced to the software pulling the mission time from the Atlas V before terminal count and then not pulling it a second time once the terminal count started… leading to the timer being off by 11 hours.)
To this end, Boeing will now perform a multi-day launch-to-docking end-to-end software test for missions going forward.
Conversely, the second software issue — the jet mapping issue that would have resulted in Starliner’s Service Module not being able to fire its thrusters to perform a critical evasive burn to get itself away from the Crew Module after Service Module/Crew Module separation following deorbit burn — was not caught because the emulator used to verify this part of the software code was legacy from other Boeing programs, not Starliner, and was not updated with the correct jet mapping software for Starliner.
The piece of hardware that had the correct Starliner Service Module jet mapping software code — and was marked to perform this software qualification — was pulled from the test and sent to New Mexico for the Service Module hot fire engine test instead.
The substituted emulator was then not updated with the correct jet mapping software code.
When the jet mapping software verification test was re-run on the ground with the hardware that had the correct software coding — done while Orbital Flight Test was preparing for deorbit, reentry, and landing — the issue was discovered and the correct code was transmitting up to Starliner to avoid what NASA classed as a “Loss Of Vehicle” situation.
Speaking to this, Mr. Mulholland noted that Boeing has and is updating all software testing procedures to not only include checklists for the software but also all of the specific hardware needed for such qualification tests to ensure there is never a hardware/emulator mismatch again.
“Going forward, we’re going to define what hardware is needed to make sure it’s there for the qualification testing,” said Mr. Mulholland.
Additionally, Boeing will now perform a full-duration undocking-to-landing integrated software tests to ensure this issue does not repeat on future missions.
Moreover, increased discipline and requirements will be added to all review and approval processes for software testing sequences going forward.
Mr. Mulholland also stated that Starliner teams have begun sharing the results of the software testing lessons learned — including the identified weak point in “chunk testing” — to other departments and divisions within Boeing and has also shared the results with outside companies and the space industry as well.