ASAP reviews Boeing failure, positive SpaceX success ahead of Commercial Crew announcement

by Chris Gebhardt

As NASA prepares to provide updated launch date targets for the uncrewed and crewed Commercial Crew demonstration missions from both SpaceX and Boeing – as well as flight crew assignments for each provider – the agency’s Aerospace Safety Advisory Panel (ASAP) held its quarterly meeting last Thursday, during which they outlined a failure on Boeing’s part that could potentially lead to a redesign of a critical element of Starliner.  The ASAP also outlined multiple points of positive progress on SpaceX’s part.

Boeing suffers critical failure with Starliner:

As was first reported by Eric Berger on Ars Technica, Boeing suffered a test stand failure of Starliner’s critical pad abort thrusters in late-June, a failure that reportedly ended with the leaking of volatile propellant from the thruster system.

In multiple statements to numerous outlets thereafter, Boeing stated that they were “confident we found the cause and are moving forward with corrective action.”

But that wasn’t quite the take-away from the ASAP meeting that occurred days after the company issued its statement.

“Boeing recently conducted a hot fire test for their low-altitude abort milestone for the CST-100,” noted a member of the ASAP panel.  “And there was an anomaly on that test that we need to better understand in terms of its potential impact on the design and operation and the schedule.

Abort testing Starliner – by Nathan Koga for NSF/L2

“And so although there’s a lot of interest in this issue, Boeing has asked for some additional time to step back and understand that a little better.”

During its admittedly brief discussion of Boeing, ASAP members made no mention of any potential corrective action, instead indicating that Boeing was still in the process of understanding exactly what happened with the failure.

It is possible that ASAP was referring to information that predated the Boeing statement; however, it is also possible that the panel – which provides a cautious and measured approach – had not yet been presented evidence of such corrective action for the abort thruster issue.

Regardless, one thing was clear from ASAP: Boeing’s current schedules for both their uncrewed and crew test flights, known as the Orbital Flight Test (OFT) and the Crew Flight Test (CFT), respectively, are in a state of flux and not well understood.

“We can expect some uncertainty and their near-term schedules at least for the Boeing provider while they go through that,” noted the ASAP.  “And then we should have a much better sense of the Orbital Flight Test and the Crew Flight Test for that provider.”

Starliner launches atop an Atlas V N22 rocket from Cape Canaveral Air Force Station, FL, carrying new crewmembers to the International Space Station (Credit: Nathan Koga, for NSF/L2).

This statement is telling, especially ahead of NASA’s plan to announce new target launch dates for the OFT and CFT missions for Boeing (as well as for both of SpaceX’s uncrewed and crewed missions) this Friday, 3 August, at an event at the agency’s Johnson Space Center.

With this ASAP statement, it is now understood that those dates for Boeing to be announced Friday might not be completely realistic as the company continues to work through mitigation and potential redesign, as stated by ASAP, resulting from the test stand issue.

Praise for SpaceX, but still work to do:

Contrasting quite significantly from the above, the ASAP panel had high praise for SpaceX regarding a number of areas related to the production of the crew Dragon and the Falcon 9 boosters that will be used to launch the crew craft to the International Space Station.

Of particular note in the panel’s praise for SpaceX was the company’s strides both at innovation and adaptability while also developing procedures and computer tracking systems to greatly aid in Systems Engineering & Integration (SE&I) to properly track all changes to the vehicle and understand how those changes relate to the overall integrated design for crew Dragon and the Falcon 9 rocket.

In fact, perhaps no better praise for these tools and systems came from an ASAP member who said they “would have been great to have for the Space Shuttle program” – which ASAP member, and former NASA astronaut Lieutenant General (Ret.) Susan Helms noted “was a great ‘attaboy’ for the program’s current visions of how the tools are not just evolving but how the SpaceX force is adapting to using them.”

Crew Dragon heads uphill on the Falcon 9 – via Nathan Koga for NSF/L2

ASAP discussed these positive attributes for SpaceX while discussing the Commercial Crew Program’s (CCP’s) overall positive progress on strategies and approaches to meeting the panel’s SE&I recommendations.

“You can never review enough times what SE&I principals actually mean,” noted Lt. Gen. (Ret.) Helms.  “It means understanding the margins of the integrated system designs, verifying that those margins exist through test and analysis, and then controlling both the configuration and the operation of the system to ensure those margins exist when flown and that these principles are both absolutely essential for culture and practice to achieve the best possible outcomes for the safety of space flight.”

Lt. Gen. (Ret.) Helms also had great praise for SpaceX’s “extreme transparency” with the CCP office on all of their data, “including how the tools can help them with risk management.

“Assuming that the program continues to have confidence that SpaceX’s principles are evolving in a way that gets them to meeting the intent of our recommendations, it looks like things are on a good path to what we had originally intended with our recommendations.”

Nonetheless, there were a couple of issues (more items to watch than issues) ASAP brought forward regarding SpaceX in relation to vehicle hardware.

The CRS-4 Dragon prepares for splashdown in the Pacific Ocean. Credit: SpaceX

One such area is an upcoming change to the vendor that provides the parachute reef line cutters for Dragon.  The original vendor selected by SpaceX was the only NASA-qualified vendor for reef line cutters, and it had to serve Dragon, Starliner, and Orion.

That vendor is an extremely small company, and challenges in ensuring the quantity and – most importantly – the quality of the delivered product due to increased demand from the three crewed vehicles developed.

As such, SpaceX made the decision to switch vendors to ensure a continuous and quality supply of reef line cutters.  That new vendor is currently going through the qualification process, which is nearing completion.

What SpaceX is now in the process of deciding is on which mission they will switch to the new vendor’s hardware – the uncrewed DM-1 flight or the crewed DM-2 mission.

The parachutes for the DM-1 mission have already been packed inside the crew Dragon capsule, and changing out the reef line cutters would delay the DM-1 flight.  SpaceX is actively engaged in risk management and risk posture discussions for potentially switching to the new reef line cutters on the crewed DM-2 flight, with ASAP noting that they are eagerly awaiting the results of those discussions.

Another area of discussion related to the COPVs, Composite Overwrap Pressure Vessels, that will be used in the Falcon 9 rocket, new COPV designs set to debut on the uncrewed DM-1 mission with Falcon 9 core B1051.

According to ASAP, “We’ve learned a lot this week about a very detailed and structured approach for certification of the COPVs used on the SpaceX design.  There’s been a lot of testing and a lot of follow-up and investigations to try to understand the physics of things like what could cause potential ignition mechanisms or other failure mechanisms.  

“And the panel really wants to commend the deep dive team [for their work], but we do recognize that there’s still a lot of work to do.  The jury is still out on this.  And we look forward to seeing what the results of what this work will mean in terms of final characterization of the risk and whether that risk will be acceptable.  And if not, what further risk mitigation measures might be necessary concerning those COPVs.”

Another area to watch – one that ASAP believes is on course to a “satisfactory conclusion” – relates the Falcon 9’s Merlin 1D engines and their upgrades for crew configuration.  “The first two engine started to go through the qualification procedure, and upon tear down there were observations of some anomalies in the hardware that were considered potentially dangerous – certainly not desirable,” noted the ASAP.

“There was a program undertaken to try to modify the design and to correct these anomalies.  SpaceX and NASA agreed on a requalification plan that actually includes six engines – two ground test configurations and one in-flight test configuration.”

The Block 5 Falcon 9 takes a Crew Dragon to space during one of SpaceX’s upcoming Demonstration Missions. (Credit: Nathan Koga for NSF/L2)

As work on this issue continued, SpaceX developed two short-term corrective actions that are deemed worthy and safe to fly on the uncrewed DM-1 mission.  At the same time, SpaceX began designing and implementing two longer-term solutions to the anomaly seen on the test stand.

The ASAP noted that the two short term actions could also potentially be used on the crewed DM-2 flight should they prove safe enough during post-flight and post-test stand teardown and inspections.  A decision on whether those short-term actions can fly on the crew demonstration mission will not be taken until a better understanding of the short-term actions are known.

Nonetheless, the ASAP is “optimistic that those courses of action will result in a satisfactory conclusion.”

Final Certification – a discussion:

Through all of this, the ASAP was primarily quite pleased with the overall status of the Commercial Crew Program and the current final drive toward certification of the products being delivered by the two providers.

There was a specific note that no schedule pressure was observed during the latest round of oversight meetings, with both providers acutely aware of the safety and risk factors involved and that those take precedence over any perceived/looming desire to see the programs up and running.

With a Starliner docked safely to ISS, a crew Dragon approaches the Station using the C2V2 radio. (Credit: Nathan Koga for NSF/L2)

However, “as we approach the certification milestones, we still have to be assured that not only will the component hardware design be ready for certification, but that the overall integrated risk of human spaceflight missions can be effectively managed in both the design of the hardware and the operational practices of using the hardware,” noted the ASAP.

To that end, one ASAP member took time to discuss, in detail, what certification of these products actually means.  During his statements, Dr. Don McErlean sought to clarify for everyone what the final certification process entailed – as it is often referred to as a “paper process.”

“It needs to be noted by everyone, and we’re especially interested in making sure that all of the external stakeholders realize this, that while the concluding process of certification has sometimes been described as a paper process, that is really just a shorthand clarification and in reality it could not be further from the truth,” noted Dr. McErlean.  

In reality, the process is as follows.  “In a certified design, the design agent – the contractor or partner in this case – performs the design and in the certification plan, the design agent and the certification agency (NASA) agree on the submittal of certification evidence.

“This could be measurements, it can be test data, it can be analysis, but it almost always involves the submittal of detailed technical data, not simply paper descriptions or forms.  Sometimes it involves witness testing and sometimes it involves physical inspection. But it almost always wraps around important technical submittals.

“The technical arm of the certifying agency – NASA in this case – then reviews and analyzes that data to validate that in fact the design meets the stated requirements with the expected margins.  This validation activity extends to many aspects of the design and also to the operation of the integrated mechanism or the entire system. Once the certifying agency agrees that these validations are correct and acceptable, then it certifies the design and then we can state that we have run to conclusion of the certification process.

“It is this technical review of certification data and products which in fact constitutes the work required to conclude the certification process, and if successful, results in the agency in fact certifying the design.

“In short, we want everyone to realize that this is not a paper sign off process.  It involves considerable detailed technical activity by both the certifying agency – NASA in this case – and the design agency or the contractor or partner as well.

“In addition, it should be stated that the design is certified for a given range of operations.  This is assumed as part of the process. And we all realize that should his range of operations be significantly exceeded, the data would have to be reevaluated and reexamined to determine if the design was adequate for the new regime.

“Now this does not seem to be the case in our vision at the moment.  I just bring that up so that everyone is aware that that is the case.  And so in front of both the agency and the contractor is a considerable amount of detailed technical review process wherein the data, backing up the design itself, is reviewed and assessed against the certification criteria.

“Once that process is complete, then the design is eligible for certification and would be signed off by the certification agency.”

Related Articles