Boeing, NASA return to SLS Green Run test preps after latest hurricane interruption

by Philip Sloss

After taking precautions against a fifth approaching hurricane, the Space Launch System (SLS) Green Run test team is back on site in the B-2 Test Stand at NASA’s Stennis Space Center in Mississippi to continue preparations for the final two critical tests of the year-long campaign on the first Core Stage.  Prime contractor Boeing is conducting Stage Green Run test operations, completing the sixth of eight test cases at the beginning of October just before the approach of Hurricane Delta forced the latest weather-related work stoppage.

During test case six, the team practiced a set of simulations of the critical sequences in the upcoming Wet Dress Rehearsal (WDR) and Hot-Fire tests, which are the seventh and eighth test cases.  Now back at work, Boeing is moving into final preparations and reviews with NASA before the WDR tanking test at end of October and the Hot-Fire test in mid-November.

Propulsion system testing and checkouts completed in the middle of hurricane season

Hurricane Delta was the fifth tropical weather system of the 2020 Atlantic hurricane season to force the Green Run workforce to secure the stage and the test stand at Stennis. Tropical Storm Cristobal in the first half of June delayed the post-COVID resumption of the Green Run before Boeing could begin powered vehicle testing.

With no tropical storm interruptions from mid-June until mid-August, Boeing and NASA were able to complete test cases two, three, and four before Hurricanes Marco and Laura both passed close enough in late-August to postpone the beginning of test case five for several days until the very end of the month. Two weeks later, Hurricane Sally again suspended testing in the B-2 Stand in mid-September just after test case five was completed, slowing down preparations for test case six.
Test case five was the final functional, integrated test of Core Stage hardware and ground systems prior to proceeding into the big tests on the vehicle while loaded with propellant.

The focus of test case five objectives were to demonstrate that the stage hydraulic systems and the four RS-25 engines were working together with the vehicle’s computers systems and other Main Propulsion System elements in rest of the stage.

One of the first test objectives was to cycle the stage’s thrust vector control (TVC) actuators that move the RS-25 engines. The four RS-25 engines are uprated Space Shuttle Main Engines that last fired during Shuttle launches in the last decade, and each one is gimbaled by pitch and yaw actuators.

“During the final cycling test, both actuators (one engine at a time) were driven simultaneously,” said Jonathan Looser, NASA SLS Core Stage Propulsion Lead. “This is the test that looked like swinging church bells.” He also said that another one of the tests involved commanding all eight actuators simultaneously.

Credit: NASA/SSC.

(Photo Caption: A view of the first Core Stage flight article in the B-2 Test Stand on February 9 less than a month into the Green Run campaign. At the end of the campaign, this stage will be barged to Kennedy Space Center where it will fly with the rest of the SLS and Orion hardware for the Artemis 1 launch.)

The TVC actuator checks were performed using hydraulic power from the test stand, as were the RS-25 engine checkouts that were completed after that. “We [ran] the first part of that test on hydraulic power from the stand,” John Cipoletti, Boeing SLS Green Run deputy test director, said in an interview early this year.

“The RS-25s [went] through a series of tests to make sure that they [could] go through their start sequence, stop sequence, make sure the igniters work, make sure all the valving is working within their calibration parameters.”

“[We then went] through both the hydraulic and pneumatic shutdown procedure. And when the engines [were] finished with that, the whole sequence really is a flight readiness test,” Cipoletti added, “then [we switched] over to concentrate on the hydraulics on the Core Stage.”

The reservoirs in each of the four hydraulic systems (one for each engine) were then topped off and the vehicle hydraulics were configured for the upcoming fueled tests of the stage.

With the stage hydraulics in the flight configuration needed for the hot-fire test at the end of the Green Run campaign, test case five operations moved to the Core Stage Auxiliary Power Units (CAPU) that provide vehicle hydraulic power. The APUs are heavily modified Shuttle hardware; the turbines are no longer hydrazine-driven, but instead use a ground-supplied helium spin start system first and then hydrogen gas tapped off the running RS-25 engines after they start.

In test case five, all four CAPUs were spin-started using the B-2 Test Stand’s helium system; the CAPUs were then run for approximately a minute. The hydraulic circulation pumps, which help maintain thermal conditioning of the hydraulic lines, were also checked out.

The final functional checkouts in test case five were completed successfully in mid-September. “The vehicle is essentially ready for Wet Dress Rehearsal,” Looser said. “There’s some minor closeouts and final test preparations to be done, but overall Core Stage is ready to go into Wet Dress Rehearsal and then into hot-fire testing.”

Test team training was the focus of test case six

Test case six was then completed at the beginning of October after a week’s worth of simulations. “It was really a very good test, we learned a lot,” said Mark Nappi, Boeing Green Run Test Manager.

“[We] went in with a good process, but we tweaked a few things and we’re going to apply those to Wet Dress [Rehearsal], but it was a fantastic test. And I’ll tell you, listening to the test team work through their problems was really impressive. They are very well practiced and [we had] a lot of confidence going into that test that if we had any problems they’d go to address them very efficiently with all the focus on safety.”

Looser added in an October 1 interview while the training sessions were in progress, that “[test case six] consists of a lot of different simulations. The vehicle is powered during that time, and they’ll go through several different test team scenarios to test the team on different contingencies or different anomalies that might occur during a tanking or a hot-fire.”

The test team training sessions in this case were the final practice runs before the final two critical Green Run tests (test cases seven and eight) where the Core Stage will be loaded with propellant for both.  The test case six simulations culminated in a practice countdown that included the mission management team that will participate in the last two big tests.

“We ended that with just a great management sim where the test team threw failures at themselves and the management team so that we could simulate how we would go and address problems that the management team would have to weigh in on,” Nappi noted.

As plans for the Green Run were refined last year, the final set of training practice runs were moved into the test case six timeline. “Last year we had several training exercises with the control room test team here at Stennis where they came in for a couple of weeks at a time and ran portions of the software against emulators to make sure they understood their procedures and they understood how the system was going to work. And that ended with two separate sessions where they had hot-fire runs,” Nappi explained.

“We were going to do a third one in January, and we made the decision that because there was so much time between that last hot-fire run and when we’d actually do Wet Dress that we would cancel that third one and really try to do it during this simulation. The only difference being that instead of being hooked up to an emulator, we would be hooked up to the vehicle and we would have to use red cards or green cards to throw failures at the test team.”

“So it really is very equivalent to those hot-fire training runs that we ran last year except we’re doing it now closer to the actual event itself,” he added.

Credit: NASA/SSC.

(Photo Caption: The B Test Stand at Stennis Space Center on July 14, with the first SLS Core Stage installed in the B-2 position on the left. The B-1 position on the right is used for single-engine acceptance firings of Aerojet Rocketdyne RS-68A engines.)

The Core Stage vehicle in the stand played a role in the tests, but test case six was more focused on test team training. “All the work that is physically being done on the stage in the volumes is getting ready for the Wet Dress,” Looser said. “For the simulation, really the only vehicle configuration is being powered on. There’s nothing unique or special to the physical configuration of the vehicle for test case six.”

Right on the heels of finishing the test case simulations, the forward schedule to the fueled tests was again clouded by the threat of close passage by Hurricane Delta.  Early, long-range storm prediction models forecast the center of the storm to pass close to the broader New Orleans area where Stennis Space Center is located.

Hands-on work on the Core Stage and B-2 Test Stand shifted from test preparations to securing the hardware for potential impacts from Delta shortly after test case six was completed; in the end, the storm stayed far enough to the west that the space center did not have to fully close. Stennis returned to normal operations on October 10, although still under the ongoing COVID-19 restrictions.

“We’ve had a lot of practice this year with exercising the contingency plans, so we’ve got it down to exactly how long it takes,” Nappi said. “All of that [storm preparation] takes about three days to do.”

“So you have that three days of prep when you’re not doing anything else other than prepping for the storm, and then of course you’re down for the storm, and then you come back and you’ve got another three days that you’ve got to do to reconfigure the vehicle so that you can go back to work.”

Nappi also said the Green Run team was able reduce the schedule impact in this instance to only a couple of days because they were able to work on some critical path tasks for the upcoming tests in parallel with hurricane preparations and post-storm reconfiguration.

In a media phone briefing on October 13, John Shannon, Boeing’s Vice President and Program Manager for SLS, said that NASA and Boeing are currently planning to execute test case seven, the Wet Dress Rehearsal that includes the first propellant loading of the stage, at the end of October. “The plan right now, if we don’t have a sixth hurricane that would cause us to shut down, is that we would do the loading of the propellants into the vehicle on October 30th,” he said.


“What we’ll do is we’ll set up a couple of days beforehand. The vehicle will be all powered up and the team will be on console and then we’ll start the loading of the 537,000 gallons of liquid hydrogen and the 196,000 gallons of liquid oxygen.”

“We’ll spend about two weeks looking at data to make sure that all the systems behaved as expected. We’ll go out and inspect the vehicle, make sure there are no surprises, and that’ll set us up well for a hot-fire that right now we’re anticipating will be in the middle of November,” Shannon added. “The tentative schedule is November 14th for that hot-fire.”

Green Run “Phase 2” software updated for Test Case Six

One of the deliverables ahead of the simulations in test case six was an update of the Boeing Stage Controller software following verification and validation (V&V) testing. The Stage Controller is the ground control system that manages vehicle and facility activities for most of the test operations.

For the WDR and hot-fire tests, the Stage Controller will conduct critical propellant loading sequences and finally the terminal countdown sequence, monitoring vehicle and ground systems and enforcing all the criteria necessary for the stage to start the engines in the hot-fire test. Once the Core Stage takes over vehicle control and fires its engines, the Stage Controller remains in charge of ground systems and continues to ensure that critical facility systems are functioning and responsive.

Final development of Boeing’s Stage Controller software was divided into two phases earlier in the process; Phase 1 supported the functional testing through test case five and Phase 2 contained updates for test cases six, seven, and eight. “Verification and Validation of the Stage Controller software formal release 2.0.3 has completed,” Lisa Espy, NASA SLS Core Stage Avionics Lead, said.  “Formal release 2.0.3 was recently installed on the Stage Controller at Stennis.”

Integrated testing of NASA’s SLS Green Run vehicle software with Boeing’s Stennis ground control software was performed in the Avionics and Software System Integration Lab (SIL) at Marshall Space Flight Center in Huntsville, Alabama. Testing was completed at the end of September.

Credit: NASA.

(Photo Caption: The Avionics and Software SIL at Marshall is supporting both Green Run and SLS integrated avionics and software testing. The Core Stage computers and software control SLS Block 1 launches through Main Engine Cut Off; in addition to the Green Run testing supported by the SITF-Q, another set of Core Stage avionics in the “ring” labeled SIL/SITF-D is being tested with the SLS flight software and Northrop Grumman SLS Booster avionics in the Hardware in the Loop Qualification Line.)

The V&V testing was conducted with NASA’s Green Run Application Software (GRAS) running on Core Stage flight computers in the Software Integration Testing Facility Qualification ring (SITF-Q) of the SIL while connected to Boeing’s Stage Controller Unit 1 in a nearby room in Building 4205 at Marshall.  A set of test control center consoles were also looped in from another room in the building.

Marshall’s Advanced Real-Time Environment for Modeling, Integration, and Simulation (ARTEMIS) real-time simulation environment and Managed Automation Environment for Simulation, Test, and Real-Time Operations (MAESTRO) test case automation tool ran the independently developed NASA flight software and Boeing ground system software through dozens of Green Run test cases, predominantly focused on off-nominal runs.

“Stage Controller qualification and validation testing was performed in SITF-Q utilizing ARTEMIS and MAESTRO,” Espy noted. “The majority of Stage Controller SITF-Q testing utilized automated test cases.”

The Core Stage flight computers are running version 14.2.6 of the GRAS, which is a variation of the SLS flight software called Flight Computer Application Software (FCAS). The SITF-Q avionics ring at Marshall will remain in support mode through the Green Run Hot-Fire test.

“SITF-Q will remain in the Green Run configuration until hot-fire has been performed to support any debugging or Stage Controller/GRAS software updates,” Dan Mitchell, NASA’s Technical Lead for SLS Avionics and Software Engineering, said. “One potential GRAS change is in work as contingency planning. After Wet Dress Rehearsal post-test data analysis, this GRAS version may be utilized during Hot Fire to mitigate minor differences seen at cryogenic temperatures.”

Mitchell noted that possible software updates based on lessons learned during the Green Run WDR and Hot-Fire tests will be carried into future updates of the main SLS flight software. “GRAS will not be updated after Green Run,” he said. “Any software updates needed based on Green Run results will be implemented only in FCAS to support the Artemis I mission.”

Lead image credit: NASA/SSC.


Related Articles