Boeing powered up the Core Stage for the first Space Launch System on the B-2 Test Stand at the NASA’s Stennis Space Center in Mississippi last week following a long stand down due to the coronavirus pandemic.
The first Core Stage is at the outdoor facility for a Green Run test campaign that culminates in first-time propellant loading followed by an inaugural test-firing.
Teams got within a week of the power up milestone in mid-March when the COVID-19 emergency forced much of the United States to shelter at home and stay there for several weeks. Stennis was one of multiple NASA facilities where all on-site work was halted.
Before walking away indefinitely and leaving the Core Stage to an occasional visit from a skeleton ride-out crew, work had to take a few steps backward to leave things in a stable configuration for long-term storage.
After two months away from the test stand, about two-thirds of the NASA and contractor team gradually began returning to on-site work in groups with the remainder working remotely to help resume testing.
Power up testing is the second of eight test cases in the Green Run campaign; beginning with the initial power-on, Boeing hopes to follow a revised schedule that would lead to the Wet Dress Rehearsal (WDR) propellant loading test countdown in the early autumn, followed by the hot fire test a few weeks later.
Stage powered up for the first time in the test stand
Test Case 2 started early last week in the B-2 Test Stand during bad weather, which slowed down the initial power up of the test stand and vehicle computer systems. “We got started on June 22 and had pretty bad weather at the beginning of the week, had a few problems that slowed us down with some of the power supplies coming from Mississippi Power, and so we really didn’t get into powering up the vehicle until [Thursday the 25th],” Mark Nappi, Boeing Green Run Test Manager, said on June 26.
“We bring up the Stage Controller, make sure it’s working properly, charge our batteries, and then go into a slow power up of the vehicle. It’s kind of one box at a time. The team thinks that by the end of second shift [Friday the 26th] they’ll have all of the line replaceable units (LRU), the black boxes, all powered up so that we can do a couple of functional tests.”
Boeing’s Stage Controller is a computer system that conducts the final seven of the eight Green Run tests of the Core Stage. It interfaces with the stage flight computers and software directly and provides interfaces to the test control team for the systems of the stage and the test stand.
After the Stage Controller was booted up, the Core Stage flight computers were activated, and the stage batteries tested. “One of the first things we did was make sure that we had good comm to both the stand and the Stage Controller, and the first series of tests that we did were focused on the batteries,” John Cipoletti, Launch Team Lead and Green Run Deputy for Boeing, explained.
(Photo Caption: A color-coded schematic diagram of the different avionics boxes distributed throughout the SLS Block 1 vehicle. The flight software runs on the three flight computers in the forward skirt of the Core Stage. During vehicle power-up last week the flight computers were brought up first, followed by verifying that the four battery units in the intertank could be charged and that power for the stage could be transferred from ground supply to the vehicle batteries.)
“We made sure we could charge the batteries and discharge the batteries, that we could do a power transfer between ground power and vehicle power. And that was all done with the flight computers up. When all three flight computers are on and they’re talking to the ground and supplying data, it’s called a Flight Computer Operating Group (FCOG), and so the FCOG has been running for a while now.”
“I’d say we’re probably a good sixty to seventy percent there right now, and we’ll work through the weekend to see when we can finish,” Nappi noted last Friday. “We’d like to finish on Sunday (June 28). If there’s too much work that remains, we’re going to give the team Sunday off and come back in on Monday and finish up on Monday and Tuesday.”
In addition to powering up all the avionics boxes on the stage, the team is performing a few functional tests as well, such as with the vehicle’s Redundant Inertial Navigation Unit (RINU). “I think they just had our inertial unit, the RINU, up and they’re running some tests for its gyro (gyroscope) compass align (alignment), and then they’ll take it inertial and make sure it’s functioning,” Cipoletti said.
“That restricts some of the work that we do on the vehicle because we don’t want to perturbate the results on those alignments with any vibration coming in.”
Mr. Nappi added, “I think the systems are acting very well. The responses have been very good. This is the first time the team has done this, but it hasn’t resulted in any non-conformances. The systems are running good and the team is learning. I think that’s the best way to characterize it.”
The SLS structural qualification test campaign concluded on June 24 as @NASA_Marshall engineers purposely pushed a liquid oxygen tank test article to its limits. Water gushed out of the tank as it buckled within 2% of the predicted failure value. MORE >> https://t.co/L6mOUHhnkJ pic.twitter.com/AEHv4haucW
— NASA_SLS (@NASA_SLS) June 25, 2020
Boeing spokesperson Patricia Soloveichik said today, 29 June, that power-up data is now collected and the vehicle is powered down. NASA will next review the data with Boeing and do not consider the test complete until after the results are analyzed.
“We have time set aside for detailed data reviews,” Cipoletti said. “Then our test director will have a break of configuration meeting which is essentially ‘hey, is there anything that you saw in any of the data that would indicate that we’re not clear to move on?'”
This is the first time the Core Stage computer software, avionics, and network have been up and running since they were tested in December 2019 during final assembly at Michoud Assembly Facility (MAF). The power up test case is focused on verifying the health of all the computer systems and the connectivity between them.
In contrast to functional bench testing performed at MAF prior to its delivery, the stage is communicating with a ground control infrastructure at Stennis similar to the one that will be used at the Kennedy Space Center (KSC) for launch operations.
“We purposefully are checking out the vehicle at Stennis with the same kind of parameters and criteria that we [will] when it [comes] to Kennedy for the launch. Up until this point we’ve been working with analysis and predictions and supplier data, but this is the first time in an integrated vehicle with the ground system like this that we’re seeing how everything responds,” Cipoletti noted.
The stage is being commanded by the Stage Controller, either manually from console operators in the test control center (TCC) or through automated sequences. The flight computers perform the requested commands and stream real-time data back to the Stage Controller, including the result of commands and instrumentation readings.
For this critical test campaign, other NASA centers are monitoring the progress of activities, and the initial power up is an opportunity for control rooms at the Huntsville Operations Support Center at Marshall Space Flight Center (MSFC), the Launch Control Center at KSC, and in Mission Control Center at Johnson Space Center to verify their remote connectivity and receive test data.
“As we get further into the test sequences, past [Test Case 2], we’ll power up faster because then we’re really just bringing up all the avionics to support other purposes. But this first run through is slow and deliberate because we’re bringing up each one of the LRUs to make sure that it’s functioning,” Cipoletti noted.
The Green Run campaign at Stennis is the final, major pre-launch development test of the newest piece of NASA’s new launch vehicle. A generic “green run” is an acceptance test firing of new rocket engine hardware, and the test campaign at Stennis will be capped by a test-firing of the stage; however, the Stage Green Run also refers to the whole development test campaign for the Core Stage at Stennis.
The hot-fire test is the first time in the program that a Core Stage will ever been fired with all its subsystems working in unison. Ground-firing the stage before it flies gives NASA and Boeing the opportunity to measure how the interoperating subsystems perform together and their operating margins.
The program has developed analytical models to predict the behavioral dynamics of the different subsystems of the stage, from propulsion and hydraulics to structures and thermal protection; the test-firing on the ground is the SLS Program’s only opportunity to anchor those models with real test data before its first test launch.
NASA decided to keep the test in the first-flight development campaign last year, but dropped plans to acceptance test any future stages.
Working with COVID restrictions
Core Stage-1 arrived at Stennis in mid-January and was installed in the B-2 Test Stand position later that month. Following the initial Modal Test, final production work on the stage was performed in parallel with finishing the first two phases of Stage Controller development.
Work to fully connect the liquid hydrogen (LH2) Main Propulsion System (MPS) feedlines and fill and drain line inside the engine compartment was originally intended to be performed while the engine section was oriented vertically at MAF. Because delivery of the stage from Michoud was behind schedule, the LH2 feedline work was moved to Stennis to speed up delivery by completing final assembly of the stage while horizontal.
The MPS connections were completed, the Stage Controller passed its readiness reviews, and the test team got within a week of starting power up tests in mid-March. “We had some production work that we needed to complete on the vehicle prior to going into this set of test cases,” Nappi explained. “Prior to the COVID shutdown, we had gotten within five days of finishing up that amount of work.”
“We left work, and we didn’t come back until the 14th of May, so we were down for exactly 58 days. And when we came back, we said we’re not going to go back full steam ahead, we’re not going to go back with a hundred percent of our workforce, so we need to bring things up slowly.”
Before leaving the vehicle and the test stand in March due to the COVID-19 public health emergency, the team had to back out of some of the preparations for Test Case 2, so there was more than five days worth of work to do when they returned. With social distancing guidelines in place to reduce health risks, it was also not going to be possible to bring the workforce back quickly or in full.
(Photo Caption: A Boeing employee looks at protective coverings at the base of the Core Stage boattail in mid-May when the initial wave of personnel was allowed to return to work. The nozzles of the four RS-25 engines are covered in yellow bags along with engine-mounted heat shield (EMHS) blankets and temp coverings above them. Purge air is run through the engines to keep them dry.)
“We had about 14 days of work to do to get into position where we could power up,” Nappi explained. “We allowed ourselves 30 days to get that 14 days of work done, and we brought the workforce back in some waves.”
“The first wave was essentially just our hands-on workforce with a minimal management team so that we could start doing some configuration work to get ready for the test. We did that for about 10 days.”
Mr. Nappi added, “When you pick up a problem, our ship-side support needed to be there to resolve problems more efficiently than trying to do it from home. So we brought wave two in, which was our support staff, and then after about another 10 days or more we brought in wave three, which was our power-up control room team so they could get ready for power up.”
With some of the workforce connected remotely, response times can be longer. “We’re probably operating at about 60 to 70% of the people you would normally have on the stand or here on Stennis,” Nappi said.
“The rest of the people are still supporting from home, so there are some — call it inefficiencies. But for the most part, we’re operating with all our hands-on workforce, with a great deal of our support staff. [For] difficult problems that are unexpected that we don’t have people here for, it might take a little bit longer to get their response or to get them here to help us resolve those problems. Those problems have been sort of few and far between.”
Schedule to the two big tests
The schedule for launch readiness of Artemis 1 currently depends on the timing and results of the hot-fire test. After the test is completed satisfactorily, the Core Stage can be refurbished and transported by barge to its launch site at KSC where NASA’s Exploration Ground Systems program is waiting to assemble the Orion spacecraft with the whole SLS vehicle.
Test case 2 checked out the computers, and subsequent test cases will check out mechanical and propulsion systems to get ready for the critical tests with propellant onboard, the wet dress rehearsal (test case 7) and the hot-fire (test case 8).
In preparation for powering up the stage and resuming testing, NASA and Boeing looked at the checkout test cases as a set. “We’ve run one test already, that was Test Case 1, and that was the Modal Test that we did right when we got to Stennis,” Nappi said. “Prior to that, we did a test readiness review for that one test case, and so that one is behind us.”
(Photo Caption: Syncom Space Services (S3) employees Cheley Carpenter and Shelton Dunn work on Level 19 of the B Test Stand at Stennis in mid-May when the initial wave of personnel was allowed to return to work. Work progress was halted and the stage was backed out of preparations for Test Case 2 in mid-March when COVID-19 forced most people to stay at home for several weeks.)
“As we got the production work done that I talked about a little earlier, we knew that we were going to have to do a test readiness review. And what was decided was we’d do something called a series test readiness review, so it’s a big test readiness review that would cover from test case 2 all the way through test case 6.”
“We had that test readiness review, it went extremely well. I think it was a five-hour review that we took three actions on, and they were very small actions, so it was a very, very well run review. We went into it very cleanly and the management team gave us the go to proceed.”
With management approval to proceed through the test cases up to the WDR, the process to proceed into the next test should be shorter. “Once we got that, prior to every test case, two, three, four, five, and six, we do a mini test readiness review, which is basically just saying ‘OK, how did the last test go? Did it go OK?’,” Nappi explained.
“And if it did, what are our constraints that we had identified for this next test. If they’re clear, then we’re going to go into the test. And that mini test readiness review is just about an hour or so and it’s conducted right in the control room.”
Test case 3 will validate safing routines that the Stage Controller would execute if a problem is identified that requires backing out of an operation in progress. The Stage Controller would send commands to the Core Stage and test stand systems to put them back in a safe configuration.
Test case 4 is a Main Propulsion System (MPS) leak and functional test that will validate the operation and pressure integrity of propellant lines and valves in the engine section at ambient temperatures. Test case 5 is functional checkout of the vehicle’s hydraulic and thrust vector control (TVC) systems.
Our first teams have returned to @NASAStennis and #NASAMichoud with COVID-19 safety protocols in place. They’re resuming @NASA_SLS Artemis I Green Run testing, building the next Artemis core stages, and accelerating Exploration Upper Stage development. pic.twitter.com/ezdxMHxbxl
— Boeing Space (@BoeingSpace) May 21, 2020
“We’re testing out the vehicle hydraulics, TVC, and also the Core Stage engines, the RS-25s,” Cipoletti said about test case 5. “That is our biggest integrated test, we’ll have our MPS systems up, our hydraulic systems up, and the engines turned on and dealing with the pneumatics and the hydraulics and the pre-start checks and that kind of stuff.”
Test case 6 is a dry-run of the countdown for the WDR and hot-fire tests, and a software update is expected for both the vehicle and the Stage Controller before that test. An update of NASA’s Green Run Application Software (GRAS) that runs on the flight computers for the test campaign at Stennis and Boeing’s Phase 2 software update for the Stage Controller will be loaded for the simulated countdown, WDR, and hot-fire tests.
“Phase 2 development work at Titusville is going well, and there’s a lot of margin between when their product needs to be done and when it’s needed,” Nappi said. Boeing’s Space and Launch Division headquarters is located in Titusville, Florida.
Integrated verification and validation testing of both software packages is being done at the Software Integration Test Facility (SITF) at MSFC; the hardware-in-the-loop lab has a working set of avionics for both Core Stage and Stage Controller tied together with emulators for the rest of the vehicle.
MSFC is still in Stage 4 of NASA’s COVID response framework and clearing support personnel on-site to troubleshoot issues is slowing progress.
“The problem resolution hasn’t gone as quickly as we had hoped it would, primarily due to the lack of people that are allowed to go onto the center,” Nappi said. “We’ve been asking, and as a matter of fact just yesterday we got approval to be able to bring more people onto the center, and we’re asking NASA to do the same thing.”
“As the schedule stands today, the delivery of the Phase 2 software is right on top of Wet Dress Rehearsal, so it’s definitely a race on which one is going to be ready first.”
While the new software is being loaded, work will also start to configure the vehicle and the stand for WDR and hot-fire. Internal platforms will be removed from the forward skirt, intertank, and engine section. Ground test instrumentation will be configured for the test and the stage will be closed out for propellant loading and firing.
(Photo Caption: Core Stage-1 is lifted into the B Test Stand during installation operations at Stennis on the morning of January 22. It is expected to be at Stennis for the Green Run test campaign for most or all of the year. Following a successful hot-fire test and some refurbishment it will be removed from the stand, placed back on NASA’s Pegasus barge, and transported to its KSC launch site. The space agency hopes the Core Stage can be transported to KSC some time around the beginning of 2021.)
As things currently stand, the WDR is schedule for autumn of 2020 with a test-firing a couple of weeks later. “When we went through the stand down, we knew immediately that we were going to get impacted by this thing, so we built a couple of scenarios for our customer and our senior management at Boeing and they were kind of bookends,” Nappi explained.
“They said if we go back to work and we hit the ground running because we weren’t gone very long, this is when we’ll finish and it looks like it’s the soonest we can finish based on X number of days being off work. If we come back at a slower pace, it looks like this and that would be our right bookend.”
“We ended up with a wet dress [rehearsal] at around the end of September and then the hot-fire would be around the second week or so of October,” he said. The schedule still includes allowances for some weather delays and issues that come up during work.