Impacted by significant disruptions in recent months, a wide range of assembly and production work for the Space Launch System (SLS) Core Stage nevertheless continues at the Michoud Assembly Facility (MAF) in New Orleans, Louisiana, to complete the test and flight hardware in parallel with facility repairs and now investigation of a serious mishap.
MAF Status Overview:
Manufacturing work on the propellant tanks going into this year had been slowed down by welding issues, and then stopped altogether by a large February tornado. Just as the SLS Program received approval for the recovery plan for the liquid hydrogen tanks from those issues, a liquid oxygen tank dome was destroyed while being processed for the next weld in the Vertical Assembly Center (VAC), temporarily suspending manufacturing again.
At the same time as NASA and Core Stage prime contractor Boeing work through these issues, in other areas of the plant workers are nearing completion of structural assembly of the intertank and engine section and finishing development of the remaining processes in the production sequence.
Tornado repairs and expenses:
The EF-3 tornado took a direct path through MAF on February 7. Building 350 suffered the greatest damage; the U.S. Department of Agriculture’s (USDA) National Finance Center (NFC) was the tenant there and the tornado displaced the approximately 1300 employees from the now-uninhabitable workplace.
Although none of the Core Stage vehicle hardware was damaged, the SLS production and assembly areas adjacent to Building 350 took a significant hit almost simultaneously from the one-third of a mile wide tornado.
Damage to buildings and infrastructure such as power and gas effectively shut down all operations and the tornado also left a few hundred cars severely damaged or destroyed across both NFC and SLS parking lots.
In a NASA Advisory Council (NAC) Human Exploration and Operations Committee meeting at the end of March, Exploration Systems Development (ESD) Deputy Associate Administrator Bill Hill noted that the SLS Program covered the immediate costs of beginning repairs and recovery: “We’re expending SLS funds to do the repairs except for Building 350.”
During a media event on April 25, MAF Director Keith Hefner said it was estimated that the cost of damage to the SLS areas of MAF was a little above $100 million and the total overall cost from the tornado to the Government-owned, contractor-operated facility was $300 million.
The U.S. federal government was being funded through a set of continuing resolutions up until May 5, when President Trump signed into law an “omnibus” appropriations bill passed by Congress earlier that week. The measure appropriated a one-time supplemental amount of $109 million to be used “for repairs at National Aeronautics and Space Administration (NASA) owned facilities that directly support NASA’s mission which were damaged as a result of recent natural disasters.”
The measure also designated the money as an “emergency requirement” so that it would not be subject to the “sequestration” spending caps imposed on discretionary spending in prior Congresses.
The appropriations bill also provided limited funding relief to the USDA for temporary relocation expenses, but the decision on how best to permanently deal with the loss of Building 350 is still under study and discussion. For now, MAF is working on a plan to provide interim, on-site workspace to allow all of the NFC workforce to return there in the next several months.
Production operations resume:
The tornado shut down all SLS operations at MAF for about a week to allow for debris to be cleared and critical repairs to be made to some of the backup power generators. Areas not directly impacted were re-opened to allow assembly work to resume on the engine section articles and the intertank, but the almost all of the SLS areas on the west end of Building 103 and clustered around it remained offline.
While some building repairs could be made, a full assessment of the impact to moving parts like door mechanisms had to wait for temporary power to be established. Although none of the vehicle hardware was damaged, some of it was underneath buildings with roof damage.
Once the door mechanisms were re-validated and the building doors could be opened, those test and flight articles were shuffled from areas where roof repairs were needed to areas where temporary or permanent repairs were complete.
“[In places] where [they were needed to do] roof repair over the tank, we really didn’t want them working [with the hardware nearby],” Steve Doering, manager of the SLS Stages Element Office, explained during an April 25 interview with NASASpaceflight.com. “There were access issues for one and we really didn’t want them working over the assets, so we pulled [them] out.”
In early March, the liquid oxygen (LOX) weld confidence article (WCA) was moved out of Cell N in Building 131 into the final assembly area of Building 103. Shortly thereafter, the flight LH2 tank was moved out of Area 6 in Building 103 into the same final assembly area. The LOX WCA is being used to develop and validate the processes for applying primer and thermal protection system (TPS) foam to the two propellant tanks. Both articles had close calls during the tornado.
Openings in the roof caused by the tornado allowed debris to fall on the flight LH2 tank. Had operations for a pre-planned move of the LOX WCA started before the tornado arrived, it could have been directly exposed to flying debris or turned into debris itself.
The work to bring the critical production areas in Building 110 back online took more time. Also known at MAF as the “Vehicle Assembly Building,” large chunks of the building’s siding broke off and were blown inside.
Building 110 was originally constructed in the 1960s for final assembly of Saturn V first stages, and the asbestos-based composition of the siding used in that era slowed down recovery efforts. Access to the area to assess other damage to the building and its equipment was restricted until after the necessary hazardous materials clean up could be completed.
The area has its own power substation and backup infrastructure, but both were knocked offline for weeks because the tornado damaged both the primary station and the fueling equipment for the backup generators. The subsequent fuel spill also increased the prerequisite cleanup work.
The other SLS work area that took a big hit was Building 131 where the primer and foam sprays on the tanks will be done; that building lost a lot of its roof.
After three months of work, most of the SLS facilities are back up and running. “From a production standpoint, we are fully operational with respect to the tornado damage,” Doering said.
“There’s still a lot of repair work that has to be done, as Keith [Hefner] was talking about, but they have either gotten temporary fixes in place or in some cases like the roof over the 103 engine section area they’ve got permanent roofing complete.
“But in most other areas, all of 110, the big holes in the siding those have all got temporary covers on and there still is no roof in Cell M, which is where all the electrical panels and the pot room are for Cell N’s foam spray area, but we have temporary shelter inside the building to cover for weather. There’s nothing critical out there that [has] to be inside in a climatized environment. That cell wasn’t air conditioned anyway.
“So everything else is longer term repairs for a while, but we’re back to full production in all areas,” he continued.
“There’s still a few areas in which we have to fully activate them, just because it wasn’t their turn. We will likely find a few gremlins left over because we haven’t turned everything back on yet.
In the VAC for example, after the first weld [on the LOX tank qualification article] we couldn’t get the clamp ring to raise up and it turned out we had a connector that had some corrosion in it on that part of the assembly, most likely because of all the water intrusion during the tornado and then with the rains afterwards there’s a big hole in the siding right next to the VAC. So that’s [an example of] little gremlins we’ll end up finding as we go forward.”
Intertank, Engine Section assembly work elsewhere around the factory:
Even before the tornado, the SLS Program was coordinating development of assembly and production processes in conjunction with application of finalized processes to the first flight vehicle and structural test elements. While manufacturing of the propellant tanks continues on one end of Building 103, in other parts of the building assembly of the dry structures of the rocket goes on in parallel.
Structural assembly of the flight intertank is nearing completion; the element is still being bolted up in the structural assembly jig, but the anticipation is that it will be ready to come out soon.
“We should be finishing up structural assembly by the end of [May],” Doering said at the time of the interview. “They’re in the process now of getting it all bolted together and they’ve got the primary structure done and they’re doing the secondary structure now.”
The next phase of production for the intertank is its TPS foam sprays, which are done differently than the propellant tanks. “The intertank has ribs on the outside, so the pocket fills are all manual,” Doering noted. “So that first spray to fill all the pockets is manual and then after that it’s an automated spray.”
The intertank foam sprays are done in Cell G of Building 114, which adjoins Buildings 103 and Building 110, but has a separate roof line. The tornado significantly damaged 114’s roof, forcing cleanup and repairs, but spared critical equipment inside.
“We lost a lot of that roof, but the control room in the cell inside was unaffected,” Doering said. “We did have a lot of moisture intrusion in Cell G, so we had a lot of issues electrically to get back in operation but they got all those fixed and we’re in the process of doing our development sprays for the intertank in there now.”
Leftover Space Shuttle External Tank (ET) hardware is being used to complete foam spray development for the intertanks. “We’ve got two old ETs [intertanks] – an engineering development unit and the GVTA that we’re using to do development spray work with,” he added. (The other parts of the ET GVTA, the Ground Vibration Test Article, were being stored outside when the tornado came through and were destroyed.)
After the foam sprays are complete, the intertank will begin final subassembly integration activities. Avionics boxes, instrumentation, and wiring will be added on the inside along with sections of the two LOX feedlines.
“When the intertank and the LOX tank get stacked the [LOX tank] sump will meet up with the feedlines that are in the intertank,” Doering explained. “You’ll see two big holes in the side of the intertank where they come out and run down the side of the vehicle.”
Structural assembly of the flight engine section is also nearing completion. “We should be done with structural assembly at the end of May,” Doering said. “There will be a few more things to do – brackets and wiring harnesses and stuff like that. And then it will come off the structural assembly jig and go on to a big platform and then all of the integration work will get done there.
“And that’s the plumbing, the orbital tube welding, the TVC [thrust vector control] system, the APUs [auxiliary power units], all of that stuff gets integrated there, and then the boattail gets integrated there as well.”
Doering explained that the integration work itself is very intricate: “The industrial engineers and the mechanical engineers have put together the sequence of events that have to happen in order to be able to put this very packed component together,” he said. “There are certain areas in which I’ll put something on the skin, for example an avionics shelf or a pressure sensor that I may come back a month later and cover up with a helium tank, which means that after that I can’t get to that thing anymore.
“And so, it’s a big puzzle, [the items that go in the engine section] all get lined up so that I’m sure that I’m not putting something on the vehicle that I can’t get to if I haven’t finished qualification for it. Or [in] other cases, I may take and put a dummy unit in there that I know I’m going to take out so that I can get to the piece behind it because I have a risk to getting that piece behind it. It’s a pretty big choreographical nightmare for all of those pieces.”
The boattail fits on the bottom of the engine section and provides protection from the heat generated during launch and ascent.
“It’s a bunch of pieces that get bolted together. And they start showing up in middle of May,” Doering noted. “It’s about a dozen pieces. A dozen main structural pieces and then there’s a whole lot of other stuff that goes in it.”
During integration, the outside of the engine section and the boat-tail will be covered in cork for thermal protection and then paint over the top to keep moisture out while the vehicle is on the ground. Prior to launch, the first two Core Stages are planned to be statically hot-fired in a full-duration ground test, also called a “Green Run.”
Doering noted that the vehicle would have a different TPS setup for the ground test because the heat flux during Green Run is worse than during launch and would erode significant parts of the engine section TPS in its launch configuration.
“The baseline was we put the cork on, we put a layer of this silver tape over the top and then we spray about two and a half inches of sacrificial TPS over the top of that, just so that we’re not destroying as much of the cork when you’re doing the firing for Green Run.”
In any event, some post Green Run refurbishment will be required before launch and a team is studying ways to minimize the amount of work required to remove the leftover Green Run TPS after the firing and apply the TPS for flight.
“We’ve got a team looking at a study that says I’ve got to go take all that TPS off of there and as I take it off I’m probably going to destroy some of the cork,” Doering explained.
“We need to have it on there, because I don’t want to burn all the cork off in the Green Run environment. So we’ve got a study that’s going on right now that says if I lay a layer of a special kind of tape over the top of the cork, I don’t have to put any [sacrificial] TPS on it because its heating characteristics are such that it will withstand whatever the engines are going to do to it and have minimal damage to the cork. That’s going to save me a bunch of refurb time if that pans out.”
Structural testing at Marshall:
Meanwhile, the engine section structural test article (STA) arrived on dock at Marshall Space Flight Center on the Pegasus barge on May 15, completing an approximately 1200 mile trip from MAF that began on April 28. Two days later, the STA and its structural simulator were moved to Building 4619 at Marshall where the test stand for the engine section is located. The STA will be prepared for testing over the next several months.
The STA is a version of the engine section outfitted just for structural testing. It includes the primary structural elements of the flight article – the barrel, a welded ring at the forward end, and a thrust structure where the engines attach. The STA is not fully outfitted as the flight article will be, including elements of only one of the two LOX feedline “downcomers” which will provide sufficient loads data during the test.
“There [are] other things that are different from the flight vehicle and that structural test articles,” Doering also noted. “There’s only two full-up engine interfaces. Inside of the STA is the thrust beam that the RS-25s mount to, but there’s only two [fully outfitted interfaces] – all the hardware, mounting associated with it – because you don’t need all four of them. You’re going to do the test representative in two locations and then you’ll interpolate for the rest. It’s a way to get all the data you need without going through all the extra cost of making it identical.”
An earlier plan for structural testing included a test case to structural failure for some of the elements, but Doering said that plan is still being studied.
“The idea is we’re going to test them to understand where our maximum margin is,” he explained. “You can do that in one of two ways. You can test them to failure or the other philosophy is you can test them until you get a full factor of safety – a safety factor of 2.0. Both of those cases will tell you where you have margin in the system and where you can potentially take mass reductions. Testing to failure, depending on how you do it, may only give you your exact margin in one location depending on how it fails. It may not give you a full understanding of all your margin across the whole element.”
The minimum factor of safety for the vehicle is 1.4 – “1.4 is where we’re at, so where you’re looking at where’s my envelope, if I get to 2 and I can go beyond that, I know I’ve got margin there,” Doering explained. “So then I can start talking about design decisions based on that. But we won’t go below 1.4, not for a manned vehicle.”
Four of the five Core Stage elements will be structurally tested at Marshall before the first launch. Originally a structural test article for the forward skirt was planned, but analysis said that wasn’t necessary.
“We had planned on it – where the LOX tank is going to be tested, it was designed to test the LOX tank or the forward skirt,” Tim Flores, integration manager for the SLS Stages Element Office, said during the April 25 media event for the engine section STA at MAF. “So we went in thinking we were going to test both of those. Then we showed that we had high enough margins that said, we don’t have to test that.”
Mishap latest disruption to completion of welding:
Back in Building 110 another setback to finishing manufacturing of the propellant tanks occurred before the agency could formally sign off on the plan to recover from the previous one. The final details of the SLS Program’s recovery plan from the change to the welding pin tool in the VAC were reviewed and approved in an agency-level meeting on May 5, but welding was once again stopped by a mishap two days before that destroyed a LOX tank dome during final preparations for the next weld.
The aft dome of the LOX qualification tank, which will be used for structural testing, was loaded into a tool called an “in feeder” that positions domes and barrels underneath the existing structure as they are welded together in the VAC. Final preparations for welding the dome to the rest of the LOX tank were being completed when the mishap occurred on May 3.
“We were processing the qualification oxygen tank and we were ready to move [the dome] into position,” NASA Associate Administrator for Human Exploration and Operations Bill Gerstenmaier noted in a media teleconference on May 12 regarding the Exploration Mission-1 (EM-1) crew study.
“As we were getting ready to move it in underneath the tank, to remove some protective covers on that dome a technician lowered the dome down and when they did that it essentially ran the dome into eight fixtures that hold the dome in the device [and] caused significant damage to the dome. It’s probably not repairable.”
“This was a significant event for us, we call it a Class B mishap,” Gerstenmaier continued. “Boeing has formed a team to go look at it, NASA has formed a team to go look at the event, we’ll learn from this event and we’ll move forward.” Although there were no reported injuries, property damage values in Type B mishaps are categorized as losses of “at least $500,000, but less than $2,000,000.”
“Following a careful review of the Vertical Assembly Center, NASA has released the VAC for normal operations to the SLS Core Stage Element office,” NASA spokesperson Kim Henry said in an email with NASASpaceflight.com. “NASA and Boeing are working together on incorporating corrective actions and procedures before resuming welding along with evaluating the hardware and production schedule.”
Gerstenmaier noted that when welding is allowed to resume, the immediate plan is to bring forward the aft dome earmarked for the flight LOX tank: “We have another dome available – a flight unit – and another one also available, so we’ll roll that into the sequence and we’ll go back and take a look at it.”
After welding, the next step for the LOX tanks will be for proof testing hydro-statically in Cell F in Building 110.
The weld pin tool issue had already delayed welding the LOX tanks and changed plans for the hydrogen tanks that were welded before the unintended consequences of the pin changes were understood.
After re-validating the weld schedule using the original pin design with a second LOX weld confidence article, the tornado hit MAF as they were getting ready to start welding the first LOX tank, the structural qualification article. Facility cleanup, repairs, and power restoration took away an additional two months before welding started in April.
For the already-welded hydrogen tanks, the SLS program decided to use the qualification tank, serial number 1 (S/N 1) as-is, only pressure testing that tank up to levels to be used in upcoming structural testing. It was also decided to weld the next hydrogen tank (S/N 3) and fly that on EM-1. The tank parts for the next SLS flight were already on-hand at MAF; welding of the metal panels into domes and barrels for S/N 3 was already underway and the plan was to begin welding those into a complete tank immediately after the two LOX tanks are completed.
The LOX dome mishap has clouded the picture somewhat. Most of the new LH2 tank parts were complete, but at least one dome still need to be welded together. At the time of publication, it was unclear whether the replacement LOX tank dome for the one destroyed in the mishap had already been welded or not. If not, it was also unclear whether the gore panels and ring for the replacement LOX dome would need to be welded in the Gore Weld Tool and the Circumferential Dome Weld Tool before the remaining dome for the new LH2 tank would need to be welded.
Spokesperson Henry reported in her email that “NASA and Boeing are now evaluating on-site assets and determining the revised production schedule for hardware to be welded in the VAC.”
The plan for the original LH2 flight tank (S/N 2) in the near-term is to use it for validation of new production processes. “I’m going maintain its flight pedigree, but I’m going to use it to buy down some risk in our process as we go forward,” Doering explained. “I’ve got to go put primer on it, I’ve got to put foam on it, I’ve got to put it in the clean cells, all of which have not been fully wrung out from a process perspective because my first one hasn’t gone through there yet.”
“And I can do that sooner than waiting for the qual tank to be ready to be put in there,” Doering said. “So while I’m putting instrumentation on it, getting the covers on it and getting it ready where you saw it today (the April 25 media event), in the meantime I can take that former flight tank and do that now and that’ll buy me some schedule back for when I ship the qual tank.”
The S/N 2 tank would first go into the internal cleaning cell, Cell E in Building 110. All the tanks will eventually be put into the cell, which washes them with water. “It’s a spray, it’s like a big dishwasher,” he explained. “We put this wand down the middle of it and it [the water] drains out the bottom.” That fits inside the tanks through the manholes in the center of the forward and aft domes.
The former flight tank for EM-1 is currently back in Area 6 of Building 103 after the temporary move to allow temporary roof repairs to tornado damage. More permanent roof repairs were seen underway when the engine section structural qualification article was rolled out to the Pegasus barge on April 27 for shipment to Marshall.
The LH2 qualification tank (S/N 1) completed pathfinding of the proof test facility in Building 451 and was moved back to Area 47 of Building 103 in mid-April; in addition to replacing the proof test instrumentation and manhole covers, it will have its welds reinspected after the proof testing before it would be ready for its turn to go into Cell E for internal cleaning.
“They’re in the process of removing all the instrumentation and the proof covers, they’ll do that first and then they’ll do the PAUT,” he said, “the phased array ultrasonic testing on the welds. You saw the PAUT [tool] was staged right behind where you guys were sitting today, it’s that big white thing with the robotic arm on it.”
The former LH2 flight tank may also be used in the near-term for some additional risk reduction work in the primer and foam spray cells in Building 131.
“The other pathfinder thing we’re thinking of doing with the [former] flight tank is [to] roll it into Cell N because you’ve got to get it lined up just right so that when it tracks, it doesn’t corkscrew,” Doering explained. “It spins pretty fast when you do the [foam] sprays and the first time we put the weld confidence article in there – the LOX weld confidence article, the short stubby tank – it took us two weeks to get the thing lined up. Well now we’re going to put one in there that’s three times as long, that’s another risk reduction opportunity that we’re looking at.”
In parallel with the risk reduction activities, work to test and evaluate techniques to repair the tank will continue without being in the critical path for EM-1. “It will not be used on the first Core Stage, but if the repair techniques work it will fly,” Doering said.
Finalizing production processes:
While the tanks are being proof tested and cleaned, development of the subsequent processes to support those are nearing completion. After tanks are cleaned, they will be sprayed with primer for corrosion protection, then with thermal protection system (TPS) foam.
The primer and foam sprays will be done in Building 131, which was perhaps the most severely damaged by the tornado. Development of the processes to apply the desired amount of primer and foam has employed the original LOX weld confidence article (WCA). The tank helped to develop and verify the primer spray setup and procedures in Cell P last year and has been serving the same purpose in the foam spray cell (Cell N). It is back in Cell N after temporary roof repairs were made.
The foam that will be sprayed on the propellant tanks is done using different processes; the acreage foam will be sprayed on the cylindrical section of the tanks with an automatic spray system. The foam for the hemispherical domes will be sprayed manually. The LOX and LH2 tanks will be covered with different thicknesses of foam and development of the spraying techniques to apply the desired thickness was done first.
“What they do is they spray on paper,” Doering explained. “So they wrap the whole tank in paper, they do the spray, [and] they validate their thicknesses – they do core samples and make sure that it’s nice and homogenous and [the spray] works at different thicknesses for LOX and for hydrogen.
“Then they tear the paper off, throw it all away, and do it again. They do it up to three or four times until they get the sprayers and the tracking system and the whole control system set up so it’s consistent and you’re getting the product on the tank that you need.”
Doering said those development sprays are wrapping up and they are getting ready to do verification sprays on the bare metal: “They ought to be getting their verification sprays started – they do all their development sprays [and] once they get all their parameters down, they do verification sprays. The operators are doing it, the same guys that will do it for the real hardware – they’ll do core tests and pull tests on that to verify that their process is correct. And they’ll be doing that probably in the next month.
“What they did on the paper is they got the control system ready, now they’ve got to go validate adhesion of the spray to the bare metal, so they’re spraying an eighteen-inch wide swath along the length of the tank.”
Doering noted that the same process of spraying on paper first then bare metal is being done for the sprays on the tank domes. Different thicknesses of foam will also be applied between the tank acreage and the domes.
NASA announced in the teleconference on May 12 that the EM-1 flight would remain as baselined – a three-week, uncrewed, lunar orbit flight. The programs across ESD researched the possibility during the study, but the impact of flying crew on EM-1 for the Core Stage would have been lower than for elements of the Orion or Exploration Ground Systems (EGS) programs.
“The Core Stage right now is agnostic to what’s on top of it,” Doering said during the April 25 interview. “As part of the study we had to go look at what do you do to the hardware. I know of no structural design issues that we [would] have to go deal with, there are no human rating issues that we have to go deal with with the exception of four waivers.
“These were waivers that we took to the vehicle design – because we designed it to be a human-rated vehicle – that we waived because we didn’t have to do them for the first flight.”
As a part of the standing reviews from inside and outside NASA of the ESD programs, NASA recently acknowledged that the old November 2018 target date for EM-1 was unachievable. ESD is still working with the SLS, Orion, and EGS programs to figure out a new target date sometime in 2019.
The disruptions from welding issues to the tornado made the Core Stage more critical to the overall EM-1 schedule, bringing it “neck and neck” with the Orion European Service Module (ESM).
At the time of the interview on April 25, Doering estimated that within the Core Stage schedule, “my supposition will be that the engine section is probably close to the critical path – it’s staying where it is – but the hydrogen tank is probably neck and neck with the engine section.
“Remember we’re bringing this other hydrogen tank up and I would have had it done already, so bringing the third one up has put its schedule out far enough for the flight tank that it’s probably neck and neck with engine section.”
In the May 12 teleconference, Gerstenmaier didn’t believe the recent LOX dome mishap would have a great schedule impact: “I don’t think overall that impacts much on the schedule. It’ll put a little delay because we essentially have stopped work for several days on the facility until the investigation teams had completed their work.”
Although manufacturing of the propellant tanks is likely to be more critical to the schedule when welding is allowed to resume, there are other critical areas of assembly and production and Henry noted that “work continued on the core stage in other areas of the factory.
“This includes the assembly areas for the intertank and engine section, welding of domes and barrels, as well as the preparation areas for the liquid hydrogen tank test article – the next test article scheduled for shipment to Marshall for testing.”
(Images: NASA and L2 which includes, presentations, videos, graphics and internal – interactive with actual SLS engineers – updates on the SLS and HLV, available on no other site. Additional renders by L2 Artist Nathan Koga. The full gallery of Nathan’s (SpaceX Dragon to MCT, SLS, Commercial Crew and more) L2 images can be *found here*))
(L2 is – as it has been for the past several years – providing full exclusive SLS and Exploration Planning coverage. To join L2, click here: http://tm.nasaspaceflight.com/l2/)