SLS Core Stage team recovering from consequences of weld pin change

by Philip Sloss

NASA and Space Launch System (SLS) Core Stage prime contractor Boeing recently resumed welding elements for the launch vehicle’s first flight after a technical issue suspended welding last year.  A change to a welding tool in the Vertical Assembly Center (VAC) at the Michoud Assembly Facility (MAF) in New Orleans, Louisiana, had unintended consequences that in part disrupted the assembly and production schedule for the Core Stage and helped push the forecast target date for the first SLS launch on Exploration Mission-1 (EM-1) into 2019.

Weld strength issue:

The Core Stage consists of five major structural elements – the liquid oxygen (LOX) and liquid hydrogen (LH2) propellant tanks are the “wet” structures, while the engine section, intertank, and forward skirt are “dry.”  Four of the five elements are welded, the bolted intertank structure being the only exception.

Welding of the structures is performed using a process called “self-reacting friction stir welding,” where a spinning pin is pushed into and moved through the material (an aluminum alloy) at high forces.  In contrast to conventional friction stir welds, self-reacting friction stir welding uses two shoulders on opposite sides of the weld rather than a large structure/anvil on the opposite side of the pin.

“Basically what you have is…a root shoulder, which bolts to the weld head, you have the pin that runs through the middle of it, and you’ve got a crown shoulder and so the whole thing spins but the root gets pushed into the article and the crown that gets attached to the pin gets pulled into the article – so it’s self-reacting, it squeezes it together,” Steve Doering, manager of the SLS Stages Element Office, explained in an interview with  “So it’s a very complicated thing.”

“And then inside, the part that’s in the metal piece, in between the root and the shoulder has a thread pattern on it like a bolt – similar to a bolt,” he continued.  “And what that thread pattern does is stir the material up when it plasticizes.  So it mixes it and gets it uniform and you don’t get pockets of alloy, pieces of the alloy components leeching out and moving in places.  It keeps it homogeneous.”

In the case of the Core Stage structures, the technology needed to weld the thickness of the material for the two propellant tanks is pushing the state of the art.  “There’s going to be a lot of PhD’s coming out of this, in the weld arena,” Doering noted.  Both the Core Stage LH2 and LOX propellant tanks are thicker welds than anyone is currently doing.  The engine section and forward skirt dry structures are of a more typical thickness seen elsewhere in the industry.

Welding of the domes, barrels, and rings that make up the major structural elements is done in the large VAC in Building 110 at MAF, and by the beginning of 2016 the first weld confidence articles (WCA) were being completed.

The full-scale welding is done in a multi-step process, as Doering explained:  “The way we qualify our weld schedules is we do a whole series of panel welds, about three or four feet long.  [We] pull them apart, break them, do tensile testing on them, do the metallography breakdown, then we do a weld confidence article – and that gets us the full-scale weld and then we cut out a bunch of pieces from that weld and we check all those.  And all those came back good.”

The VAC welds around the circumference of the structures, joining domes, rings, and barrels to other barrels.  On inspection and evaluation of the early, completed full-scale welds a couple of issues were identified, one that affected the welding pin and one that affected the welded material.

“After we did the weld confidence article for the oxygen tank, which is our thickest material, we found two things that we learned as we’re pushing the state of the art for these welds, self-reacting friction stir,” Doering explained.

“You’ve got two barrel panels that are welded together each made up of eight panels.  And so [at] the vertical intersection as the circumferential weld comes around, we were seeing some indications [of] some really small but significant voids in the weld land at the intersection.  And so we were trying to figure out what’s causing that.

“The other thing that we found, which is related to the pin was we were getting microcracks at the base of the pin where it meets the root shoulder.”

Doering explained the rationale for making the change:  “With these microcracks at the pin at the root shoulder we had a higher propensity for potentially breaking a pin.  Think of this as a big drill bit which you’re moving it sideways through the metal or through the wood – whatever material it is – and it’s easy to break.  You break drill bits all the time when you’re at home and you’re drilling through something and you’ve got a little torque and it snaps because it’s very brittle.”

“Breaking a pin is not a huge deal in a circumferential weld, but it requires a repair, it’s time,” he continued.  “When you don’t do a controlled shutdown of the weld head then it locks it in there, so you have to drill it out.  It poses some problems that we would rather not have out in front of us to go work.  That’s what drove the need for the change.”

“So we made a very small change to the threading,” Doering noted, “where we knocked off the very points of two of the threads and we tapered them a little bit to give more material back there at the root.  And then we did another series of test panels and they were testing good.  And then as we were going through the process after that, we found a way to solve intersection voids, which in the end became unrelated to the pin.”

After the modifications to the pin were made to address the risk of breaking the pin, a new set of test welds were performed and passed testing, and full-scale welding in the VAC resumed, with most of the welded elements being completed by the end of the summer last year with the two LOX tanks, the thickest of the welds, up next.  By that time, though, the Core Stage team had begun seeing random, infrequent strength failures in test panels welded with the modified pin tool.

“We were finding random areas in which the strength of the weld is below the design requirement,” Doering said.  “And it is not predictable and it’s not something you can find through any kind of NDE, non-destructive evaluation technique.  When you dissect the weld itself and do the metallography, the materials analysis on it, you can see what the issue is – it creates a little brittle layer at the very top which you rely a lot for your strength on elongation of the material.

“It’s kind of like a rubber band,” he explained, “you stretch it and stretch it and it does great and then all of a sudden it gets to the point where it doesn’t want to stretch any more and [snap] it fails immediately.  Because one little spot starts to open up and once it opens up the rest opens up like a zipper.  It’s the way the strength issue works here with these.”

The plan to weld the LOX tanks and finish the VAC welding of all the flight and structural qualification articles necessary for the first flight last year was suspended in order to investigate the issue.  Lots of test panels were welded to try to characterize the problem, which would appear unpredictably.  “About every fifteen panels or so we would get some bad data,” Doering explained.  “And it’s really not repeatable – it’s not like it’s every fifteenth one, it’s on average about every fifteenth one.

“And we were spending a long time researching it and creating samples,” he noted, “[because] we have to make a lot of samples if we’re only getting failures one out of every fifteen.”

After doing all the test panel welds and research, the current understanding is that the original pin design is the only one known to meet the design strength requirements.

“We still don’t know enough about self-reacting friction stir welding at these thicker welds, what the sensitivities are that affect the strength of the weld,” Doering explained, “we have a point solution [where] we have data that says that it works – that’s the RPMs (revolutions per minute), the forces, and the…pin design.”

“But it’s a point solution,” he noted.  “So [if there is] any deviation from that, there’s not enough either industry data or research data at production levels or anything out there to tell us what the real sensitivities around that are.  So because of that, the only way to guarantee that you have a good weld on your tank is to destroy the tank – there’s not an NDE technique to verify that this particular random mechanism we saw on the [modified] pin exists anywhere in the weld.  You can’t tell unless you take it apart.”

In addition to going back to the original pin design, they have added an extra step to the process before making the full-scale weld.  “On the VAC prior to doing the weld we are doing what we call pre-qualifying the pins,” Doering explained, “because there’s always some variability in your tooling.  [If] I have two different pins, there will be some variability in them – [because] they’re manufactured items.”

“And because we don’t yet know what all the sensitivities are, I’m pre-qualifying my pins.  So I’ve got a handful of pins, I go do welds on the VAC just prior to doing my weld and then I pull [test] all the samples I get back.  If they pull good…then now I’ve got a good pin, now I’ve got a good weld schedule, my risk to having a bad weld when I actually weld the tank becomes very low.”

Although research on doing these high-thickness self-reacting welds will continue it’s not known how long it will take to better understand the dynamics of the issue.

The issue does not impact the strength of welds at smaller thicknesses, where there is more industry and research data, and so does not affect the engine section or the forward skirt.  The welding of two rings to the forward skirt barrel was done in the VAC last fall to allow production work on that element to move forward.

Given the desire to continue assembly and production and that the original pin design produces strong welds, VAC welding resumed early this year with a second, full-scale LOX weld confidence article demonstrating the dome to barrel and barrel to barrel welds at that thickness.  That article was completed a week before the tornado that hit MAF on February 7 shutdown operations again.

After cleanup and infrastructure repairs, welding of the LOX qualification article began in April.

“I’ve got a dome and two barrels sitting in the VAC right now,” Doering said at the time of the interview.  “We’ll be done with the LOX qual tank towards the end of May – I’ll be done with the last weld May 4th, then it’s going to be a week or so before I get it out of the VAC into Cell A, get the break-over brackets on it and get it out of there.  And then we’ll start loading the flight tank and so it’s going to be a little over a month and a half to weld the full flight tank.”

Hydrogen tank impact:

Although the weld strength issue stopped welding the qualification and flight articles of the LOX tank before it could start, the issue wasn’t caught until after both LH2 tanks were welded with the modified pin tool last summer.  The implications of the two tanks possibly having below design strength welds disrupted the original, post-weld plans.

The LH2 qualification tank, which will be used for structural testing at Marshall Space Flight Center in Huntsville, Alabama, was welded first and after setup and configuration was taken to Building 451 in December of last year both for proof testing of the welds and to qualify the test facility and procedures for subsequent flight tanks.  Hydrogen tanks are proof tested by pressurizing them with nitrogen gas while a hydraulic test rig applies loads to the structure.

“We wanted to wring out…the control system – 451 was another building that was made bigger to fit the hydrogen tank,” Doering said.  “The control system is all new, the reaction fittings are all new, along with all the actuators.  We didn’t want to put the flight asset in there to try to use it for the first time, so [using] the qual article [first] was also trying to wring out the pressurization and the actuation of the control system in 451.”

Originally, the plan included a test case to pressurize the qualification tank to slightly above flight pressure to help as a part of that “pathfinding” work; however, the discovery that the welds may be below design strength forced plans to be reconsidered.

“We couldn’t say with any real degree of certainty that these welds would make it to [flight pressure],” Doering said.  “In a pneumatic test, pressurizing it like that, it’s like a balloon…there’s a good portion of the community that thinks it will survive, there’s another portion of the community that says you don’t know enough to be able to say that, [and] there’s another portion of the community that says…’no way.’

“We do know from the testing that we can support lower strength limits, but not the full design strength limits for flight,” he continued.  “For structural qualification, we are not pressurizing the tank to full flight pressure.  We don’t need to because structural qualification is primarily going through the buckling and compression loading.  The proof test that we do out here in 451 for the pneumatic proof for the hydrogen tank pressurizes to flight pressure, plus margin above that.”

As a result, the program decided instead to proof test the qualification tank to the pressure it will see during the structural testing at Marshall in the future, which is lower than flight pressure.  “We just need to ensure that it will survive the pressure that we use for structural qualification, we do not need to do the full tension case, which is what you get in proof,” Doering explained.

“For structural qual, since it’s just compression and buckling, we don’t pressurize it all the way.  And so we got comfortable, based on the data that we had that says this tank will survive without any question at [structural test pressure], there’s no reason for us to not utilize this programmatic critical asset to go through qual.

“We’re going to get everything we’re going to get independent of the weld strength, because the weld strength is the tension strength, not anything else.  It doesn’t affect buckling, it doesn’t affect compression, it doesn’t affect any of those other things.  We going to go through a full qualification series for compression and buckling in the test stand with the qual tank.  So for what it’s needed for, it’s got no issue.”

Lower pressure isn’t an option for the LH2 flight tank, which must perform at flight pressures both in testing and in flight.  The SLS Program developed and is working on multiple, parallel options for consideration that include repairs and/or replacement of the already-welded flight tank.

“We’re looking at use as-is – can I get to the point where I’m comfortable using that flight tank?” Doering said.  “The answer to that is probably not, just because the analysis tools don’t exist yet to do this.

“The industry says your weld is either brittle or ductile; we’ve got this ductile weld with a real small piece of brittleness in it and there is currently no analysis tool that allows you to [analyze] this hybrid between a brittle and ductile weld – it’s either fully brittle, which means it doesn’t support the loads, or it’s fully ductile, which if it was, it would [support the loads].  So we don’t have a way to get in between, without destroying the tank.”

Proof testing the flight tank as-is would also carry additional risk – as was noted in the aftermath of the tornado, Building 451 is designed to dissipate the energy of a blast in case of a tank failure while it is pressurized with nitrogen gas and that scenario would also effectively sideline the proof test facility until it could be rebuilt and re-qualified.

“So the use as-is is probably unlikely,” Doering added.  “We could take it out to the parking lot and pressurize it to [flight pressure] and if it didn’t break, we’d say ‘OK, what do we do now?’  Well, we know it didn’t break, but I don’t know if I’ve done anything to the tank that [might cause it to break] the next time I did it.  So it’s a circular argument that you can never get behind because you just don’t have the data.”

Another option is to repair the flight tank first.  “We have a couple of options that look promising to be able to repair this tank,” he said.  “If you get rid of that…brittle surface on the outside of the weld, then you’re back to a full ductile system [and] your tank will work – that’s the theory.

“Initial testing has validated that theory, but we’ve got to put it into full-scale production.  So we’re in the process of going through the development of a repair technique for that tank, the flight tank, so that I’m not wasting this program asset.”

However, the repair option carries significant schedule uncertainty associated with the research and testing of repair techniques.

The favored option at this point is to use the next hydrogen tank, serial number three (S/N 3), for EM-1.  (For reference, S/N 1 is the qualification tank, S/N 2 is the original flight tank.)  The constituent panels and gores for the second Core Stage have been at MAF since last year, and the work to weld those into the two domes and five barrels that make up a hydrogen tank is about half complete.

“It’s being welded up now on the Vertical Weld Center (VWC),” Doering noted.  “Two of the five barrel panels are done, one dome is finished and [for] the other dome the gores are being welded up now.”

Welding the full S/N 3 hydrogen tank will begin as soon as both LOX tanks have cleared out of the VAC.  “I will probably start welding the first dome to barrel [in the VAC] while I’m still pulling the last pieces off the dome weld tools and the VWC,” Doering said.

These options and program recommendations were expected to be reviewed at an agency-level meeting last Friday, but the outcome of that meeting is unknown at this time.

(Images: NASA and L2 which includes, presentations, videos, graphics and internal – interactive with actual SLS engineers – updates on the SLS and HLV, available on no other site. Additional renders by L2 Artist Nathan Koga. The full gallery of Nathan’s (SpaceX Dragon to MCT, SLS, Commercial Crew and more) L2 images can be *found here*))

(L2 is – as it has been for the past several years – providing full exclusive SLS and Exploration Planning coverage. To join L2, click here: //

Related Articles