Secret weld: How shoddy parts disabled a $2.7 billion submarine
By David B. Larter
GROTON, Conn. (Jan. 10, 2013) The Virginia-class attack submarine USS Minnesota (SSN 783) completes a two-day transit from Norfolk, Va., to its new permanent homeport at Naval Submarine Base New London in Groton, Conn. (U.S. Navy photo by Mass Communication Specialist 1st Class Jason J. Perry /Released)
In early 2015 engineers on a brand-new submarine made a troubling find: A pipe joint near the innermost chamber of its nuclear-powered engine showed signs of tampering.
The defective elbow pipe, used to funnel steam from the reactor to the sub's propulsion turbines and generators, showed evidence of jury-rigged welding that could've been designed to make it the part appear satisfactory. But the part was already installed, the sub was already commissioned.
These defective parts, each probably valued on the order of $10,000 or less, have kept the $2.7 billion attack submarine Minnesota languishing in an overhaul for nearly two years, while engineers attempt to cut out and replace a difficult to reach part near in close proximity to the nuclear reactor. Meanwhile, Navy engineers are scouring aircraft carriers and other submarines for problems and criminal investigators are gathering evidence.
The unauthorized parts are impacting sidelining three new Virginia-class attack submarines, likely extending the post-shakedown overhauls for the other two subs and adding greatly to the final tab months or years at a time these fearsome vessels are needed around the globe to defend carrier groups and strike America's adversaries. It's also trapped its crew in limbo as repair deadlines come and go, while other subs must take their place.
But excitement soon began to give way to concern, then outright frustration and anger.
The Minnesota, the 10th Virginia-class attack boat, was delivered 11 months ahead of schedule. But it has been in the shipyards at Electric Boat in But the $2.7 billion ship that three years ago was being held up as a symbol of a robust and effective program is languishing in the shipyards at Electric Boat in Grotton, Connecticut for two years — more than twice as long as a normal post-shakedown availability. It still has months to go. The plankownering crew has spent only a handful of days at sea since joining the fleet and experts say they're likely to forfeit their whole deployment cycle, forcing fleet bossescommanders to make tough decisions about whether to extend deployments or withhold forces from missions overseas.
News of the lousy parts first emerged in August, a month after the Minnesota was to have finished its overhaul. Since then, a Justice Department-led investigation is examining the quality control issues that led the shoddy part to be installed in the $2.7-billion sub. Minnesota has been in it’s post-shakedown availability, a major, post-delivery overhaul that fixes problems found during a sub's initial service period, for two years and the Navy projects it still has months to go. The ship has been underway no more than a few days since joining the fleet and experts say the loss has forfeited nearly a whole deployment cycle, stripping desperately needed attack boat presence from the fleet.
The same shoddy elbow jointsfittings were installed aboard attack subs in both the attack submarine North Dakota and John Warner, forcing the Navy to spend millions of dollars and many more months to repair them. If these pipes ruptured, they would leak steam and force the submarine to take emergency measures that would impaireverely reduce its combat effectiveness. which will both need to have them replaced, significantly increasing the workload and cost for their post-shakedown overhauls.
The culprit: shoddily constructed and improperly certified pipe fittings, welded into a part of the state-of-the-art propulsion system that was never supposed to be accessed.
The fittings are part of the ship's main steam system, which takes steam from the reactor compartment to the propulsion turbines and generators. If the fittings failed, the results wouldn't be catastrophic but the ship couldn't operate normally until it was fixed, according to three sources familiar with the repairs. A retired submarine engineering officer who spoke to Navy Times on background to protect business ties.
Minnesota’s repairs should be completed sometime this summer, according to a statement from Naval Sea Systems Command, but for many of the officers and crew that may be too late. They'll have to report to their next tour of duty without having deployed, which they worry could hurt their careers, Many of them will have to rotate before making a deployment, which many fear will negatively impact their careers, said Brian Skon, the head of the Minnesota Navy League, who helped sponsor the commissioning ceremony and stays in touch with the crew.
"They're frustrated," Skon said. "They want to be underway, they want do a deployment. I spoke with the chief of the boat and he's been very clear: he wants to be a COB on deployment."
Complicating matters further, the company At the center of the debacle is pipe-maker Nuflo Inc., a Jacksonville, Florida-based manufacturer that is the focus of the investigation into a Florida-based company called NuFlo, is included in a Justice Department investigation into quality control issues, according to two Navy sources familiar with the inquiry. The investigation has delayed the repairs so that agents can recover evidence, sources said. so as to Engineers have had to Because of the investigation, the Navy has had to be careful about how it has conducted the repairs because of fears of destroying evidence or hampering investigators, further extending the delays, two Navy sources said.
With 120 employees, the pipe maker bills itself as "the primary manufacturer of fittings for U.S. Navy Aircraft Carriers and Submarines," according to their website. Nuflo has provided parts for the carrier Theodore Roosevelt's recent mid-life refueling overhaul, complex overhaul and refueling, as well as for the new carrier Gerald R. Ford, according to various news reports. Neither the Nuflo's CEO or spokespersonwoman responded to repeated calls and emails for comment by March 25.
The setback for what has been the Navy's most successful shipbuilding program is startling because News that three boats from one of the Navy's most celebrated program will need to be ripped up to fix deficient pipe fittings is startling because Virginia-class has been in production for more than 15 years, according to a defense acquisitions expert. said Dan Goure, an analyst with Lexington Institute, an Arlington-based think tank.
"This is an unusual situation, especially since this is a relatively mature program," said Dan Goure an analyst with the Lexington Institute, based in Arlington, Va.-based think tank. "It's also surprising that the yards would have had this problem."
Making matters worse are concerns that the flawed pipe fittings may extend well beyond the three identified attack submarines. In a statement, NAVSEA, which oversees ship construction and maintenance, said it has sent inspectors across the fleet to test Nuflo-made fittings on other ships, according to the statement. said it was still trying to get their arms around the full extent of the issue.
A 2013 image from one of the few days that the Minnesota has been underway since it was built.
Photo Credit: Mark D. Faram/Staff
"As part of an ongoing investigation into a quality control issue with a supplier, General Dynamics Electric Boat and Huntington Ingalls Incorporated, Newport News, determined that fittings supplied by the vendor in question required additional testing and repair due to incorrect test documentation, incorrect testing, or unauthorized and undocumented weld repairs performed on these fittings," a NAVSEA spokeswoman said in the statement. statement reads"The fittings, which are used in various piping applications aboard new construction submarines, are also installed on other ships. Therefore, out of an abundance of caution, the Navy, in coordination with its industry partners, has been performing additional inspections and surveys throughout the fleet to fully bound the issue."
The full scope of the problem remains unclear. NAVSEA declined to comment on whether any other shoddy parts had been found on other ships, citing the would not comment on which other ships were affected, citing the ongoing investigation.
"NuFlo has been doing business with the Navy's nuclear enterprise for some time now," said one industry source who asked to speak anonymously due to the could not speak on the record, citing the ongoing Justice Department investigation.
The Virginia-class submarine is a joint project between General Dynamics Electric Boat and Huntington Ingalls. A spokeswoman for HIIngalls declined to comment because of the ongoing investigation. A spokesman for Electric Boat said he was at a conference and would have to deferred all questions to NAVSEAaval Sea Systems Command. NuFlo did not respond to multiple calls and an email seeking comment
All eyes were on the Minnesota when it was commissioned at a 2013 ceremony in Norfolk, Va. The sub was delivered 11 months early but extended maintenance has prevented the crew from starting their pre-deployment work-ups.
Photo Credit: MCSN Andrew Schneider/Navy
The Minnesota's plankowners aboard the attack submarine Minnesota in the late summer of 2013 were eager to take one of the fleet's most lethal ships out for a spin. the ship out for a spin.
After months of intense work getting one of the Navy's newest and deadliest ships off the line and into the fleet.
"I think it will be one of those defining moments in our careers," said Senior Chief Machinist's Mate (SS/DV) Jody Reynolds in a Navy release, marking all the effort to establish a the crew put in to setting the foundations of a great command.
At its commissioning ceremony, the brass took a victory lap. The sub was delivered 11 months ahead of schedule and they cited it as as proof that the Virginia-class program was the "gold standard" in defense acquisitions.
Then Minnesota entered the yards. It was supposed to last less than a year, before ballooning into a two-and-a-half year quagmire.last just less than a year has ballooned into a more than two-and-a-half year quagmireendurance challenge.
"Minnesota's sea trials are a testament to the success of the Virginia-class program's ability to deliver high-quality and extremely capable boats early and on cost," said then-head of Submarine Group 2 Rear Adm. Ken Perry, referring to the .
The post-shakedown availability would repair problems identified at inspections and in sea trials. The work, valued at a and scheduled to According to the initial contract award announcement, Minnesota’s post-shakedown availability was scheduled to last a little less than a year. The $57.2 million, would be completed by contract was awarded in March 2014 and was scheduled to be completed by February 2015. the following year.
That was extended to July, which became public a month later when the deadline was missed and But in August, when news of the deficient pipe fittings first broke, Navy Times' sister publication Defense News reported that the joints were sidelining three submarines.
NAVSEA's latest completione estimate is sometime "this summer," according to their statement. Minnesota had apparently been extended to July. Naval Sea Systems Command says the boat should be rejoining the fleet in the summer. This means Minnesota’s post-shakedown repairsyards period will have lasted more than two years — as much time as it takes to refuel and overhaul a Los Angeles-class attack sub.
By contrast, the post-shakedown availability for the the Virginia-class attack submarine California, the eighth of the class, was completed in 2013 in just 11 months.
All of this is ending up on the shoulders of the crew. If the PSA had gone off without a hitch, Minnesota would be nearing the end of the work-up cycle and closing in on its first deployment, said Bryan Clark, a retired submarine officer and analyst with the Center for Strategic and Budgetary Assessments. To top it off, a big chunk of the plankowners are likely might never get a chanceto deploy with their boat.
"For the crew it sucks because most of them came on not long before commissioning with the understanding that they would be doing a post-shakedown period in the yards, then work-ups then a deployment," said Clark, a defense expert at the Center for Strategic and Budgetary Assessments. "Now you've got a whole crew of people who will spend their whole time in the shipyards or work-ups but never deploy."
The force is only able to meet a little over half of the requests for attack submarines in the Pacific. Here, the Los Angeles-class attack submarine Chicago sails near Guam.
Photo Credit: Lt.j.g. Jonathan Peterson/Navy
Beyond the crew,Meanwhile, the demand for attack boats, capable of running spy missions or delivering stealthy special operations teams against well guarded adversaries, against well overseas is nearing Cold War levels. In a recent column in Breaking Defense, House Armed Services Committee member Rep. Randy Forbes, R-Virginia, said the fleetNavy was only meeting 62 percent of the combatant commander demand for the boats.
In February, U.S. Pacific Command head Adm. Harry Harris, whose forces must respond to the growing tensions between China and its neighbors, testified that told lawmakers that attack subs were among his most pressing needs; the fleet was only meeting 62 percent of his demands for attack boats, he said. In October, the 6th Fleet commander, Vice Adm. James Foggo, said he needed more attack boats in Europe in part to counter Russia moves.told Navy Times in October that Russia was steadily improving its undersea capabilities and that he needed attack boats in Europe.
"The Russians have always fully funded their submarine capabilities and as they've evolved, they've become better," he said. "They've become quieter and more capable adversaries. So we need to watch that more carefully and we need to watch our presence in the undersea domain."
Spokespeople for the Navy and NAVSEA declined to provide an estimated cost for Minnesota's extra year in the shipyards or to say how much it will likely cost to fix the John Warner and North Dakota. The Navy spokesman acknowledged that maintenance delays affect what ships are sent on deployment, but declined to go into any specifics about how other crews were and how they could be affected. A Navy spokesman declined to say how much the The Navy would not comment on what specific impact Minnesota’s delays were having, nor would they comment on the projected cost overruns for the Minnesota’s extra year in the shipyards or who would be paying for them. In a statement, however, Navy spokesman Lt. Cmdr. Tim Hawkins acknowledged that maintenance delays do impact operations.
"Generally speaking, delays in maintenance periods will impact the overall operational availability of the submarine force," Lt. Cmdr. Tim Hawkins said. "Leaders regularly review operational schedules and adjust them based on force availability and presence requirements. Attack submarines, which are always in high demand, will continue to be deployed when and where they are needed most."
The Navy would similarly not comment on how long the required repairs for John Warner or North Dakota would take, how much they would cost or who was paying for them.
The statement from NAVSEA said No subships have been recalled from deployment for related repairs, NAVSEA said. But the parts must be replaced within a few years of its commissioning to reduce the risk that the joint will leak or even burst in a combat scenario. and that the parts were generally safe to operate for now. John Warner recently returned from its shake-down and is headed into its own yards period. Warner's pipe fittings will likely be replaced in its PSA, but Navy officials would not say if the availability would need to be extended as a result of the faulty fittings.
It the pipe joint were to rupture, it would not cause a radioactive incident. But it could effectively render the submarine unable to operate for weeks or months until fixed. The crew of the attack submarine Jefferson City discovered a water leak in the propulsion plant; finding and fixing that kept the sub stuck in Guam for five months in 2014.
[Add stuff here about how this kind of casualty would be addressed by the crew/sf]
would be profoundly
The Minnesota is being repaired at the Electric Boat shipyard in Groton, Conn. The contract for the one-year overhaul was $57 million; officials were unable to say how much the cost has risen due to the repair delays.
Photo Credit: General Dynamics Electric Boat
The flawedproblematic fittings are joints in the elbows in 10-inch steam pipes that direct steam heated by the reactor core to the propulsion turbines and electrical generators that power the sub. These parts are to designed to maneuver by obstructions and around corners and often resemble pieces of metal macaroni. take the steam from the plant to the propulsion turbines. The elbows are used to lead the pipes around corners and obstructions.
Defense News reported in August that the Minnesota had one of the bad elbows installed; John Warner has three and North Dakota has six. According to the senior Navy official, The Nuflo-made parts pipe elbows initially failed magnetic test inspections that showed "minor surface indications," then successfully passed ultrasonic test inspections after minor repairs.
But further testing by Electric Boat using acid etch inspections, which can reveal cracks in metal, showed the unauthorized welds that prompted the investigation.
When parts are delivered to the builder, one industry source explained on background, they have to be certified with documentation showing, among other things, who made it, with what tools and where, and how it was tested to meet the standards. what was done to the part and with what tools. So when the undocumented welds were discovered, red flags went up.
Two sources familiar with the repairs said The repairs to Minnesota arewere time-consuming and expensive, according to two sources familiar with the work. The reactor must go through a lengthy process to set the right conditions before a repair worker can enter the compartment, which was designed never to be refueled. For this reason, these parts were builtdesigned to last 35 years or beyond, the full life of the submarine. And these fixes require highly Partly this is because setting the right reactor conditions to make the repairs in places that were never supposed to be accessed takes time. The fittings around the pipes, as well as the pipes themselves, were designed to t for 35 years or more, adding to the expense and difficulty of the repairs. The repairs require also require highly skilled technicians to work in areas where radiation exposure limits how long they can be in the space.to work in areas where radiation exposure may limit the amount of time they can be in the space.
"This is a really complicated and difficult cut and weld job," one Navy source said.
What's not clear is how long the repairs of John Warner and North Dakota will take, how many other ships have these deficient fittings, and what the total cost will be in terms of money and lost operational time.
The Navy refuses to comment while the investigation grinds on., citing the ongoing investigation.
Defense News Staff Writer Chris Cavas contributed to this report.
About David B. Larter
David B. Larter was the naval warfare reporter for Defense News.