CSI: NASA -- Deciphering today's technological failures to prevent future problems

They analyze readouts from precision tools, devise ways to test large pieces of rocket hardware without damaging the rocket itself, and burn, blow up or vaporize leftover fragments in an effort to find out why something failed.

Think of it as CSI: KSC.

NASA's Kennedy Space Center in Florida is home to a failure analysis lab system whose ancestral roots extend back to the 1960's when failures were not uncommon during early days of rocket development.

These days, the stakes are far greater for engineers and designers and a significant failure on a launch can ground a rocket fleet for more than a year, let alone an afternoon. One of the first teams called into action is the failure analysts.

"Everyone's looking to you to come up with the answer," said Chad Carl, who leads the Materials and Processes Engineering Section of the Failure Analysis and Materials Evaluation Lab at Kennedy.

Their analyses cover such a wide range of failures in everything from tiny valves in processing equipment to nosecones that the lab is nicknamed "Malfunction Junction."

"It was like solving puzzles all the time," said Rick Rapson, a retired engineer who examined everything from quick-disconnect valves on shuttle components to a propane tank that exploded on a turkey farm in Iowa. "Like a policeman solves crimes by looking at the evidence, you're looking for the piece of the puzzle that caused the event to occur and sometimes you had to look a long time to find the right piece. But it was pretty rare that you got stumped."

Making the work much harder was the fact that when rockets fail, there isn't often much left to study.

"When something fails, it's usually a long way away and it's not coming back, so we won't get to look at it," said Todd Campbell, an engineer with NASA's Launch Services Program, which is responsible for sending many of the space agency's flagship missions into space.

The failure analysts and the engineering teams consider themselves a critical element in minimizing the disruption by tracking down what went wrong, finding out if more rockets have the same problem and going with a way to fix it.

"If there's something that's supposed to fly and it can't because it has a crack in it or there's some unknown, we're called in to solve the mystery, figure out what happened and why and what we can do to get flying again," said Bryan Tucker, an engineer in the lab.

Working at the agency's primary launch site means Kennedy's analysis teams work mostly with rockets and ground support equipment, though there are occasional times when the spacecraft is also evaluated to determine its role in a problem.

"At the end of the day it's all about Earth to space," said Dave Sollberger, deputy chief engineer of NASA's Launch Service Program and the person who determines that a rocket is ready to go from an engineering standpoint. "Our job is not the science of what the satellite does, our job is altitude and velocity to get the spacecraft either to low earth orbit or on a deep space trajectory."

He depends on his team of engineers and data drawn from stringent evaluations of a rocket's components and materials to help him feel comfortable that a launcher will perform correctly.

Sometimes a launch does not go well, such as two recent occasions in which payload fairings did not separate correctly from around the spacecraft and the missions were lost.

In the past, Sollberger did not have the Kennedy lab to call on since it spent the vast amount of its time studying space shuttle components. With the shuttle program winding down and then retiring, though, the LSP engineer found a sound source of expertise to help his work.

"The labs have been around for decades, but our involvement with using the labs has increased dramatically over the last year," Sollberger said. "They have some real top-notch technical expertise. And they've got some phenomenal equipment so clearly we want to take advantage of that and we have been taking advantage of that."

Before, the LSP engineers farmed testing to outside labs, but that meant the engineers and analysts didn't meet face-to-face much and the communication back-and-forth was often very formal, Sollberger said. With this approach, if something comes up the engineers can simply walk over and share a finding or new theory easily.

"There's historical ways to do tests and then there's capabilities that they've developed that are really unique that are shuttle driven that we didn't even know were available until we had that communication," Sollberger said.

Much of the work this year has centered on making sure the payload fairing problems did not extend to other rockets and missions. While engineering boards determine a categorical cause for the failure, NASA still had upcoming missions to launch.

Although rocket designs fly dozens and dozens of times successfully, to an engineer certifying that a rocket is ready to safely deliver a cutting edge spacecraft into orbit, there still are plenty of things that can go wrong on each flight simply because launchers are not reused. Ground tests routinely are performed on components and engines are test-fired, but on launch day, the rocket carrying a satellite into space is doing something it has never done before.

"On the LSP side, every launch vehicle is brand new," Carl said. "So every single time you are dealing with a set of parts coming together to make a launch vehicle that have never flown before, and essentially it's an all-new vehicle every single time."

"We can't test absolutely every component that we fly because a lot of the tests to verify proper mechanical properties, they're destructive and you can't destroy everything you're trying to fly, obviously," Tucker said.

So what does it take to successfully analyze spaceflight failures aside from an engineering degree?

For one thing, it requires a desire to deal with different problems every day without becoming stressed out.

"I can't think of anybody out here that's got a really excitable, short attention span, ADHD kind of personality," Carl said.

"You're more likely to find OCD (obsessive compulsive disorder)," Tucker joked.

Also, knowing how to place the details that are found in to the larger scope of a rocket or spacecraft is important.

"I think it's important in that position to have a big picture orientation, but also be detail-oriented to pick up the clues," Carl said.

"Sometimes we have to ignore the forest and look at the trees," said Tucker. "You can't miss anything, but you realize the forest is there, but you've got to look at every tree. You have to be able to see all those details because those details lead you to a solution." Understand, also, that even when they are not thinking about a problem or test result, these engineers still arrive at a solution. "I've popped awake at 3 o'clock in the morning, bam, there it is, the a-ha moment" Tucker said. "I thought I was peacefully sleeping." Rapson came up with the cause of the drag chute door popping off the shuttle at launch of STS-95 while driving into work, six weeks after they'd starting looking into the issue. Turned out the sheer pins holding the door on were not strong enough for the design. They were strengthened to solve the issue.

"We looked at everything we could think of," Rapson said. "I figured it out driving in. The one piece of the puzzle, it can be a thing where all of the sudden a light comes on."

Long airplane flights can also do the trick.

"I can't tell you how many times I was on a plane and have been working on my laptop, maybe cleaning up email and out of nowhere it just hits you, we need to look at that, that's going to be where the answer is," Carl said.