Sandia Supercomputer To Be World's Fastest, Yet Smaller And Less Expensive Than Any Competitor

The first quarter of the $90 million, 41.5 teraflops (trillion operations/second) machine should be installed at Sandia by the end of September and fully up and running by January, says Bill Camp (Sandia's Director of Computation, Computers, Information and Mathematics), who heads the effort to design and assemble the innovative machine.

Performance testing will begin in early 2005. By the end of 2005, the machine should be capable of 100 teraflops, after each single-processor chip is replaced with a new chip that contains two independent processors, each running 25 percent faster than the original chip.

Japan's Earth Simulator, currently the world's fastest supercomputer, has a peak eight megawatts of power compared to Red Storm's projected two megawatts and takes up approximately three times the space.

Red Storm, an air-cooled supercomputer, is being developed by Sandia and Cray Inc. using mostly off-the-shelf parts.

Design innovations permit the machine, from concept to assembly, to be completed with unusual rapidity. While manufacturers typically require four to seven years from concept to first product on a new supercomputer, Cray says Red Storm will begin testing at Sandia less than 30 months after conceptual work began.

The main purpose of the machine is work for the U.S. nuclear stockpile: designing new components; virtually testing components under hostile, abnormal, and normal conditions; and helping in weapons engineering and weapons physics. The machine is expected to run ten times as fast as Sandia's ASCI Red computer system on Sandia's important application codes. (ASCI Red held first place on the top-500 list of the world's supercomputers for three-and-one-half consecutive years.)

But the machine, because of its uniquely inexpensive design, may become the center of Cray's future supercomputer line, says Camp. "From Cray's point of view, the approach we're pioneering here is so powerful they may want their next supercomputers to follow suit."

The machine has unique characteristics: it is scalable from a single cabinet (96 processors) to approximately 300 cabinets (30,000 processors). In addition, the system was designed with a unique capability to monitor and manage itself. Much of the cost incurred for the machine is non-recurring engineering design costs.

"We couldn't afford a 'Rolls Royce' -- an entirely custom-designed machine," says Camp. "The way Red Storm is designed, we don't have to shut down to replace a part. We work around failed components until we decide to fix them -- all without shutting down."

Cray was chosen because the company was "forward-looking, flexible, willing to work with us to design a new architecture, and had the lowest cost proposal."

The machine itself -- a few facts

The machine has 96 processors in each computer cabinet, with four processors to a board. Each processor can have up to eight gigabytes of memory sitting next to it. Four Cray SeaStars -- powerful networking chips -- sit on a daughter board atop each processor board. All SeaStars talk to each other "like a Rubik cube with lots of squares on each face," says Camp. "Cray SeaStars are about a factor of five faster than any current competing capability."

Messages encoded in MPI (the Message Passage Interface standard) move from processor to processor at a sustained speed of 4.5 gigabytes per second bidirectionally. The amount of time to get the first information bit from one processor to another is less than 5 microseconds across the system. The machine is arranged in four rows of cabinets. There are a total of 11,648 Opteron processors and a similar number of SeaStars.

The SeaStar chip includes an 800 MHz DDR Hypertransport interface to its Opteron processor, a PowerPC core for handling message-passing chores, and a seven-port router (six external ports). SeaStars are linked together to make up the system's 3-D (X-Y-Z axis) mesh interconnect.

IBM is fabricating the SeaStar chips using 0.13-micron CMOS technology.

Visualization will occur inside the computer itself -- a capability unique to Red Storm among supercomputers.

###

Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin company, for the U.S. Department of Energy's National Nuclear Security Administration. With main facilities in Albuquerque, N.M., and Livermore, Calif., Sandia has major R&D responsibilities in national security, energy and environmental technologies, and economic competitiveness.