Record Set Simulating Seismic Wave Propagation Across The Earth

Results of these latest seismic wave simulations were announced at SC08, the international conference for high-performance computing, where the research by the SDSC-led team was a finalist for Gordon Bell prize, awarded annually for outstanding achievement in high-performance computing applications. The SDSC team includes researchers from the California Institute of Technology; Université de Pau and INRIA, France; and the Institut Universitaire de France, France.

The record runs were completed on the ‘Jaguar’ system at Oak Ridge National Laboratory (ORNL). The record run broke the two-second barrier by achieving a shortest period of 1.15 seconds and 161 teraflops, using 149,784 cores.

That is the shortest wave period ever obtained in seismic wave propagation, as well as the highest level of parallelism and the first sustained performance of seismic wave propagation exceeding 160 TFlops. (One teraflop equals one trillion calculations per second. To put that in perspective, it would take a person operating a hand-held calculator more than 30,000 years to complete one trillion calculations.)

The latest supercomputer simulations, led by SDSC researcher Laura Carrington, focused on overcoming a key challenge in global seismology: modeling the propagation of seismic waves with frequencies as small as one second. The team broke the previous record set in 2003, which achieved frequencies of 3.5 seconds.

“In breaking the two-second barrier, we were able to model wave propagation clear through the Earth to more accurately predict its structure,” said Carrington, who is with SDSC’s Performance Modeling and Characterization (PMaC) laboratory. “More significantly, by achieving a frequency just above one second, we were able to duplicate the same frequencies that occur in nature, providing an unprecedented level of resolution that will greatly enhance our understanding of the physics and chemistry of the Earth's interior.”

Researchers used a spectral finite-element application called SPECFEM3D_GLOBE for the latest and largest simulations ever done in this area of research. Waves at frequencies of between one and two seconds, generated when major earthquakes with magnitudes measuring 6.5 or more occur, help reveal new information about the 3D structure of the Earth because they are the highest frequency signals that can propagate, or travel, all the way through the Earth, particularly near the core-mantle boundary (CMB), the inner core boundary (ICB), and the enigmatic inner core, which is comprised of solid iron.

There is no need to simulate wave periods of less than one second for seismographic comparisons, because such frequencies signals do not propagate across the entire globe, according to the researchers.

Moreover, the team’s research is crucial in helping seismologists better understand the dramatic differences in the complex structure of the Earth's inner core, which appears to be anisotropic, or having unequal physical properties along different axes, with dramatic differences between its eastern and western hemispheres.

Earlier, the research team conducted simulations using a wide range of resources provided by the TeraGrid, the nation’s largest open scientific discovery infrastructure linking compute resources among 11 partner sites across the U.S. Runs were conducted using approximately 32,000 cores on the ‘Ranger’ supercomputer at the Texas Advanced Computing Center (TACC) at The University of Texas in Austin. That run achieved a seismic period length of 1.84 seconds and at a sustained 28.7 TFlops.

Prior to that, the team successfully completed several simulations using TeraGrid supercomputer resources, including 29,000 cores on the ‘Jaguar’ system at ORNL’s National Center for Computational Sciences, achieving 1.94 seconds at 35.7 TFlops. Runs were also conducted on the Cray XT4 ‘Franklin’ system at the National Energy Research Scientific Computing Center (NERSC) at the Lawrence Berkeley National Laboratory in Berkeley, Calif.; (12,150 cores with a shortest seismic wave of 3 seconds at 24 TFlops), and on the Cray XT4 ‘Kraken’ system at the University of Tennessee-Knoxville (17,496 cores at 2.52 seconds at 22.4T Flops. The TFlops number in these and subsequent runs were measures using PSiNSlight, a performance measurement and tracing tool developed by SDSC’s PMaC Lab.

The researchers also made a number of radical structure and memory management changes to the SPECFEM3D_GLOBAL application to enable what researchers call “peta-scale-ability”, or using processors numbering in the hundreds of thousands. Efforts focused on the application to overlapping communications using non blocking MPI calls, optimizations to reduce cache misses, optimization to reduce I/O, large restructurings to reducing memory consumption, and work to reduce memory fragmentation.

In addition to Carrington, SDSC researchers on the team include Allan Snavely, director of the PMaC Lab; and researchers Michael Laurenzano and Mustafa Tikir. Additional team members include Dimitri Komatitsch, David Michéa, and Nicolas Le Goff of the Université de Pau, CNRS and INRIA Magique-3D, Laboratoire de Modélisation et d'Imagerie en Géosciences, Pau, France; and Jeroen Tromp, formerly a professor of geophysics with the California Institute of Technology’s Seismological Laboratory. Tromp recently joined Princeton University’s geosciences and mathematics departments. Komatitsch is also affiliated with the Institut Universitaire de France, in Paris.

Komatitsch and Tromp developed SPECFEM3D_GLOBE over 10 years ago, in order to do 3D whole Earth models that account for geological variations in and under the Earth’s crust. These variations have a dramatic effect on how earthquakes propagate. Komatitsch’s work on the application was vital in the collaboration that accomplished this record breaking run.

About the run, Tromp and Komatitsch had this to say: "One of the long-term goals of seismology is to be able to routinely simulate 3D seismic wave propagation in the whole Earth following earthquakes at frequencies of about 1 Hz, the highest frequency signal that can be seen clear across the planet. Very large numerical simulations performed on the new Cray XT5 system at Oak Ridge will enable us to get increasingly closer to this lofty goal."

Research for the SPECFEM3D_GLOBE applications was supported in part by the National Science Foundation (NSF); the Performance Engineering Research Institute (PERI); and the U.S. Department of Energy (DoE); in addition to resources from the NSF-funded TeraGrid program.

The SC08 conference was held in Austin, Texas, November 15-21.