Supercomputing the transition from ordinary to extraordinary forms of matter

But "melting" atoms and their subatomic building blocks is far more difficult than taking an ice cube out of the freezer on a warm day. It requires huge particle accelerators like the Relativistic Heavy Ion Collider, a nuclear physics scientific user facility at the U.S. Department of Energy's (DOE) Brookhaven National Laboratory, to smash atomic nuclei together at close to the speed of light, and sophisticated detectors and powerful supercomputers to help physicists make sense of what comes out. By studying the collision debris and comparing those experimental observations with predictions from complex calculations, physicists at Brookhaven are plotting specific points on the nuclear phase diagram to reveal details of this extraordinary transition and other characteristics of matter created at RHIC.

To plot the key points where the transition takes place, the scientists are looking for large fluctuations in the excess of certain kinds of particles produced from collision to collision (for example, more protons than antiprotons, or more positively vs. negatively charged particles).

"At RHIC's top energy, where we know we've essentially "melted" the protons and neutrons to produce a plasma of quarks and gluons -- similar to what existed some 13.8 billion years ago -- protons and antiprotons are produced in nearly equal amounts," said Frithjof Karsch, a theoretical physicist mapping out this new terrain. "But as you go to lower energies, where a denser quark soup is produced, we expect to see more protons than antiprotons, with the excess number of protons fluctuating from collision to collision."

By looking at millions of collision events over a wide range of energies -- essentially conducting a beam energy scan -- RHIC's detectors can pick up the fluctuations as likely signatures of the transition. But they can't measure precisely the temperatures or densities at which those fluctuations were produced -- the data you need to plot points on the phase diagram map.

"That's where the supercomputers come in," says Karsch.

Supercomputers can simulate the types of fluctuations you would expect for the wide range of temperatures and densities at RHIC. They start by mathematically modeling all the possible interactions of subatomic quarks and gluons as governed by the theory of Quantum Chromodynamics, or QCD, which includes variables such as temperature and density in the equations.

Because the number of values for these and many other variables in the equations of QCD is very large, only supercomputers can handle the complex calculations. To simplify the problem, the computers look at interactions of quarks and gluons placed at discrete points on an imaginary four-dimensional "lattice" that accounts for three spatial dimensions plus time. The lattice consists of about 300,000 grid points, and on each point the values of 48 variables need to be adjusted to characterize a specific configuration of the interacting quarks and gluons. Supercomputers use Monte Carlo sampling -- more or less trying random numbers, like rolling a pair of dice -- to find the most probable configuration of these values.

"But there are many such configurations and we have to explore them all to allow for the many possible ways those quarks and gluons can interact," said Karsch, leader of the nuclear physics lattice QCD group at Brookhaven and director of a DOE-sponsored "Scientific Discovery through Advanced Computing" (SciDAC-3) partnership program, "Computing Properties of Hadrons, Nuclei and Nuclear Matter from Quantum Chromodynamics."

To build these lattice QCD configurations, the scientists used Blue Gene supercomputers of the New York State Center for Computational Science, hosted by Brookhaven, as well as two new prototype racks of the Blue Gene/Q supercomputers at Brookhaven and at the RIKEN/BNL Research Center -- a center founded and funded by the Japanese RIKEN laboratory in a cooperative agreement with Brookhaven. The machines turned out over 10,000 of these most probable configurations for each temperature.

The scientists then loaded the lattice configurations onto a different kind of supercomputer -- the Graphic Processing Unit (GPU) cluster operated by the US based lattice QCD consortium, USQCD, at DOE's Thomas Jefferson Accelerator Laboratory and another GPU cluster at Bielefeld University in Germany.

"GPUs are the kinds of computers that were invented to make video games," said Brookhaven theoretical physicist Swagato Mukherjee, who coordinated the simulations and analysis. "They have very fast processors that can perform many simultaneous operations and draw every single pixel at the same time. That's what you need to see fast-moving graphics, but it's also very useful for these complex physics problems where we need to perform many simultaneous repetitive operations on each of the stored configurations of values to calculate the fluctuations of the excess particle numbers."

The scientists used 800 GPUs at Jefferson Lab and at Bielefeld University to analyze their 10,000 most probable configurations at each temperature and calculated the fluctuations of excess particle numbers for various combinations of temperature and density relevant for collisions at RHIC. By matching the fluctuations measured in real RHIC collisions at a given beam energy with the calculated values, they could use the supercomputed calculations to identify the temperature and density at which those fluctuations took place -- the coordinates they needed to plot a point on the phase diagram map.

Repeating the process for many experimentally measured values of fluctuations over the wide range of beam energies available at RHIC is helping scientists trace the line on the map that shows how the transition from quark soup to ordinary matter changes with temperature and density. These studies will reveal the conditions under which the transition is abrupt, with a sharp dividing line between phases like the distinct forms of liquid water and vapor (a first-order phase transition), and where it is smooth and continuous with no distinct phases -- as they expect is the case at RHIC's highest energies. They may even be able to identify whether there is a "critical point," the specific temperature and density at which the type of transition switches from continuous to abrupt.

"If found, this would define the starting point for an even richer phase structure at higher densities and lower temperatures," said Karsch. "The existence of a 'critical point' is a prerequisite for the existence of many exotic phases of the nuclear matter that may exist within the core of compact stellar objects, such as neutron stars, so finding one means RHIC gives us access to studying these exotic forms of matter in a controllable way with our experiments."

To be confident of their findings and explore other aspects of quark-gluon plasma, the scientists are using the same approach to study fluctuations in the excess number of particles containing heavier "strange" quarks.

"Different hadrons -- bound states of three quarks or a quark and an antiquark pair -- can melt at different temperatures," Karsch said. "We have to show that it happens within a very narrow temperature range for these different kinds of particles to identify the critical point."

The analysis of strange quarks has revealed that, like the ordinary protons and neutrons consisting of lighter up and down quarks, hadrons containing the heavier strange quarks also melt in the same temperature region. In this analysis the scientists have also found that for the temperatures achieved at top RHIC energies, the strange quarks remain strongly interacting even within the soup of quarks and gluons -- a piece of evidence for the strongly coupled nature of the nearly perfect fluid created at RHIC.

The lattice QCD group at Brookhaven now prepares for a new round of simulations. The plan is to use new software developed under the SciDAC-3 partnership program and computing resources applied for by USQCD through the DOE's Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program -- and ideally, the world largest supercomputer, TITAN, at Oak Ridge National Laboratory, which combines more than 18,000 GPUs.

With that processing power analyzing existing data and future collisions at RHIC, scientists will continue to narrow the search for landmarks on the nuclear phase diagram and expand our understanding of how the matter of the early universe transformed into the stuff of our familiar everyday world.

The supercomputing analyses of QCD and RHIC data are funded by the DOE Office of Science.