Code Speedup Strengthens Researchers’ Grasp of Neutrons

English physicist James Chadwick's 1932 discovery of the electrically neutral neutron filled a big gap in scientists' understanding of the structure of the atomic nucleus. Instead of a nucleus containing positively charged protons and negatively charged electrons, Chadwick demonstrated that protons and neutrons, collectively known as nucleons, clustered together in the nucleus to give atoms the majority of their mass.

In 1935, Japanese physicist Hideki Yukawa predicted how this clustering was possible, identifying an atomic force separate from the electromagnetic force that links protons and electrons. Discovery of the so-called strong force led scientists even further down physics' rabbit hole, revealing the quantum underpinning of matter.

In addition to holding neighboring protons and neutrons together into a nucleus, the strong force acts to combine elementary particles known as quarks into protons and neutrons. Each proton and neutron has three quarks that are linked via the exchange of force carriers called gluons.

Today, nuclear physicists are using supercomputers to develop theories that explain how and why nuclei stick together and decay as they do. Devising such a framework could help scientists shed light on the basic physics of some complex questions such as how the universe formed, how stars burn, and why they explode. Improved understanding in this field also would come in handy here on Earth, where increasing energy needs could be met via nuclear fission and fusion.

Part of solving the atomic nucleus depends on gaining a better grasp of neutrons. A team led by James Vary of Iowa State University simulated clusters of neutrons called "neutron drops" to understand their properties better. The ab initio calculations, or calculations based on fundamental forces and principles, were performed on the Titan supercomputer at the US Department of Energy's (DOE's) Oak Ridge National Laboratory. Titan is the flagship machine of the Oak Ridge Leadership Computing Facility, a DOE Office of Science User Facility. Leveraging Titan's massive memory and computing power, the team was able to determine the ground-state energies and other properties of systems of up to 40 neutrons. The results were published in the December 2014 issue of Physics Letters B.

To maximize its time on Titan, Vary's team took advantage of Titan's GPUs to achieve a two- to threefold speedup of the initial matrix-building portion of its Many Fermion Dynamics-nuclear (MFDn) application. Thus far, the project has used 42 million core hours on Titan, scaling to 18,424 of Titan's nodes (about 99 percent of the machine).

The team's results, which closely match the calculations of complementary methods, can guide researchers' ongoing development of a universal framework called nuclear density functional theory that describes atomic nuclei and other forms of nuclear matter based on the density of protons and neutrons in space.

"Simulating a system of only neutrons is a bit simpler than simulating a normal nucleus that contains protons and neutrons, but it could help us learn about nuclei that have many more neutrons than protons," said Hugh Potter, a team member and Iowa State graduate student. "This methodology also provides insight into the part of the strong force that is specific to neutrons."

Memory and GPU muscle

Tracking the interplay between nucleons creates an enormous computational problem that grows with the size of the nucleus. Vary's team gained access to Titan, a Cray XK7 with a peak performance of 27 petaflops (or 27 quadrillion calculations per second), through a 2014 Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program allocation.

Using MFDn, Vary's team calculated the interactions of each neutron drop system, accounting for the forces that exist between pairs of neutrons (the two-body force) and groups of three (the three-body force). An outside force known as a harmonic oscillator trap was imposed on each neutron drop system to hold the nucleons in place.

MFDn reconstructs nuclei in three stages. First, the code generates a large billion-by-billion matrix representing the strong force acting between nucleons. A specialized algorithm reduces this matrix to a useful set of 10 vectors, each with about a billion values. From this data set, researchers can derive various properties of the nucleus in question, including its energy, size, and density.

Titan possesses 700 terabytes of memory. Despite this figure -- equivalent to around 90,000 laptops -- memory is one of the major constraints of calculating nuclear systems. As the number of nucleons in the system grows, the memory demands on the computer also increase. To reduce memory usage, Vary's team packaged part of MFDn's input data in a compressed form. "Unpacking" this data can be time consuming, but by moving this process to Titan's GPUs, the team sped up its code by 20 to 40 percent.

"Having each nucleon interact with every other nucleon gets computationally expensive very quickly," Potter said. "What the GPUs do is decompress our input files to help us save time when we save space. Without Titan's leadership class memory, we wouldn't be able to calculate problems up to the size that we did. And without its GPUs, we wouldn't have been able to get as much science done as efficiently as we did."

The team is continuing its investigation of neutron drops, among other nuclei, under a 2015 INCITE allocation.

Related publications:

D. Potter, et al., "Accelerating Ab Initio Nuclear Physics Calculations with GPUs," in Proceedings of the International Conference on Nuclear Theory in the Supercomputing Era -- 2013, eds. A. M. Shirokov and A. I. Mazur (Khabarovsk, Russia: Pacific National University, 2014), 263-271. doi: http://arxiv.org/abs/1412.5989.

D. Potter, et al., "Ab initio study of neutron drops with chiral Hamiltonians," Physics Letters B 739 (2014): 445-450. doi: http://arxiv.org/abs/1406.1160.