Cosmologists create largest simulation of galaxy formation, break their own record

However, the vastness of our galaxy -- let alone our entire universe -- means experiments to understand our origins are expensive, difficult, and time consuming. In fact, experiments are impossible for studying certain aspects of astrophysics, meaning that in order to gain greater insight into how galaxies formed, researchers rely on supercomputing.

In an attempt to develop a more complete picture of galaxy formation, researchers from the Heidelberg Institute for Theoretical Studies, the Max-Planck Institutes for Astrophysics and for Astronomy, the Massachusetts Institute of Technology, Harvard University, and the Center for Computational Astrophysics in New York have turned to supercomputing resources at the High-Performance Computing Center Stuttgart (HLRS), one of the three world-class German supercomputing facilities that comprise the Gauss Centre for Supercomputing (GCS). The resulting simulation will help to verify and expand on existing experimental knowledge about the universe's early stages.

Recently, the team expanded on its 2015 record-breaking "Illustris" simulation -- the largest-ever hydrological simulation of galaxy formation. Hydrodynamic simulations allow researchers to accurately simulate the movement of gas. Stars form from cosmic gas, and stars' light provides astrophysicists and cosmologists with important information for understanding how the universe works.

The researchers improved on the scope and accuracy of their simulation, naming this phase of the project, "Illustris, The Next Generation," or "IllustrisTNG." The team released its first round of findings across three journal articles appearing in the Monthly Notices of the Royal Astronomical Society and are preparing several more for publication.

Magnetic modelling

Just as humanity cannot envision exactly how the universe came to be, a computer simulation cannot recreate the birth of the universe in a literal sense. Instead, researchers feed equations and other starting conditions -- observations coming from satellite arrays and other sources -- into a gigantic computational cube representing a large swath of the universe and then use numerical methods to set this "universe in a box" in motion.

For many aspects of the simulation, researchers can start their calculations at a fundamental, or ab initio, level with no need for preconceived input data, but processes that are less understood -- such as star formation and the growth of supermassive black holes -- need to be informed by observation and by making assumptions that can simplify the deluge of calculations.

As computational power and know-how have increased, so too has the ability to simulate larger areas of space and increasingly intricate and complex phenomena related to galaxy formation. With IllustrisTNG, the team simulated 3 different universe "slices" at different resolutions. The largest was 300 megaparsecs across, or roughly 1 billion light years. The team used 24,000 cores on Hazel Hen over the span of 35 million core hours.

In one of IllustrisTNG 's major advances, the researchers reworked the simulation to include a more precise accounting for magnetic fields, improving the simulation's accuracy. "Magnetic fields are interesting for a variety of reasons," said Prof. Dr. Volker Springel, professor and researcher at the Heidelberg Institute for Theoretical Studies and principal investigator on the project. "The magnetic pressure exerted on cosmic gas can occasionally be equal to thermal (temperature) pressure, meaning that if you neglect this, you will miss these effects and ultimately compromise your results."

While developing IllustrisTNG the team also made a surprising advance in understanding black hole physics. Based on observational knowledge, the researchers knew that supermassive black holes propel cosmic gases with a lot of energy while also "blowing" this gas away from galaxy clusters. This helps to "shut off" star formation in the biggest galaxies and thus imposes a limit on the maximum size they can reach.

In the previous Illustris simulation, researchers noticed that while black holes go through this energy transfer process, they would not shut off the star formation completely. By revising the black holes' physics in the simulation, the team saw much better agreement between the data and observation, giving researchers greater confidence that their simulation corresponds to reality.

A long-standing alliance

The team has been using GCS resources since 2015 and been running the IllustrisTNG simulation on HLRS resources since March 2016. Considering that IllustrisTNG's dataset is both larger and more accurate than the original, the researchers are confident their data will be used far and wide while they apply for more time to continue refining the simulation. The original Illustris data release garnered 2,000 registered users and resulted in more than 130 publications.

During that time, the researchers have relied on GCS support staff to help with several low-level issues related to their code, specifically related to memory crashes and file system issues. Team members Drs. Dylan Nelson and Rainer Weinberger also both benefitted from attending 2016 and 2017 machine-level scaling workshops at HLRS. The team's long-standing collaboration with HLRS has resulted in winning 2016 and 2017 Golden Spike awards, which are given to outstanding user projects during HLRS' annual Results and Review Workshop.

Nelson pointed out that while current-generation supercomputers have enabled simulations that have largely overcome most fundamental issues related to massive-scale cosmological modelling, there are still opportunities for improvement.

"Increased memory and processing resources in next-generation systems will allow us to simulate large volumes of the universe with higher resolution," Nelson said. "Large volumes are important for cosmology, understanding the large-scale structure of the universe, and making firm predictions for the next generation of large observational projects. High resolution is important for improving our physical models of the processes going on inside of individual galaxies in our simulation."