Titan targets tumors

Surgery, chemotherapy, and radiation therapy all play important roles in cancer treatment, and sometimes the best successes come from combining all three approaches.

Doctors usually do the most common form of radiation therapy with x-rays, which can penetrate tissue, killing the cancerous cells in deep-seated tumors. Unfortunately, these same x-rays can also damage healthy tissue surrounding the tumor.

Thus, in recent years, the use of beams of heavy particles, such as protons or ions, has come into focus. These beams can deposit most of their energy inside the tumor, while at the same time leaving the healthy tissue unharmed. Unfortunately, these beams come from bulky particle accelerators, which make the treatment cost prohibitive for many patients.

At the German research laboratory Helmholtz-Zentrum Dresden-Rossendorf (HZDR), researchers are looking into replacing particle accelerators with high-powered lasers. The electromagnetic fields of the laser can accelerate ions in a very short time, thus effectively reducing the distance needed to accelerate the ions to therapeutic energies from several meters to a few micrometers.

As a scientist experienced in accelerator research and laser physics, HZDR researcher Michael Bussmann aims for understanding and controlling this new method of particle acceleration to make it available for patient treatment. "I'm coming from accelerator research and laser physics, and what my team and I have been looking at is how we make best use of the high-power lasers so they can replace accelerators for applications like treating cancerous tumors," Bussmann said.

"This is fundamental physics on the one hand, as the laser pulse rips apart all the matter found in a target, usually a very thin metal foil or a tiny sphere, separating the building blocks of atoms--negatively charged electrons and positively charged atomic nuclei, ions--from each other. This state of matter is called a plasma," Bussmann explained. "On the other hand, it has real applications as well. Simulations play a role that is unique, as experiments are still not very reproducible and we can't really diagnose what's happening in a few femtoseconds."

To observe these atomic-scale interactions happening many times per second, Bussmann's team is using the Cray XK7 Titan supercomputer located at the Oak Ridge Leadership Computing Facility (OLCF), a US Department of Energy (DOE) Office of Science User Facility located at DOE's Oak Ridge National Laboratory (ORNL).

Bussmann explained that without access to leadership-class supercomputing resources, his team would not be able to run such complex simulations efficiently. "I need to simulate a huge volume of atoms over a very long time scale," Bussmann said. "The only way to do this comes through supercomputing, because the large volume needs a lot of memory, and the long time scales mean I need a lot of computational power, and that is where the GPUs come into play."

Titan uses traditional CPUs along with high-speed GPUs, or graphics processing units, to accelerate simulations. Bussmann's team does all of its calculations on Titan's GPUs at a rate 10-100 times faster than what is possible on CPU-only machines. "We no longer think of simulations in terms of CPU hours but rather frames per second," Bussmann said, describing the effect this speed-up has had on the team's research.

Particles to laser pulses

When using lasers to treat tumors, researchers must work within extremely small parameters. Typically, a team must aim a laser pulse at a target about the same size as the laser's focus -- no more than several micrometers long. Researchers then suspend a thin foil material; next, they use the laser to excite electrons connected to its atoms and drive the electrons away from the atoms. In the end, the strong separation of negatively charged electrons and now positively charged atoms, or ions, creates the forces that accelerate an ion beam toward the tumor.

Until recently, lasers that provided pulses with enough power to reach high ion energies were large and costly. Now, new technologies such as energy efficient laser diodes--which help increase the laser power -- have made these systems much more compact. Moreover, as more of the electrical energy used to operate these lasers can be converted into laser light power, researchers can now take several hundred laser shots per minute. This is a huge gain compared with several shots per day that were possible with the first laser systems.

With this gain, researchers can produce laser-accelerated ion beams at the rate needed for treating patients, reducing the potential treatment time to several minutes, thus saving costs and making the procedure easier for patients to endure. Still, researchers must perform extensive research not only to increase the energy of the ions to reach even deep seated tumors but also to really control the beam well enough to be used routinely in a medical treatment. Due to power demands, this technology is still in its infancy, and may require several years before it can be used in clinical applications.

Ions are ideal to target the cancerous tissue precisely. Unlike x-rays, ions do not release a lot of their energy into the tissue before coming to their final stop inside the tumor. There, they release a sudden burst of remaining energy, depositing the lethal dose of radiation directly into the tumor. Using ion beams allows medical professionals to calculate exactly where ions need to stop; with precise aiming, they can kill only the tumor cells and spare the vast majority of nearby healthy cells.

Successful student simulations

In August 2008, the team got an unlikely boost in its computational capabilities in the form of 17 year old high school student Heiko Burau. He won a German "Jugend forscht," or youth research award, allowing him to work with experts at a prestigious German research institution. This award allowed Burau to work with Bussmann's group, and the results were immediately noticeable-- within 6 weeks, the student GPU programmer had created the team's first GPU accelerated two-dimensional simulation. Burau is now starting his diploma thesis -- still as part of the HZDR team.

Burau's initial work has been picked up by an ever-growing team of young researchers. Axel Huebl, who joined the team a year after Burau, has pushed the code to make optimal use of all of Titan's GPUs, winning him time on Titan through DOE's Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program for his PhD research on laser-driven radiation therapy of cancer. The team has been developing its code, PIConGPU (particle in cell on graphics processing units) for 8 years, with undergraduate students doing much of the development work. Although some might see this as a sign of inexperience, Bussmann loves his team's youthful approach to problem solving.

"Our group is full of youth; otherwise we couldn't have done this research," Bussmann said. "One of the main reasons is because we weren't afraid of doing it. If you're sure things are going to get rocky down the road, maybe an older group wouldn't go there, but we said, 'let's just give it a try,' and it worked out."

For its work on PIConGPU, the Bussmann team was named a finalist for the 2013 Gordon Bell Prize -- one of the most prestigious honors in supercomputing.

In addition to student developers, OLCF veterans have also helped optimize PIConGPU for Titan. ORNL's Scientific Data Group, led by Scott Klasky, has helped the team drastically improve its ability to write out petabytes of data from the simulation and move it between Titan and other ORNL computing resources. To address I/O concerns, the team has been using the Adaptable I/O System, or ADIOS, developed by the Scientific Data Group. ADIOS has enabled a tenfold improvement for the team's I/O capabilities.

ORNL visualization expert David Pugmire is also collaborating with the team to develop a full-scale visualization of its data, which is a huge help. "We've done this kind of visualization on other projects, but that in itself is a huge effort. That's an area where we get help from Oak Ridge quite a lot, and we're impressed by how far this help goes," Bussmann said.

Because of its various collaborations and diligent work on Titan, the team was able to create some of the most realistic three-dimensional simulations of high power laser interactions with targets that are on the scale of the laser focus. With the calculation power available, the team could use targets at densities close to what is used in experiments.

Moreover, the team can include effects that happen on much longer time scales than the interaction of the laser pulse itself and that strongly influence the outcome of experiments. One of the main advances dealt with the creation of plasma caused by laser energy deposited into the target well before the main part of the laser pulse hits the target. This so-called preplasma can be much larger than the initial target, meaning the team has to simulate a much larger volume. These effects are present in experiments as well as in large-scale simulations, so the team is able to offer a much more detailed comparison between simulation and experiment.

Though the team has made major advances using Titan, there is plenty of room for improvement. Bussmann noted that further improving the atomic-level physics in his team's simulations would help improve accuracy. Ideally, the team could use a database to describe every atom with each atom's respective electronic configurations.

Describing billions of atomic level interactions in such small time scales, while also needing to reference the atomic database, is currently too computationally expensive. Summit, the OLCF's next-generation supercomputer, set to be delivered in 2018, will address some of these issues. "Summit would be an almost ideal system for this task," Bussmann said. "We could leverage the power of the CPU to go through the database quickly and bring the information we need to the GPUs using the very fast interconnects. Right now, we're basically relying on GPUs, but heterogeneous computing makes sense -- to let each part of the system do that part it's best at and let them work things out together."