Why radiation transport dominates the cost
AREPO-RT treats radiation as a photon fluid. For each frequency bin it evolves the radiation energy density and flux, closes the system with the M1 approximation for the Eddington tensor, and solves the resulting hyperbolic equations with a second-order finite-volume scheme on the same Voronoi mesh as the gas. Absorption, photoionization and photoheating are coupled through a non-equilibrium hydrogen–helium network. Unlike ray tracing, the M1 closure is local and remains efficient for very large numbers of sources, which is essential when every star particle emits.
The price is the time step. An explicit radiation update must satisfy ΔtRT ≤ η Δx / (c̃ + |vcell|), where c̃ is the (reduced) speed of light, which is much larger than the sound speed. AREPO-RT therefore subcycles: it performs many radiation steps, typically 64, per hydrodynamic step, with the mesh held fixed. Each subcycle updates the thermochemistry, computes gradients and fluxes, and exchanges primitive variables, gradients and fluxes with neighboring MPI tasks. In THESAN-1, which used about 28 million CPU-hours on SuperMUC-NG, the radiation solver took 53 percent of the total run time, and more than half of that was MPI communication. This cost is what the GPU port addresses.
Radiation transport on the GPU
Because the mesh is static during the subcycles, the solver is a good match for GPUs: the mesh and cell data are copied to the GPU once per radiation step and copied back only at the end. All radiation calculations run on the GPU; the CPU is responsible for the MPI exchange. Only boundary data that must go to other tasks cross between host and device in each subcycle. The main design choices were:
- Overlapping communication with work. Each task uses four CUDA streams: two for host–device copies, one for cells and interfaces in the interior of the domain, and one, with higher priority, for those on the domain surface. Interior work proceeds while boundary data are exchanged.
- Race-free fluxes. One thread per interface computes and stores the flux; a second kernel, parallel over cells, sums the fluxes of each cell. Interfaces between tasks are computed on both sides, so most fluxes never need to be communicated.
- Register pressure. Gradients use one thread per cell and frequency bin, because one thread per cell exceeds the 255-register limit with three or more bins.
- Thermochemistry without a library. One thread per cell integrates the semi-implicit scheme with adaptive substeps that limit changes of the thermal energy and of the ionic abundances to 10 percent per step.
- Memory layout. Data that do not change between subcycles are stored as structures of arrays. Cells and interfaces are sorted by whether they are active and whether they touch another task, and cells also by their number of interfaces to reduce warp divergence. The CUDA Multi-Process Service lets several MPI tasks share one GPU.
Validation
The GPU and CPU versions were compared in an expanding H II region with the full hydrogen–helium network and three frequency bins, and in a cosmological box at the mass resolution of THESAN-2 evolved from z = 49 to 5. The ionization profiles agree except for slight shifts at the ionization fronts, and the reionization histories show small deviations; both are attributed to the different integrators for the chemical network and, in the cosmological run, to randomness in the galaxy-formation model.
Performance
All measurements in the paper compare the GPU solver with the original CPU solver on the same Raven nodes: two 36-core Intel Xeon IceLake Platinum 8360Y CPUs and four NVIDIA A100 GPUs per node.
- Benchmark speed-up
- ≈ 15×
- GPU vs original CPU RT solver on standard tests
- Cosmological run, RT only
- 17×
- z = 49 → 5, one node
- Cosmological run, total
- 3.7×
- 120.2 → 32.7 node-hours
- Test hardware
- 72 cores + 4 A100
- per node of the Raven GPU partition (MPCDF)
- Weak scaling tested to
- 4,608 cores
- and 256 GPUs
- GPU saturates at
- ≳ 105 cells
- per MPI task (single-bin transport test)
In the expanding H II region with three bins and full chemistry, the GPU version is about 15 times faster for large problems. In the cosmological run, the radiation solver became 17 times faster and its share of the run time fell from 78 to 17 percent, so the whole simulation ran 3.7 times faster without any change to the rest of AREPO.
In weak scaling from z = 5.8 to 5.7, the radiation solver was 17.2 times faster on one node and about 13 times faster on several, and the full simulation about 3.5 and 3 times faster. Strong scaling is the weak point: at 64 nodes only about 18,000 cells per task remain, too few to fill the GPU, and the radiation speed-up drops to 6.
From the paper to Lumina
The paper was written on NVIDIA hardware in CUDA, with a HIP version for AMD GPUs. For Lumina the solver was further optimized and rewritten in HIP; on Frontier it is about 20 times faster than the original CPU solver. This factor refers to the later version on different hardware and is not the benchmark result of the 2024 paper. Lumina holds the mesh fixed for 64 radiation subcycles per hydrodynamic step during hydrogen reionization and 256 afterwards.
The paper also identifies the next steps for unstructured meshes on GPUs: a single Voronoi mesh per node to remove ghost cells between tasks on the same node, single-precision primitive variables, and fewer host–device copies.
Paper
-
Adapting AREPO-RT for exascale computing: GPU acceleration and efficient communication
Zier, Kannan, Smith et al. · MNRAS 533, 268–286 · 2024
The GPU port and the validation and scaling tests described on this page.
-
Zier, Smith, Shen et al. · MNRAS 552, stag1469 · 2026
Lumina, run with a further-optimized HIP version of the GPU solver.