At a glance
- Problem
- Isolated SIDM halos
- Equal-mass particles; elastic, isotropic scattering with a constant or tabulated velocity-dependent cross section.
- Hardware
- CPUs and GPUs
- Shared-memory CPU mode; GPU kernels compile to CUDA or HIP from one source.
- Gravity
- Tree, monopole or quadrupole
- Morton-ordered octree built bottom-up at every step; up to 8 particles per leaf by default.
- Reproducibility
- Bitwise identical
- Across CPU and CUDA runs with different numbers of ranks, GPUs and segments (tested).
- Per-step speed
- Up to 60× AREPO-2
- One node with four A100 GPUs vs. AREPO-2 on one 112-core CPU node; 108 particles in r200, global time steps in both.
- Largest run
- 1.47 × 109 particles
- Benchmark on one four-A100 node; data replication limits the code to about 2 × 109.
NORI stands for N-body code Optimized for Random Interactions. The code is described in Zier et al. 2026 (submitted), written with Rüdiger Pakmor, Xuejian Shen, Rongrong Liu, Lucas Schleuss, Mark Vogelsberger and Lars Hernquist. To our knowledge it is the first SIDM code that runs on GPUs.
What it is for
Gravothermal collapse gets harder to simulate as it proceeds: time steps shrink as the scattering rate in the core grows, the core loses mass and particles, and small energy errors shift the collapse time. A large particle number and careful energy conservation are essential.
General-purpose codes such as AREPO-2 distribute the volume over many processors and use individual time steps, which suits cosmological simulations. Isolated SIDM simulations, however, typically contain fewer than 108 particles, which fit into the memory of one compute node or even one GPU. NORI exploits this: every node, GPU or group of GPUs holds a complete copy of the particles and of the tree, so tree walks need no MPI communication and the work can be divided freely.
Design
Tree construction and gravity
After every drift, particles are sorted along a Morton curve and the octree is rebuilt from the bottom up. Each node is determined by a few neighboring keys or nodes, so CPU ranks or GPU threads build different parts of the tree without coordination, and the result does not depend on how the work was divided. Gravity uses a one-sided tree walk with monopoles or with quadrupoles and the relative opening criterion of the GADGET codes. Quadrupoles cost more per node but conserve energy markedly better at the same force accuracy, so they are the default.
Scattering: frozen decisions, ordered update
NORI separates deciding which pairs scatter from applying the scatterings. All scattering probabilities are evaluated in parallel from a common particle state, with velocities held fixed. The accepted events from all ranks are then sorted by a random 64-bit priority and applied one after another in that single global order, each using the velocities left by the events before it. Every event conserves momentum and kinetic energy to round-off, a particle can scatter several times per step, and no event has to be cancelled. Only the accepted events, a small fraction of all trials, need this serial step.
Freezing the velocities for the decisions is an approximation of second order in the pair probabilities, the same order as the sequential processing in AREPO-2, and it vanishes as the time step goes to zero.
Time integration
All particles share one global time step, set by the strictest gravitational or SIDM limit and integrated with a kick–drift–kick leapfrog on a power-of-two timeline. The SIDM limit is the per-pair criterion introduced for AREPO-2. A global step keeps both partners of every event synchronized; its cost during runaway collapse is discussed below.
Parallel layout and reproducibility
On CPUs, all ranks of a node share one copy of the particles and the tree, and take chunks of target particles from a shared counter. In the replicated GPU mode, each GPU holds a complete copy; in the partitioned mode, the GPUs of one node split the particles and read each other's data by peer-to-peer access, which raises the memory limit. With a fixed evaluation order, counter-based Philox random numbers, fused multiply-adds only where requested, and no transcendental library functions in the simulation arithmetic, runs are bitwise identical across CPU and CUDA configurations, so the GPU code can be debugged on CPUs.
Validation
- Force accuracy. Against direct summation for a Hernquist sphere with 106 particles, the opening parameters are chosen so that the 90th-percentile relative force error, 2.0 × 10−3, matches that of the gravity settings used in AREPO-2.
- Collisionless equilibrium. An NFW halo of 1010 M☉ with 107 particles inside r200 keeps its profile for 50 Gyr outside the innermost 0.2 kpc, where two-body relaxation lowers the density. With quadrupole gravity, the relative drift of the total energy stays below 10−4.
- Gravothermal collapse. The same halo with σ/m = 30 cm2 g−1 collapses in close agreement with our AREPO-2 run. The quadrupole runs stay within 10−4 of their initial energy up to the highest central densities reached.
- A reference solution. With 108 particles inside r200, about three times as many as the largest previous isolated collapse simulation (Fischer et al. 2025), we followed the halo to 104 times the NFW scale density. Two runs with independent random numbers reach a given central density within about 0.1 percent of each other in time. The curve is shown on the overview page.
Performance
The benchmarks ran on Harvard's Cannon cluster: CPU nodes with 112 Intel Sapphire Rapids cores, and GPU nodes with four NVIDIA A100 GPUs (80 GB each). For a halo with 108 particles inside r200 and global time steps in both codes, one time step takes 15 s with NORI and 60 s with AREPO-2 on one CPU node, a speed-up of about 4 that grows to 7.7 on 16 nodes. Part of this comes from the tree settings: with the same gravity settings as AREPO-2, NORI is 2.8 times faster on one node. One GPU node takes 0.99 s per step, 15 times faster than NORI and 60 times faster than AREPO-2 on one CPU node. Adding GPU nodes helps less, by a factor of 1.5 from one to four nodes, partly because every GPU repeats the sorting and tree construction.
In production, AREPO-2 uses individual time steps, so the fairer comparison is the time to solution for a full collapse. For the halo with 107 particles inside r200, NORI on one GPU node reaches 103 and 104 times the scale density after 3.8 and 5.2 hours; AREPO-2 on one CPU node needs 5.3 and 5.4 days, a speed-up of 34 and 25. The advantage shrinks during the runaway, to 9 at 105 times the scale density, because the global time step advances the whole halo at the pace required by the core.
Limitations
The choices that make NORI fast also narrow its range. The whole tree must fit into the memory of each node or GPU, which limits runs to about 2 × 109 particles. Global time steps become inefficient once a small core sets the pace for the whole halo. The code handles isolated systems of equal-mass particles with elastic, isotropic scattering, without periodic boundaries, external potentials, gas or galaxy formation; such simulations still need a general-purpose code like AREPO-2. Other isolated problems, such as dark-matter-only collisions of galaxy clusters, need no changes to the code.
Paper
-
NORI: a GPU-accelerated code for simulating self-gravitating and self-interacting dark matter
Zier, Pakmor, Shen et al. · submitted · 2026
Design, validation and performance of NORI, and a reference collapse simulation with 10⁸ particles inside the virial radius.