Skip to main content

Code

Submitted 2026

NORI

An N-body code for isolated halos of self-gravitating, self-interacting dark matter that runs on multi-core CPUs and on GPUs. It is built to make high-resolution simulations of gravothermal collapse affordable on a single GPU node.

At a glance

Problem
Isolated SIDM halos
Equal-mass particles; elastic, isotropic scattering with a constant or tabulated velocity-dependent cross section.
Hardware
CPUs and GPUs
Shared-memory CPU mode; GPU kernels compile to CUDA or HIP from one source.
Gravity
Tree, monopole or quadrupole
Morton-ordered octree built bottom-up at every step; up to 8 particles per leaf by default.
Reproducibility
Bitwise identical
Across CPU and CUDA runs with different numbers of ranks, GPUs and segments (tested).
Per-step speed
Up to 60× AREPO-2
One node with four A100 GPUs vs. AREPO-2 on one 112-core CPU node; 108 particles in r200, global time steps in both.
Largest run
1.47 × 109 particles
Benchmark on one four-A100 node; data replication limits the code to about 2 × 109.

NORI stands for N-body code Optimized for Random Interactions. The code is described in Zier et al. 2026 (submitted), written with Rüdiger Pakmor, Xuejian Shen, Rongrong Liu, Lucas Schleuss, Mark Vogelsberger and Lars Hernquist. To our knowledge it is the first SIDM code that runs on GPUs.

What it is for

Gravothermal collapse gets harder to simulate as it proceeds: time steps shrink as the scattering rate in the core grows, the core loses mass and particles, and small energy errors shift the collapse time. A large particle number and careful energy conservation are essential.

General-purpose codes such as AREPO-2 distribute the volume over many processors and use individual time steps, which suits cosmological simulations. Isolated SIDM simulations, however, typically contain fewer than 108 particles, which fit into the memory of one compute node or even one GPU. NORI exploits this: every node, GPU or group of GPUs holds a complete copy of the particles and of the tree, so tree walks need no MPI communication and the work can be divided freely.

Design

Tree construction and gravity

After every drift, particles are sorted along a Morton curve and the octree is rebuilt from the bottom up. Each node is determined by a few neighboring keys or nodes, so CPU ranks or GPU threads build different parts of the tree without coordination, and the result does not depend on how the work was divided. Gravity uses a one-sided tree walk with monopoles or with quadrupoles and the relative opening criterion of the GADGET codes. Quadrupoles cost more per node but conserve energy markedly better at the same force accuracy, so they are the default.

Scattering: frozen decisions, ordered update

NORI separates deciding which pairs scatter from applying the scatterings. All scattering probabilities are evaluated in parallel from a common particle state, with velocities held fixed. The accepted events from all ranks are then sorted by a random 64-bit priority and applied one after another in that single global order, each using the velocities left by the events before it. Every event conserves momentum and kinetic energy to round-off, a particle can scatter several times per step, and no event has to be cancelled. Only the accepted events, a small fraction of all trials, need this serial step.

Three-panel schematic. Left: a particle i with its smoothing sphere of radius h_i; each neighbor is tested with a Philox random number and one trial is accepted; velocities stay unchanged. Middle: accepted pairs from three ranks are merged and sorted by random priority into a single list e1 to e7. Right: the events are applied in that order; vertical bars connect the two partners of each event, and open circles mark particles that already scattered earlier in the step.
The scattering step in NORI. Left: each particle tests all neighbors within its smoothing length, with velocities frozen. Middle: the accepted events of all ranks are merged into one order. Right: the ordered update applies them serially, so a particle that scatters twice (open circles) uses its updated velocity the second time. Figure: Zier et al. 2026.

Freezing the velocities for the decisions is an approximation of second order in the pair probabilities, the same order as the sequential processing in AREPO-2, and it vanishes as the time step goes to zero.

Time integration

All particles share one global time step, set by the strictest gravitational or SIDM limit and integrated with a kick–drift–kick leapfrog on a power-of-two timeline. The SIDM limit is the per-pair criterion introduced for AREPO-2. A global step keeps both partners of every event synchronized; its cost during runaway collapse is discussed below.

Parallel layout and reproducibility

On CPUs, all ranks of a node share one copy of the particles and the tree, and take chunks of target particles from a shared counter. In the replicated GPU mode, each GPU holds a complete copy; in the partitioned mode, the GPUs of one node split the particles and read each other's data by peer-to-peer access, which raises the memory limit. With a fixed evaluation order, counter-based Philox random numbers, fused multiply-adds only where requested, and no transcendental library functions in the simulation arithmetic, runs are bitwise identical across CPU and CUDA configurations, so the GPU code can be debugged on CPUs.

Three-panel schematic of data layouts. Left, CPU shared-memory node: one particle replica and one shared tree, with ranks claiming chunks of target particles from a shared atomic counter. Middle, replicated GPU mode: each of four GPUs stores all particles and the full tree and walks one interval of targets. Right, partitioned GPU mode: four GPUs each store a quarter of the particles but a full tree copy, inside a dashed frame marking one replica.
The three parallel configurations. Blue marks the particles a rank or GPU stores, orange the particles whose tree walks it carries out. Every GPU holds a complete tree in both GPU modes. Figure: Zier et al. 2026.

Validation

  • Force accuracy. Against direct summation for a Hernquist sphere with 106 particles, the opening parameters are chosen so that the 90th-percentile relative force error, 2.0 × 10−3, matches that of the gravity settings used in AREPO-2.
  • Collisionless equilibrium. An NFW halo of 1010 M☉ with 107 particles inside r200 keeps its profile for 50 Gyr outside the innermost 0.2 kpc, where two-body relaxation lowers the density. With quadrupole gravity, the relative drift of the total energy stays below 10−4.
  • Gravothermal collapse. The same halo with σ/m = 30 cm2 g−1 collapses in close agreement with our AREPO-2 run. The quadrupole runs stay within 10−4 of their initial energy up to the highest central densities reached.
  • A reference solution. With 108 particles inside r200, about three times as many as the largest previous isolated collapse simulation (Fischer et al. 2025), we followed the halo to 104 times the NFW scale density. Two runs with independent random numbers reach a given central density within about 0.1 percent of each other in time. The curve is shown on the overview page.

Performance

The benchmarks ran on Harvard's Cannon cluster: CPU nodes with 112 Intel Sapphire Rapids cores, and GPU nodes with four NVIDIA A100 GPUs (80 GB each). For a halo with 108 particles inside r200 and global time steps in both codes, one time step takes 15 s with NORI and 60 s with AREPO-2 on one CPU node, a speed-up of about 4 that grows to 7.7 on 16 nodes. Part of this comes from the tree settings: with the same gravity settings as AREPO-2, NORI is 2.8 times faster on one node. One GPU node takes 0.99 s per step, 15 times faster than NORI and 60 times faster than AREPO-2 on one CPU node. Adding GPU nodes helps less, by a factor of 1.5 from one to four nodes, partly because every GPU repeats the sorting and tree construction.

In production, AREPO-2 uses individual time steps, so the fairer comparison is the time to solution for a full collapse. For the halo with 107 particles inside r200, NORI on one GPU node reaches 103 and 104 times the scale density after 3.8 and 5.2 hours; AREPO-2 on one CPU node needs 5.3 and 5.4 days, a speed-up of 34 and 25. The advantage shrinks during the runaway, to 9 at 105 times the scale density, because the global time step advances the whole halo at the pace required by the core.

Wall-clock time in hours, logarithmic, versus central density in units of the scale density for three runs: AREPO-2 with individual steps on one CPU node (black, top), NORI with global steps on one CPU node (orange, middle), and NORI on one GPU node (green, bottom). All curves rise steeply at first and then flatten; the GPU curve lies about an order of magnitude below the others and turns up again above about ten thousand.
Cumulative wall-clock time to reach a given central density in the collapsing halo with 107 particles inside r200. The GPU run (green) is more than an order of magnitude cheaper up to 104 times the scale density; its cost rises in the runaway, when the global time step becomes small. Figure: Zier et al. 2026.

Limitations

The choices that make NORI fast also narrow its range. The whole tree must fit into the memory of each node or GPU, which limits runs to about 2 × 109 particles. Global time steps become inefficient once a small core sets the pace for the whole halo. The code handles isolated systems of equal-mass particles with elastic, isotropic scattering, without periodic boundaries, external potentials, gas or galaxy formation; such simulations still need a general-purpose code like AREPO-2. Other isolated problems, such as dark-matter-only collisions of galaxy clusters, need no changes to the code.

Paper

  1. NORI: a GPU-accelerated code for simulating self-gravitating and self-interacting dark matter

    Zier, Pakmor, Shen et al. · submitted · 2026

    Design, validation and performance of NORI, and a reference collapse simulation with 10⁸ particles inside the virial radius.

    arXiv ADS