exoplasim/docs/performance.md

Simulator performance

Assessment and optimization date: 2026-10-10. This report concerns the ExoPlaSim simulator, not SBDART or Python postprocessing.

Current result

Keep the checked build for existing experiments. The local working copy uses that mode and a freshly rebuilt serial T21 executable.

The retained Fortran changes reduced median CPU time by 2.2% in the final checked-build comparison. They produced byte-identical output and restart files. The final native-performance comparison showed no reliable additional whole-model gain from the source changes.

Performance mode is explicit and optional. Its compiler settings can change weather trajectories. Faster completion does not establish climate-statistical equivalence.

Retained changes

exoplasim/plasim/src/legmod.f90 splits the FP64 serial fc2sp transform into symmetric and antisymmetric loops. It computes paired-latitude sums and differences once per zonal mode instead of testing parity for each spectral coefficient. The compiler now vectorizes both loops. Each output coefficient retains its latitude summation order.

The FP32 serial path retains its original loop after mixed parity-split benchmark results. The distributed fc2sp branch is unchanged.

The checked GNU Fortran build uses four local wind accumulators in dv2uv. The lanes contain the real and imaginary parts of both wind components. The code writes results after each spectral sum instead of repeatedly updating large output arrays. It retains spectral summation order and restores the planetary-vorticity adjustment to the input array.

The packed wind implementation regressed in native microbenchmarks. Performance builds therefore retain the original wind loop. EXOPLASIM_CHECKED selects the checked implementation. This is a measured build-mode distinction, not an unconditional replacement.

exoplasim/configure.sh adds checked and performance modes. Checked remains the default. Performance removes GNU Fortran runtime checks. Explicit native tuning enables host instructions and disables floating-point contraction. Debug settings, traps, initialization, and precision selection remain available. See docs/build-modes.md.

No fast-math, precision reduction, threading, FFT replacement, or physics-frequency changes were introduced.

Whole-model measurements

The host exposes four logical CPUs from an AMD Ryzen 5 PRO 3400G. Other workloads were active. Tests use serial T21, ten levels, and FP64 default reals.

Each simulator run receives identical namelists, input data, and a saved restart from a copied runtime template. The benchmark requests exactly 1,440 steps and keeps output enabled. It rotates execution order and repeats each variant three times. Compilation, setup, Python processing, and output comparison are outside the timed section.

Final checked-build batch:

Source	Median user CPU seconds	Median wall seconds
Pinned upstream	58.61	59.07
Current source	57.32	57.67

Median CPU time fell by 2.2%. Median wall time fell by 2.4%. Current source consumed less CPU time in all three paired comparisons. One wall-time observation had additional scheduling delay. These are modest results from a shared host, not isolated-hardware guarantees.

Final native-performance batch:

Source	Median user CPU seconds	Median wall seconds
Pinned upstream	51.95	52.30
Current source	52.09	52.38

The difference is within observed run variation. Do not claim a native whole-model source speedup. These batches occurred at different times, so do not use their medians to calculate a cross-build speedup.

An earlier combined batch measured about 7% less CPU time for native-performance upstream than checked upstream. Its compiler settings also changed the weather trajectory. That is an experimental timing result, not permission to replace an existing experiment's build.

Numerical acceptance

All repeated runs within each variant produced the same output hash. Current source matched upstream output and restart files byte for byte when flags matched, in both final whole-model batches. Parsed output fields also matched exactly and contained finite values.

The kernel harness tested T21, T42, and T85 with ten levels. It covered full and half-latitude partitions, FP32 and FP64, and checked and native settings. Every executable exercised deterministic nonzero coefficients, zero inputs, and nontrivial filters with a different planetary-vorticity value. All 24 combinations matched upstream bit for bit. The harness also verified restoration of the input vorticity array.

These results cover the tested source changes under matched flags. They do not cover other planetary settings, complete MPI execution, other model families, or long-run climate statistics. Half-latitude kernel tests are not MPI acceptance.

Generic and native builds did not match each other. In the combined 1,440-step test, the largest local surface-temperature difference reached about 17.6 K after contraction was disabled. The test compares weather trajectories, not equilibrated climate means. Disabling contraction did not eliminate cross-build differences, so it is not a complete numerical-equivalence fix. The exact causes of all cross-build differences remain unisolated.

A final Python API smoke test forced compilation of the current checked source. It produced 98 postprocessed fields with finite surface temperatures from 189.74 K to 304.10 K. This is a startup check, not a reference climate.

Kernel timing and interpretation

In the final FP64 serial checked harness, fc2sp speedups were approximately 1.27, 1.64, and 1.31 times at T21, T42, and T85. The checked dv2uv speedups were approximately 1.78, 1.96, and 2.03 times. Short synthetic measurements have noise and do not predict whole-model savings directly.

The native FP64 harness showed larger fc2sp gains but no intentional wind-kernel change. Native baseline and current wind timings varied despite equivalent implementations. This reinforces the need for whole-model measurements and sufficient timing intervals.

The earlier instrumented baseline profile located the following approximate self-time groups:

Group	Approximate sampled self-time
Shortwave and longwave radiation	26%
Listed Legendre kernels	22%
Listed rain and convection kernels	21%
Listed FFT kernels	10%
Vertical diffusion	4%

That gprof run accounts for about 37.47 sampled seconds against 56.90 user CPU seconds. Use it as hotspot evidence, not exact accounting. Compiler inlining, instrumentation, sampling, and host contention limit the result.

Execution path

plasim.f90 contains the sequential timestep loop. Each step calls gridpointa, spectrala, gridpointd, and spectrald. Dynamics use Fourier and Legendre transforms. Gridpoint physics call flux, radiation, rain, surface, and carbon modules.

Python prepares inputs and starts the compiled executable. It is not the numerical timestep hot path. The simulator uses its own FFT implementation. Gridpoint arrays generally store horizontal points contiguously, which favors horizontal SIMD. Vertical recurrences still require ordered work within each column.

MPI support exists. Several collective operations run separately for each level. The serial executable uses MPI stubs. The inspected source has no OpenMP, OpenACC, or CUDA backend. This optimization did not add one.

GPU assessment

Radiation and column physics contain independent horizontal work. FFTs can use batched libraries. Legendre transforms can use batched reductions or matrix operations. These are plausible accelerator targets.

T21 has 2,048 columns and 20,480 atmospheric cells at ten levels. One FP64 3D field occupies only 160 KiB. Many small launches and repeated host transfers can consume the compute savings. A useful GPU backend should keep state and scratch buffers resident across timesteps and transfer only necessary diagnostics and checkpoints.

T42 has 8,192 columns, and T85 has 32,768. Higher resolution, more levels, and batched independent planets give an accelerator more work. Legendre work grows roughly cubically with truncation at a fixed level count. FFT work grows roughly as grid size times log longitude count.

A GPU must have adequate FP64 throughput. FP32 or mixed precision requires separate scientific validation. No accelerator is exposed to this container, and no GPU speedup was measured.

The baseline profile also limits partial offload. If radiation occupies 26% of runtime, infinitely fast radiation permits only about 1.35 times whole-model speed. Infinitely fast FFTs alone permit about 1.11 times. Transfers would reduce those gains. A major speedup needs a coherent device-resident dynamics and physics path or effective ensemble batching.

Rust assessment

A language port does not reduce transform, reduction, or physics work by itself. The hot path is already compiled Fortran. Rust could match, improve, or worsen performance depending on layout, vectorization, bounds checks, math libraries, and batching.

The PlaSim directory contains about 44,600 Fortran lines, including optional modules. A full port must also preserve restart formats, calendars, MPI behavior, physical thresholds, and exoplanet changes.

Rust may help state ownership, embedding, testing, or independent model instances. Those are separate goals. Attribute performance gains to algorithms, layout, SIMD, or parallelism, not the language name. Stay with Fortran for the current speed objective.

Reproduction and next work

Use docs/transform-tests.md for benchmark commands and test scope. Use docs/build-modes.md for compiler configuration and cache rebuilding. Evidence is in docs/work/2026-10-10/optimization/. Baseline profiling evidence remains in docs/work/2026-10-10/performance/.

Further work should target representative radiation and convection workloads, or ensemble throughput. Benchmark at the intended resolution and physics configuration before changing those kernels. Any substantial numerical change needs conservation checks, restart tests, energy budgets, and long-run climate statistics. Bitwise long-trajectory agreement is not the sole acceptance criterion for deliberate numerical changes.