perf(batch-ssa): compile the model once for the whole CPU pool

[akutuva21]

perf(batch-ssa): compile the model once for the whole CPU pool

CppBatchSsaSimulator::simulateMultiCore in cpp/engine/BatchSsa.cpp built one
OdeIntegrator per worker plus one on the calling thread. Every batch call paid
numThreads + 1 full model compiles (16 on a 15-CPU host) and held numThreads
copies of the compiled reaction network alive simultaneously. This change shares a
single integrator across the pool: numThreads + 1 compiles becomes 1.

Scope: one file, 18 insertions, 8 deletions. cpp/engine/BatchSsa.cpp only.
No change to cpp/engine/OdeIntegrator.* (three operators plus perfEngine hold
regions of it) and none to cpp/bindings/ (perfTooling's lane).

---

Why it is safe

The previous code carried the comment // Independent OdeIntegrator per thread for thread safety. Sharing one is sound because the SSA path is re-entrant on
OdeIntegrator:

• integrate() dispatches method == "ssa" to integrateSSA()
  (cpp/engine/OdeIntegrator.cpp:1541).
• integrateSSA() (cpp/engine/OdeIntegrator.cpp:2360) reads only the compiled
  reaction/group tables and writes exclusively to its own locals and the
  OdeResult it returns.
• Every helper it reaches is const and returns its output through an
  out-parameter: outputTimes (:1351), parseStopIf (:1391),
  updateGroups (:1340), stopConditionMet (:1404), computePropensity
  (:2320).
• The only mutable member, groupValues_, is scratch for derivs()
  (cpp/engine/OdeIntegrator.hpp:119), which the SSA path never calls.

memoryUsageBytes is corrected to match: it previously reported
numThreads * sizeof(OdeIntegrator).

---

Measured — deterministic metric (EXEMPT CLASS)

Instructions retired, from /usr/bin/time -l. This is a counter, not a
duration: it cannot move when nothing changed. 3 batch reps per process, 2-3
process runs per arm, 15 threads, batch_size <= 20000.

config	before	after	delta
models/egfr_net.bngl batch=60, t_end=0.02	202.76e9	129.01e9	-36.4%
models/egfr_net.bngl batch=500, t_end=0.02	1048.45e9	974.66e9	-7.0%
models/gene_expr_simple.bngl batch=20000, t_end=2000	2.713e9	2.596e9	-4.3%
models/isomerization.bngl batch=20000, t_end=20	3.983e9	3.914e9	-2.0%

Within-arm spread across process runs: 0.11% or less on every row. The arms
separate far outside that.

Peak RSS, /usr/bin/time -l, egfr_net batch=500:
before 60,617,280 / 63,832,064 / 68,173,824 B; after 27,901,952 / 29,442,048 /
28,408,656 B — roughly halved, because 15 compiled networks are no longer
resident at once.

The effect scales as expected: it is a fixed per-batch cost, so it is largest
when the batch is small relative to the compile, and vanishes as the batch grows.

---

Measured — wall clock (NOT EXEMPT, reported with full context)

Interleaved A/B in one session, 12 alternations x 2 reps per arm = 24 samples
per arm, via /tmp/bng-bench-lock/benchlock (slot 0, held).

• R = 53 runnable; loadavg band ~88-119 (coarse band only, per memWatch's
  calibration of loadavg as non-responsive on this host).
• Co-tenants, complete list (ps -axo pid,etime,command, including my own):
  perfOracle verify/identity_check.py + its bng_cpp_changed model.bngl
  children; perfNetGen bench/netgen_bench.py --full; two pytest tests/validation; shared-main bng_cpp isingspin_localfcn.bngl;
  correctness bng_cpp hybrid_test.bngl; swarmCache /tmp/cachebench/bng_cpp_base big_1000.bngl; perfNetGen build/cpp/bng_cpp .../e6_gen.bngl; perfEngine
  held benchlock slot 0 for the NFsim baseline.
• Within-arm spread: 45% - 138%.

config	A (before) min traj/s sim	B (after) min traj/s sim	ratio at minima
egfr_net batch=200, 15 threads	37.5	57.3	1.53x
isomerization batch=20000, 15 threads	780,490	832,182	1.07x

I am claiming the wall-clock result as CORROBORATION, not as the result. With
45-138% within-arm spread the minima are the only usable statistic and they
overlap the other arm's range. The instruction counts are the number to rely on:
they are deterministic and reproduce to 0.11%.

Thread scaling, isomerization batch=20000, sim_ms minimum over 4 reps:
1 thread 115.5 ms, 2 threads 64.0, 4 threads 35.2, 8 threads 23.7, 15 threads
19.4. Efficiency collapses past 4 threads (90% / 82% / 61% / 40%). This is not
load imbalance in the partition — see candidate D below — and it is not fan-out
(candidate B). It is host contention plus allocator pressure inside
integrateSSA, which is not my file.

---

Candidate population, including the losers

#	candidate	verdict	number
A	share one compiled OdeIntegrator across all workers	KEPT	-36.4% instructions at batch=60, -7.0% at batch=500; peak RSS halved
B	persistent worker pool instead of std::async per batch	REJECTED — measured no-win	isolated fan-out cost is 0.09-0.12 ms per batch for 15 no-op std::async tasks + join (--mode fanout, 8 reps). That is 0.4% of a 24 ms batch. Nothing to amortise.
C	std::thread + condvar instead of std::async	REJECTED — measured no-win	std::async with launch::async already costs one thread per task; candidate B's measurement covers it.
D	dynamic work-stealing chunks	REJECTED — measured no-win AND would break bit-identity	benchmarks/batch_ssa/parse_dump.py on the per-trajectory event counts: heaviest chunk / mean chunk = 1.0047x, so static partitioning costs 0.47% on isomerization batch=20000/15 threads. Separately, the observable means are summed per worker and reduced in worker order, so ANY change to the partition count or boundaries changes the floating-point summation order and breaks the bit-identity gate. Load balancing and bit-identity are mutually exclusive here; 0.47% is not worth trading it for.
E	false-sharing padding on trajectoryEventCounts / finalSpecies	REJECTED — no addressable work	workers write disjoint contiguous b ranges, so only the 14 chunk-boundary cache lines of batchSize/16 lines are ever shared (14 of 1250 at batch=20000).

Profiled bottleneck (sample <pid> 6, macOS, top-of-stack), which is why the
pool layer was the right place to look at all — and where the ceiling is:

• egfr_net batch=500: integrateSSA 50,644 of ~42,000 aggregated
  top-of-stack samples dominates; main thread blocked in future::get is
  6,140; OdeIntegrator::OdeIntegrator (the per-worker compile) is 85 samples,
  1.95% of main-thread samples.
• isomerization batch=20000: integrateSSA 20,419; allocator
  (_xzm_free + _xzm_xzone_malloc* + operator new) ~4,699; the pool lambda
  body outside integrateSSA 484.

Where the remaining time goes, and why it is not mine — reported, not edited:
integrateSSA builds and discards, for every one of the 20,000 trajectories, a
full numPoints x numSpecies concentration history
(appendScheduledThrough does result.concentrations.push_back(y) at
cpp/engine/OdeIntegrator.cpp:2393 and :2401) plus outputTimes()
(:2362) and parseStopIf() (:2363) per call. CpuBatchSsaSimulator reads
only res.concentrations.back() and res.observables. An option that suppresses
the history would be the next large win, and it lives in
cpp/engine/OdeIntegrator.*, which is perfEngine's / swarmCrossover's /
swarmCache's / swarmSerial's. Flagging it rather than touching it.

---

Correctness gate

Bit-identity against the pre-change binary — two binaries, separate processes,
same base_seed, --mode dump writes every output array as raw little-endian
bytes, compared with cmp:

IDENTICAL  isomerization_20_20000_15            (481,481 B)
IDENTICAL  gene_expr_simple_500_20000_15        (401,211 B)
IDENTICAL  gene_expr_simple_2000_5000_8         (101,211 B)
IDENTICAL  toy-jim_50_5000_15                   (703,519 B)
IDENTICAL  egfr_net_0.02_500_15                  (754,492 B)
IDENTICAL  performance_test_models/egfr_net_0.02_400_15  (601,926 B)
IDENTICAL  egfr_net_0.02_800_1                   (1,200,892 B)

Each dump covers finalSpecies (int32), finalObservables (float),
trajectoryEventCounts (uint32), observableMeans / observableStdDevs (float),
and meanSpecies / stdSpecies / meanObservables / stdObservables (double) —
so final states AND observable means/stds are both proven bit-identical, at 15
threads, 8 threads, and 1 thread.

Gates, all from this worktree's own build (Release -j4, BUILD_TESTS=ON,
bindings ON):

$ ctest --test-dir build -j4 --output-on-failure
100% tests passed out of 461
Total Test time (real) =   7.35 sec      [peak RSS 634,929,152 B]

$ python3 benchmarks/batch_ssa/run_under_test.py tests/python -q
UNDER TEST pkg: /Users/akutuva/Documents/BioNetGen/BNG3-batch/python/bionetgen/__init__.py
UNDER TEST ext: /Users/akutuva/Documents/BioNetGen/BNG3-batch/build/cpp/_bionetgen_cpp.cpython-314-darwin.so
862 passed, 28 skipped, 1388 warnings in 15.17s    [peak RSS 430,669,824 B]

$ python3 tests/test_batch_ssa_statistical_parity.py
...
ALL STATISTICAL PARITY TESTS PASSED SUCCESSFULLY!

The parity gate covers conservation identities, seed reproducibility, and
threads=0 vs threads=1 equality; on this Mac it also ran the Metal GPU arm.
My evidence is macOS-local only — the parity gate's CPU reference arm is
hardware-independent, but I have no Linux run, so I say nothing about the
cpp/CMakeLists.txt extension-linking change in flight (PR #30).

benchmarks/batch_ssa/run_under_test.py exists because bionetgen is installed
here as a scikit-build editable whose meta_path finder beats sys.path: without
it, pytest tests/python in this worktree silently tests the shared main
tree. The two UNDER TEST lines above are the proof of which tree ran.

Build numbers, /usr/bin/time -l: configure 302 MB / 23.3 s; full
Release -j4 build (bindings ON, tests ON) 471 MB / 157.2 s; incremental relink
of BatchSsa.cpp 463 MB / 23.5 s. All far under the 4 GiB cap. Every batch run
above is batch_size <= 20000; peak RSS 634 MB is the largest single figure and
belongs to ctest, not to a batch.

---

Base and provenance

Base commit 6889fba. Branch agent/perf-batch, two commits: the benchmark
harness first (5d2c038), then the change (f49406e) — the harness predates the
change so the baseline is reproducible from the branch itself.

Both A/B binaries were linked in this worktree from its own build/ tree, one
before the edit and one after, via benchmarks/batch_ssa/build_driver.sh, which
reuses CMake's own flags.make / link.txt for bng_cpp rather than adding a
target to cpp/CMakeLists.txt (perfTooling's file).

git stash was never used. The baseline binary was produced by copying the
built binary aside, per Main's standing rule.

Reproducing

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=ON -DBUILD_PYTHON_BINDINGS=ON
cmake --build build -j4
benchmarks/batch_ssa/build_driver.sh "$PWD" /tmp/bench_before
# ... apply the change, rebuild ...
benchmarks/batch_ssa/build_driver.sh "$PWD" /tmp/bench_after

# bit-identity
for b in before after; do /tmp/bench_$b --model models/isomerization.bngl \
  --batch 20000 --t-end 20 --threads 15 --reps 1 --mode dump --dump /tmp/d_$b.bin; done
cmp /tmp/d_before.bin /tmp/d_after.bin

# deterministic work removed
/usr/bin/time -l /tmp/bench_before --model models/egfr_net.bngl --batch 60 \
  --t-end 0.02 --threads 15 --reps 3
/usr/bin/time -l /tmp/bench_after  --model models/egfr_net.bngl --batch 60 \
  --t-end 0.02 --threads 15 --reps 3

# partition balance, load-independent
/tmp/bench_after --model models/isomerization.bngl --batch 20000 --t-end 20 \
  --threads 15 --reps 1 --mode dump --dump /tmp/d.bin
python3 benchmarks/batch_ssa/parse_dump.py /tmp/d.bin 20000 15

# pool fan-out cost, isolated
/tmp/bench_after --mode fanout --threads 15 --reps 8
```---

## Post-rebase re-verification

Rebased onto `origin/main` at `1c78a72` (was `6889fba`). The rebase hit a directory-vs-directory conflict on `bench/` — the shared fitness harness landed on main — so the harness commits were replayed onto main rather than merged, keeping `benchmarks/batch_ssa/` as its home. The production file re-applied with no conflict.

**`git diff --name-only origin/main HEAD` at the rebased tip — five harness files plus one production file:**

benchmarks/batch_ssa/bench_batch_ssa.cpp
benchmarks/batch_ssa/build_driver.sh
benchmarks/batch_ssa/parse_dump.py
benchmarks/batch_ssa/run_ab.sh
benchmarks/batch_ssa/run_under_test.py
cpp/engine/BatchSsa.cpp

**Two commits, harness before change:**

e37bcf0 perf(batch-ssa): compile the model once for the whole CPU pool
f7b393d bench(batch-ssa): CPU pool throughput + bit-identity harness

### Verification tokens

Measured on this branch. A merge or conflict-pick that reverted the change returns the pre-change values:

$ grep -c 'Independent OdeIntegrator per thread' cpp/engine/BatchSsa.cpp
0 # pre-change value: 1
$ grep -c 'OdeIntegrator integrator(model_, network_)' cpp/engine/BatchSsa.cpp
2 # pre-change value: 1

`2` is one integrator in `simulateSingleWorker` (unchanged by this PR) and one in `simulateMultiCore` (the shared one). `git diff HEAD -- cpp/engine/BatchSsa.cpp` is empty, so the working tree carries exactly what is committed.

### Re-entrancy survives the rebase — re-checked, not assumed

`integrateSSA` is the property this change rests on, and `perfEngine`'s SSA change is merged into this base. Re-checked on the rebased tree:

- `OdeIntegrator.hpp` — `groupValues_` is still the single `mutable` member, still documented as `derivs()` scratch.
- `OdeIntegrator.cpp` — `updateGroups`, `computePropensity`, `outputTimes`, `parseStopIf` and `stopConditionMet` all still carry the trailing `const` on the definition, each writing only through its out-parameter.
- The empirical check is stronger than any of it: the batch-SSA parity gate and the `cmp` identity below both pass on this base.

### Bit-identity, re-run against a baseline built from the REBASED `origin/main`

Not the old 6889fba binary. `git checkout origin/main -- cpp/engine/BatchSsa.cpp`, rebuild, snapshot the driver, then restore from a `cp` copy (no `git stash`). Full output dumps, `cmp` empty:

IDENTICAL isomerization_20_20000_15 (481,481 B)
IDENTICAL gene_expr_simple_500_20000_15 (401,211 B)
IDENTICAL gene_expr_simple_2000_5000_8 (101,211 B)
IDENTICAL toy-jim_50_5000_15 (703,519 B)
IDENTICAL egfr_net_0.02_500_15 (754,492 B)
IDENTICAL performance_test_models/egfr_net_0.02_400_15 (601,926 B)
IDENTICAL egfr_net_0.02_800_1 (1,200,892 B)

15, 8 and 1 thread configurations. Covers `finalSpecies`, `finalObservables`, `trajectoryEventCounts`, `observableMeans` / `observableStdDevs`, and the four double `mean*` / `std*` matrices.

### Gates, re-run on the rebased tree

Counts rose because `origin/main` merged other lanes' tests between my two runs; the pass/fail line is what matters.

$ ctest --test-dir build -j4
100% tests passed out of 473 [was 461 at 6889fba]

$ python3 benchmarks/batch_ssa/run_under_test.py tests/python -q
UNDER TEST pkg: /Users/akutuva/Documents/BioNetGen/BNG3-batch/python/bionetgen/init.py
UNDER TEST ext (submodule): /Users/akutuva/Documents/BioNetGen/BNG3-batch/build/cpp/_bionetgen_cpp.cpython-314-darwin.so
958 passed, 28 skipped, 1380 warnings in 23.10s [was 862 passed, 28 skipped at 6889fba]

$ python3 tests/test_batch_ssa_statistical_parity.py
ALL STATISTICAL PARITY TESTS PASSED SUCCESSFULLY!

Rebuild after the rebase: 454,541,312 B peak RSS / 101.3 s real, `cmake --build build -j4`.

### Three caveats stated rather than buried

1. **`origin/main` currently does not configure on a fresh tree**, and it did not when I finished. `git show origin/main:tests/cpp/CMakeLists.txt | grep -n test_correctness_regressions` returns the `add_executable` block twice, at lines 427 and 435, so a fresh `cmake -B` dies with `add_executable cannot create target "test_correctness_regressions"`. Found independently by `irSnapshot`, `swarmCache` and `memWatch`; the duplicated content belongs to `correctness`'s PR #59. **My `ctest 473/473` and `pytest 958` numbers above therefore demonstrate the state of THIS branch at THIS commit, not that current main configures.** My build tree was configured before the duplicate landed, so an incremental build cannot see it. `@irSnapshot` has withdrawn its own 473/473 on exactly this basis and is right to.
2. **`correctness`'s batch-SSA seed derivation (`2294cfd`) is not on my base.** When it merges, per-trajectory seeds become a SplitMix64 of `(base, traj)` rather than `base + b`, so **any** batch-SSA byte-identity check spanning that merge shows a diff that is not a regression — including mine. The identity claim above is scoped to this branch against `origin/main` as it stands. `correctness` asked me to mirror their formula at `BatchSsa.cpp:235` and `:365`; I have not, because putting a value-changing edit in a PR whose central claim is that it changes no values would make the identity gate unfalsifiable.
3. **`tests/python/conftest.py` on main (PR #53) strips `sys.path` entries containing `BioNetGen` except the worktree's own `build/cpp`.** My worktree has its own build, so the guard is satisfied here and `run_under_test.py` is belt-and-braces. In a worktree without one the guard raises loudly rather than substituting another tree — `sciStochastic`'s table is the evidence for that arm, not mine.