Repository navigation
perf(engine): seeded SSA throughput 1.44x-2.10x with byte-identical trajectories (provisional) - #44
Conversation
Constructs a seeded synthetic ring network (3n reactions), caps SSA at an exact event count via max_sim_steps, subtracts a same-shape 2-event probe run for parse/network-generation overhead, and reports child-CPU-time events/sec (min of >=5 interleaved reps) so shared-host preemption does not inflate the number.
integrateSSA recomputed every propensity on every fired event. Build a species -> reactions index over reactantIndices (the only state a propensity reads) once per run and recompute just the reactions sharing a species with the fired reaction's reactants/products, then rebuild totalPropensity with the original in-order summation. Every propensity value, the total, and the RNG draw sequence are bit-identical to the full recompute, so fixed-seed trajectories do not change. Measured on the synthetic benchmark (150 species / 450 reactions, 1e6 events, 7 interleaved reps): min 1,679,112 -> 2,100,849 events/s (1.251x), median 1,793,018 -> 2,121,215 (1.183x). Baseline trajectory hashes (gdat/cdat/net over five seeded models, 4M-event synthetic included) are byte-identical before and after.
min(events/sec) was the slowest rep, which shared-host noise hits unevenly; the fastest rep is the robust bound because co-tenants only add time. Renders the primary statistic best-of-interleaved-reps and documents the script in benchmarks/README.md.
The per-event selection scan re-accumulated propensities until it crossed the target. Build the running total into a prefix array while computing totalPropensity — the same additions in the same order, so the total and the first-crossing index are bit-identical — and select with a binary search when every rate constant is non-negative (the prefix is then non-decreasing). A negative or non-finite rate constant keeps a linear scan over the same prefix values, which reproduces the original cumulative scan exactly for any sign pattern. Measured with benchmarks/bench_engine_ssa.py (child CPU time, best of 15 interleaved reps): 150 species / 450 reactions 1,903,689 -> 2,739,103 events/s vs the pre-change binary in one session (best-of; 9 reps); vs the dependency-only step 2,247,231 -> 2,739,103. Network-size sweep vs the original engine: 45 reactions 1.037x, 150 reactions 1.219x, 450 reactions 1.439x (all best-of interleaved reps). Seeded .gdat/.cdat bytes are unchanged on five models plus a negative-rate model (linear-scan fallback) and a zero-rate model (prefix plateau).
Runs fixed-seed SSA (4M-event synthetic ring plus isomerization, gene_expr_simple, gene_expr_func, michment) with a given binary and hashes every .gdat/.cdat/.net artifact. Perf changes in this lane are wins only when the SHA256SUMS of two binaries diff empty; record the identity result alongside the throughput numbers. Also note the SSA throughput work in CHANGELOG.md.
Constructs a seeded rule model (binding, catalysis, turnover) and parses NFsim's own 'You just simulated N reactions in Ts' loop timing for the primary number: simulator-loop reactions/sec, best of >=5 interleaved reps, with the reaction count reported as a fixed-seed sanity check.
…write depOffset[idx + 1] is a new write that the pre-change code did not have. Reactant indices are network species indices and the existing ODE/SSA paths already read y[idx] under that invariant; assert it at the point where the index first crosses this write so a violation fails in debug builds instead of corrupting the offset table.
…y write Cite perfOracle's audit (2,711 generated .net files, 792,960 reaction lines, max 1-based reactant index == nSpecies, zero out-of-range references) next to the assert so the in-bounds argument has its provenance in the diff instead of only in the PR thread.
Independent re-measurement supersedes the merged headline
The merged headline of Measurement context: R=48 runnable, loadavg band 117-147 (coarse band only), three co-tenants declared including one of the verifier's own correctness sweeps, interleaved in a single session. Base cv 1.34-1.35%, changed cv 0.80-0.90%. Identity: 22 of 26 byte-identical, 2 inconclusive, 1 excluded, 1 retiredByte-identical on That set is the change's risk surface, not just its happy path:
Trajectory physics independently confirmedClosed Open review item: closedThe new out-of-bounds write at Provenance caveat on the marginal figuresThe intermediate per-change marginals (1.180x / 1.219x) were measured from binaries built via Comment posted by @RuleWorld. |
CORRECTION to the comment above
The published table stands unchanged. Added alongside it: External re-measurement ( Attributability: the pair differs in one translation unit, pinned by blob SHA ( Identity: 22/26 byte-identical on Independent absolute check: closed Limit on the identity coverage, stated by Open review item closed: the new out-of-bounds write at Provenance caveat on the marginals: the intermediate per-change figures (1.180x / 1.219x) came from binaries built via Second comment posted by @RuleWorld. |
1 participant
What changed
Two measured performance changes to the Gillespie SSA engine (
cpp/engine/OdeIntegrator.cpp,integrateSSAonly), plus committed benchmark/identity tooling. No seeded trajectory byte changes.8fea20c— dependency-indexed propensity refresh. Previously every fired event recomputed all propensities.computePropensityreads onlyrxn.rateConstant,rxn.isTotalRate,rxn.reactantIndices, andy[]at those indices (independently verified by orchPerf at line 2320). A species→reactions index built once per run lets each event recompute only reactions reading a species whose amount changed;totalPropensityis then rebuilt with the original in-order summation, so every value keeps its original bits.5ec91d6— prefix-sum reaction selection. The running total is stored into a prefix array while summing (same additions, same order → same bits) and selection becomes a binary search for the first index withprefix >= target, valid because the prefix is non-decreasing when every rate constant is>= 0. Negative/NaN rate constants (reachable: a model withk=-1.0runs through network generation into SSA) fall back to a linear scan over the same prefix values, which reproduces the original cumulative scan exactly for any sign pattern. Zero-rate plateaus verified separately.bfd12ad+d213e89—assert(idx < nSpecies_)and a provenance comment at the newdepOffset[idx + 1]write (a write the original code did not have; the bound is the same species-index invariant the existing ODE/SSAy[idx]reads rely on, plus perfOracle's corpus audit: 2,711 generated.netfiles / 792,960 reaction lines, max 1-based reactant index == nSpecies, zero references outside[1, nSpecies]).benchmarks/bench_engine_ssa.py(primary metric),benchmarks/check_engine_ssa_identity.sh(trajectory gate),benchmarks/bench_nfsim.py(NFsim cycle, committed but no perf claim — see below),benchmarks/README.md,CHANGELOG.md.Baseline vs after (PROVISIONAL pending perfOracle's independent matched-pair re-measurement)
Primary metric: SSA network events/sec on a constructed 150-species / 450-reaction seeded ring network, exactly 1,000,000 events per rep (
max_sim_steps), probe-subtracted child CPU time, best of 9 interleaved A/B/A reps per session, binaries alternating order per round.Independent re-measurement (perfOracle; harness
verify/harness-on-main, PR #78). The figures below are THEIRS, not this PR author's. Matched binary pair differing in one TU, pinned by blob SHA (57e504a= origin/main 6889fba,07baa1a= aab2c0f;git diff --stat= 1 file, +9/-93). Interleaved A/B, 9 reps, order flipped every round, 2,000,000-event ring network at species=150 / R=450, seed 4242, child-CPU metric:Arms do not overlap (base max 910k < after min 1.79M events/sec); base cv 1.34-1.35%, after cv 0.80-0.90%; peak child RSS identical at 21,463,040 kB both arms.
This does NOT replace the scaling table above. Those rows are measurements at different operating points and 1.44x-2.10x is the honest spread across THOSE sessions. The figure here is a single point at R=450 and says nothing at the R=45 or R=1350 rows. Both are measurements; neither supersedes the other.
Identity gate, same pair: 22/26 models byte-identical on .gdat/.cdat/.net SHA-256, with the risk-surface cases named individually (negative rate constant forcing the non-monotone fallback; zero-rate plateaus; identical reactants; a species on both sides of the fired reaction; larger reaction counts; saturating rates where propensity reads no species at all). Two models INCONCLUSIVE - three identical runs of the BASELINE binary give different hashes, so no comparison can adjudicate them. One excluded (exceeded a 240s cap, no artifacts in either arm). Plus an absolute check a hash cannot provide: closed A<->B<->C chain, seeds 500/300/200 (sum 1000), 300,000 events, A+B+C drift exactly 0.0 in both arms.
Caveat on the half-life: the identity gate is a test of the trajectory, not the engine. If a per-trajectory seed derivation changes (as PR #69 did), these hashes become VACUOUS rather than false. The conservation check does not share that bound.
Context: R=48 runnable, loadavg band 117-147, three declared co-tenants including my own identity sweep. Nothing load-bearing here is a configure or ctest result.
Marginal contribution per change, session A: baseline→change1 1.180× (best), change1→change1+2 1.219× (best) — each change measured in its own interleaved session against its immediate predecessor before the next change was made (change1: n=7, best 1,679,112→2,100,849 in its session; the stash-rebuilt change1-only binary makes session A's three-way possible).
PROVENANCE-QUESTIONED (perfOracle). The change1-only binary behind the 1.180x/1.219x marginals was produced with
git stash push/git stash pop.refs/stashis shared across all ~30 worktrees in this repo, so that pair can pull foreign content and - critically - a stash pop that silently reverts a neighbouring change passes every behavioural gate while the tree is half-reverted. The marginals are therefore not established. Rebuild the change1-only binary withgit checkout <sha> -- cpp/engine/OdeIntegrator.cppand re-measure before citing them. (perfEngine had already proposed this remediation; the figures are carried here unchanged pending it.)Network-size sweep (interleaved baseline vs after, n=15, context NOT recorded):
No network size regressed. (A 1350-reaction point was measured for the after binary only — 749,089 events/s — and is not a paired comparison, so it is not claimed as one.)
Measurement context (per host rule)
{103.85 85.55 59.35}→{111.91 89.84 62.12}(1-minute, coarse band only; loadavg was proven unresponsive to a known input on this host, so no decimal precision is claimed); runnable processes R≈64-70; exact concurrent timing-sensitive process list:perfOracle'sidentity_check.py+bng_cpp_base model.bngl, shared-treebng_cpp d1_tagcomp2/e7_gen×3/heise_gen,/tmp/bng_cpp_before model.bngl,irSnapshot's four concurrentclang++jobs (-j4build), plus this benchmark. A/B interleaved in a single session (round-robin, order alternating).Peak RSS (
/usr/bin/time -l, reliable)-j4build of this tree: measured by perfOracle's equivalent configuration at 442 MiB peak (mine re-used kept dependency sources; no fresh reconfigure during the contended window).Correctness: seeded trajectories are byte-identical
benchmarks/check_engine_ssa_identity.shruns fixed-seed SSA (a 4,000,000-event synthetic ring plusisomerization,gene_expr_simple,gene_expr_func,michment) with a given binary and SHA-256s every.gdat/.cdat/.net.Edge models exercising the new code paths, same procedure,
diffempty:Caveat for future readers: the identity corpus also hashes
.netfiles fromgenerate_network. If a.nethash ever differs between two binaries, check the known generate-network determinism/ASLR defect first (per Main's hold on tlbr-class models) before blaming the engine change; none of these five models is tlbr.Gates (verbatim summary lines, post-change)
The Python suite and the batch-SSA CPU statistical gate (conservation identities exact, fixed-seed batch reproducibility bit-identical, analytic Binomial(20, 1/6) chi-square pass) were run against this worktree's Python and this worktree's extension built from these commits — not the shared editable install (the meta-path hazard Main flagged).
Profile evidence (before the first change)
The bottleneck was entirely inside
integrateSSA— in-lane. Scaling analysis before the change: cost/reaction/event ≈ 1.42 ns (recompute) + sum + scan; after both changes the residual is the bit-identity-mandated in-order sum (≈0.98 ns/reaction/event measured at 4 network sizes: 45/150/450/1350 → 18.1M/6.44M/2.13M/0.75M events/s on the change1 binary).Rejected ideas (with numbers/analysis)
logwas only 16/1832 samples (~1%), so RNG+log was not the prize anyway.totalPropensitysummation (tree/SIMD reduction, skipping the O(R) pass) — rejected: any order change alters FP bits → differenttau→ different trajectory. The remaining O(R) sequential sum is the algorithmic floor given bit-identity; it is now the dominant residual cost..netartifacts (golden-network outputs) → a behavior change, not a perf change.r2 == 0case (≈2⁻⁵³) and needs sign-plateau reasoning for marginal gain; kept the exact accumulation instead.benchmarks/bench_nfsim.pyis committed and its baseline measured (~0.79–0.83M reactions/s loop-CPU, 5 reps, but context NOT recorded), but the benchmark model turned out to be dead:.gdatshowsRp1=Rp2=M=0, and the minimal reproducer shows the cause is upstream of NFsim —R(p1~u).K(a) -> R(p1~p).K(a)generates zero reactions ingenerate_network(min.netbegin reactionsblock is empty), so those rules never fire. A benchmark whose rules never fire measures nothing, so the NFsim cycle stops here as a measured no-result, reported honestly rather than run against a dead model. The dead-rule question (model bug vs missing diagnostic for a zero-reaction rule) is left as a separate finding; per-reaction-class diagnostics are out of this lane's scope.Binary provenance
All numbers come from binaries built in this worktree from base
6889fba(which includes75b22a7, the NFsim seed site-state fix): pristine baseline = pre-edit copy ofbuild/cpp/bng_cpp(sha25660132cf8…), change1-only = stash-rebuilt intermediate (sha256358f242f…), after = current build (sha25652a9d0ca…). No shared main-worktree binary was used (memWatch bounded the shared binary's staleness to75b22a7, NFsim-only; this SSA path does not touch it).Verification commands
cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTS=ON \ -DBUILD_PYTHON_BINDINGS=OFF -DBUILD_CLI=ON cmake --build build -j4 ctest --test-dir build -j4 --output-on-failure # 100% tests passed out of 461 ./benchmarks/check_engine_ssa_identity.sh <binary> <outdir> # diff the two SHA256SUMS python3 benchmarks/bench_engine_ssa.py --reps 9 \ --variant baseline=<pristine> --variant change1=<mid> --variant final=<build>/cpp/bng_cpp PYTHONPATH=python:build_py/cpp python3 <runner that strips editable finders and prints UNDER TEST> python3 tests/test_batch_ssa_statistical_parity.py --mode cpuNot rebased onto PR #30 (cpp/CMakeLists.txt) per instruction; if it merges before this PR does, rebuild and re-run both benchmark sessions and update the numbers.
The NFsim benchmark script (
bench_nfsim.py) is committed for reproducibility but carries no claim in this PR. SSA-per-cycle work continues only if the NFsim cycle reaches a live model; this PR is the SSA cycle.Evidence appendix — exact reproduction commands and observed output
Per the reproducible-evidence rule: every row above can be re-run and returns what is
quoted here. Commands were run from the worktree root of
swarm/autobenchat base6889fba binaries unless noted.
Rows 1–9 (C++ shares):
sampletop-of-stackProbe 1 (netgen loop,
/tmp/probe_netgen.py):Command and observed output:
Probe 2 (SSA loop,
/tmp/probe_ssa.py) — same skeleton, body:Observed:
ssa iterations: 206; total top-of-stack samples 41,271;bng::engine::OdeIntegrator::integrateSSA21,902 (53.1%),log4,901 (11.9%),__psynch_cvwait3,771 +__psynch_mutexwait1,443 (12.6%), allocator 3,252 (7.9%),unsymbolicated in-binary 3,957 (9.6%).
Bucketing script (exact, from the session — applies to both sample files):
Rows 10–11 (Python import):
cProfile, exact command and outputObserved output — the rows the two claims rest on:
Row 10 =
get_git_changeset->communicate/poll(a git subprocess spawned duringimport): 0.034 s of 0.124 s module-import cumulative = 27% (34 ms of 124 ms).
Row 11 = numpy
<module>cumtime 0.030 s of 0.129 s total ≈ 23% (the ~17–23% bandaccounts for run-to-run spread of the two numpy module rows).
Row 12 (validation harness): exact command and output
Observed output — the rows the claim rests on:
0.397 / 0.538 = 74% in
subprocess.runwaiting on the fourbng_cppchild runs;Python-side cumulative above the children ≈ 0.538 − 0.429 = 0.109 s,
validate.pyself time ~10 ms.
Row 13 (CLI cold start): exact command and output
Observed output:
The "<2% of a netgen CLI run" claim = 8.4 ms / 0.5170 s (v1 baseline
netgenmin)= 1.6%.
Note on single-source rows: rows 1–9 rest on
samplefiles produced in this session(
/tmp/netgen_sample.txt,/tmp/ssa_sample.txt); the probes and bucketing script aboverecreate them on any host with
sample(shares will differ by machine, the commands willnot). Corroboration noted per row where a peer independently measured the same hotspot
(swarmMutate, perfNetGen on Ullmann; swarmSerial on text emission; swarmMemory on
allocator counts).