Repository navigation
Benchmark qualification and host-wide timing lock - #92
Merged
Merged
Conversation
Caps how many timing-sensitive benchmarks run at once on one host, across every worktree. Written during a session in which measured contention came from concurrent BENCHMARKS rather than from builds: 1-minute loadavg ranged 23-131 while peak resident memory never moved and no process exceeded its memory cap. The discriminator is whether a run's RESULT depends on machine timing, not whether it consumes CPU. An A/B or a benchmark harness needs a slot; a correctness run, a single-model invocation, a ctest or a determinism census does not. Counting allocations, instructions or events is deterministic and cannot move under load, and that exemption must be declared rather than quietly folded into a speedup claim. Uses fcntl.flock rather than a directory lock, because on a host where agents are routinely culled a stale counter lock wedges the protocol permanently while the kernel releases flock on process death for any reason. On release each holder receives a co-tenant ADVISORY in two classes. NEW is the obvious one - processes started during the hold. PERSIST covers processes alive at release that were already alive at acquire, which a start-time diff misses entirely: a long-running correctness sweep that began before the A/B took its slot never starts during the window yet loads the machine throughout it. The advisory judges nothing and blocks on nothing; it exists so that an undeclared co-tenant becomes a declared one, because a reader who was not told cannot weigh the number. Tests: 21 + 9 assertions plus a 6-way race, all using a unique lock directory per invocation. That isolation is load-bearing rather than housekeeping: every assertion concerns exact slot occupancy, so a shared path makes concurrent runs fight and report slot leaks that do not exist. Verified with 16 simultaneous full-suite instances on a loaded host, 16/16 clean, where 8 concurrent instances previously produced 8 failures. race_test.py asserts the invariant that matters - never more than MAX held at any single instant - not 'only MAX racers ever win', which is not an invariant, since a holder that finishes legitimately frees its slot. No production code touched. Adds tools/benchlock/ only.
Reject SSA runs unless they produce fresh nonempty network output and a complete numeric time grid. Persist per-case validity and action-derived sampling expectations, then have the comparator verify case coverage, required files, hashes, and grid contents. Add regressions for missing, empty, truncated, and wrong-endpoint outputs and mismatched coverage.
Replace the toolchain-sensitive comparison of two small batch-mean samples with a direct check of the 32-bit trajectory RNG keys. The old additive mapping reuses 511 of 512 keys for adjacent base seeds. Centralize the existing low-32-bit and zero-to-one conversion so the test covers the same key construction used by single-worker, multi-worker, and integrated batch SSA. The SplitMix mapping and resulting seed streams are unchanged.
…g3-20261002-performance
akutuva21
marked this pull request as ready for review
October 4, 2026 01:14
This was referenced Oct 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
This branch qualifies the remaining performance and benchmark-measurement work against the current corrected BNG3 source. It adds the host-wide benchmark lock and robustness tests, reproducible network-generation and seeded-SSA harnesses, exact-source evidence for PR44/PR58, and the PR67 documentation source-anchor correction. It does not change production simulator code.
Findings
9601746f8884ed19ab2acea49fe87fc4660ace46, 10 fixtures passed andfceri_ji_gen.bnglwas explicitly unsupported by that oracle because of a dangling numbered bond; there were 0 mismatches among supported cases. The old TLBR nondeterminism exemption was removed after main's determinism fix.3471714032bd880b03d776b922a02d16c4317ebb. Independent seeded identity checks passed 8/8 cases. On one synthetic ring fixture over five pairs, raw median throughput ratio was 1.45294x (probe-corrected estimate 1.48524x), with essentially unchanged peak RSS. This is one-fixture evidence, not a general BNG3 speed claim.michmentremains outside this measurement pending the parity lane's correction.Local verification
bench/evidence/.Hosted checks are reported below against this PR's exact head SHA. This is a draft for review; no merge, close, or release is requested.
Exact-head hosted checks
All runs are terminal for head
cd81d5ed5f417b268aae33339156410eb0e0cc01against basec345f3aa635aac0048681b79c6a4ff1e75a298f7:test_mm_free_substrate_root.py,test_observable_type_support.py), the Windowsunistd.hinclude intest_gdat_output_format.cpp, and all 18 Python 3.9–3.14 / Ubuntu–macOS–Windows jobs failing to import_bionetgen_cppduring collection oftest_thermodynamic_parse_path.py. No files in those failure paths are changed in this branch, and the exact-base logs show the same failure signatures. No patch-introduced CI failure was identified.nvccis absent, so no CUDA backend was compiled or exercised.tests/python/conftest.pybecausebionetgen._bionetgen_cppis unavailable after resolving the source tree; the exact-base parity run 36812376327 has the same failure. Scheduled NFsim historical validation was skipped.The PR remains a draft for review. No merge, close, or release is requested.