Skip to content
This repository was archived by the owner on Nov 21, 2026. It is now read-only.

Benchmarks: 2026-11 is the only instrument on the measure side - #429

Merged
tae898 merged 2 commits into
mainfrom
instrument-2611-only
Oct 21, 2026
Merged

tae898 merged 2 commits into
mainfrom
instrument-2611-only

Conversation

@tae898

@tae898 tae898 commented Oct 21, 2026 •

Copy link
Copy Markdown

Closes #428

Measure side of DECISIONS #196 ("let's go for November directly without October"): 2026-11 is the only instrument the harness can produce. This PR plus the one-stage-generator PR and the publish-side PR finish the remaining scope of the umbrella #410; I am not closing #410 here.

BENCH_INSTRUMENT defaults to 2026-11; any other value exits with a message naming the retired instruments. The stamp on every row and the INSTRUMENTS / instrument_at_least selector stay so a later instrument can be added (instrument_at_least keeps callers in memory_probe.py and dense_beam.py, and the generated 2026-11 stages grep for it). NEXT_INSTRUMENT, next_instrument_default(), large_result_path(), embedded_pins_xms/embedded_xms/embedded_jvm_args and client_sensitivity_enabled() are deleted. The runner now forwards its own BENCH_INSTRUMENT to every cell, so the container cannot disagree with the launcher.

Deleted OFF paths: unset sparse warm-up and the 2026-10 seeded-sample draw (warmup_indices), no CPU idle hold, no dense id index (id_index_enabled, BENCH_DENSE_ID_INDEX), pinned embedded -Xms at the nine call sites, the rows read of large l4 results (BENCH_LARGE_RESULT_PATH; the _plain sensitivity timing stays), the client sensitivity switch. LeanSession2, the requests client in lean_http.Session() and test_lean_http2.py are gone; BENCH_ARCADEDB_HTTP_CLIENT now takes official or lean only, lean being what the runner sets for the twin cell, and it is no longer forwarded from the launcher.

Env overrides kept: BENCH_SPARSE_WARMUP=N (count) and BENCH_CPU_DMA_LATENCY_US=N (target), because the PROTOCOL.md section 7 table documents them with explicit values; off and 0 no longer turn the method off. PQ arms (#419) keep their own BENCH_DENSE_PQ=1 gate; its instrument half becomes trivially true.

Tests: _under_2026_10 tests deleted, _under_2026_11 ones are plain default tests (test_next_instrument_switches.py renamed test_instrument_method.py), test_instrument_selector.py rewritten (unset is 2026-11, the retired values are refused). Docs: CAMPAIGN rows 81, 82, 84, 89 and the two switch paragraphs become one "What the 2026-11 instrument measures" paragraph, PROTOCOL override and heap rows lose the "under BENCH_INSTRUMENT=2026-11" qualifiers and the lean2 row, FAIRNESS, COMPARATORS, PAGE-SPEC, and the bindings benchmark pages (a new "What the Instrument Does" section in protocol.md) describe 2026-11 as the method.

Checked locally (no Docker, no cells): the benchmark suite on origin/main with the untracked result dirs copied in is 1 failed, 1007 passed, 24 skipped; after, 2 failed, 938 passed, 24 skipped (the count is lower by the deleted lean2 and _under_2026_10 tests). The 2 failures: test_dense_ef_sweep.py::test_export_adds_the_matched_columns_only_for_2026_11 fails the same way on origin/main, and test_stage_generator_2611.py::test_check_refuses_another_instrument_in_the_environment expects the word "mix" in the message that bench_common's refusal now prints first (that file is rewritten by the one-stage-generator PR). uvx pre-commit run --files and the strict mkdocs build the way docs-build.yml runs it pass.

Merge order: the one-stage-generator PR, this PR, then the publish-side PR. The publish-side tools use BENCH_INSTRUMENT=2026-10 as their own selection and import bench_common: checked, export_web, equivalence_check and resource_check exit at import under that value, and degenerate_check and fairness_check import it lazily, so they fail when they reach it. The publish-side PR has to stop setting the variable to 2026-10; until it merges, the 2026-10 page cannot be re-exported.

🤖 Generated with Claude Code

tae898 and others added 2 commits October 22, 2026 01:15
BENCH_INSTRUMENT defaults to 2026-11 and any other value is refused. The OFF
path of every switch the 2026-11 instrument turned on is deleted (sparse
warm-up, CPU idle hold, dense id index, embedded -Xms, large-result Arrow
path, served client sensitivity), LeanSession2 and the requests client path
go with their tests, and the runner forwards its own BENCH_INSTRUMENT to every
cell. CAMPAIGN, PROTOCOL, FAIRNESS, COMPARATORS, PAGE-SPEC and the bindings
benchmark pages describe 2026-11 as the method.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nt gate is moot under 2026-11 only) (#428)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Benchmarks: 2026-11 is the only instrument on the measure side

1 participant