This repository was archived by the owner on Nov 21, 2026. It is now read-only.
Repository navigation
Benchmarks: 2026-11 is the only instrument on the measure side - #429
Merged
Merged
Conversation
This was referenced Oct 21, 2026
Merged
tae898
force-pushed
the
instrument-2611-only
branch
from
October 21, 2026 15:40
3c1d52e to
8107052
Compare
BENCH_INSTRUMENT defaults to 2026-11 and any other value is refused. The OFF path of every switch the 2026-11 instrument turned on is deleted (sparse warm-up, CPU idle hold, dense id index, embedded -Xms, large-result Arrow path, served client sensitivity), LeanSession2 and the requests client path go with their tests, and the runner forwards its own BENCH_INSTRUMENT to every cell. CAMPAIGN, PROTOCOL, FAIRNESS, COMPARATORS, PAGE-SPEC and the bindings benchmark pages describe 2026-11 as the method. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nt gate is moot under 2026-11 only) (#428) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tae898
force-pushed
the
instrument-2611-only
branch
from
October 21, 2026 16:16
8107052 to
5286481
Compare
This was referenced Oct 21, 2026
Closed
Merged
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #428
Measure side of DECISIONS #196 ("let's go for November directly without October"): 2026-11 is the only instrument the harness can produce. This PR plus the one-stage-generator PR and the publish-side PR finish the remaining scope of the umbrella #410; I am not closing #410 here.
BENCH_INSTRUMENTdefaults to2026-11; any other value exits with a message naming the retired instruments. The stamp on every row and theINSTRUMENTS/instrument_at_leastselector stay so a later instrument can be added (instrument_at_leastkeeps callers inmemory_probe.pyanddense_beam.py, and the generated 2026-11 stages grep for it).NEXT_INSTRUMENT,next_instrument_default(),large_result_path(),embedded_pins_xms/embedded_xms/embedded_jvm_argsandclient_sensitivity_enabled()are deleted. The runner now forwards its ownBENCH_INSTRUMENTto every cell, so the container cannot disagree with the launcher.Deleted OFF paths: unset sparse warm-up and the 2026-10 seeded-sample draw (
warmup_indices), no CPU idle hold, no dense id index (id_index_enabled,BENCH_DENSE_ID_INDEX), pinned embedded-Xmsat the nine call sites, the rows read of large l4 results (BENCH_LARGE_RESULT_PATH; the_plainsensitivity timing stays), the client sensitivity switch.LeanSession2, therequestsclient inlean_http.Session()andtest_lean_http2.pyare gone;BENCH_ARCADEDB_HTTP_CLIENTnow takesofficialorleanonly,leanbeing what the runner sets for the twin cell, and it is no longer forwarded from the launcher.Env overrides kept:
BENCH_SPARSE_WARMUP=N(count) andBENCH_CPU_DMA_LATENCY_US=N(target), because the PROTOCOL.md section 7 table documents them with explicit values;offand0no longer turn the method off. PQ arms (#419) keep their ownBENCH_DENSE_PQ=1gate; its instrument half becomes trivially true.Tests:
_under_2026_10tests deleted,_under_2026_11ones are plain default tests (test_next_instrument_switches.pyrenamedtest_instrument_method.py),test_instrument_selector.pyrewritten (unset is 2026-11, the retired values are refused). Docs: CAMPAIGN rows 81, 82, 84, 89 and the two switch paragraphs become one "What the 2026-11 instrument measures" paragraph, PROTOCOL override and heap rows lose the "under BENCH_INSTRUMENT=2026-11" qualifiers and the lean2 row, FAIRNESS, COMPARATORS, PAGE-SPEC, and the bindings benchmark pages (a new "What the Instrument Does" section in protocol.md) describe 2026-11 as the method.Checked locally (no Docker, no cells): the benchmark suite on origin/main with the untracked result dirs copied in is 1 failed, 1007 passed, 24 skipped; after, 2 failed, 938 passed, 24 skipped (the count is lower by the deleted lean2 and
_under_2026_10tests). The 2 failures:test_dense_ef_sweep.py::test_export_adds_the_matched_columns_only_for_2026_11fails the same way on origin/main, andtest_stage_generator_2611.py::test_check_refuses_another_instrument_in_the_environmentexpects the word "mix" in the message that bench_common's refusal now prints first (that file is rewritten by the one-stage-generator PR).uvx pre-commit run --filesand the strict mkdocs build the waydocs-build.ymlruns it pass.Merge order: the one-stage-generator PR, this PR, then the publish-side PR. The publish-side tools use
BENCH_INSTRUMENT=2026-10as their own selection and importbench_common: checked,export_web,equivalence_checkandresource_checkexit at import under that value, anddegenerate_checkandfairness_checkimport it lazily, so they fail when they reach it. The publish-side PR has to stop setting the variable to2026-10; until it merges, the 2026-10 page cannot be re-exported.🤖 Generated with Claude Code