Skip to content
This repository was archived by the owner on Nov 21, 2026. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 3 additions & 25 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -641,14 +641,9 @@ benchmarks/experiments/results/**
!benchmarks/experiments/results/runs_paper.csv
!benchmarks/experiments/results/sparse_cliff.jsonl
!benchmarks/experiments/results/web_benchmarks.json
# The October campaign's freeze and payload (DECISIONS #84). Tracked for the
# same reason the September pair is: the page serves the payload and the
# freeze is the rows it was built from, and neither can be re-derived later
# from an append log that keeps growing. Named here because the allowlist
# above means `git add` REFUSES an unnamed path, which is where a landing
# would stop: the site pushed, the frozen rows in no commit.
!benchmarks/experiments/results/runs_paper_oct.csv
!benchmarks/experiments/results/web_benchmarks_next.json
# The freeze, the payload and the generated set are tracked: they are the rows the page was built from and what it serves, and
# neither can be re-derived from an append log that keeps growing. The allowlist makes `git add` refuse an unnamed path, so a
# landing that skipped them would push the site with the frozen rows in no commit.
!benchmarks/experiments/results/e3_q17/
!benchmarks/experiments/results/e3_q17/*.json
!benchmarks/experiments/results/ingest_ab/
Expand All @@ -661,23 +656,6 @@ benchmarks/experiments/results/**
benchmarks/experiments/results/generated/figures/*.pdf
benchmarks/experiments/results/generated/results_manifest.json
benchmarks/experiments/results/generated/benchmark-results-*.tar.gz
# The October campaign's generated artifacts (make_paper_tables.GENERATED_NAME).
# One directory per campaign, for the reason runs_paper_oct.csv is its own
# file: results/generated is SEPTEMBER'S PUBLISHED SET, page_check pins the
# live page against it, and an October freeze used to rewrite it in place.
# Tracked, and named here because under this allowlist an unnamed path is
# ignored and `git add` on it commits nothing silently -- which is where a
# preview landing would stop, the site pushed and the tables in no commit.
!benchmarks/experiments/results/generated_oct/
!benchmarks/experiments/results/generated_oct/**
benchmarks/experiments/results/generated_oct/figures/*.pdf
benchmarks/experiments/results/generated_oct/results_manifest.json
benchmarks/experiments/results/generated_oct/benchmark-results-*.tar.gz
# generated_skeleton is deliberately absent: BENCH_SKELETON=1 is a laptop
# placeholder run and has nothing to put under version control. It is ignored
# by the rule above, which is the whole point of it being its own directory --
# until 2026-09-19 a skeleton run left September's GATE_STATUS.txt and
# withheld_recall.json dirty, and a dirty tree on the bench host is an outage.

# Backward-compatibility fixtures: databases written by older releases (#9265)
!engine/src/test/resources/compat/*.zip
Expand Down
69 changes: 17 additions & 52 deletions benchmarks/experiments/CAMPAIGN.md

Large diffs are not rendered by default.

84 changes: 39 additions & 45 deletions benchmarks/experiments/PAGE-SPEC.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion benchmarks/experiments/PROTOCOL.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ The admission filter is `make_paper_tables.load_canonical()`. Every test below r
- Dedupe key is `(lane, scale, n_docs, workload, backend, gav is not False, rep, durability_class)`, latest `ts_utc`, never `run_id`. Without the `gav` term the no-view ablation overwrites T3, and without the class term the strict arm of a write cell overwrites its relaxed twin (DECISIONS #90; a row with no class counts as relaxed, which is what it was). **GATE** `load_canonical`
- One released engine line per table. Delete preference cascades, never reorder them. **GATE** T4 refusal in `make_paper_tables` and five of f8's six bars in `make_paper_figures`; **REMEMBERED** for f8's dense bar, whose `_dense_multipass` source reads no version field, and for every other table (F5)
- Lane scripts publish; bespoke drivers investigate. Both known protocol violations were bespoke-driver rows promoted to cells, and T5's dense half is still an overlay from `dense_multipass_driver.py`, run through `runner.py --driver`. **GATE** `fairness_check.report_producers` (F6b) for `runs.jsonl` lanes, which sees no overlay record; **REMEMBERED** for every overlay-fed cell
- Every cell is N=5, or its shortfall is disclosed in `EXPECTED` and in the caption, and the disclosure is deleted when the shortfall is fixed. **N=5 is reached in two passes on the 26.10.1 measurement (DECISIONS #164): reps 1 to 3 for every arm first, reps 4 and 5 in a later pass, one whole lane at a time so every engine of a lane is topped up within days. A table landed between the passes states n=3 in its caption and in `EXPECTED`, and the later landing deletes both. Medians of three carry no rank test (the smallest two-sided Mann-Whitney p at 3 versus 3 is 0.1), so a claim under about 1.3x waits for the top-up, and the two batches are compared before they are pooled.** **GATE** `provenance_check.caption_n` over the generated `.tex` tables, skipped on a lane-scoped landing (`BENCH_ONLY_LANES`) and on a skeleton; **REMEMBERED** for T5's dense half and for the deletion half, which `if ns == {5}: continue` short-circuits
- Every cell is N=5, or its shortfall is disclosed in `EXPECTED` and in the caption, and the disclosure is deleted when the shortfall is fixed. **N=5 is reached in two passes on the 26.10.1 measurement (DECISIONS #164): reps 1 to 3 for every arm first, reps 4 and 5 in a later pass, one whole lane at a time so every engine of a lane is topped up within days. A table landed between the passes states n=3 in its caption and in `EXPECTED`, and the later landing deletes both. Medians of three carry no rank test (the smallest two-sided Mann-Whitney p at 3 versus 3 is 0.1), so a claim under about 1.3x waits for the top-up, and the two batches are compared before they are pooled.** **GATE** `provenance_check.caption_n` over the generated `.tex` tables, skipped on a lane-scoped landing (`BENCH_ONLY_LANES`); **REMEMBERED** for T5's dense half and for the deletion half, which `if ns == {5}: continue` short-circuits
- Never publish a scale where comparators have rows and we do not; print what was withheld and why at export time. **GATE** withheld-scale guard in `export_web`
- Every artifact the page cites lives in this repository, and the superseded run is not left where it can be compared against. **GATE (report only)** `provenance_check.FEEDS`: an empty feed prints "(no result files)" and counts nothing. **GATE**: every pinned reader refuses when `<name>_<pin>` is missing
- Every overlay directory and top-level result file feeding a table records run conditions and a real version; "feeds a table" means "is opened by `make_paper_tables`". **GATE** `run_conditions`, `lane_files` and `_is_real_version` for the two lane files; **REMEMBERED** for the overlay directories T4 and T5 print, whose audit accepts any non-empty string
Expand Down
Loading
Loading