Skip to content
This repository was archived by the owner on Nov 21, 2026. It is now read-only.

Benchmarks: every publish gate passes on the 2026-11 fixture payload, with a test that runs them (#433) - #434

Merged
tae898 merged 1 commit into
mainfrom
publish-gates-2611
Oct 21, 2026
Merged

tae898 merged 1 commit into
mainfrom
publish-gates-2611

Conversation

@tae898

@tae898 tae898 commented Oct 21, 2026

Copy link
Copy Markdown

Closes #433

Every publish gate now runs on the 2026-11 fixture payload and passes, and test_publish_gates_2611.py runs them in the suite (make_paper_tables with --strict, equivalence_check, resource_check, override_registry, degenerate_check, fairness_check, page_check, version_consistency_check, version_pin_check, provenance_check, claims_check --regen, make_paper_figures). Two negative tests keep the roster and the equivalence gate alive. Fixture rows change only through build_fixture.realistic_2611 (also python build_fixture.py --patch, since the real results are archived) and the committed runs.jsonl.gz plus the ArangoDB overlay files it touches; the builder's docstring names each patch and its reason.

Gate, failure, cause, fix (before counts are from the fixture as merged in #432):

  • make_paper_tables internal fairness_check (307 failures): the copied September and October rows lack the override stamps a 2026-11 row carries (index_kinds, arcadedb_add_hierarchy, server_query_max_heap_elements, server_http_body_max_bytes, duckdb_threads, es_*, neo4j_*, ts_*), served SurrealDB still reads "no sync setting" (retired by BUGS F165), the e4 row records no durability, three dense arms lack index_s. Cause: unrealistic fixture. Fix: realistic_2611 stamps them from override_registry.OVERRIDES and the runner's own values.
  • fairness_check F3 and F15 at deep10m: the overlay files carry the client's 8g and none of the stamps, and the fixture had no deep10m campaign rows to inherit them from. Cause: both. Fixture: one deep10m campaign row per overlay arm. Gate: _dense_rows now inherits every l3d override stamp from the campaign row, embedded arms included, as it already did for the server envelope (the driver cannot write them, so every real deep10m cell would have failed NOT STAMPED).
  • equivalence_check (8 failures): DuckPGQ rows were the September adapter's (different answers, no digest for the two edge writes), and the e4 row had no res_limit* digests. Cause: unrealistic fixture. Fix: DuckPGQ rows take the answer digests of the ArcadeDB embedded row of the same cell; e4 rows get digests.
  • resource_check (2): FalkorDB's disk growth 0.0 was a reading taken before the RDB flushed (fixture fix); SurrealDB's lifecycle memory read as far below "the others" because ArcadeDB's embedded and served arms counted as two of the three peers (gate fix: three engines, not three arms).
  • provenance_check (26): the N=5 rule and its 2026-10 disclosures (3 to 5 repetitions is the 2026-11 range, make_2611_stages --reps; N=0 rows have no cells and are not a count), the paper's T3 pooled the strict and relaxed durability classes (graph_table now takes the headline class), and 12 version findings came from rows under the 2026-10 pass-1 pin d36b4ca beside the fixture pin (fixture: one pin, one version per engine, ArangoDB 3.12.12).
  • version_consistency_check (4): the same mixed pins and ArangoDB 3.12.11 beside 3.12.12, in the rows and in the overlay files. Cause: unrealistic fixture. Fix: one pin and one version per engine.
  • page_check (roster, A1, A2, crash): the roster owed every registered arm, but the 2026-11 subset leaves out the arms in make_2611_stages.OUTSIDE, the dense table is the multipass overlay's (it owes the overlay arms), and tables of lanes with no rows (restart, l2olap) are not owed; the PQ arms are registered only under BENCH_DENSE_PQ=1 and read as STRAY (page_check now registers them whenever the payload carries PQ rows); pq_clusters, pq_subspaces, pq_index_bytes are declared not-printed with a reason; and Benchmarks: the publish side renders 2026-11 rows only #432 deleted the j_bad computation and left or j_bad in the return, so every run ended in a NameError (call restored: the image-defaults arm is in the 2026-11 subset). Fixture: 14 roster arms the subset owes (cloned from a donor arm under their own name and version, synthetic like the PQ arms), plus the image-defaults JVM stamps.
  • make_paper_figures: crashed on the missing sparse_cliff.jsonl (a one-shot artifact of the retired instruments; f3 is not built). Gate fix: f3 skips with a printed reason when the file is absent. f6 draws once the deep10m rows exist.
  • override_registry and degenerate_check passed before and pass now. claims_check's default mode (111 of 120 claims disagree) pins the typed numbers of the retired paper to the rows behind them and its docstring says nothing runs it as a gate, so it is documented as no gate and its --regen mode is what runs.
  • Not in scope: pycost has no tableId in DBBench's lanes.ts (a humem.ai change; the exporter already omits the table), and refresh step 3's table coverage reads that checkout.

Docs: PUBLISHING.md (gate list, procedure, what the fixture is and how it is patched), CAMPAIGN.md row 52.

Suite before: 899 passed, 24 skipped, 0 failed. After: 913 passed, 25 skipped (the figures test needs matplotlib, pdftotext and Ghostscript and skips with that reason; it passed locally with matplotlib), 0 failed.

🤖 Generated with Claude Code

… with a test that runs them (#433)

Each failure fixed at its cause: fixture rows that no 2026-11 run could write (realistic_2611 in build_fixture.py), and gates that still encoded the retired instruments or the all-arms roster. page_check also crashed on an unset j_bad since #432.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898
tae898 merged commit 0dd3495 into main Oct 21, 2026
3 checks passed
@tae898
tae898 deleted the publish-gates-2611 branch October 21, 2026 17:21
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Publish gates: run equivalence, resource, paper-table and page checks on the 2026-11 fixture and fix what they flag

1 participant