This repository was archived by the owner on Nov 21, 2026. It is now read-only.
Repository navigation
Benchmarks: every publish gate passes on the 2026-11 fixture payload, with a test that runs them (#433) - #434
Merged
Merged
Conversation
… with a test that runs them (#433) Each failure fixed at its cause: fixture rows that no 2026-11 run could write (realistic_2611 in build_fixture.py), and gates that still encoded the retired instruments or the all-arms roster. page_check also crashed on an unset j_bad since #432. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Oct 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #433
Every publish gate now runs on the 2026-11 fixture payload and passes, and
test_publish_gates_2611.pyruns them in the suite (make_paper_tables with--strict, equivalence_check, resource_check, override_registry, degenerate_check, fairness_check, page_check, version_consistency_check, version_pin_check, provenance_check, claims_check--regen, make_paper_figures). Two negative tests keep the roster and the equivalence gate alive. Fixture rows change only throughbuild_fixture.realistic_2611(alsopython build_fixture.py --patch, since the real results are archived) and the committedruns.jsonl.gzplus the ArangoDB overlay files it touches; the builder's docstring names each patch and its reason.Gate, failure, cause, fix (before counts are from the fixture as merged in #432):
index_kinds,arcadedb_add_hierarchy,server_query_max_heap_elements,server_http_body_max_bytes,duckdb_threads,es_*,neo4j_*,ts_*), served SurrealDB still reads "no sync setting" (retired by BUGS F165), the e4 row records no durability, three dense arms lackindex_s. Cause: unrealistic fixture. Fix:realistic_2611stamps them fromoverride_registry.OVERRIDESand the runner's own values._dense_rowsnow inherits every l3d override stamp from the campaign row, embedded arms included, as it already did for the server envelope (the driver cannot write them, so every real deep10m cell would have failed NOT STAMPED).res_limit*digests. Cause: unrealistic fixture. Fix: DuckPGQ rows take the answer digests of the ArcadeDB embedded row of the same cell; e4 rows get digests.make_2611_stages --reps; N=0 rows have no cells and are not a count), the paper's T3 pooled the strict and relaxed durability classes (graph_tablenow takes the headline class), and 12 version findings came from rows under the 2026-10 pass-1 pin d36b4ca beside the fixture pin (fixture: one pin, one version per engine, ArangoDB 3.12.12).make_2611_stages.OUTSIDE, the dense table is the multipass overlay's (it owes the overlay arms), and tables of lanes with no rows (restart, l2olap) are not owed; the PQ arms are registered only underBENCH_DENSE_PQ=1and read as STRAY (page_check now registers them whenever the payload carries PQ rows);pq_clusters,pq_subspaces,pq_index_bytesare declared not-printed with a reason; and Benchmarks: the publish side renders 2026-11 rows only #432 deleted thej_badcomputation and leftor j_badin the return, so every run ended in a NameError (call restored: the image-defaults arm is in the 2026-11 subset). Fixture: 14 roster arms the subset owes (cloned from a donor arm under their own name and version, synthetic like the PQ arms), plus the image-defaults JVM stamps.sparse_cliff.jsonl(a one-shot artifact of the retired instruments; f3 is not built). Gate fix: f3 skips with a printed reason when the file is absent. f6 draws once the deep10m rows exist.--regenmode is what runs.pycosthas notableIdin DBBench'slanes.ts(a humem.ai change; the exporter already omits the table), and refresh step 3's table coverage reads that checkout.Docs: PUBLISHING.md (gate list, procedure, what the fixture is and how it is patched), CAMPAIGN.md row 52.
Suite before: 899 passed, 24 skipped, 0 failed. After: 913 passed, 25 skipped (the figures test needs matplotlib, pdftotext and Ghostscript and skips with that reason; it passed locally with matplotlib), 0 failed.
🤖 Generated with Claude Code