Skip to content
This repository was archived by the owner on Nov 21, 2026. It is now read-only.

Benchmarks: the publish side renders 2026-11 rows only - #432

Merged
tae898 merged 7 commits into
mainfrom
publish-2611-only
Oct 21, 2026
Merged

tae898 merged 7 commits into
mainfrom
publish-2611-only

Conversation

@tae898

@tae898 tae898 commented Oct 21, 2026

Copy link
Copy Markdown

Closes #427

DECISIONS #196: no page renders a 2026-10 or September row again, and DBBench publishes first with November data. The exporter and the page tooling get one mode. Based on #429 (measure side) because that PR makes BENCH_INSTRUMENT other than 2026-11 an import error; merge order is PR-B (#431), #429, then this one, and GitHub retargets this PR to main when #429 merges.

export_web.py goes from about 10,100 to 7,600 lines: no _OCTOBER_ENV, no September or skeleton branches, no notes that describe 2026-10 rows (GAV, dense unindexed delete, sparse whole-record, releases-behind line, ts-doc OOM, top-parts refusal, pinned-heap jvm_heap text). New publish_instrument.py holds the rule once (PUBLISH_INSTRUMENT, refuse_other_instruments, results_dir()), and every freeze, exporter and gate refuses a row of another instrument with a message that says so. One set of names: runs_paper.csv, web_benchmarks.json, generated/. The preview route, the October campaign and the skeleton publish are deleted, with runs_paper_oct.csv, runs_skeleton_laptop.csv, web_benchmarks*.json and generated_oct/.

What 2026-11 rows needed that the publish side lacked: the served ArcadeDB lean twin (arcadedb_http_client_role = sensitivity) was collapsed into the headline cell by the freeze's dedupe, so the key now includes the role, the freeze keeps the twin rows, the tables and gates read headline rows, and the payload attaches the twin to the headline entry as client_twin. The l4 plain-read columns and the idle-state, sparse warm-up, dense id index and official-driver sentences are generated from what the rows stamp.

refresh_web_page.py has one target: the payload goes to src/data/dbbench-benchmarks.json in the humem.ai checkout and the tableId coverage check reads src/lib/dbbench/lanes.ts. The prose-number checks against the old ArcadeDB page are gone because humem.ai removed every number. land_stage.py, publish_results_asset.py, page_check.py, claims_check.py and version_consistency_check.py are 2026-11 only; campaign_switch_check.py is now a re-pin checklist.

Proof is test_publish_2611_e2e.py: it freezes and exports a small fixture (fixtures/publish_2611, real 2026-10 rows restamped 2026-11 plus the 2026-11 row fields, built by build_fixture.py) and checks the payload, the matched-recall dense columns, the per-phase memory column, the twin, the paper tables and the refusals. conftest.py points the whole suite at that fixture, so no test reads an untracked results/ directory; the suite passes from a clean checkout (960 passed, 24 skipped, 1 failed: test_stage_generator_2611.py::test_check_refuses_another_instrument_in_the_environment asserts the word "mix" in a message that #429 changed; it belongs to the stage-generator PR).

Docs rewritten to the one mode: PUBLISHING.md, PAGE-SPEC.md, READING-RESULTS.md, the publish rows of CAMPAIGN.md and the bindings benchmark pages (strict mkdocs build passes).

🤖 Generated with Claude Code

@tae898
tae898 changed the base branch from instrument-2611-only to main October 21, 2026 15:36
@tae898
tae898 force-pushed the publish-2611-only branch 2 times, most recently from 7177974 to 31660a7 Compare October 21, 2026 16:23
tae898 and others added 7 commits October 22, 2026 01:40
…re and its end-to-end publish test (#410)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… developer's results/ (#427)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ir docs render 2026-11 rows only (#427)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… and freezes are deleted, the allowlist names the one set (#427)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…#427)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… from one suffix; stale skeleton and export_web comments go (#427)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sion row; the fixture carries synthetic PQ rows (#427)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tae898
tae898 force-pushed the publish-2611-only branch from 31660a7 to aebed1e Compare October 21, 2026 16:42
@tae898
tae898 merged commit 11f71c3 into main Oct 21, 2026
4 checks passed
@tae898
tae898 deleted the publish-2611-only branch October 21, 2026 16:49
tae898 added a commit that referenced this pull request Oct 21, 2026
… with a test that runs them (#433) (#434)

Each failure fixed at its cause: fixture rows that no 2026-11 run could write (realistic_2611 in build_fixture.py), and gates that still encoded the retired instruments or the all-arms roster. page_check also crashed on an unset j_bad since #432.

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
tae898 added a commit to humemai/mmdbbench that referenced this pull request Nov 20, 2026
…rs write this repository's paths (#100)

* experiments: port the 4 lines of origin PR 590 that this repository lacked (refs #25, #26)

Origin: humemai/arcadedb-embedded-python@6eff4f1 (PR 590, branch docs/hygiene-bench-20261111, a draft there)

PR 590 fixes drift in CAMPAIGN.md, PROTOCOL.md and PAGE-SPEC.md. CAMPAIGN.md stays in the origin, so its 13 lines are not here. The other 4 lines were absent
from this tree, in 2 files, and each is true of this tree's code, checked by grep:

- PROTOCOL.md, instrument sentence: export_web._instrument_of is now publish_instrument.instrument_of (publish_instrument.py:24; no _instrument_of exists)
- PROTOCOL.md, tolerance sentence: page_check._check_prose is now page_check._pin_check (page_check.py, derives 0.5 * 10 ** -decimals)
- PROTOCOL.md, NOT CHECKED sentence: the (0, 1) return is page_check._check_page_atomicity, when its results log is missing
- PAGE-SPEC.md, the re-run paragraph: STALE_UNTIL_RERUN and LIFECYCLE_STALE are not in export_web.py, so the paragraph now says the list was removed

Line numbers differ from the PR (main here moved 18 lines in PAGE-SPEC.md), so each old line was matched by its text, once, not by position. One deviation
from the PR text: the PAGE-SPEC.md paragraph says "#432", a PR of the origin, and here that would read as an issue of this repository, so it is written
humemai/arcadedb-embedded-python#432, as PAGE-SPEC.md already writes #396. PROVENANCE.md gets row 10.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* experiments: the stage generators write the paths of this repository (refs #92, #26)

Phase D of the retirement of humemai/arcadedb-embedded-python switches the mini's harness checkout to this repository at a stage boundary. It was blocked
because make_2611_stages.py wrote REPO=$HOME/repos/humemai/arcadedb-embedded-python and R=$REPO/benchmarks/experiments into every stage (HEAD, and the pair
install PAIR). A stage generated after the switch would have run the old harness.

- REPO is BENCH_STAGE_REPO at generation, default $HOME/repos/humemai/mmdbbench (the mini clones this repository as mmdbbench). A value that bash could read as more
  than a path (space, quote, semicolon, parentheses, `..`) is refused, because it is written unquoted. R is $REPO/experiments.
- This repository has no bindings/python, so a stage that read the pinned wheel from $REPO/bindings/python/dist would have aborted at its first wheel guard. The wheel
  directory is now $BINDINGS_REPO/bindings/python/dist, the variable build_c25_wheel.sh and verify_pair_c25.sh already need, defaulting to the stage's own REPO (a
  worktree that holds its own dist/ keeps working). verify_pair_c25.sh gets it exported by the stage.
- make_post1_rerun_stages.py follows: its REPO anchor and leak check read stage_repo(), and its wheel stage uses the same BINDINGS_REPO default, so post1 stages
  built from a tree still patch cleanly (its --selftest passes).
- stage_drafts_2611.sha256 regenerated (every draft changes: 139 lines); test_stage_generator_2611.py drops BENCH_STAGE_REPO from the generator's environment, and
  adds tests: no generated stage (comparator drafts and the release chain) names the old repository or benchmarks/experiments, they pass bash -n, BENCH_STAGE_REPO
  is honoured, six bad values are refused.
- README.md "Paths that stay as they are" no longer lists the generators; PROVENANCE.md row 11.

Not changed on purpose: stage scripts that exist as records, and the BENCH_LAND_REMOTE and merge_campaign --remote defaults, which must flip on the day of the switch,
not before (the stages still running from the old checkout land from there). Not run here: pytest, which the laptop guard blocks; the generator's own --check and
the post1 --selftest pass, the drafts regenerate, and bash -n passes on a generated release chain. CI runs the suite.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* experiments: refuse a bad BENCH_STAGE_REPO before any stage is written; the override test checks the default path, not every citation (refs #92)

CI of the first push failed in two new tests. The generator checked the path inside the per-file loop, so it refused only after it had written the first stage (stage_repo() is now called right after the arguments are parsed). The override test asserted that no generated stage contains humemai/mmdbbench, but the stage template cites #62 in a comment; it now asserts that the default checkout path is gone.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Benchmarks: the publish side renders 2026-11 rows only

1 participant