Skip to content
This repository was archived by the owner on Nov 21, 2026. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion benchmarks/experiments/CAMPAIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,7 @@ The detail behind item 8:

**The next campaign's stages are generated by `make_2611_stages.py`.** One template carries every guard the campaign learned (cap table, shadowed definitions, budgets against the cap, whole-cell cost, images present and current, comparator digests frozen at generation, the JPype pin read out of the image, the pinned wheel by sha256, the cap rule, the ALL-DONE trap); the instrument, `REPS` and the banner are its parameters. The generator selects the multi-model subset. The subset is `ENGINES` in the file: ArcadeDB (embedded and served), SurrealDB (embedded and served), ArangoDB, Memgraph, PostgreSQL with pgvector, AGE and TimescaleDB (and plain PostgreSQL where a lane uses it), MongoDB with MongoDB Search, DuckDB with VSS and DuckPGQ, SQLite with sqlite-vec, the composed Qdrant plus Neo4j stack, Neo4j and Qdrant on their own (graph and dense; sparse on the second list), and LadybugDB on every lane it supports. Every registered arm is either in the subset or in `OUTSIDE`, so a newly registered arm forces a decision (`--check`). The core lanes are graph (interactive and analytics), cross-model, documents and dense; `--second-list` adds lifecycle, then sparse and time series, after them. TimescaleDB runs whenever the time-series lane does; `--without-timescaledb` leaves it out.
Every generated stage exports `BENCH_INSTRUMENT=2026-11` before its first Python call, reads the selection back from `bench_common` and aborts if the tree does not select it, writes `runs_page_<pin>_2611.jsonl` (and overlay directories with the same suffix) so one rows file never holds two instruments, and fixes `REPS` at generation (`--reps`, default 3, at most 5). The generator re-executes itself with the variable set and refuses a different value in the environment. `land_stage.py` reads only 2026-10 rows today, so landing the new rows is a change of its own, made when the first 2026-11 tables land.
The comparators are the default mode and run on the release pair already installed (`ARCADEDB_WHEEL`, `ARCADEDB_SERVER_IMAGE`, `ARCADEDB_ENGINE_COMMIT`): `python3 make_2611_stages.py --second-list --after <last stage of the previous chain> --id-prefix qS --out DIR`. `--after` is required (a stage id, or `none`), and each stage then waits for the previous one with `grep -qx "<id> ALL-DONE"`. The ArcadeDB pass is `--arcadedb-release X.Y.Z` with `--engine-commit`, `--image-digest sha256:...`, `--wheel-name`, `--wheel-sha256` and `--wheel-url`; it refuses unless all five are given, because the release may not exist when the drafts are made. It writes `<prefix>00`, the pair install (official image by digest, wheel by sha256, `build_matched_pair.sh`, `verify_pair_c25.sh`), and ArcadeDB's lane stages behind it.
The comparators are the default mode and run on the release pair already installed (`ARCADEDB_WHEEL`, `ARCADEDB_SERVER_IMAGE`, `ARCADEDB_ENGINE_COMMIT`): `python3 make_2611_stages.py --second-list --after <last item of the queue on the machine> --status-file STATUS.txt --id-prefix qS --out DIR`. `--after` is required, with no default (a marker name, or `none`), and the first stage waits for `grep -qx "<marker> ALL-DONE"`, each later one for the stage before it. The marker must be the LAST item still to run, one that writes its line whether it runs or is skipped: a marker already in the machine's STATUS.txt makes the first stage start at once, beside the running jobs (2026-10-23: `qMA06 ALL-DONE`, written by a skipped study). So `--status-file` is required with a marker (a copy of that STATUS.txt, or `none` when there is no queue) and the generator refuses a marker the file already holds; `--check --drafts DIR --status-file STATUS.txt` makes the same test on drafts that exist. The subset is emitted REP-MAJOR (2026-10-23): with `--reps N` the chain is pass 1 of every stage (core lanes, then the second list), then pass 2 of every stage, then pass 3, and each stage runs ONE repetition (`--only-reps $PASS`). If time runs out, every engine has the same n; repetitions days apart show drift. A stage whose environment declares fewer reps (`REPS=k`) is left out of the later passes. Inside a lane the engines run in `ENGINE_PRIORITY` (SurrealDB, ArangoDB, PostgreSQL, MongoDB, Redis 8, composed Qdrant+Neo4j, Neo4j, Qdrant, DuckDB, SQLite, Memgraph, LadybugDB, CrateDB; ArcadeDB's arms lead it when they join; Redis and CrateDB are skipped until their adapters are in `ENGINES`). The rep-1-first rule and the cap rule hold across passes: a pass above 1 runs a cell only when rep 1 of it was clean, and only a timeout of rep 1 censors the larger sizes. WHICH REPS EXIST is `~/rep-state-2026-11-<pin>.tsv` on the host (one `label, rep, ok|fail` line per finished repetition; the rows themselves carry the rep number in the runner's results file): stop after any pass, and the next pass, or a restarted stage, picks up from it with no regeneration; N can go to 3 or stop at 2. Delete that file to start a chain over. The ArcadeDB pass is `--arcadedb-release X.Y.Z` with `--engine-commit`, `--image-digest sha256:...`, `--wheel-name`, `--wheel-sha256` and `--wheel-url`; it refuses unless all five are given, because the release may not exist when the drafts are made. It writes `<prefix>00`, the pair install (official image by digest, wheel by sha256, `build_matched_pair.sh`, `verify_pair_c25.sh`), and ArcadeDB's lane stages behind it.
The ArcadeDB pair's guards are split by what a stage runs. Every stage exports the pinned wheel (`ARCADEDB_WHEEL`, with `ARCADEDB_SERVER_IMAGE` and `ARCADEDB_ENGINE_COMMIT`) before `build_images.sh`, which bakes it into every generic image (client, duckdb, dense), and so every stage guards that the file exists at the pinned path and has the pinned sha256; `build_images.sh` copies it from that path and nothing in it or the Dockerfiles reads `dist/` itself. Only a stage that runs an ArcadeDB arm (the `--arcadedb-release` pass) also guards the rest: exactly one wheel in `dist/` (it protects `verify_pair_c25.sh`), the server image, the wheel and JPype read back out of `dbbench:arcadedb`, and `verify_pair_c25.sh`. A comparator stage on a host whose `dbbench:arcadedb` is the previous release's therefore runs. `build_images.sh` has no PyPI fallback: it refuses to build the `arcadedb` image (named or by default) without `ARCADEDB_WHEEL`, and refuses a wheel that is not a file, and refuses a wheel whose `arcadedb_embedded/core.py` lacks `_INSERT_MANY_CHUNK` (`wheel_guard.py`: before 26.10.2 `insert_many` parsed the whole input into one Java array, which the kernel killed on l4/ts1000/arcadedb_ts_doc on 2026-10-08; use a 26.10.2 or newer wheel); the images that take no wheel can be built alone by naming them. Every ABORT hands on: the `ALL-DONE` trap is set right after the wait, before the first guard, so a stage that aborts before its START line still writes its marker and the next stage runs (its own guards then run too). The ABORT line is in `STATUS.txt`, where `mini_watch.sh` reports it; to resume, fix the cause and run the aborted stage's script again by hand after the chain, and landing reads only the rows that exist. `make_2611_stages.py --check --drafts DIR` lints both rules.
The default order is the core block first, then the second list. Inside the core block the large tiers (deep10m, tpch10, the full SF1 network) run after the small ones, and the cells that dominate the hours (Memgraph's deep10m dense cells, SurrealDB's documents at tpch1, the full SF1 network's analytics) run last of all in stages of their own, where they are easiest to drop if time runs short; `BIGGEST` in the file names them, and `--order paper` keeps the table order. Splitting moves stage boundaries only, and `--check` proves the cells are the same. `--project TABLE` prints projected hours per drafted stage from a per-cell hours table (tab separated: lane, workload, scale, backend, class, hours at 3 repetitions, hours at 5 repetitions; a cell with no clean rep carries its one failed attempt). `--check --drafts DIR` lints written scripts: the instrument exported and never 2026-10, the 2611 rows file, only subset arms, each stage waiting for the one before it. Drafts are written to a folder outside the tree and copied to the bench host by hand on the owner's go. Which stage is running is state, not method: it is in `STATUS.txt`'s tail and in HANDOFF, never here. `test_stage_generator_2611.py` holds the written drafts byte for byte against `stage_drafts_2611.sha256`, so a stage cannot move without the list changing in the same commit.

Expand Down
191 changes: 191 additions & 0 deletions benchmarks/experiments/fixtures/mini_status_20261023.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,191 @@
# Excerpt of mini's ~/STATUS.txt, read-only copy 2026-10-23: every ALL-DONE marker line it held, then its last 12 lines. test_stage_generator_status.py feeds it to make_2611_stages.py --status-file. Not a log to extend.
[2026-08-10T13:18:47+00:00] STAGE ALL-DONE 2026-08-10T13:18:47+00:00
[2026-08-14T15:58:14+00:00] STAGE ALL-DONE rc=0
[2026-08-14T17:36:19+00:00] q84 ALL-DONE
[2026-08-15T01:35:29+00:00] q86 ALL-DONE
[2026-08-15T04:12:03+00:00] q90 ALL-DONE
[2026-08-15T04:18:19+00:00] q90 ALL-DONE
[2026-08-15T04:44:40+00:00] q91 ALL-DONE
[2026-08-15T04:49:11+00:00] q92 ALL-DONE
[2026-08-15T05:59:04+00:00] q93 ALL-DONE
[2026-08-15T07:02:26+00:00] q93 ALL-DONE
[2026-08-15T08:06:04+00:00] q93 ALL-DONE
[2026-08-15T15:05:34+00:00] q94 ALL-DONE
[2026-08-19T18:34:40+00:00] qI ALL-DONE
[2026-08-19T19:47:25+00:00] qJ ALL-DONE
[2026-08-22T10:45:34+00:00] qO ALL-DONE
[2026-08-22T10:57:25+00:00] qP ALL-DONE
[2026-08-22T12:20:30+00:00] qQ ALL-DONE
[2026-08-22T12:29:44+00:00] qR ALL-DONE
[2026-08-23T01:22:13+00:00] qS ALL-DONE
[2026-08-23T06:51:11+00:00] qT ALL-DONE
[2026-08-23T23:01:22+00:00] qU ALL-DONE
[2026-08-24T00:34:47+00:00] qV ALL-DONE
[2026-08-24T04:43:27+00:00] qW ALL-DONE
[2026-08-25T06:54:34+00:00] qZ ALL-DONE
[2026-08-25T10:33:59+00:00] qZ2 ALL-DONE
[2026-08-25T10:46:44+00:00] qZ3 ALL-DONE
[2026-08-25T10:50:27+00:00] qZ4 ALL-DONE
[2026-08-26T02:06:35+00:00] qAA ALL-DONE
[2026-08-26T03:28:21+00:00] qAB ALL-DONE
[2026-08-26T03:46:11+00:00] qAC ALL-DONE
[2026-08-26T04:17:39+00:00] qAD ALL-DONE
[2026-08-26T04:46:56+00:00] qAE ALL-DONE
[2026-08-26T04:49:38+00:00] qAF ALL-DONE
[2026-08-26T04:56:33+00:00] qAF ALL-DONE
[2026-08-26T05:08:26+00:00] qAG ALL-DONE
[2026-08-26T15:14:42+00:00] qAH ALL-DONE
[2026-08-27T00:27:41+00:00] qBA ALL-DONE
[2026-08-27T00:30:00+00:00] qBC ALL-DONE
[2026-08-27T00:41:08+00:00] qBD ALL-DONE
[2026-08-27T01:20:36+00:00] qBF ALL-DONE
[2026-08-27T02:23:42+00:00] qBG ALL-DONE
[2026-08-27T19:05:47+00:00] qBHR ALL-DONE
[2026-08-27T20:19:57+00:00] qBHR ALL-DONE
[2026-08-28T12:51:43+00:00] qBK ALL-DONE
[2026-08-28T12:53:24+00:00] qBL ALL-DONE
[2026-08-28T13:41:16+00:00] qBI ALL-DONE
[2026-08-28T23:00:12+00:00] qBJ ALL-DONE
[2026-08-29T08:35:56+00:00] qBO ALL-DONE
[2026-08-29T08:39:35+00:00] qBM ALL-DONE
[2026-08-29T13:22:02+00:00] qBM ALL-DONE
[2026-08-29T13:26:35+00:00] qBM ALL-DONE
[2026-08-29T13:41:04+00:00] qBN ALL-DONE
[2026-08-29T13:41:28+00:00] qBL ALL-DONE
[2026-08-29T15:40:41+00:00] qBN ALL-DONE
[2026-08-29T16:35:58+00:00] qBP ALL-DONE
[2026-08-29T16:45:54+00:00] qBN ALL-DONE
[2026-08-31T15:23:54+00:00] qCA ALL-DONE
[2026-08-31T22:17:46+00:00] qCB ALL-DONE
[2026-08-31T23:15:07+00:00] qCC ALL-DONE
[2026-09-01T09:34:25+00:00] qCD ALL-DONE
[2026-09-01T09:44:55+00:00] qCE ALL-DONE
[2026-09-03T10:34:18+00:00] qCG ALL-DONE
[2026-09-03T12:54:32+00:00] qCF ALL-DONE
[2026-09-03T14:21:52+00:00] qCH ALL-DONE
[2026-09-04T03:13:44+00:00] qCI ALL-DONE
[2026-09-05T23:55:09+00:00] qCJ ALL-DONE
[2026-09-06T01:19:15+00:00] qCK ALL-DONE
[2026-09-06T04:54:15+00:00] qCL ALL-DONE
[2026-09-06T09:04:32+00:00] qCM ALL-DONE
[2026-09-06T21:36:55+00:00] qCN ALL-DONE
[2026-09-07T06:09:02+00:00] qCO ALL-DONE
[2026-09-07T06:50:08+00:00] qCP ALL-DONE
[2026-09-07T06:58:49+00:00] qCR ALL-DONE
[2026-09-07T08:00:48+00:00] qCS ALL-DONE
[2026-09-07T08:17:37+00:00] qCT ALL-DONE
[2026-09-07T10:37:04+00:00] qCQ ALL-DONE
[2026-09-07T13:22:23+00:00] qCU ALL-DONE
[2026-09-07T18:15:01+00:00] qCV ALL-DONE
[2026-09-07T18:33:15+00:00] qCW ALL-DONE
[2026-09-07T19:27:59+00:00] qCX ALL-DONE
[2026-09-08T01:00:48+00:00] qCY ALL-DONE
[2026-09-08T01:05:18+00:00] qCX2 ALL-DONE
[2026-09-08T02:42:27+00:00] qCX3 ALL-DONE
[2026-09-08T03:37:46+00:00] qCY2 ALL-DONE
[2026-09-10T05:45:27+00:00] qDA ALL-DONE
[2026-09-10T08:20:34+00:00] qDA ALL-DONE
[2026-09-10T11:21:15+00:00] qDB ALL-DONE
[2026-09-10T12:27:44+00:00] qDC ALL-DONE
[2026-09-10T14:49:23+00:00] qDD ALL-DONE
[2026-09-10T15:21:43+00:00] qDG ALL-DONE
[2026-09-10T15:33:58+00:00] qDF ALL-DONE
[2026-09-11T17:47:46+00:00] qDE ALL-DONE
[2026-09-11T18:33:08+00:00] qDH ALL-DONE
[2026-09-12T02:04:43+00:00] qDI ALL-DONE
[2026-09-12T09:40:58+00:00] qDK ALL-DONE
[2026-09-12T11:28:31+00:00] qDJ ALL-DONE
[2026-09-12T12:43:43+00:00] qDL ALL-DONE
[2026-09-12T12:49:00+00:00] qDM ALL-DONE
[2026-09-12T14:23:00+00:00] qDN ALL-DONE
[2026-09-14T12:59:11+00:00] qDO ALL-DONE
[2026-09-14T13:07:47+00:00] qDP ALL-DONE
[2026-09-14T13:12:58+00:00] qDQ ALL-DONE
[2026-09-14T13:18:58+00:00] qDR ALL-DONE
[2026-09-15T13:39:42+00:00] qDS ALL-DONE
[2026-09-15T13:52:28+00:00] qDT ALL-DONE
[2026-09-16T07:18:39+00:00] qDU ALL-DONE
[2026-09-17T07:19:21+00:00] qDV ALL-DONE
[2026-09-17T22:16:17+00:00] qDW ALL-DONE
[2026-09-18T02:20:54+00:00] qDX ALL-DONE
2026-09-18T04:18:45Z qDY ALL-DONE rc=1 (probe log: results/log or docker logs of cli-l3d_arangodb_dense_search_deep10m_r1; artifact results/probe_f55/mp_arango_b1.json)
2026-09-18T04:42:16Z qDY ALL-DONE rc=0 (probe log: results/log or docker logs of cli-l3d_arangodb_dense_search_deep10m_r1; artifact results/probe_f55/mp_arango_b1.json)
[2026-09-18T17:45:45+00:00] qEA ALL-DONE
[2026-09-19T00:37:32+00:00] qEB ALL-DONE
[2026-09-19T03:11:38+00:00] qCAL ALL-DONE
[2026-09-19T12:40:31+00:00] qOA ALL-DONE
qOAR ALL-DONE
[2026-09-19T12:49:46+00:00] qOA2 ALL-DONE
qOAR3 ALL-DONE
qOAR3 ALL-DONE
qOAR3 ALL-DONE
qOAR3 ALL-DONE
[2026-09-20T21:24:58+00:00] qOC ALL-DONE
qOD ALL-DONE
qOI ALL-DONE
qOD2 ALL-DONE
qOA3 ALL-DONE
qOD3 ALL-DONE
qOA4 ALL-DONE
qOE ALL-DONE
qOF ALL-DONE
qOG ALL-DONE
qOH ALL-DONE
[2026-10-05T16:21:32+00:00] overlay_guard: qOH ALL-DONE, exiting
qOB2 ALL-DONE
qRA ALL-DONE
qT01 ALL-DONE
qP01 ALL-DONE
qP02 ALL-DONE
qP35 ALL-DONE
qP03 ALL-DONE
qP04 ALL-DONE
qP05 ALL-DONE
qP06 ALL-DONE
qP07 ALL-DONE
qP08 ALL-DONE
qP09 ALL-DONE
qP10 ALL-DONE
qP11 ALL-DONE
qP12 ALL-DONE
qP13 ALL-DONE
qP14 ALL-DONE
qP15 ALL-DONE
qP16 ALL-DONE
qP17 ALL-DONE
qP18 ALL-DONE
qP19 ALL-DONE
qP20 ALL-DONE
qP21 ALL-DONE
qP22 ALL-DONE
qP23 ALL-DONE
[2026-10-18T05:01:37+00:00] qA00..qA15 installed and waiting (the user's go 2026-10-18): the ArcadeDB 26.10.2 pass, pin e854692736 (the released image's and wheel's buildNumber), qA00 builds the matched pair after qP34 ALL-DONE (DECISIONS #184, #186, #188)
qA00 ALL-DONE
[2026-10-18T05:10:07+00:00] qA01..qA15 stopped by hand before any cell ran: qA00 started early because its wait grep matched the phrase "qP34 ALL-DONE" inside the 05:01 install note (my bug). qP24..qP34 continue; the 26.10.2 pass is re-installed after the fix.
qP24 ALL-DONE
qP25 ALL-DONE
qP26 ALL-DONE
qP27 ALL-DONE
qP28 ALL-DONE
qP29 ALL-DONE
qP30 ALL-DONE
qP31 ALL-DONE
qMA00 ALL-DONE
qMA01 ALL-DONE
qMA02 ALL-DONE
qMA03 ALL-DONE
qMA04 ALL-DONE
qMA05 ALL-DONE
qMA06 ALL-DONE
[2026-10-23T06:01:59+00:00] qMA05 finished
qMA05 ALL-DONE
[2026-10-23T06:02:49+00:00] qMA06 SKIP: conf/skip-qMA06 exists
qMA06 ALL-DONE
[2026-10-23T06:03:44+00:00] fb-jvm-aot: qMA06 is done: starting jvm-aot as a single-lane stage (/home/tk/ab-2610/repros/jvm-aot-ab-20261022-fb/mini_driver.sh)
[2026-10-23T06:03:44+00:00] jvm-aot: stage mode (single lane, CPUs 2,4): waiting for the machine to be alone
[2026-10-23T06:03:56+00:00] jvm-aot START (stage mode): lane 1 CPUs 2,4 siblings 3,5, helpers 16-19, cool-down 70 C cap 600 s, results /home/tk/ab-2610/out/jvm-aot-ab-20261022
[2026-10-23T06:03:56+00:00] jvm-aot: warm-up of the uv cache (network, helper cores 16-19)
[2026-10-23T06:13:40+00:00] native-scale: waiting for the alone slot: process alive: 3175933/cmdline: bash /home/tk/ab-2610/lanes/fallback_stage.sh jvm-aot
[2026-10-23T06:51:12+00:00] jvm-aot finished rc=0 (completed rounds 0..24, min clean pairs 5): /home/tk/ab-2610/out/jvm-aot-ab-20261022/DONE written
[2026-10-23T06:51:12+00:00] fb-jvm-aot: jvm-aot driver exited rc=0: rc=0 2026-10-23T06:51:12+00:00 completed rounds 0..24, min clean pairs 5
[2026-10-23T06:51:12+00:00] fb-native-ab: queued: waits for the main queue (qMA06) and for other fallback jobs
Loading
Loading