diff --git a/benchmarks/README.md b/benchmarks/README.md index 2462822f84b..be8fbdb1749 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -11,9 +11,10 @@ in scope and in where they publish, not in the kind of thing they measure. against specialist and multi-model engines on documents, graph, dense and sparse vectors, time series, and cross-model queries. It is what every campaign runs, and its results are the project page at - . There is no paper; the page is the - only artifact, and every number on it is generated from the frozen rows by - the exporter and checked by gates before it can be published. + . The page is the living artifact, and + every number on it is generated from the frozen rows by the exporter and + checked by gates before it can be published; a paper written from the same + rows pins a stable release where the page may pin a commit (DECISIONS #42). - `python-bindings/` is the **SciPy 2026 paper's artifact**, frozen with the paper: the package as published on PyPI against four embedded Python specialists on one corpus, plus `jpype_overhead/`, which times the binding @@ -22,8 +23,8 @@ in scope and in where they publish, not in the kind of thing they measure. extended; new work goes in `experiments/`. The one exception is `jpype_overhead/results/mini_results_.csv`, the bench-host run that feeds the page's Python-cost table: `export_web.py` reads the file named by the pin it - publishes, only the September pin's exists, and the October payload omits the - table until it is re-measured. Its own README carries its versions and layout. + publishes, only the September pin's exists until the 26.10.1 measurement's + Python-cost stage lands, and the October payload omits the table until then. Its own README carries its versions and layout. ## Where to start in `experiments/` @@ -38,6 +39,7 @@ its detail. Read it first; the rest are the references it points to. | `PROTOCOL.md` | How a row is produced, and what enforces each rule (a gate, a check, or a person) | | `FAIRNESS.md` | When two numbers may be compared: the invariants and the disclosed overrides | | `COMPARATORS.md` | What every comparator is pinned to, and why it runs the way it does | +| `QUANTIZATION.md` | What each dense and sparse engine ships, and which quantized arm the page runs | | `PAGE-SPEC.md` | What the page contains: tables, rows, columns, and what each cell must satisfy | | `PUBLISHING.md` | How the numbers get from the frozen rows onto the page, the preview route, and the results asset | | `READING-RESULTS.md` | How to read what came out without misreading it | diff --git a/benchmarks/experiments/CAMPAIGN.md b/benchmarks/experiments/CAMPAIGN.md index d9201eb52c1..72a0ddb1c5a 100644 --- a/benchmarks/experiments/CAMPAIGN.md +++ b/benchmarks/experiments/CAMPAIGN.md @@ -8,7 +8,7 @@ Per campaign, once: 1. Settle the instrument before any cell runs: the query set, the measurement set, the durability rule, the fairness invariants, and the answer checking. A change mid-campaign splits the rows, so this is the only free moment (DECISIONS #82d, #88, #89, #90; PROTOCOL.md). The method pages in the package documentation (`bindings/python/docs/benchmarks/`) describe the same instrument without numbers, so they change in this step too, never later. 2. Smoke every adapter, one cell per engine per lane, and keep the evidence (the smoke's rows and logs). Smoke wherever a machine is free: nothing in the fairness contract touches a smoke, and the bench host is reserved only while a published cell runs, so a smoke on the real corpora there is both faster and a better test than a capped slice on the laptop. One runner per host, always: `runner.py` sweeps every `dbbench=1` container on the machine at startup, so it takes a host-wide lock first (`$TMPDIR/dbbench-runner.lock`, whichever checkout it runs from) and a second runner is refused at startup (BUGS F58). -3. Pin ArcadeDB: sync upstream to the release the campaign runs and build the wheel by the routine in `bindings/python/docs/development/upstream-pr.md`, run our Java reproductions on the built engine to confirm every fix it should carry is intact, and record the engine commit inside that wheel as the campaign's pin (the pin is the commit, not the version string, DECISIONS #49). The served arm's matched pair is built from the same image by `build_matched_pair.sh` (HANDOFF.md). One wheel, one pin, held start to finish. +3. Pin ArcadeDB: sync upstream to the release the campaign runs and build the wheel by the routine in `bindings/python/docs/development/upstream-pr.md`, run our Java reproductions on the built engine to confirm every fix it should carry is intact, and record the engine commit inside that wheel as the campaign's pin (the pin is the commit, not the version string, DECISIONS #49). For a stable release the pin is the release's own jar commit (26.10.1: `d36b4ca3a`), the wheel is the official jars plus this tree's bridge and package, and the pair is placed by the recipe in `.notes/bench/repin-round5/REPIN-START.md`. The served arm's matched pair is built from the same image by `build_matched_pair.sh` (HANDOFF.md). One wheel, one pin, held start to finish. 4. Re-pin every comparator to its latest stable release, smoking the risky jumps on their own (COMPARATORS.md, DECISIONS #87). 5. Publish a skeleton to the preview route from laptop data, so the page's shape can be reviewed before the numbers exist (DECISIONS #86, PUBLISHING.md). The skeleton is written from nothing: its tables and the sentences under them are generated from the rows, and its prose describes the instrument only. Nothing is copied from the previous page, because a sentence that was true on last campaign's rows is the one that goes stale unnoticed (DECISIONS #102, PAGE-SPEC.md "What the October page may say"). 6. Run the campaign, landing each stage to the preview rather than to the live page (section 6 below). @@ -97,9 +97,9 @@ deep10m sits at 0.67 because a single fp32 dense build peaks well above the 18g ```sh export ARCADEDB_WHEEL= ARCADEDB_SERVER_IMAGE=arcadedb-c25: # set together, or runner.py refuses -BENCH_ALLOW_DEV=1 bash build_images.sh # Phase A; without the flag it refuses a pre-release pin, which every commit-pinned wheel is. It installs jpype1==1.7.1 with the wheel (Dockerfile.bench, section 7 row 73); the repo venv needs `uv pip install --python .venv jpype1==1.7.1` for the same reason +BENCH_ALLOW_DEV=1 bash build_images.sh # Phase A; without the flag it refuses a pre-release pin, which every commit-pinned wheel is (a stable release's wheel, the 26.10.1 measurement's, builds with BENCH_ALLOW_DEV=0). It installs jpype1==1.7.1 with the wheel (Dockerfile.bench, section 7 row 73); the repo venv needs `uv pip install --python .venv jpype1==1.7.1` for the same reason export ARCADEDB_ENGINE_COMMIT= BENCH_HOST=mini # a paper-tier run refuses to start without the host, and without the commit when it runs an ArcadeDB arm -source campaign_env.sh # the BENCH_* dataset switches for a hand run; no queue script sources it (the October generator and the September scripts export them inline, PROTOCOL section 6) +source campaign_env.sh # the BENCH_* dataset switches for a hand run; no queue script sources it (the generators, `make_october_stages.py` and `make_2610_stages.py`, and the September scripts export them inline, PROTOCOL section 6) campaign_env_check # asserts every corpus is present python3 -u runner.py --lanes ... --scale ... --reps 5 ``` @@ -119,8 +119,8 @@ vector lanes' ingests, and the lifecycle lane -- runs once, at the relaxed default. The strict cell writes its own `run_id` (a `_dstrict` suffix), its own raw artifact, and its own canonical key, so the two never shadow each other; `fairness_check` F10b fails a write cell that exists in only one class, and the -four engines with no knob (Neo4j, DuckDB, LadybugDB, the SurrealDB 3.2.4 -server) are exempt because they run once and declare it. +three engines with no knob (Neo4j, DuckDB, LadybugDB) are exempt because they +run once and declare it. Rules that have each cost a run: @@ -166,162 +166,124 @@ Finished scripts move to `~/queue_archive` on mini. The chain holds its pin star **Sizes, every campaign (#108).** Each table carries two sizes wherever a second one is affordable and informative: graph interactive SF1 and SF10, documents transactional and analytical TPC-H SF1 and SF10, time series 100 and 1,000 hosts, cross-model 50k and 500k products, dense 1M and 9.99M, sparse 100k / 1M / 8.84M. Graph analytics is the single exception at one size, the full SF1 network, because the full network at SF10 is not feasible. Plan the matrix at two sizes from the start; if it does not fit, the transactional documents table's SF10 is the first size to drop, and dropping it is recorded rather than done silently. -**THE OCTOBER CAMPAIGN STARTED 2026-09-19 05:17Z** with `qOA`; the chain below runs in order, each stage -waiting on its predecessor's `ALL-DONE` marker. Which stage is running is state, not method: it is in -`STATUS.txt`'s tail and in HANDOFF, never here. Generated by `make_october_stages.py`; installed in the -bench host's home, not the repo. The pair was built first from upstream's published snapshot -`arcadedata/arcadedb@sha256:ccb0ad1f6e...`, whose jars read `buildNumber 417314c18da782620463bc7c09ac6bd34ac6fbda` -(read out of the image, not taken from a plan), giving `arcadedb-c25:417314c18` and a 26.10.1.dev0 wheel over the -same 86 jars. The September wheel was deleted at that point rather than left beside it. +**The 26.10.1 measurement (started 2026-10-06, pin `d36b4ca3a`, instrument 2026-10).** `make_2610_stages.py` generates the queue scripts, one per stage, as a profile of `make_october_stages.py`, so every guard below reaches them. It refuses to emit without `ARCADEDB_WHEEL`, `ARCADEDB_SERVER_IMAGE`, and `ARCADEDB_ENGINE_COMMIT`, and each stage carries the comparator digests as they stood at generation and refuses to start if `runner.py` now says otherwise. `--order tiers` (the default) queues every ArcadeDB arm first, the other engines already in the harness second, and new versions and new arms last; `--rep-pass N` runs repetitions 1 to N of every cell before the rest (section 7, row 45; DECISIONS #163 to #165). Each stage waits on its predecessor's ` ALL-DONE` marker in `~/STATUS.txt`; the scripts are installed in the bench host's home, not the repo. Which stage is running is state, not method: it is in `STATUS.txt`'s tail and in HANDOFF, never here. -The first stage's preflight is the record of what "ready" means here: it rebuilt `dbbench:arcadedb` with the -pinned wheel, read the version back OUT of the rebuilt image, and only then ran the pair check, which reported -`jars=image+wheel+embedded-image` where the same check reported `jars=both` an hour earlier and was wrong. The -first rows confirm it end to end: `instrument 2026-10`, `engine_commit 417314c18`, and BOTH ArcadeDB arms on one -engine line (embedded `26.10.1.dev0`, served `server:26.10.1-SNAPSHOT (build 417314c18...)`), which is the -defect of BUGS F69 verified fixed on real data rather than in principle. - -**The October chain on mini, at pin `417314c18`.** The stages `make_october_stages.py` generates from one table, plus the hand-written `qOL` and `qOM`, each waiting on its predecessor's ` ALL-DONE` marker in `~/STATUS.txt`: +**The October chain on mini, at pin `417314c18` (superseded: the 26.10.1 measurement replaces it table by table, DECISIONS #129 and #160).** `make_october_stages.py` generated these stages from one table, plus the hand-written `qOL` and `qOM`, each waiting on its predecessor's ` ALL-DONE` marker in `~/STATUS.txt`: `qOA` -> `qOA2` -> `qOB` -> `qOC` -> `qOD` -> `qOI` -> `qOD2` -> `qOA3` -> `qOD3` -> `qOA4` -> `qOE` -> `qOF` -> `qOG` -> `qOH` -> `qOB2` -> `qOJ` -> `qOK` -> `qOL` -> `qOM` -> `qOA5` -> `qOA6` -> `qOA7`. -**`qOA6` was stopped before it started** (2026-10-02, DECISIONS #134): its process was killed in its wait loop, before its exit trap could write a marker, so the chain ends with `qOA5`'s ALL-DONE. It stays in `make_october_stages.STAGES` and in the chain above, as `qOB` does, because it was generated and queued; the SurrealDB lifecycle cells it would have re-run are re-run in the 26.10.1 measurement. - -**`qOA7` re-runs ArangoDB's relaxed cross-model rows with its `pid` index** (2026-10-03, BUGS F168, the user's go-ahead the same day): drawn after `qOA6` because the generator lists it there, it waits on `qOA5`'s ALL-DONE, not on `qOA6` (which never writes one), runs the relaxed class of `e2` hybrid for `arangodb_e2` alone at both catalog sizes (10 cells), and its rows supersede the 20 pre-index relaxed rows under the canonical key, so /next's withdrawn ArangoDB row returns at its landing. - -**`qOK` re-runs the deployment decomposition** because its rows record no durability, and F10 fails any 2026-10 row that does not. - -**`qOJ` re-runs the time-series lane** because its rows are stale: `qOC` measured it with DuckDB holding no index while every other arm on the lane reached a host's points through one, so DuckDB's two host-filtered queries read every row of every host (BUGS F98, FAIRNESS F14). It runs after `qOH` and `qOB2` rather than next because the chain is serial by F2 and a stage beside a running one would break the invariant the campaign rests on; the `e4` table can land before it finishes, the time-series table waits for it. Its rows will also be the first to carry `index s` beside `ingest total s`. - -**Read this order off `make_october_stages.STAGES`, not off the chain drawn above** (BUGS F92); `structure_check.py` compares the two. - -**Every page table needs a stage that produces it, and E4 did not have one.** The live page carries the deployment decomposition and `export_web` refuses to build a page without `e4decomp_`, but no October stage ran the `e4` lane and no archived script ever had -- September's artifact was made by hand. `qOI` is that stage. Before a campaign, walk the payload's table list and name the stage that feeds each one (BUGS F84). +`make_october_stages.py` is still the template `make_2610_stages.py` reuses, which is why `structure_check.py` keeps holding this drawing against `make_october_stages.STAGES`. **Read an order off that list, not off the chain drawn above** (BUGS F92). `qOA6` was stopped before it started (DECISIONS #134), and stays in the list and the chain because it was generated and queued; the SurrealDB lifecycle cells it would have re-run are re-run in the 26.10.1 measurement. -**Every stage checks that its images carry their pins before it runs (BUGS F90, F76).** The build step and the existence check that follows it both passed for nine days while `dbbench:pg-age` sat on PostgreSQL 17.11 under an instrument pinned to 18.6: "images present" is true of an image built a week before the pin moved. It was not a labelling problem. AGE 1.8.0 builds edges about 65x faster than 1.7.0, so the cross-model cell at `e2_500k` spent its entire 7,200 s cap on edge creation, never reached the HNSW build, and was recorded as a censoring that reads like an engine limit. The ArcadeDB arm has been verified by content since August -- `verify_pair_c25.sh` reads the jar's own build number -- and `version_pin_check --runtime` now gives every comparator the same treatment, running each `dbbench:*` image and reading its versions back out before cell 1. An image that is absent, or that cannot state its versions, is a failure and not a pass; pins it cannot resolve (`PKGS[arcadedb]` is a shell variable) are reported rather than skipped silently. +What the October chain taught, kept as rules: -**Every stage checks that its whole cell fits before it runs (BUGS F74, #109).** The budget sum above bounds what ITERATIONS may cost; it cannot see a query's FIRST TOUCH, which #82b makes always complete. That is the gap that censored October's first analytics cell at its 7,200 s cap while its budgets summed to 48% of it, because one query spent 1,384 s on its first touch alone. Each stage now runs `cell_cost_check` for every tier it will use -- ingest plus measured first touches plus the budget ceiling, against the cap -- and aborts only on a SCORED projection over the cap. A tier whose query roster has no rows yet is reported and not blocked, because the tool refuses to score what it cannot see rather than guessing (it once scored 5 of 14 queries and reported ok). +**Every page table needs a stage that produces it** (BUGS F84). Before a campaign, walk the payload's table list and name the stage that feeds each one: the e4 decomposition had none, because `export_web` refuses a page without `e4decomp_` and September's artifact was made by hand. `make_2610_stages.check_coverage()` now proves it for every registered arm of every page lane, and the e4 decomposition has its own stage. -**Every stage with a timed write runs it twice, once at each durability class (#90, FAIRNESS F10b).** `make_october_stages.STRICT_WORKLOADS` (derived from `fairness_check.WRITE_CELLS`) names the three (lane, workload) pairs that get the second pass: documents and graph `oltp`, and cross-model `hybrid`; `atomicity` is not a timed write and stays at one class. Reads are untouched and bulk ingest stays at one setting (#90a). THE CLASS IS A CLI FLAG, NOT AN ENVIRONMENT VARIABLE: `runner.py` assigns `os.environ["BENCH_DURABILITY"]` from `--durability`, default `relaxed`, so exporting the variable is overwritten before any lane reads it and the cell runs relaxed while claiming nothing (BUGS F81). qOA3 and qOD3 add the pass for the two stages that ran before this landed; they are queued early because fairness_check F10 blocks the landing of the tables they feed. +**Every stage checks that its images carry their pins before it runs (BUGS F90, F76).** "Images present" is true of an image built a week before the pin moved: `dbbench:pg-age` sat on PostgreSQL 17.11 under an instrument pinned to 18.6 for nine days, and AGE 1.8.0 builds edges about 65x faster than 1.7.0, so the cross-model cell at `e2_500k` spent its entire 7,200 s cap on edge creation and was recorded as a censoring that reads like an engine limit. The ArcadeDB arm is verified by content (`verify_pair_c25.sh` reads the jar's own build number), and `version_pin_check --runtime` gives every comparator the same treatment: it runs each `dbbench:*` image and reads its versions back out before cell 1. An image that is absent, or that cannot state its versions, is a failure and not a pass; pins it cannot resolve (`PKGS[arcadedb]` is a shell variable) are reported rather than skipped silently. -**A repair runs directly after the stage it repairs**, not behind the long stages queued after it: a repair queued behind them leaves its table unlandable for all of them, and the campaign is sequenced by uncertainty, where an arm known to be broken is the most certain thing on the board to fix. +**Every stage checks that its whole cell fits before it runs (BUGS F74, #109).** The budget sum bounds what ITERATIONS may cost; it cannot see a query's FIRST TOUCH, which #82b makes always complete, and one query spent 1,384 s on its first touch alone while its budgets summed to 48% of the cap. Each stage runs `cell_cost_check` for every tier it will use (ingest plus measured first touches plus the budget ceiling, against the cap) and aborts only on a SCORED projection over the cap. A tier whose query roster has no rows yet is reported and not blocked, because the tool refuses to score what it cannot see rather than guessing (it once scored 5 of 14 queries and reported ok). -The repairs in that line are `qOA2` (DuckPGQ, after a duplicate class shadowed the working one, BUGS F73), `qOB2` (graph analytics re-run under #109's tier exclusions) and `qOD2` (the two cross-model arms that ran against images nobody was rebuilding, BUGS F76). A repair is a new stage at the tail, never an edit to a running one: bash reads a script by byte offset, so a mid-run edit resumes mid-token, and a stage waiting in its marker loop can be replaced only by killing its process group and relaunching it. +**Every stage with a timed write runs it twice, once at each durability class (#90, FAIRNESS F10b).** `make_october_stages.STRICT_WORKLOADS` (derived from `fairness_check.WRITE_CELLS`) names the three (lane, workload) pairs that get the second pass: documents and graph `oltp`, and cross-model `hybrid`; `atomicity` is not a timed write and stays at one class. Reads are untouched and bulk ingest stays at one setting (#90a). THE CLASS IS A CLI FLAG, NOT AN ENVIRONMENT VARIABLE: `runner.py` assigns `os.environ["BENCH_DURABILITY"]` from `--durability`, default `relaxed`, so exporting the variable is overwritten before any lane reads it and the cell runs relaxed while claiming nothing (BUGS F81). -**A stage builds the images its backends name (#110).** `_images_for()` reads `image` and `server_image` off each backend in `runner.BACKENDS`, so a new arm brings its image with it; the stage then verifies each one exists before the first cell. The typed list this replaced had gone stale in three places at once -- `dbbench:mongo-search` never built, `pg-age` and `dense` predating the re-pin -- and none of the three announced itself as a missing image. MongoDB's arm reported `server_not_ready`, which reads like a slow engine. +**A repair runs directly after the stage it repairs**, not behind the long stages queued after it: a repair queued behind them leaves its table unlandable for all of them, and the campaign is sequenced by uncertainty, where an arm known to be broken is the most certain thing on the board to fix. A repair is a new stage at the tail, never an edit of a running one (section 5 says why). -## 7. October: the skeleton, then the comparators, then one switch +**A stage builds the images its backends name (#110).** `_images_for()` reads `image` and `server_image` off each backend in `runner.BACKENDS`, so a new arm brings its image with it; the stage then verifies each one exists before the first cell. The typed list this replaced had gone stale in three places at once, and none of the three announced itself as a missing image: MongoDB's arm reported `server_not_ready`, which reads like a slow engine. -**Skeleton first, on the laptop.** Once the instrument branch is verified arm by arm, the whole of it runs on the laptop at micro and sweep scales, one repetition, every lane and every backend, into its own results file, and publishes to the preview route as the October page's shape: every table, every column, every condition, and every prose slot, with placeholder numbers, weeks before mini measures anything (DECISIONS #86). `refresh_web_page.py --skeleton` implies `--preview`, refuses any row from the bench host or at paper tier, stamps the payload and every table's conditions as placeholders, waives F1 and F3 by name in the payload because both describe the bench host, and runs every other gate exactly as October will. A live publish refuses a payload stamped skeleton. Nothing from the skeleton is merged into the campaign's results, and the first real stage overwrites the preview payload. +## 7. The re-pin checklist, and the rules a campaign taught -**Harness fixes that land INSIDE this campaign (2026-09-23).** Four adapter defects were fixed after their lanes were measured, and each rides a stage that re-measures that lane anyway, so none splits rows and none waits for the re-pin: - -* **l4, via `qOJ`** (already queued to re-run l4 for F98's DuckDB index): F115, the time-series document arm loading a row at a time while every comparator used its engine's bulk loader -- 2.24x on ingest, now `insert_many`. -* **e2, via `qOL`** (hand-written, waits on `qOK`, refuses to start unless all three are in the tree it pulls): F112 the post-filter (published `filtered recall@10` 0.013 against everyone else's 1.000), F113 the write-ahead log off during the bulk load, and F116 the build handing JPype Python lists instead of Java arrays (4.29x on the vertex load). The three were verified together on the pin's own engine after syncing upstream: `recall_filtered` 1.0, no durability relaxation, and the ingest/index timers still partition their total. - -**Three more, 2026-09-26 (DECISIONS #117), all against a COMPARATOR, found by the re-pin binding audit:** - -* **e2, via `qOM`** (hand-written on mini, waits on `qOL`, refuses to start unless the three fixes are in the tree and `dbbench:composed` carries Qdrant 1.19.1): F132, SurrealDB's candidate ranking scanned every product instead of addressing record ids (442 ms published, 15.7 ms fixed on the laptop); F133, the composed stack's vector half was the Qdrant client's pure-Python local mode, now the pinned Qdrant server in one container with Neo4j (`Dockerfile.composed`); F134 for the served e2 arm (the build waits until a KNN probe's latency settles, `surreal_common.await_hnsw_settled`, as on the dense lane); the composed image runs Qdrant with `RUN_MODE=production` as its own image does (`6c6c5102e8`; without it Qdrant bound to localhost). Re-runs SurrealDB embedded at e2, SurrealDB served at both sizes, the composed stack at both sizes with its atomicity trial, REPS=5 (`d3f790fa84`). Until it lands, `/next` withholds SurrealDB's graph-filtered search and its served e2_500k retrieval and withdraws the composed stack's e2 row (its atomicity row stays). - -**The re-pin checklist: everything the next measurement owes the harness, not just the engine (kept current; last swept 2026-10-02; the user decided items 1, 2, 3, 4, 6, and 7 in DECISIONS #116, items 12-15 in #119-#121, row 23 in #125, row 24 in #126, row 26 in #128 with its timing in #131, rows 27-29 and 37-43 in #131, row 44 in #132; row 25 is BUGS F163).** This is the ONE list of re-pin changes; HANDOFF points here rather than keeping a copy (HANDOFF keeps only the chores after the chain, which change no instrument). Nothing on it changes the running chain: each item touches a lane already measured, and a mid-run change splits the rows. Every item is settled before the first cell of the next measurement (step 1 of the routine). An item that moves ArcadeDB's numbers is the user's call (the "free for them, decisive for us" rule). Once decided, it becomes a DECISIONS entry naming what it amends or retires, changes PROTOCOL.md, FAIRNESS.md, and the method pages in the same commit, and leaves this table. **Scope** says whether an item fits a re-pin, which re-measures only ArcadeDB's arms and carries the comparators forward (DECISIONS #42), or changes comparator arms too, which then need re-measuring with it or wait for a full campaign. +**The re-pin checklist: everything a measurement owes the harness, not just the engine (last swept 2026-10-06).** This is the ONE list of re-pin changes; HANDOFF points here rather than keeping a copy. Each row names the decision behind it. A row that is built and verified is condensed to a line that keeps its number, because code comments, tests, and DECISIONS cite rows by number: never renumber, reorder, or delete one. An open row, or one that carries an instruction a future run needs, stays whole. Nothing on the list changes a running chain, because a mid-run change splits the rows; every item is settled before the first cell of the next measurement (step 1 of the routine). An item that moves ArcadeDB's numbers is the user's call (the "free for them, decisive for us" rule). Once decided, an item becomes a DECISIONS entry naming what it amends or retires, changes PROTOCOL.md, FAIRNESS.md, and the method pages in the same commit, and is condensed here once built. **Scope** says whose numbers an item moves: ArcadeDB's alone, or comparators too, which then need re-measuring with it. | # | change | scope | status | evidence | |---|---|---|---|---| -| 1 | Both graph arms load through ArcadeDB's bulk path: `graph_batch(use_wal=True, expected_edge_count=N)` embedded, `POST /api/v1/batch?wal=true&expectedEdgeCount=N` served, the WAL kept on | ArcadeDB only | **decided 2026-09-26 (DECISIONS #116): adopt at the re-pin.** Recommended by the maintainers (ArcadeData/arcadedb#8287). Warm gap 3.2-4.0x embedded, 6-7x served. **Prototyped** (2026-09-24): laptop micro smoke, every answer digest identical to the current loaders; the message half still needs a smoke on the bench host (LDBC corpus) | BUGS F123 | -| 2 | Every timed statement binds its values: the graph template, the served documents CRUD, the cross-model transaction, the served dense search | **comparators too**: every Cypher engine on the graph table, SurrealDB, PG+AGE | **decided 2026-09-26 (DECISIONS #116): adopt at the re-pin, on EVERY engine**, not ArcadeDB alone (recommended on ArcadeData/arcadedb#8132, #8286; binding is every engine's recommended path, so ArcadeDB-only would tilt the comparison). **Prepared on `repin-prep` (2026-09-26):** l1tpc served and SurrealDB OLTP (`2fb1f2b10c`); the graph template on every engine that can bind (`d124332a6a`: ArcadeDB x2, Neo4j, Memgraph, FalkorDB, LadybugDB prepared once per text, SurrealDB with RecordID; 9 engines, all 8 digests identical; DuckPGQ cannot take a parameter in any GRAPH_TABLE query, cwida/duckpgq-extension#75, and says so on its class); e2 (both ArcadeDB arms, SurrealDB, PG+AGE's Cypher half), the dense deletes and SurrealDB's query vector, and the served mutate insert: written, smoke running. Found on the way: F130 (SurrealDB graph write was two transactions), F131 (ArcadeDB's e2 update one statement per product; served, one round trip each). **Not affected:** the time-series lane (constants, the text repeats) and the lifecycle lane (every timed statement is the first after an open, parsed whatever its form; its probe values are constants). **Two exceptions, to be disclosed under their tables at the re-pin (not yet in the page generator):** DuckPGQ binds nothing inside `GRAPH_TABLE` (cwida/duckpgq-extension#75), and PG+AGE keeps its pid LIST in the text because a bound list loses the expression index (reported as apache/age#2582, 2026-09-27; bind it once a pinned AGE plans a list parameter as `= ANY`) | BUGS F114, F125; the two bullets below | -| 3 | Dense ingest through `graph_batch(use_wal=True)` with Java float arrays | ArcadeDB only | **decided 2026-09-26 (DECISIONS #116): adopt at the re-pin** (1.4x). **Prepared on `repin-prep`** (laptop, 20k: ingest 1.84 -> 1.46 s, recall level; the ingest sentence and its pin follow) | DECISIONS #116 item 3 (the vector-lane audit it settled is `.notes` commit `b1c4e8c`, HANDOFF before its 2026-09-29 sweep) | -| 4 | A served bulk path for DOCUMENT and vector types (served document and vector arms load through `sqlscript` with the values in the text; `/batch` is vertex/edge JSONL). Measured 2026-09-25 on the laptop at `5f8b817097`, four paths, WAL on in all: plain rows 13k rows/s `sqlscript`, 21k bound `INSERT ... CONTENT :rows`, 12.6k Postgres wire `executeBatch`, 20.6k gRPC `BulkInsert`; with a 96-float vector 4.8k, 8.3k, 12.1k, 7.9k | ArcadeDB only | **answered on #8337 (2026-09-25):** documents through HTTP `INSERT INTO T CONTENT :rows` with the batch bound as one parameter (gRPC `BulkInsert` equally fine); rows with a vector through the Postgres wire with `float4[]` (no packed vector encoding on HTTP or gRPC yet, tracked upstream as an enhancement); try 2k/5k/10k batches on the bench host and keep the best per arm (2k-5k for vector rows); all four paths keep the WAL on. **Decided 2026-09-26 (DECISIONS #116):** at the re-pin, switch the served document arms and the served vector arms and sweep the batch size on mini. **Amended the same day:** the served VECTOR arms use `CONTENT :rows` too, because from Python it beats the Postgres wire (laptop, 50k x 96: 4,980 rows/s against psycopg's 3,990; #8337's 12.1k Postgres figure is from Java's driver). Prepared on `repin-prep`: l1tpc served loader (`54f9f22da0`, smoke: digests identical, ingest 52.1 -> 43.4 s at SF0.1), l3d served build and the ingest sentences (`34484cd66d`, smoke: recall 1.0 both, 2.71 -> 1.36 s). Also the sparse served build (`CONTENT :rows`, on `repin-prep`; `a6006bc51f`, its NumPy pass removed in `fe74bf0392`, BUGS F151). Measured 2026-09-28 on the laptop, BigANN 100k, `c44f6604e8` pair: served build 56.3 / 62.0 s against October's loader 63.2 / 70.0 s (embedded 23.6 / 26.4 s), query time unchanged (one server, both loaders, interleaved: 15.1 against 15.4 ms p50), and every sampled weight stored exactly, where October's 9-decimal text is off on 80 of 101 records (F152). e2's served graph load also goes through `/batch` now (`569402dca8` on `repin-prep`: build 40.5 -> 17.4 s at 50k, digests equal the embedded arm's, the embeddings exact where sqlscript sent six digits). Item 4 is fully prepared. The lifecycle lane's served loader is untimed setup (no published column reads it), so it stays. Fairness: every comparator already loads through its vendor's recommended bulk path (COPY, insert_many, import_bulk), so this gives ArcadeDB's served arms parity, not an edge | `.notes` repros/served-doc-ingest | -| 5 | The strict durability setting: ArcadeDB `txWalFlush=2` is an fsync per commit, SQLite `synchronous=FULL` an fdatasync, which is `txWalFlush=1` | ArcadeDB only | **decided 2026-10-04 (the user, #138): adopt `txWalFlush=1` for ArcadeDB's strict class at the re-pin, disclosed on the page**; the re-pin's laptop smoke checks the row records 1. **Built on `repin-prep-2`, 2026-10-06** (`bench_common.ARCADE_STRICT_TX_WAL_FLUSH` = 1: the embedded JVM argument, the async executor's `yes_nometadata`, the served `JAVA_OPTS`, the read-back, the strict string and its classification, with the October string kept for the rows measured at 2; `test_strict_durability.py`; PROTOCOL section 7 has the row; the laptop smoke that the row records 1 is in REPIN-REHEARSAL) **Rehearsed 2026-10-06** through the runner at micro on both ArcadeDB documents arms: the strict rows record `txWalFlush=1` and the relaxed ones `txWalFlush=0`, and the answers agree across the classes | found 2026-09-24, straced on the laptop: `txWalFlush=2` is `force(true)`, one fsync per commit; SQLite `synchronous=FULL` in WAL mode one fdatasync; `txWalFlush=1` is `force(false)`. On the laptop NVMe all three cost ~6 ms a commit; on mini the documents table has ArcadeDB strict at ~8 ms against SQLite's 2.5, unexplained and not probed mid-campaign. Pick the value that matches SQLite's sync by effect; the knob-free-for-them rule keeps 2 unless 1 and 2 measure as equivalent guarantees. Settled by upstream's own docs (`concepts/transactions.adoc`, docs commit c52695ba66): `1` is "Safe against power loss. Recommended for production." and `2` has "No additional recovery value over `1`: `fdatasync` already flushes the metadata needed to read back the appended records (POSIX). Benchmarks show no measurable performance difference either." The server's production mode defaults to 1 (`ArcadeDBServer.java` L397-L411 at 417314c18). 1 is the setting matched by effect to SQLite's `synchronous=FULL` in WAL mode, so it is not the flattering choice the knob-free-for-them rule guards against | -| 6 | Served SQL arms over the Postgres wire (psycopg) instead of HTTP (`requests`) | ArcadeDB only | unblocked: #8288 fixed in ArcadeData/arcadedb#8308, and pgjdbc's default prepare in #8315 (#8244), both merged for 26.10.1 and verified; **decided 2026-09-26 (DECISIONS #116): measure on the bench host at the re-pin and switch if it is the faster read path** (the laptop cannot time below a millisecond; the maintainer already says it is, on #8332) | `.notes` repros/protocol-latency | -| 7 | What the memory column means: the peak, the working set after a full GC, or the peak with the heap printed beside it | every JVM arm | **decided 2026-09-26 (DECISIONS #116): the peak, what an operator has to provision, with the heap setting printed beside it for JVM engines**; no new instrumentation. **Prepared on `repin-prep`:** a `JVM heap` column beside the peak, from each cell's rows (dry landing: 4g/12g beside ArcadeDB and Neo4j on the graph table, every gate passing). Found on the way and fixed on main for October: the JVM memory note was missing from October's tables while the PostgreSQL one was there (BUGS F135). **At the re-pin, also read every generated sentence that mentions memory: none may phrase the column as efficiency ("uses less memory"), because a JVM's peak is the heap it was given; the like-for-like answer is the memory sweep below.** **The memory sweep, planned for after the chain and the re-pin (the user, 2026-09-27), is an instrument change:** it answers what the work needs rather than what an operator provisions, as the smallest memory each arm completes a cell with. First-cut design, settled before it runs: JVM arms (ArcadeDB both deployments, Neo4j) sweep the heap at a fixed cap; non-JVM arms sweep the container cap; "completes" means the same answer digest within a stated slowdown of the full-memory run; start with one or two tables (documents SF10, graph SF10) rather than the campaign; mini only, one pin. It costs days of bench time. Until it runs, no page sentence phrases the memory column as efficiency | DECISIONS #116 item 7 (the HANDOFF text it settled is `.notes` commit `b1c4e8c`) | +| 1 | Both graph arms load through ArcadeDB's bulk path: `graph_batch(use_wal=True, expected_edge_count=N)` embedded, `POST /api/v1/batch?wal=true&expectedEdgeCount=N` served, the WAL kept on | ArcadeDB only | **built (DECISIONS #116), in the 26.10.1 measurement**: `l2_graph.py`, both arms; recommended by the maintainers (ArcadeData/arcadedb#8287); every answer digest equals the old loaders' on the laptop smoke and in the SF1 rehearsal | BUGS F123 | +| 2 | Every timed statement binds its values: the graph template, the served documents CRUD, the cross-model transaction, the served dense search | **comparators too**: every Cypher engine on the graph table, SurrealDB, PG+AGE | **built (DECISIONS #116), in the 26.10.1 measurement, on every engine that can bind** (recommended on ArcadeData/arcadedb#8132 and #8286; the adapters in `graph_common.py` and the lanes show which). **Still owed: the page generator does not yet disclose two exceptions under their tables.** DuckPGQ binds nothing inside `GRAPH_TABLE` (cwida/duckpgq-extension#75, said on its adapter class), and PG+AGE keeps its pid list in the text because a bound list loses the expression index (apache/age#2582; bind it once a pinned AGE plans a list parameter as `= ANY`). Not affected: the time-series lane (constants) and the lifecycle lane (every timed statement is the first after an open) | BUGS F114, F125, F130, F131 | +| 3 | Dense ingest through `graph_batch(use_wal=True)` with Java float arrays | ArcadeDB only | **built (DECISIONS #116; 1.4x on the laptop)**: `l3d_dense.py` | DECISIONS #116 item 3 | +| 4 | A served bulk path for DOCUMENT and vector types: the served document and vector arms load through `INSERT ... CONTENT :rows` with the batch bound as one parameter, in place of `sqlscript` with the values in the text | ArcadeDB only | **built (DECISIONS #116, the maintainers' answer on ArcadeData/arcadedb#8337)**: `l1_tpc.py` (`ArcadeServerTPC.load_batch`, 2,000 rows a request, `BENCH_SERVED_LOAD_BATCH`), `l3d_dense.py`, `l3_sparse.py`, and e2's served graph load through `/batch`; the WAL stays on in every path, and every comparator already loads through its vendor's bulk path, so this is parity, not an edge. **No batch-size sweep is recorded** (the maintainers suggested 2k, 5k, and 10k on the bench host, keeping the best per arm) | `.notes` repros/served-doc-ingest; BUGS F151, F152 | +| 5 | The strict durability setting: ArcadeDB `txWalFlush=2` is an fsync per commit, SQLite `synchronous=FULL` an fdatasync, which is `txWalFlush=1` | ArcadeDB only | **built (the user, DECISIONS #147; own issue #138)**: `bench_common.ARCADE_STRICT_TX_WAL_FLUSH` = 1 for the embedded JVM argument, the async executor's `yes_nometadata`, the served `JAVA_OPTS`, the read-back, and the strict string (the October string is kept for rows measured at 2); `test_strict_durability.py`; PROTOCOL section 7 has the row. Upstream's docs call `1` "recommended for production" and say `2` adds no recovery value, and `1` is matched by effect to SQLite's `synchronous=FULL`, so it is not the flattering choice the knob-free-for-them rule guards against | `concepts/transactions.adoc` in upstream's docs; strace of the three settings on the laptop, 2026-09-24 | +| 6 | Served SQL arms over the Postgres wire (psycopg) instead of HTTP (`requests`) | ArcadeDB only | unblocked: #8288 fixed in ArcadeData/arcadedb#8308, and pgjdbc's default prepare in #8315 (#8244), both merged for 26.10.1 and verified; **decided 2026-09-26 (DECISIONS #116): measure on the bench host at the re-pin and switch if it is the faster read path** (the laptop cannot time below a millisecond; the maintainer already says it is, on #8332). **Still open:** the served arms read through the lean HTTP client (row 72) now, and the comparison on mini is still to run (DECISIONS #162 item 3) | `.notes` repros/protocol-latency | +| 7 | What the memory column means: the peak, the working set after a full GC, or the peak with the heap printed beside it | every JVM arm | **decided (the user, DECISIONS #116 item 7): the peak, what an operator has to provision, with the heap setting printed beside it for JVM engines; built** (a `JVM heap` column from each cell's rows in `export_web`; the JVM memory note on every table that shows a JVM arm, BUGS F135). **No generated sentence may phrase the column as efficiency ("uses less memory"): a JVM's peak is the heap it was given; the like-for-like answer is the memory sweep.** **Still open, the memory sweep (the user, 2026-09-27), planned after the chain and the re-pin; it is an instrument change:** it answers what the work needs rather than what an operator provisions, as the smallest memory each arm completes a cell with. First-cut design, settled before it runs: JVM arms (ArcadeDB both deployments, Neo4j) sweep the heap at a fixed cap; non-JVM arms sweep the container cap; "completes" means the same answer digest within a stated slowdown of the full-memory run; start with one or two tables (documents SF10, graph SF10); mini only, one pin; it costs days of bench time | DECISIONS #116 item 7 | | 8 | The sparse corpus generator yields NumPy, and each adapter converts in its own timer | comparators too (same work, moved) | measured 3.7% of ArcadeDB's build; do it with the next sparse measurement | bullet below | -| 9 | The re-pin build: 26.10.1 with every fix we filed verified by its repro (#8232, #8260/#8265/#8266 and the follow-up #8312, #8285, #8288, #8332, #8333, #8335, PR #8269; all verified on the laptop, the last three at `ab91649b43`; #8429 at `ff4e96b9ef` on 2026-09-27, whose workaround in `l1_tpc.py`, the served TPC-C scripts ending with their UPDATE, comes out with it), comparators to latest stable. Expect ArcadeDB's documents analytics and graph analytics to move at the re-pin: full scans 1.7-2.4x faster (#8312), Q1 5.4x and Q6 5.6x faster through the `l_shipdate` index at 4M rows (#8333, `repros/shipdate-index/verify-8339-*`), one-hop GAV aggregations 3-5x faster (#8335). **Expect some cells to get SLOWER, and say so:** the embedded cross-model strict cells, because an `LSM_VECTOR` build switched that thread's WAL off and those commits were never durable (#8129, fixed by #8221; the durability table already says why the cost read as zero). #8352 does not touch the harness (`txWalFlush` is set as a JVM property, not through the per-thread setter). **PR #8524 (`c44f6604e8`, fixing our #8523 and #8492, verified 2026-09-28 on both JDKs, laptop cores 0-3):** SQL scans with a WHERE clause and aggregations now run in parallel, single-bucket types included (page ranges), with per-worker partial aggregates; at 2M lineitem rows Q1 6.0 -> 1.3 s, Q6 1.55 -> 0.36 s, top parts 4.4 -> 1.2 s, the filtered sums 4-5x, on four cores (mini gives twelve, so expect more on `docs_olap`); a new-order's commit is 2.2x cheaper and the transaction 1.6-1.8x (`docs_oltp`). Every lane with a SQL scan moves with it, the graph analytics' SQL paths included. **The dense library moves too:** JVector 4.0.0-rc.9 -> 4.0.1 (`05cbdc20ad`, #8518, in the snapshot from 2026-09-28), under `LSM_VECTOR` on the dense and cross-model tables, so a move there at the re-pin can be the library's, not ours or a fix of ours; compare recall at the same settings before reading a latency change. **Ranks can hold while times drop:** item 2 binds values on every engine, and Neo4j gained 2-4x from it in the smoke. Laptop ratios at repro sizes, not projections of the page; only the re-measure decides. **Fixed 2026-09-28 evening, verified on the laptop before the re-pin (`.notes` repros/verify-2026-09-28.sh):** #8594 (PR #8599, a satisfied LIMIT releases its scan), #8576 (PR #8582: sparse searches rescore INT8/FP16 candidates from the records' own weights, `rescoreOversample` default 2, so int8 recall rises and upstream measured 10-20% more query time: the sparse table moves both ways), #8574, #8553, #8537 (PR #8586: time-series tag skipping and `COMPACT TIMESERIES TYPE`, the packed-key sparse cursor heap, edge-free Cypher hops), #8333's follow-up (PR #8602, index ranges served by the parallel scan). Also re-verify #7146, #7183, #7184, #7190 at 10M on the new pair (the orphan and rebuild-deferral fixes, last verified below 10M). **Fixed 2026-09-29 to 10-01 and expected to move published cells:** #8626 (lifecycle clean close), #8852 (lifecycle vector first search after reopen), #8797 and #8725 (Cypher aggregation and filtered label scans in parallel workers: graph analytics) | routine | steps 3 and 4 of the routine | `.notes` issues-trail | -| 10 | ArcadeDB's bucket count. The fairness rule runs every engine at its defaults, and ArcadeDB's type default is one bucket, so its SQL scans never run in parallel. Until ArcadeData/arcadedb#8265 that was harmless (four buckets made analytics 24-35% slower); with #8265 merged, four buckets made a full-scan `GROUP BY` about 1.4x faster than one on 4 cores; with #8312 merged too (both in 26.10.1), one bucket caught up and four gain only 1.01x to 1.09x (laptop, 2M rows, `5f8b817097`; a filtered `count(*)` gains nothing either way) | ArcadeDB only | **The query side is ANSWERED by PR #8524 (`c44f6604e8`): a single bucket is split into page ranges across the workers, so four buckets gain little (Q1 1,307 -> 1,148 ms at 2M rows, laptop) and the types stay at the engine default, which is also what #8523's reply told upstream.** Before that: asked upstream on #8260 (2026-09-25 00:04Z, then told them it is low priority after #8312); re-measure on the bench host at the re-pin and keep the default unless the gain comes back; tuning ArcadeDB alone is the user's call. **The INGEST side (BUGS F141, 2026-09-27):** the documents lane loads one thread into the one default bucket; async `createRecord` gains nothing there and 2.5x at 8 buckets (Java, current snapshot, reads unchanged), 4.2x through the package. Asked which path and bucket count upstream recommends on #8478; **answered 2026-09-28** (async `createRecord`, `BUCKETS` = k x the executor's parallel level with k 1 or 2, set at `CREATE TYPE`, the executor's own WAL flush matched to the class, indexes after the load). **Prepared on `repin-prep` (`077e6deee9`), NOT adopted:** laptop SF0.1, 3 writers, `c44f6604e8`: ingest 1.6x faster at both classes with every digest identical, but a control on the same harness with only the bucket count varied puts the five analytics queries 17-47% slower at 3 buckets than at 1, against the answer's "no longer costs on scans". **Java repro, 2026-09-28 (`.notes` repros/doc-ingest-paths/BucketScanCost.java, 2M rows, 4 cores):** async load 30.0k -> 80.2k rows/s at 3 buckets; the five queries without an index within 7% of one bucket at 3 buckets, 15-29% slower at 8. So k = 1, and the lane's 17-47% is not the engine's until shown otherwise (it is not reproduced in Java; to be separated on mini at the re-pin with `BENCH_ARCADE_LINEITEM_BUCKETS=1` against the rule). Replied on #8478 once with all of it. Adopt at the re-pin if mini's SF1 and SF10 cells agree; the bucket count is stated on the page | `.notes` repros/groupby-scan, repros/doc-ingest-paths | +| 9 | The re-pin build: the official 26.10.1 with every fix we filed verified by its repro, comparators to latest stable (DECISIONS #129, relaxed by #163 to #165) | routine (steps 3 and 4 of the routine) | **built and verified.** The pin is the release's own jar commit `d36b4ca3a`; the fixes we filed (#8232, #8260, #8265, #8266, #8285, #8288, #8312, #8332, #8333, #8335, #8429, #8492, #8523 by PR #8524, #8537, #8553, #8574, #8576, #8594, and #8602; the full list is `.notes` issues-trail) were each verified by their repro on the laptop, and #8626 (lifecycle clean close), #8852 (lifecycle vector first search after reopen), and #8797 and #8725 (Cypher aggregation and filtered label scans in parallel workers) landed after and are expected to move published cells. **Read the rows with these in hand, and say so on the page:** some cells get SLOWER, the embedded cross-model strict cells (an `LSM_VECTOR` build switched that thread's WAL off, so those commits were never durable: #8129, fixed by #8221) and the sparse table's int8 arms (#8582 rescores candidates from the records' own weights, 10 to 20% more query time); the dense and cross-model tables moved with the vector library (JVector 4.0.0-rc.9 to 4.0.1, #8518), so compare recall at the same settings before reading a latency change; ranks can hold while times drop, because item 2 binds values on every engine (Neo4j gained 2 to 4x in the smoke). Laptop ratios are not projections of the page; only the re-measure decides. **Still owed:** re-verify #7146, #7183, #7184, and #7190 at 10M on the 26.10.1 pair (last verified below 10M); #7260, where the rebuild after an admitted one is deferred again at 10M, is fixed in 26.11.1 (PR #9289) and not in 26.10.1 | `.notes` issues-trail | +| 10 | ArcadeDB's bucket count. The fairness rule runs every engine at its defaults, and ArcadeDB's type default is one bucket, so its SQL scans never run in parallel. Until ArcadeData/arcadedb#8265 that was harmless (four buckets made analytics 24-35% slower); with #8265 merged, four buckets made a full-scan `GROUP BY` about 1.4x faster than one on 4 cores; with #8312 merged too (both in 26.10.1), one bucket caught up and four gain only 1.01x to 1.09x (laptop, 2M rows, `5f8b817097`; a filtered `count(*)` gains nothing either way) | ArcadeDB only | **The query side is ANSWERED by PR #8524 (`c44f6604e8`): a single bucket is split into page ranges across the workers, so four buckets gain little (Q1 1,307 -> 1,148 ms at 2M rows, laptop) and the types stay at the engine default, which is also what #8523's reply told upstream.** Before that: asked upstream on #8260 (2026-09-25 00:04Z, then told them it is low priority after #8312); re-measure on the bench host at the re-pin and keep the default unless the gain comes back; tuning ArcadeDB alone is the user's call. **The INGEST side (BUGS F141, 2026-09-27):** the documents lane loads one thread into the one default bucket; async `createRecord` gains nothing there and 2.5x at 8 buckets (Java, current snapshot, reads unchanged), 4.2x through the package. Asked which path and bucket count upstream recommends on #8478; **answered 2026-09-28** (async `createRecord`, `BUCKETS` = k x the executor's parallel level with k 1 or 2, set at `CREATE TYPE`, the executor's own WAL flush matched to the class, indexes after the load). **Prepared (`077e6deee9`; `BENCH_ARCADE_LINEITEM_BUCKETS` in `l1_tpc.py`), NOT adopted:** laptop SF0.1, 3 writers, `c44f6604e8`: ingest 1.6x faster at both classes with every digest identical, but a control on the same harness with only the bucket count varied puts the five analytics queries 17-47% slower at 3 buckets than at 1, against the answer's "no longer costs on scans". **Java repro, 2026-09-28 (`.notes` repros/doc-ingest-paths/BucketScanCost.java, 2M rows, 4 cores):** async load 30.0k -> 80.2k rows/s at 3 buckets; the five queries without an index within 7% of one bucket at 3 buckets, 15-29% slower at 8. So k = 1, and the lane's 17-47% is not the engine's until shown otherwise (it is not reproduced in Java; to be separated on mini at the re-pin with `BENCH_ARCADE_LINEITEM_BUCKETS=1` against the rule). Replied on #8478 once with all of it. Adopt at the re-pin if mini's SF1 and SF10 cells agree; the bucket count is stated on the page | `.notes` repros/groupby-scan, repros/doc-ingest-paths | | 11 | Every engine's index decision is re-measured at the size its table runs, not at a smoke size. The documents lane's `l_shipdate` index was given to ArcadeDB on a 6.1x gain at 500k rows; at SF10 (60M) ArcadeDB's Q6 scaled 93-100x and Q1 43x for 10x the data, and a laptop repro shows the index plan collapsing once the table outgrows the page cache (at 4M rows under a 256 MB cache: Q1 5x slower than a scan, Q6 1.4x slower) | **comparators too**: every engine's index set on every lane | open. The engine half is fixed upstream (ArcadeData/arcadedb#8333, PR #8339, in 26.10.1: past `arcadedb.queryIndexMaxSelectivity`, default 25% of the type, a range scans; below it, index hits load in physical order). Verified at 4M rows: Q1 with the index now within 10% of the scan (was 5x slower), Q6 2.4-2.6x faster than the scan (was 1.4x slower). So at the re-pin ArcadeDB's by-effect answer is likely to keep the index, at the engine default; re-measure it at SF10 like every engine's | `.notes` BUGS F126, repros/shipdate-index | -| 12 | The LDBC person's age: `ldbc_snb.gen_persons` reads the epoch-millisecond `birthday` of our `LongDateFormatter` corpus as a date string, so every age is 0 (BUGS F146). Parse milliseconds, and move `3-hop filtered` from `x.age > 30` (passes 100% once ages are right: they span 36-46) to `x.age > 41` (49.7% at SF1, 49.3% at SF10) so it filters what DECISIONS #82 says it filters | every engine on the graph table (same corpus, same query); changes the answers of `point`, `1-hop`, `friend_age_by_city`, and `3-hop filtered` | **decided 2026-09-28 (DECISIONS #119): parse milliseconds, threshold 41; October's page discloses the zero ages from the rows, the live September page is not touched.** Prepared on `repin-prep`; re-baseline the graph digests at the re-pin; October's `3-hop filtered` answers are all 0. **Expect SurrealDB embedded's `3-hop filtered` to get much slower, and project its SF10 cell against the budget before the stage:** its `array::distinct` runs on core 2.3.10 (the `surrealdb==2.0.0` SDK, PyPI's latest stable; 3.x is beta), whose `Array::uniq` removes each duplicate with `Vec::remove`, quadratic in the duplicates. October's filter kept no one, so the array was empty; keeping half, the laptop smoke's cold pass took about 1.3 s per start at SF1 against October's 0.19 s p50 on mini. Fixed in SurrealDB 3.x (a hash-set rebuild), so the served arm (3.2.4) is unaffected and there is nothing to file. On the laptop that SF1 cell then hit the one-hour cap with nothing recorded, which is item 14 | BUGS F146; mini's own SF1/SF10 `person_0_0.csv` | -| 13 | The cross-model embedded ranking statement passes its query vector as a Java `float[]`, as the arm's other vector calls already do (BUGS F145) | ArcadeDB embedded only (its served arm posts JSON) | **done on `repin-prep` (`aff821012f`)**, checked against brute force 50/50 | BUGS F145; `.notes` repros/filtered-search | -| 14 | The graph table's four transactional reads each get a per-read budget spanning both passes, so one slow read censors its own column instead of timing out the whole cell (DECISIONS #120); and the answer digest rounds floats twice (twelve significant digits, then six), because averages that are exact six-digit ties split on their last bit once real ages loaded (Neo4j's running mean against eight engines on `1-hop`) | every engine on the graph table (one budget per read, never per engine); the digest change touches every lane's float columns, and only at six-digit ties | **decided 2026-09-28 (user), prepared on `repin-prep`**: `OLTP_READ_BUDGET_S` 1,800 s clamped to 540 s at SF1, `record_censored_answer` and the gate's E4b, the page's per-engine budget sentence on the graph table, `test_result_digest` tie and declaration tests; committed as `58dac7226e`. Laptop smoke at SF1, ten engines: 0 equivalence failures, 8 groups agreeing (`3-hop filtered` on real, non-zero answers), SurrealDB embedded's `3-hop filtered` censored after 111 of 500 starts (p50 2.35 s) with its other reads and writes measured and the cell done in 996 s (it had timed out whole at 3,600 s) | DECISIONS #120; BUGS F146 | -| 15 | The time-series lane settles ArcadeDB's native arms by polling `SELECT FROM schema:types` until every shard's `mutableSamples` is 0 (bounded, outside the ingest timer), instead of relying on the fixed 90 s alone, and each row records the uncompacted count at its first timed query | ArcadeDB's two native time-series arms (the other engines keep their own settle; the symmetric 90 s floor stays for every arm) | **decided 2026-09-28 (user agreed, DECISIONS #121): NOT for qOJ** (October's rows show 90 s sufficed at both sizes: the served arm's newest reading 1.6-2.7 ms at 1,000 hosts, where uncompacted data at a tenth of the size costs 17-19 ms); **prepared on `repin-prep` (`382e808d01`)** for the next campaign, where it also covers a partial fix of #8574; laptop smoke at ts100 with the lane floor at 0: the served arm waited 55.5 s and the embedded 5.1 s, both at 0 uncompacted samples; it also ends the embedded arm's double sleep (it slept the lane's 90 s itself, then the driver slept it again). **Re-decide against #8586 (2026-09-28):** upstream added `COMPACT TIMESERIES TYPE `, which seals the tail on demand and returns `mutableSamples`, and made tag lookups on the uncompacted tail skip pages without the tag; the polling settle may become one statement (DECISIONS #121, amended). **Watch at the re-pin:** on the laptop the COMPACTED newest reading went 0.11 to 0.16-0.19 ms p50 on `77451baea7` (six interleaved pairs per JVM; reported on #8574 as a lead), and this lane publishes the compacted case, so the re-pin's `l4` row says whether it holds on the bench host | ArcadeData/arcadedb#8574; `.notes` repros/ts-last-served | -| 16 | The runner records a served cell whose server fails to start instead of crashing on it, and a cell that raises no longer ends its batch: `_therm0` is initialised at the top of `run_cell` beside `cli_cid`, and `worker()` catches an exception from a cell, prints it as `RAISED `, runs the rest of the batch, and exits 1 (BUGS F150) | every lane (failure path only; a cell whose client starts takes the same readings as before) | **prepared on `repin-prep` (`fe74bf0392`), NOT for October**: the campaign holds its runner, F150 has not fired on mini (no crash anywhere in STATUS.txt), the landing's coverage gate catches a lost cell, and `mini_watch.sh` reports `Exception in thread` and `RAISED` lines within its interval; laptop check: the JDK 21 cell that crashed now records `server_not_ready` and exits 1 | BUGS F150 | +| 12 | The LDBC person's age is parsed from the epoch-millisecond `birthday` (it was read as a date string, so every age was 0, BUGS F146), and `3-hop filtered` moves from `x.age > 30` to `x.age > 41` (49.7% at SF1, 49.3% at SF10) | every engine on the graph table (same corpus, same query); changes the answers of `point`, `1-hop`, `friend_age_by_city`, and `3-hop filtered` | **built (the user, DECISIONS #119), in the 26.10.1 measurement**: `ldbc_snb.py`, `graph_common.py`; every graph digest is re-baselined, and October's `3-hop filtered` answers are all 0. **Read SurrealDB embedded's `3-hop filtered` against its per-read budget (row 14):** core 2.3.10's `array::distinct` is quadratic in the duplicates, so with real ages it runs far slower than October's empty arrays did (fixed in SurrealDB 3.x; the served arm is unaffected) | BUGS F146 | +| 13 | The cross-model embedded ranking statement passes its query vector as a Java `float[]`, as the arm's other vector calls already do (BUGS F145) | ArcadeDB embedded only (its served arm posts JSON) | **done** (`aff821012f`), checked against brute force 50/50 | BUGS F145; `.notes` repros/filtered-search | +| 14 | The graph table's four transactional reads each get a per-read budget spanning both passes, so one slow read censors its own column instead of timing out the whole cell (DECISIONS #120); and the answer digest rounds floats twice (twelve significant digits, then six), because averages that are exact six-digit ties split on their last bit once real ages loaded | every engine on the graph table (one budget per read, never per engine); the digest change touches every lane's float columns, and only at six-digit ties | **built (the user, 2026-09-28)**: `graph_common.OLTP_READ_BUDGET_S` (1,800 s, clamped to 540 s at SF1), `record_censored_answer`, the gate's E4b, the page's per-engine budget sentence, `test_result_digest.py` | DECISIONS #120; BUGS F146 | +| 15 | The time-series lane settles ArcadeDB's native arms by polling `SELECT FROM schema:types` until every shard's `mutableSamples` is 0 (bounded, outside the ingest timer), instead of relying on the fixed 90 s alone, and each row records the uncompacted count at its first timed query | ArcadeDB's two native time-series arms (the other engines keep their own settle; the symmetric 90 s floor stays for every arm) | **built (the user agreed, DECISIONS #121)**: `l4_tsbs.py`; it also ends the embedded arm's double sleep. Upstream's `COMPACT TIMESERIES TYPE` (#8586) may turn the polling into one statement. **Check on the bench host:** on the laptop the compacted newest reading went 0.11 to 0.16-0.19 ms p50 on `77451baea7` (reported on #8574), and this lane publishes the compacted case, so the first `l4` rows say whether it holds | ArcadeData/arcadedb#8574 | +| 16 | The runner records a served cell whose server fails to start instead of crashing on it, and a cell that raises no longer ends its batch: `_therm0` is initialised at the top of `run_cell`, and `worker()` prints `RAISED `, runs the rest of the batch, and exits 1 | every lane (failure path only) | **built (BUGS F150)**: `runner.py`; the JDK 21 cell that crashed now records `server_not_ready` | BUGS F150 | | 17 | The documents lane's `l_shipdate` index, re-decided per engine by measurement (it was added "so Q1/Q6 filters avoid full scans"). On `c44f6604e8` (26.10.1-SNAPSHOT) ArcadeDB's planner serves a 96% range (Q1) with the index by #8339's full-scan branch, whose rows then go through projection and aggregation on the query thread, while without the index the aggregation runs in #8524's parallel scan workers: Q1 3.5x slower with the index, Q6 (14%) 1.3-1.4x slower; a 1.2% range still 10x faster with it | every engine of the documents lane (measure each with and without, the index-decisions rule) | **open, for the re-pin; answered upstream by PR #8602 (2026-09-28):** the full-scan and physical-order branches now run in the parallel scan's workers, projection and aggregation included, and `arcadedb.queryIndexMaxSelectivity` (default 0.6) is divided by (1 + W) / 2 for W workers; Luca measured Q1 with the index 643 -> 247 ms against 211 ms without on our repro, our verification is `.notes` repros/verify-2026-09-28.sh. Measure every engine with and without the index at the re-pin either way | ArcadeData/arcadedb#8333; `.notes` repros/index-vs-parallel-scan | -| 18 | The time-series driver holds its parsed corpus at about 89 bytes a point and frees it after ingest: it parsed the whole corpus into one list of 5-tuples before the ingest timer (on purpose, so no arm's `ingest_s` covers the parse) and kept it, 281 bytes a point, until the cell exited; `TS_CHUNK` and the arms' batch sizes only slice that list, so nothing builds up during ingest (BUGS F66)| every embedded arm of the time-series lane | **built on `repin-prep` (`5f61f963fe`, 2026-10-03):** `parse_lp` shares each distinct host, timestamp, and reading (ts100 695 -> 221 MiB, ts1000 7,305 -> 2,186 MiB held after the parse, laptop, RssAnon), `main()` frees the corpus after the ingest timer and the engine's settle, DuckDB unregisters its Arrow copy, the settle floor absorbs the release time, and the row records `corpus_release_s`. No timer, sent data, or answer changed: parse identical point for point on ts100 and ts1000, six digests equal before and after on SQLite, DuckDB, ArcadeDB (both arms), and SurrealDB embedded. Not run: the served arms, a full ts1000 cell. Re-run `ts1000` after it | `.notes` BUGS F66 | +| 18 | The time-series driver holds its parsed corpus at about 89 bytes a point and frees it after ingest, where it held 281 bytes a point until the cell exited (BUGS F66) | every embedded arm of the time-series lane | **built (2026-10-03)**: `l4_tsbs.parse_lp` shares each distinct host, timestamp, and reading (ts100 695 to 221 MiB, ts1000 7,305 to 2,186 MiB held after the parse), `main()` frees the corpus after the ingest timer and the engine's settle, and the row records `corpus_release_s`; no timer, sent data, or answer changed (six digests equal before and after on five arms). Not run: the served arms and a full ts1000 cell, so the first `ts1000` rows are its check | `.notes` BUGS F66; FAIRNESS.md F3 | | 19 | The disk column measures at the build point: read the size when the build timer stops, right after the engine's own settle step; a post-query reading is a second column; `settled=False` blocks publication; the settle budget exceeds a compaction cycle (today `tries=3, settle_s=3.0`, a 9 s window) | every engine's disk column | open, NOT adopted for October; the design cautions (a barrier-and-watcher deadlock, `LSMVectorIndex.compact()` a silent no-op unless a compaction is scheduled) are in PAGE-SPEC's disk section | PAGE-SPEC.md, disk | -| 20 | The fairness audits still owed: ask each JVM engine for its live heap rather than trusting the ENV we passed (Elasticsearch reports `jvm.mem.heap_max_in_bytes`); audit the thread pools of the comparators the first audit did not cover (PostgreSQL, MongoDB, TimescaleDB, QuestDB, pgvector, PG+AGE, SurrealDB, SQLite, ArangoDB); audit every engine's memory pools the same way, for a buffer pool, cache, or memory limit sized from the host's RAM rather than the cell's cap (the class of BUGS F160: LadybugDB's buffer pool and Memgraph's memory limit were, and are fitted now), the engines above and those of the first audit alike; whether any engine's disk IO scheduling differs under the same cap | every comparator | PostgreSQL + AGE audited and fitted on `repin-prep` (`49a81672dc`, 2026-10-02): both AGE arms get parallel workers per query = cpuset - 1 and `work_mem` = (cap x 0.75) / (cpuset x 16), measured on the full SF1 network (LSQB q2 1.8x faster with the workers; temp-file spill 42 GB -> 0.7 GB with the work_mem; no OOM kill; same answers); FAIRNESS F6 carries the row. The same fit is owed to `postgres_tuned`, TimescaleDB, and pgvector at the next campaign (the `postgres` arm stays at PostgreSQL's defaults by design: it is the defaults arm). The other audits stay open | FAIRNESS.md; repros/age-dialect/resource_fit_probe.py | -| 21 | Every override is disclosed where a reader meets it: the rows marked **NOWHERE** in PROTOCOL section 7 get a row field or a page sentence, and the manifest records engine configuration, not just cpuset, memory and images | every lane with an override | **built and merged into `repin-prep` (2026-10-03, `4492066280`; reviewed: the lane adapters read back only at connect or after the build timer, the 83 tests pass, and one in-timer string copy in `ArcadeServer.command` was cut); it reaches `main` with the re-pin, because `fairness_check` F15 fails every pre-stamp row and would turn October's landings red.** `overrides.py` registers the nine overrides a table shows (Elasticsearch security off and no replica, DuckDB's thread pool and its experimental HNSW persistence, Neo4j's page cache and checkpoint interval, ArcadeDB's `addHierarchy`, the served ArcadeDB's query-memory limit, and the native time-series ingest timer's stop at acceptance). Each has a row field read back from the engine (the served ArcadeDB's limit is asked of the engine over HTTP by the runner; the served `addHierarchy` has no read-back at this pin and records the request, naming it as one), a generated sentence under every October table that shows the arm, `fairness_check` F15 (the stamp), a `page_check` section (the sentence), and `test_overrides.py`, which reads PROTOCOL.md section 7 and fails if a row says NOWHERE. The runner manifest gains `engine_config` (each backend's launch configuration and the registered overrides in force) and `runner_env`. Rows measured before the stamps existed fail F15 by design, as a `durability`-less row does, so merging before the re-pin would turn the October landings red. Not done: the page's overrides table with a column for which way each override moves a number (it needs measurements), and the release asset's manifest does not carry the table | PROTOCOL.md section 7; FAIRNESS.md F15 | -| 22 | The lifecycle lane reads the vector index's statistics in one extra untimed cycle after the timed ones, not inside every timed session: `l5_lifecycle.cycle()` calls `_vector_stats()` between the action timer and `close()`, and at the October pin that read raises the close it precedes, so the embedded dense-vector "open and close" cell is not a session that touched nothing | ArcadeDB embedded lifecycle, vector situation only | open; the lifecycle landing (after qOA6, BUGS F161) discloses it under the table (export_web `vector_stats`, no number) and the sentence retires with this row | BUGS F159: laptop, 1M vectors, clean close 6.4 -> 15.3 ms with the read before #8630 (`de966e79c5`), 3.0 -> 3.5 after (`b8ab990821`); mini's pin reads 88.5 ms and its split is not measured; `.notes/bench/repros/vector-clean-close/` | -| 23 | LadybugDB's graph arm runs with its thread pool and buffer pool fitted to the cell (`max_num_threads` from `sched_getaffinity`, `buffer_pool_size` at 0.8 of the cgroup limit), both on the row | LadybugDB, the graph lane (OLTP and analytics) | **landed inside this campaign (DECISIONS #125):** `qOB2` pulls it, and `qOA5` re-runs the OLTP cells after `qOM`, superseding qOA/qOA3's unfitted rows | BUGS F160, FAIRNESS F6's LadybugDB row, LadybugDB/ladybug#1070 | -| 24 | The SurrealDB lifecycle arm streams its build input in batches of `INGEST_BATCH` (`_batched`), as the ArcadeDB arm streams row by row; it built each situation's whole input as one Python list first, which sat in SurrealDB's published peak memory at every size (a third of doc's at 1M, three fifths of vector's) and alone filled the 28g envelope at 10M vector | SurrealDB, the lifecycle lane | the fix is in the tree; **`qOA6`, its re-run, was stopped before it started** (DECISIONS #134): every SurrealDB lifecycle cell is re-run in the 26.10.1 measurement, and the October lifecycle table withholds SurrealDB's peak memory and its 10M vector row meanwhile (`export_web.LIFECYCLE_STALE`, own PR #73) | BUGS F161 | -| 25 | Served ArcadeDB reads go through a path with no transaction. `POST /command` wraps even a SELECT in an auto-commit transaction, inside which a scan runs on one thread (ArcadeData/arcadedb#8775), while the embedded arms read outside one. Since ArcadeData/arcadedb#8792 (merged as 7f2c770697, 2026-10-01) `POST /query` runs an idempotent statement without one: verified on the merge, 1M vertices, SQL 44 ms through POST /query (45 through GET) against 315 through POST /command. So the lanes that read through `/query` (l2_graph, l3_sparse, l3d_dense, l4_tsbs) need only a pin that includes 7f2c770697; l1_tpc's `olap` still reads through `/command` and moves to `/query`. Confirm the pin's commit and an EXPLAIN on the re-pin image | ArcadeDB served: l1_tpc `olap` (the other lanes by the pin alone) | **done on `repin-prep` (2026-10-02, `44ebc18b38`):** l1_tpc's served analytics and timed point read go through POST /query, writes and the untimed verification scans stay on /command; EXPLAIN on 26.10.1-SNAPSHOT build `6c2a72417a` plans the filtered aggregate `(parallel)` over /query and sequential over /command, same answer; smoke there, 11 answer groups agreeing served and embedded. October is unaffected (no type scan ran in parallel at the pin in either arm) | BUGS F163 | -| 26 | PostgreSQL + AGE joins both graph tables (DECISIONS #128): a `pgage_graph` adapter on `dbbench:pg-age`, the lane's Cypher through `cypher()` with each rewrite recorded, indexes by measurement (#112), memory fitted like the other PostgreSQL arms, oltp in both durability classes plus olap, at sf1, sf10, and sf1full | PG+AGE only; inside the 26.10.1 measurement under its instrument (rows 2, 12, and 14 apply) | **built and smoked on `repin-prep`** (`9f741d51e2`, `49a81672dc`, 2026-10-02): micro oltp and olap, the LDBC SF1 slice, and the capped sf1full slice; every digest matches Neo4j's and LadybugDB's except the two CRUD read-backs at micro, which AGE's btree scan from a `>=` bound gets wrong (apache/age#2587, declared in `equivalence_check.KNOWN_DISAGREEMENTS`; the write and update cells publish with a note, DECISIONS #134). Package pinned (`postgresql-18-age=1.8.0~rc0-2.pgdg12+1`), Person `id` index chosen by measurement, resources fitted (row 20). Lands with `repin-prep`; runs in the 26.10.1 measurement's graph stages | DECISIONS #128, #131, #134 | -| 27 | MongoDB's graph arm: check `$graphLookup` against the chained `$lookup` form at the campaign's LDBC corpus, and use it for the three-hop read if the two agree there. At micro they agree on 50 of 50 ids at two and three hops, and `$graphLookup` takes 30.01 ms against 38.27 per three-hop operation, but only the chained form is faithful by construction, so every timed query uses `$lookup` until then | MongoDB, the graph lane: every question `$graphLookup` can express | **decided (DECISIONS #131 item 2):** rewrite with `$graphLookup` wherever it expresses the question, checked for equivalence at the campaign's corpus; keep and disclose `$lookup` only where it cannot. **Settled by row 56 on `repin-prep-2` (2026-10-06):** the graph questions are asked undirected over one stored edge per friendship, which a single-direction `$graphLookup` cannot state exactly, so the hops are chained `$lookup` again and PROTOCOL section 7 says why; the equivalence check is the SF1 digest comparison against a DuckDB reference | `l2_graph.MongoGraph` docstring; PROTOCOL.md section 7; DECISIONS #131 | -| 28 | `MongoGraph.QUERY_LANGUAGE` says what the arm runs: it stamps "the aggregation pipeline ($graphLookup)" into `query_language` on every MongoDB graph row, while no timed query uses `$graphLookup` (every hop and every analytics query is chained `$lookup`) | MongoDB graph rows (a recorded field; no number moves) | at the re-pin, since a code change on `main` reaches the running chain; with row 27. **Done on `repin-prep-2` (2026-10-06) with row 56:** `QUERY_LANGUAGE` reads "the aggregation pipeline ($lookup joins over both ends of each friendship)", which is what runs | `l2_graph.MongoGraph`; DECISIONS #131 | -| 29 | The Python-cost table (`pycost`): re-measure `jpype_overhead` at the pin (decided, DECISIONS #131 item 10). `export_web` reads `mini_results_.csv`, and only the September pin's file exists, so the October payload omits the table | the page's `pycost` table (the binding suite; no comparator) | decided; **the landing step is built on `repin-prep-2`, 2026-10-06** (`land_stage.pull_pycost` pulls `~/pycost/mini_results_.csv` from the bench host into `jpype_overhead/results/` and commits it with the landing; `test_land_stage_pycost.py`); the stage itself is `make_2610_stages` qRJ | `export_web.OVERHEAD`; PAGE-SPEC.md section 2 | +| 20 | The fairness audits still owed: ask each JVM engine for its live heap rather than trusting the ENV we passed (Elasticsearch reports `jvm.mem.heap_max_in_bytes`, now read back on its dense arm as `es_heap_max_mib`); audit the thread pools of the comparators the first audit did not cover (MongoDB, QuestDB, SurrealDB, SQLite, ArangoDB); audit every engine's memory pools for a buffer pool, cache, or memory limit sized from the host's RAM rather than the cell's cap (the class of BUGS F160); whether any engine's disk IO scheduling differs under the same cap | every comparator | **partly done**: every PostgreSQL arm except `postgres`, the defaults arm by design, is audited and fitted (`runner.PG_FIT_CMD`: parallel workers per query = cpuset - 1, `work_mem` from the cap over the cpuset), and Memgraph's memory limit and LadybugDB's buffer pool are fitted (F160); FAIRNESS F6 carries the rows. The rest stays open | FAIRNESS.md F6; `.notes` repros/age-dialect/resource_fit_probe.py | +| 21 | Every override is disclosed where a reader meets it: the rows marked **NOWHERE** in PROTOCOL section 7 get a row field or a page sentence, and the manifest records engine configuration, not just cpuset, memory, and images | every lane with an override | **built (2026-10-03), on main with the re-pin**: `overrides.py` registers the overrides a table shows, each with a row field read back from the engine, a generated sentence under every table that shows the arm, `fairness_check` F15 (the stamp), a `page_check` section (the sentence), and `test_overrides.py` (it reads PROTOCOL.md section 7 and fails on a NOWHERE); the runner manifest gains `engine_config` and `runner_env`. Rows measured before the stamps fail F15 by design. Not done: the page's overrides table with a column for which way each override moves a number (it needs measurements), and the release asset's manifest does not carry the table | PROTOCOL.md section 7; FAIRNESS.md F15 | +| 22 | The lifecycle lane reads the vector index's statistics in one extra untimed cycle after the timed ones, not inside every timed session: `l5_lifecycle.cycle()` calls `_vector_stats()` between the action timer and `close()`, and at the October pin that read raises the close it precedes, so the embedded dense-vector "open and close" cell is not a session that touched nothing | ArcadeDB embedded lifecycle, vector situation only | open; the lifecycle table discloses it under the table (`export_web` `vector_stats`, no number) and the sentence retires with this row; #8630, in 26.10.1, removed most of what the read added to the close | BUGS F159: laptop, 1M vectors, clean close 6.4 -> 15.3 ms with the read before #8630 (`de966e79c5`), 3.0 -> 3.5 after (`b8ab990821`); mini's pin reads 88.5 ms and its split is not measured; `.notes/bench/repros/vector-clean-close/` | +| 23 | LadybugDB's graph arm runs with its thread pool and buffer pool fitted to the cell (`max_num_threads` from `sched_getaffinity`, `buffer_pool_size` at 0.8 of the cgroup limit), both on the row | LadybugDB, the graph lane (OLTP and analytics) | **built (DECISIONS #125)**: `l2_graph.py`; the 26.10.1 measurement runs every LadybugDB graph cell fitted | BUGS F160; FAIRNESS.md F6; LadybugDB/ladybug#1070 | +| 24 | The SurrealDB lifecycle arm streams its build input in batches of `INGEST_BATCH` (`_batched`), as the ArcadeDB arm streams row by row; it built each situation's whole input as one Python list first, which sat in SurrealDB's published peak memory at every size and alone filled the 28g envelope at 10M vector | SurrealDB, the lifecycle lane | **built (DECISIONS #126, #134)**: `l5_lifecycle_surreal.py`; every SurrealDB lifecycle cell is re-run in the 26.10.1 measurement, and the October lifecycle table withholds SurrealDB's peak memory and its 10M vector row until then (`export_web.LIFECYCLE_STALE`) | BUGS F161 | +| 25 | Served ArcadeDB reads go through a path with no transaction: `POST /command` wraps even a SELECT in an auto-commit transaction, inside which a scan runs on one thread (ArcadeData/arcadedb#8775), while the embedded arms read outside one; since ArcadeData/arcadedb#8792 `POST /query` runs an idempotent statement without one | ArcadeDB served: `l1_tpc` `olap` (the other lanes read through `/query` by the pin alone) | **built (2026-10-02)**: `l1_tpc.py`'s served analytics and timed point read go through `POST /query`, and its writes and untimed verification scans stay on `/command`; EXPLAIN on a 26.10.1 build plans the filtered aggregate `(parallel)` over `/query` and sequential over `/command`, same answer, and 11 answer groups agree served and embedded | BUGS F163 | +| 26 | PostgreSQL + AGE joins both graph tables (DECISIONS #128): a `pgage_graph` adapter on `dbbench:pg-age`, the lane's Cypher through `cypher()` with each rewrite recorded, indexes by measurement (#112), memory fitted like the other PostgreSQL arms, oltp in both durability classes plus olap, at sf1, sf10, and sf1full | PG+AGE only; inside the 26.10.1 measurement under its instrument (rows 2, 12, and 14 apply) | **built and smoked (2026-10-02)**: `l2_graph.py`; every digest matches Neo4j's and LadybugDB's except the CRUD read-backs, which AGE's btree range scan from a `>=` bound gets wrong (apache/age#2587, declared in `equivalence_check.KNOWN_DISAGREEMENTS`; the write and update cells publish with a note, DECISIONS #134); the package is pinned (`postgresql-18-age=1.8.0~rc0-2.pgdg12+1`). Runs in the 26.10.1 measurement's graph stages | DECISIONS #128, #131, #134 | +| 27 | MongoDB's graph arm: check `$graphLookup` against the chained `$lookup` form at the campaign's LDBC corpus, and use it where it states the question | MongoDB, the graph lane | **settled the other way by row 56 (DECISIONS #131 item 2, then #151)**: the graph questions are asked undirected over one stored edge per friendship, which a single-direction `$graphLookup` cannot state exactly, so every hop and every analytics query is a chained `$lookup` and PROTOCOL section 7 says why; the equivalence check is the SF1 digest comparison against a DuckDB reference | `l2_graph.MongoGraph` docstring; PROTOCOL.md section 7 | +| 28 | `MongoGraph.QUERY_LANGUAGE` says what the arm runs, not `$graphLookup` | MongoDB graph rows (a recorded field; no number moves) | **done with row 56**: it reads "the aggregation pipeline ($lookup joins over both ends of each friendship)" | `l2_graph.MongoGraph`; DECISIONS #131 | +| 29 | The Python-cost table (`pycost`): re-measure `jpype_overhead` at the pin (DECISIONS #131 item 10); `export_web` reads `mini_results_.csv`, and until the 26.10.1 measurement's host-side Python-cost stage writes one, only the September pin's file exists and the October payload omits the table | the page's `pycost` table (the binding suite; no comparator) | **built**: `land_stage.pull_pycost` pulls `~/pycost/mini_results_.csv` from the bench host into `jpype_overhead/results/` and commits it with the landing (`test_land_stage_pycost.py`); the stage itself is the generator's host-side Python-cost stage | `export_web.OVERHEAD`; PAGE-SPEC.md section 2 | | 30 | Fit or disclose the two JVM processes that run at their own default heap: QuestDB, which `runner.JVM_BACKENDS` lists, so its rows record the tier heap although no heap reaches its server, and mongot inside `dbbench:mongo-search`, whose heap nothing sets | QuestDB (time series), MongoDB Search (dense, cross-model) | open | `runner.BACKENDS`, `docker-conf/mongo-search-entrypoint.sh`; section 4 | -| 31 | Fix the F13/F11 header mismatch in `fairness_check.py`: `check_close_cost` prints its section as F11, which FAIRNESS.md numbers F13 | the gate's output only | at the re-pin, since `fairness_check` gates every landing of the running chain. **Done on `repin-prep-2` (2026-10-06):** the three headings read F13; `test_fairness_headers.py` holds the numbering against FAIRNESS.md | FAIRNESS.md F13 | +| 31 | `fairness_check.check_close_cost` printed its section as F11, which FAIRNESS.md numbers F13 | the gate's output only | **done**: the three headings read F13, and `test_fairness_headers.py` holds the numbering against FAIRNESS.md | FAIRNESS.md F13 | | 32 | Served ArcadeDB rows record `engine_stats_after_build` as the embedded rows do: the dense lane's hook reads the engine's counters through the embedded API, and the served adapter has no HTTP `getStats()` path, so its rows carry an empty one | ArcadeDB served, the dense lane | open, between campaigns | BUGS C4 | | 33 | The page tables planned and not built: `l3d_params` (matched operating points; waits on a renderer for text cells), `l4_tentag` (one-tag against ten-tag, ratios only), `pyingest` (the write side of the Python cost), `pysweep` (where the Python tax comes from), and `ops_recovery`, `ops_failover`, and `ops_start` (measured, never published) | the page (new tables, no lane change) | open | PAGE-SPEC.md section 2 | | 34 | The lifecycle `graph_gav` situation gets a bounded seed set and a per-cycle assertion that the view was used, so it can print; until then it is withheld (`export_web.LIFECYCLE_WITHHELD`) | ArcadeDB, the lifecycle lane | open | PAGE-SPEC.md rule 7 | | 35 | Build the figures PAGE-SPEC section 3 lists as not built: `f3_sparse_perquery` once a per-query run is stamped at the pin, so the figure and the tables describe one engine (no row of `results/sparse_cliff.jsonl` carries an engine version, so `make_paper_figures` skips it); `f9_build_cost`; and `f10_lifecycle` only if `lifecycle` shows the O(stored) close shape (if close is flat everywhere, the table suffices) | the page (figures, no lane change) | open | PAGE-SPEC.md section 3 | | 36 | The sparse table's registered sentence (`export_web.OCT_PROSE["l3s"]["one_timer"]`) says every engine builds its sparse index as it ingests; pgvector builds its index after its load (COPY, then CREATE INDEX), inside the same timer, so the sentence names that exception | the page's `l3s` conditions (no number moves) | open; the October preview carries the sentence | `l3_sparse.PgVectorSparse.build` | -| 37 | Elasticsearch joins the dense tables: an fp32 arm (`index_options.type: hnsw`) and an int8 arm (`int8_hnsw`, its default), `m` and `ef_construction` at the matched operating point, both sizes | Elasticsearch, the dense lane | **built and smoked on `repin-prep` (`de82a23762`, 2026-10-02)**: `elasticsearch_dense` (hnsw) and `elasticsearch_dense_int8` (int8_hnsw, top k re-scored), smoked through the runner at micro (mutation forced on) and tiny with the other new dense arms | DECISIONS #131 item 3; `dense_vector` kNN checked on the pinned 9.5.4 image 2026-10-02 | -| 38 | Memgraph, FalkorDB, and LadybugDB join the dense tables: Memgraph's vector index (f32), FalkorDB's vector index (6.0.1 at the re-pin: 4.x segfaults on a write inside a vector statement, which the dense lane does not do), LadybugDB's official `vector` extension downloaded at connect as DuckPGQ's is | Memgraph, FalkorDB, LadybugDB, the dense lane | **built and smoked on `repin-prep` (`de82a23762`, 2026-10-02)**: `memgraph_dense`, `falkordb_dense` (6.0.1, waits for OPERATIONAL), `ladybug_dense`; micro and tiny smoke recall 0.98-0.9997 beside Qdrant's 0.9999, deleted hits 0, F14c ok | DECISIONS #131 item 3; each checked on its pinned image 2026-10-02 | -| 39 | Memgraph and LadybugDB join the cross-model table: vector top-k, hop, and update in one transaction, atomicity by rollback; DuckDB (VSS and DuckPGQ in one engine) tested, then added or listed with its reason | Memgraph, LadybugDB, DuckDB, the cross-model lane | **built and smoked on `repin-prep` (`5cc6f56cf4`, 2026-10-02)**: `memgraph_e2`, LadybugDB, and DuckDB arms; laptop smoke at e2 scale, all four graph-capable arms (these three and `pg_age_e2`) agree on every digest, atomicity 0 torn of 40 each | DECISIONS #131 item 4; the whole transaction and its rollback checked on Memgraph 3.13.1 and LadybugDB 0.20.4 2026-10-02 | -| 40 | The lifecycle table gains SurrealDB served, and each embedded engine on the page on the lifecycle workloads its model supports | SurrealDB served, DuckDB, SQLite, LadybugDB, Chroma, LanceDB, sqlite-vec, the lifecycle lane | decided; **queued last** in the 26.10.1 measurement, behind every table the paper uses (DECISIONS #133). **Built on `repin-prep`** (2026-10-02): SQLite and DuckDB (`077ed9a395`), LadybugDB, Chroma, LanceDB, and sqlite-vec (`ec1650886d`, `201d38486c`), each situation it cannot express declared with the engine's own answer, one sentence per engine on the page; laptop smoke, 64 cells at lc10k and lc100k before the rebase and 48 at lc10k after it, every documents, ten-index, graph, and time-series digest equal to ArcadeDB's. SurrealDB served is not added (DECISIONS #135) | DECISIONS #131 item 5, #133 | -| 41 | Time series gains a plain-table PostgreSQL arm, as SQLite, DuckDB, SurrealDB, and ArangoDB run on plain tables | PostgreSQL, the time-series lane | **built and smoked on `repin-prep` (`4eb14e4009`, 2026-10-02)**: plain PostgreSQL on the time-series lane; smoke digests equal TimescaleDB's and SQLite's on all six (the commit's words) | DECISIONS #131 item 6 | -| 42 | A quantization survey: for every dense and sparse comparator, what its pinned image ships, recorded per engine against its DDL (MongoDB's index sets `"quantization": "none"` today), with an int8 arm wherever the engine ships one; #53's rule extended to the sparse table | every dense and sparse comparator | decided; **built on `repin-prep`** (2026-10-02): Neo4j SCALAR (`32c0a011da`), MongoDB `scalar`, Memgraph `i8`, LanceDB's unquantized `IVF_HNSW_FLAT` beside its SQ arm, ArangoDB SQ8, and Qdrant sparse `uint8` (`8fd1efdafc`), each set in its definition and read back onto the row, the cell refused on a mismatch; smoked at 5k and 20k SIFT and Big-ANN 100k (QUANTIZATION.md) | DECISIONS #131 item 8 | -| 43 | PostgreSQL + AGE joins the multi-model capability table (`export_web.MULTIMODEL_ENGINES`) | PG+AGE, the `multimodel` table | decided | DECISIONS #131 item 9 | -| 44 | DuckDB VSS and ArangoDB split ingest from index on the dense lane: both build their index after the load inside one timer, which October discloses under the dense table | DuckDB VSS, ArangoDB, the dense lane | decided | DECISIONS #132 | -| 45 | The 26.10.1 measurement runs in the paper's priority order: graph (transactional and analytics, PostgreSQL + AGE included), cross-model, documents, and dense first; then sparse, time series, lifecycle, and the Python-cost table; row 40 last. Five repetitions stay. Projected 500-595 h without row 40; launched about Oct 8 (qOA6 stopped), the paper's tables land about Oct 21-26 | every arm | decided 2026-10-02; **order amended 2026-10-06 (the user, DECISIONS #163, #165): three tiers, core tables first.** `make_2610_stages.py --order tiers` (the default) queues, in this order: every ArcadeDB arm, embedded and served, on the core lanes (graph, graph analytics, cross-model, documents, dense, sparse, time series, e4, the Python-cost table); then the other engines already in the harness on the same lanes, the engines whose version moved since October (ArangoDB 3.12.12, FalkorDB 6.0.1, LadybugDB 0.21.2) included, still on the new instrument; then ArcadeDB's lifecycle, restart and image-defaults arms; then the lifecycle and restart comparators; then the arms with no October rows (Elasticsearch dense, Memgraph dense and cross-model, the int8 variants, PG+AGE graph, ...), which may run in November or December. The ids are new (`qT01`...) so a killed stage's ALL-DONE marker cannot start the new chain; `--order paper` regenerates the 2026-10-06 11:22Z chain (`qRA`..`qRO`). **Repetitions in two passes (the user, 2026-10-06, DECISIONS #164):** `--rep-pass 3` runs reps 1 to 3 of every cell first (ids `qP01`..), reps 4 and 5 follow lane by lane | DECISIONS #133, #134, #163; `.notes` projection-2026-10-02 | -| 46 | Both laptop smoke scripts ask LSQB: their analytics stage ran at `--scale sf1`, where the message half never loads (`ldbc_snb.loads_messages` is true only at sf1full), so every smoke since 2026-09-19 recorded `lsqb_na` and none asked LSQB's nine; both now run it at `sf1full` with the same caps | `oct_repin_smoke.sh` stage 27, `oct_skeleton_arcadedb.sh` stage 07 | fixed on `repin-prep` (`9f741d51e2`); lands with it | the PG+AGE smoke, 2026-10-02 | -| 47 | SurrealDB served runs both durability classes: `sync=never` on its storage path (relaxed) and `sync=every` (strict), the mode read back from the server's startup log before the client starts, the cell refused on a mismatch (`surreal_sync_mode` on the row). October ran its default, a sync at every commit, in both classes under a "not verified" label | the five SurrealDB served arms | fixed on `repin-prep`; the October page marks its write cells `re-run` (PR #81) | BUGS F165; 107 of 107 October serverlogs; laptop strace, one fdatasync per commit at the default and none at `sync=never` | -| 48 | The client image moves LadybugDB to 0.21.2 (0.20.4 crashes the dense arm's mutation phase; 0.21.1 carries #1074 and #1075) and pins the harness's own libraries (numpy 2.5.3, pandas 3.0.6, pyarrow 25.0.1, requests 2.34.2, psycopg 3.3.6), which were bare in every image and had already split pandas 3.0.5 / 3.0.6 between two images; the FalkorDB graph arm moves to the dense arm's 6.0.1 digest, whose pool now follows the cpuset (THREAD_COUNT 4 under a 4-CPU cpuset) and whose durability is unchanged by trace (no sync at the image default, one fdatasync per write with AOF always) | every client-image arm; `falkordb_graph` | done on `repin-prep` (`ddda7a3d95`): laptop smoke at micro, graph transactional in both durability classes, graph analytics, and LadybugDB dense with its mutation phase, 10 of 10 cells ok, 13 answer groups agreeing with Neo4j; FalkorDB 6.0.1 also answers LSQB q1's anonymous chain correctly, where 4.20.6 lost multiplicity. Mini's `dbbench:client` is rebuilt from this script at the re-pin | DECISIONS #129; `build_images.sh`; `.notes` repros/falkordb-601-durability | -| 49 | The Qdrant sparse build waits for the exact point count it sent, not only for a green status, inside the build timer (`BENCH_QDRANT_SETTLE_S`, a short collection refuses the cell); rows record `qdrant_points_at_first_green` and `qdrant_settle_after_green_s` | `qdrant_sparse`, `qdrant_sparse_uint8` | fixed on `repin-prep` (`c3393c1350`); the October rows are unaffected (recall 1.0 at 100k and 1M, 0.9999 at 8.84M where ArcadeDB's exact arm reads the same, and the search starts the moment the build timer stops) | BUGS F166; laptop: 66,064 of 100,000 points at the first green, recall 0.93 | -| 50 | Neo4j's vector search expansion is set to 1.0 and read back on every Neo4j vector arm: each quantization type brings its own `vector.default_search_expansion_factor` (NONE 1.0, SCALAR 1.5, BINARY 3.0 on 2026.08.1 and 2026.09.0), and the lane matches search effort at ef 100 (Neo4j asks for 100 candidates and keeps 10), so the default would give the SCALAR arm 150 | `neo4j_dense`, `neo4j_dense_int8`, `neo4j_e2` | done on `repin-prep` (`69d6808742`); smoke recall 0.9986 / 0.9908 at micro | QUANTIZATION.md; `.notes` repros/neo4j-202609-quant | -| 51 | The lifecycle answer gate compares embedded arms with embedded arms and served with served: its "not comparable" declaration, keyed on the query, had exempted every embedded comparator from the check | the lifecycle table's answer gate | done on `repin-prep` (`850f607644`); October's SurrealDB lifecycle answers checked by hand, all agree | BUGS F167 | -| 52 | The publish is rehearsed before the data lands: export_web and the six gates on a payload from the October skeleton rows plus a smoke of every new arm; 21 defects fixed (19 undeclared numeric row fields, the sparse table's cold column, e4 reported lost although declared, raw engine errors with unsourced digits in a note) and the lifecycle answer-check note folded from 5,273 to 756 characters | export_web, page_check | done on `repin-prep` (`1a82302f3b`, `ee058b55f9`, `850f607644`) | the remaining refusals come only from stale laptop rows or the laptop host | -| 53 | The 26.10.1 measurement's queue scripts come from a generator (`make_2610_stages.py`): eleven stages in DECISIONS #133's order, five repetitions, the e4 decomposition included, the Python-cost table host-side, the lifecycle expansion last; coverage derived from the runner's lanes and the page's tables, the release pin required, each stage refusing a comparator digest that moved | every arm | written on `repin-prep` (`b04edcd2f5`); queue_lint clean; projected 553 h; launch checklist in `.notes` HANDOFF | DECISIONS #133 | -| 54 | ArcadeDB's TIMESERIES types declare `COMPACTION_INTERVAL 1 HOURS` (the hourly aggregate's bucket), so compaction cuts sealed blocks at bucket boundaries and the aggregate answers from block statistics | ArcadeDB only (embedded and served time-series arms) | **decided 2026-10-04 (the user, #138), CONDITIONAL on upstream confirming it as the intended declaration (ArcadeData/arcadedb#9166)**; **CONFIRMED 2026-10-05 (upstream: `COMPACTION_INTERVAL 1 HOURS` with `SHARDS` at its default is the declaration for this workload; issue closed with #9246 in 26.10.1), so built on `repin-prep-2`, 2026-10-06**: both native arms create the type with it (`SHARDS 4` stays, as this row says, recorded as `ts_shards`; upstream's answer names `SHARDS` at its default, which the maintainer may choose to follow too), the row stamps `ts_compaction_interval_ms` as `schema:types` reports it, and `overrides.py` registers `arcadedb_ts_compaction_interval` (sentence under the time-series table, `fairness_check` stamp rule, `PROTOCOL.md` section 7 row), in force only for rows not measured at the October pin (`PRE_COMPACTION_PINS`: those ran without it, carry no stamp, and get neither the sentence nor the stamp rule, so `fairness_check`'s count on the October rows is unchanged by it); the bindings' docs and examples already teach it (#178); `test_ts_compaction.py` and `test_overrides.py` hold it; if confirmed: change the lane's DDL, disclose it on the page beside the time-series table, teach it in the bindings' time-series docs, and smoke the other two time-series queries (newest reading, one-minute max) for a cost; if upstream says otherwise, record the answer here and keep the default | laptop, relative only (`.notes/bench/repros/perf-timeseries-20261004`, `best-practice-20261004`): on a compacted type PROFILE goes from 0 fast-path and 9 decoded blocks to 16 fast-path and 0 decoded with the same 12 rows; the 12h aggregate's steady p50 from 8.3-10.1 ms to 0.62-1.39 ms and its first call after startup from 632-1,064 ms to 7.8-11.1 ms (pin and newest main). Only ArcadeDB has the knob, hence the user's call and the disclosure. The harness's fixed `SHARDS 4` showed no measurable effect and stays | -| 55 | The graph interactive table's reads get an untimed warm-up on ids disjoint from the timed set, for every engine, before the timed pass; the table prints the warm median and keeps the first query of a session as the cold column; the rows record `read_warmup`, and the "already-warm by construction" sentence (bench_common.NA_COLD_WARM_TXN) is retired for this lane: the l2 rows that record `read_warmup` must stop stamping `cold_warm_na`, or `export_web._cold_note` falls back to that sentence once `_graph_first_pass_note` steps aside | every engine on the graph interactive table (l2) | **decided 2026-10-04 (the user, approving my recommendation)**: build in `l2_graph.py` before the re-pin and smoke it on the laptop; **built on `repin-prep-2`, 2026-10-06** (`test_graph_read_warmup.py` runs the lane against a stand-in adapter: 1,000 untimed reads per operation on persons outside the timed set, the first read of the session is the cold number, a slow read's warm-up stops at a quarter of its budget, `read_warmup` is written and `cold_warm_na` is not; the l2 table prints a `cold first query ms` column only for rows that record `read_warmup`; the second timed pass stays as the check that the warm-up was enough; the count is a lane constant, `graph_common.READ_WARMUP_IDS`, to be tuned on mini if the second pass still moves); until then the October page says the first pass includes the JVM's warm-up (own PR from 2026-10-04) **Seen running 2026-10-06**: a read that starts before a warm-up cap finishes after it, so a slow engine's warm-up overshoots its cap by up to one read per operation (PostgreSQL + AGE spent 171 s of warm-up on the capped SF1 slice) | found 2026-10-04 by the graph point-lookup investigation (`.notes/bench/repros/perf-graph-point-20261004`): the lane times its reads twice over the same ids and the table prints the first pass (`l2_graph.py` ~3015-3020, `export_web.py` ~1507). On the October rows (SF1 point p50) ArcadeDB embedded's first pass is 3.59x its second (0.485 vs 0.135 ms), ArcadeDB served 1.46x, Neo4j 2.13x, every other engine 0.89x to 1.10x, so the published 3.2x gap to the median engine (rank 7 of 9) is mostly the JVM compiling the query path; on the second pass it is 0.90x (rank 5 of 9). A Java control (laptop, relative only) shows an untimed warm-up over disjoint ids removes 87-94% of the excess on the pin and on newest main, JDK 25 and 21. DECISIONS #89 already asks for the warm median and PROTOCOL.md requires warm queries disjoint from cold ones, so this implements a standing rule for every engine rather than flattering one | -| 56 | Every graph question is asked undirected, on every engine: each friendship stays one stored KNOWS edge (load and storage unchanged), and the graph OLTP reads (1-hop, 2-hop, 3-hop filtered, and the untimed 3-hop visited count) and the graph analytics questions (most friends, degree distribution, average friend age, friends in same city, and triangles) use each dialect's undirected or ANY form: `-[:KNOWS]-` in Cypher and the equivalents the LSQB queries in `l2_graph.py` already spell for every other dialect. A question that counts friendships or triangles still counts each one once (friends in same city with `a.id < b.id`; the six triangle spellings in `l2_graph.py` with `a.id < b.id AND b.id < c.id`). Every graph row records `knows_direction` (`undirected`), which `export_web._knows_one_way_note`, page_check's one-way disclosure check, and the page-level list of changes key on. Every graph digest is re-baselined, like F146's | every engine on the graph OLTP (l2) and graph analytics (l2olap) tables; LSQB already asks undirected and does not change | **decided 2026-10-04 (the user, DECISIONS #151)**: build in `l2_graph.py` and `graph_common.py` before the re-pin and smoke it on the laptop at LDBC SF1 against the validity hunt's DuckDB reference; **built on `repin-prep-2`, 2026-10-06** (`test_graph_undirected.py` holds every dialect's text; the laptop smoke at SF1 against a DuckDB reference of the lane's own digests is recorded per engine in REPIN-REHEARSAL; `knows_direction` is written on every graph row before the first query); until then the October page says the friendships are stored one way, what the hop reads and the triangle count therefore answer, and that the next measurement asks both directions (own PR, 2026-10-04) **Verified 2026-10-06** against a DuckDB reference computed outside the engines (Cypher's relationship-isomorphism over the same LDBC data, digested with the lane's own declarations): on the capped SF1 slice the five analytics questions and LSQB's seven agree on all twelve engines, and the five interactive reads agree on eleven (PostgreSQL + AGE's filtered 3-hop is censored at its budget, so that digest is unverified). The reference found four defects in this row's spellings, all fixed with tests: SurrealQL's integer division (`8ae6f5a21f`) and its closure scoping (`cff71eb55d`), FalkorDB's and LadybugDB's reuse of a relationship in a chain (`156cdef1af`), and MongoDB's `$lookup` `localField` (`7ecdb975ea`); see REPIN-REHEARSAL, round 5. The synthetic smoke scales store friendships twice and are not a cross-dialect gate | BUGS F169 (`.notes/bench/repros/validity-20261004`): in all 180,623 SF1 KNOWS rows `Person1.id < Person2.id`, so every directed question answers a question about ids. Triangles are 0 on every engine by construction and 387,573 undirected (ArcadeDB 775,146 = 2 x 387,573 with both directions loaded, equal to DuckDB, pin and main 98a1); 1-hop sees 53.2% of the 500 starts' friendships (`out/graph_expected.json`). Rejected: storing both directions (doubles the edges, moves ingest and storage, and is not how LDBC defines knows) | -| 57 | The documents lane loads `l_tax` and runs the full TPC-H Q1 on every engine: `l_tax` joins `LI_COLS` and every engine's load; sum_charge (`sum(l_extendedprice * (1 - l_discount) * (1 + l_tax))`), avg_price, and avg_disc join every engine's Q1 text (Q1_DUCK, Q1_SQLITE, Q1_ARCADE, and the MongoDB, SurrealDB, PostgreSQL, and ArangoDB texts in `l1_tpc.py`) and `OLAP_DIGEST["q1"]`. Every documents analytics row records `tpch_q1` (`full`), which `export_web._partial_q1` and the page-level list of changes key on. Q1's digests and timings are re-baselined | every engine on the documents analytics table (docs_olap); the load gains one column on both documents tables | **decided 2026-10-04 (the user, DECISIONS #151)**: build before the re-pin (**built on `repin-prep-2`, 2026-10-06**: `l_tax` is LAST in `LI_COLS` so every positional table keeps its order, every engine's Q1 computes all ten columns, the digest declares them, rows record `tpch_q1` = full; `test_tpch_q1_full.py` runs the real SQLite and DuckDB adapters against the specification in pandas and against DuckDB's bundled official answer at SF0.01, and the DuckDB adapter was checked against the official SF1 answer, 32 of 32 cells, on the laptop) and check the SF1 answer against DuckDB's bundled official Q1 answer (`tpch_answers()`), as the validity hunt's `scripts/tpch_reference.py` does; until then the October page says the lane's Q1 has seven of TPC-H Q1's ten output columns and that the next measurement runs the full Q1 (own PR, 2026-10-04) | BUGS F170: every engine's Q1 returns 7 of TPC-H Q1's 10 output columns and `LI_COLS` has no `l_tax`; the columns it does compute match the official SF1 answer cell for cell, and every October Q1 digest equals DuckDB's answer to the lane's text (`.notes/bench/repros/validity-20261004/out/tpch_reference.json`). Rejected: keeping the partial Q1 permanently | -| 58 | Every write the documents and graph tables time is inside an answer check: the documents post-state digest adds the stock the new-order transaction decrements (`SELECT p_partkey, stock FROM part WHERE stock <> 100`, in each engine's language); the graph post-state digests add the KNOWS edges into the written persons after the insert (the edge half of `OLTP_WRITE`) and after the delete (the edge removal of `DETACH DELETE`); a gate compares `payments_n` across engines | every engine on the documents OLTP (docs_oltp) and graph OLTP (l2) tables | **decided 2026-10-04 (mine, #150)**: build before the re-pin; new digests, so nothing is compared with October's. **Done on `repin-prep-2`, 2026-10-06 (documents half first, the graph half after)**: the graph half reads the KNOWS edges into the written persons back untimed after the insert (`res_graph_insert_edges`, one per person) and after the delete (`res_graph_delete_edges`, none) through `edge_scan` on every adapter, `test_graph_post_state.py` runs the lane against a stand-in that skips the edge write and one that leaves the edge on delete, and `equivalence_check` names AGE's missing first edge as the same open defect as its missing first person (apache/age#2587). Documents half (`test_docs_post_state.py`: every documents arm reads back `stock <> 100` untimed, the lane digests it after new-order as `stock` and the payment count as the one-row digest `payments`, which the cross-engine gate compares; SQLite and DuckDB run through the real adapters in the test) | BUGS F172: an engine that skipped the stock decrement or the edge write would print a faster write and pass every gate. For ArcadeDB both are right on the pin and on main 98a1 (SF0.1: 200 new-orders leave the expected stock on every touched part; LDBC SF1: 50 inserts leave 50 KNOWS edges into the new persons and deleting half leaves 25; `.notes/bench/repros/validity-20261004/out/tpc_diff_sf01_*.json`, `out/graph_diff_*_cmp.json`); the other engines are unchecked | -| 59 | The answer profile and the degenerate-answer check land: `bench_common.record_result` also writes `res__profile` (per declared column, the number of distinct canonical values and how many are 0 or null), and `degenerate_check.py` reports ZERO, CONSTANT, SCALE, and EMPTY answers, with a by-design allowlist and a `--self-test` | every lane (one field added to every row) | **decided 2026-10-04 (mine, #150)**: lands with the re-pin, because it adds a field to every row. The patch is `.notes/bench/repros/validity-20261004/drafts/patch-answer-profile-degenerate-check.diff` (not applied here; it updates `test_result_digest.py`, which passes on it) | the validity hunt (`.notes/bench/repros/validity-20261004/SUMMARY.md`): on October's rows it flags `l4 q_orderlimit` (SCALE), on the skeleton rows `l2/olap/sf1/triangles` (ZERO, F169), and its self-test catches the three F146 answers. Agreeing digests hid both F146 and F169 | -| 60 | LSQB's q5, q6, q8, and q9 run LSQB's own text on every engine: the node inequality (`tag1 <> tag2`, `person1 <> person3`) replaces the id comparison (`tag1.id <> tag2.id`, `person1.id <> person3.id`) in `graph_common.LSQB_QUERIES` and in LadybugDB's Cypher in `l2_graph.py`, and every other dialect's spelling of the four in `l2_graph.py` (DuckPGQ's SQL/PGQ, SurrealDB's, ArangoDB's, and MongoDB's) is checked against LSQB's text for its language, after a smoke on all nine graph engines (FalkorDB and Memgraph have not run the node form yet). DECISIONS #109's `sf1full` exclusion of q6 and q9 (`graph_common.TIER_EXCLUDED`) is re-decided, since it rested on the id form's first touches. Every graph analytics row records `lsqb_text` (`lsqb`), which `export_web._lsqb_id_form_note`, page_check's LSQB id-form disclosure check, and the page-level list of changes key on. The four queries' timings are re-baselined; their answers, and so their digests, do not change | every engine on the graph analytics table (l2olap) | **decided 2026-10-04 (the user, DECISIONS #154 items 1 and 2)**: build in `graph_common.py` and `l2_graph.py` before the re-pin; **built on `repin-prep-2`, 2026-10-06, except the `sf1full` exclusion** (`graph_common.LSQB_QUERIES` carry LSQB's node inequality, LadybugDB runs LSQB's own Ladybug text (`id(tag1) <> id(tag2)`, the form LSQB publishes for it; this row's node form also works there and counts the same), DuckPGQ's id comparison is LSQB's own PGQ text, the AQL, SurrealQL, and aggregation-pipeline spellings compare node identity; every graph analytics row records `lsqb_text`; `test_lsqb_text.py` holds the four texts against LSQB's published ones; the exclusion of q6 and q9 at `sf1full` (`graph_common.TIER_EXCLUDED`) is NOT lifted: it needs a first-touch measurement on mini and a maintainer decision, see REPIN-REHEARSAL); until then the October page says these queries compared the nodes' ids where LSQB's text compares the nodes, that on the build measured here the id form keeps ArcadeDB on a slower plan than LSQB's text gets (true at the pin for all four with and without the lane's view; q5's order reverses on newer main), and that the next measurement runs LSQB's own text (own branch `page-lsqb-and-repin-rows`, 2026-10-04). The sentence names only the queries a row without `lsqb_text` measured, so at `sf1full`, while #109 stands, it names LSQB Q5 and LSQB Q8 | BUGS F174 (`.notes/bench/repros/perf-lsqb-q9-20261004`): `d1fe3183a0` (2026-09-18) wrote the inequality on the id property; ArcadeDB's planner recognises only the inequality between two node variables for its count operators (`COUNT ANTI-JOIN CHAIN` for q9, one for q5 and q6), so the id form sent all four to the row pipeline. Config A, Temurin 25, q9: 33.59 s (pin) and 16.93 s (main) with the id form against 0.0105 s and 0.0065 s with LSQB's text; Neo4j pays 1.4-1.7x for the rewrite; the answers are identical in both forms (laptop, relative only) | -| 61 | ArcadeDB's Graph Analytical View on the graph analytics lane covers every vertex and edge type of the loaded graph, as ArcadeDB's documentation and upstream's own LSQB runner build it, instead of `VERTEX TYPES (Person) EDGE TYPES (KNOWS)` (both ArcadeDB adapters' `post_build` in `l2_graph.py`); the view build column prices the larger view, and the no-view arm (`BENCH_GAV=0`, `backend_arm` `nogav`) stays on the table as the like-for-like row. Every ArcadeDB with-view graph analytics row records `gav_types` (the types the view covers), which `export_web._gav_scope_note` and the page-level list of changes key on; the page discloses the setting beside the table | ArcadeDB only (the embedded and served with-view arms on the graph analytics table, l2olap) | **decided 2026-10-04 (the user, DECISIONS #154 item 3)**: an ArcadeDB-only setting, the user's call, recorded as such; build before the re-pin and smoke it with row 60's text; **built on `repin-prep-2`, 2026-10-06** (both ArcadeDB adapters create the view over Person plus `MSG_VERTEX_LABELS` and KNOWS plus `MSG_EDGE_TYPES` at the full-network tier, over Person and KNOWS where no message half is loaded; `Message`, the abstract supertype, is not listed, as upstream's runner does not list it; the engine's own report of the covered types is kept as `gav_types` and a narrower cover refuses the cell; `test_gav_scope.py`; the smoke with row 60's text is in REPIN-REHEARSAL); until then the October page says the view covers only the persons and their friendships, so no LSQB query can use it, and that the next measurement builds it over every type and keeps the rows without it (own branch `page-lsqb-and-repin-rows`, 2026-10-04) | `.notes/bench/repros/perf-lsqb-q9-20261004` (SUMMARY, draft own-B): the lane's view covers only Person and KNOWS and serves no LSQB query; with LSQB's text, the view over every type made q5 34-46x, q6 5-270x, and q8 37-79x faster and q9 no faster (laptop, relative only) | -| 62 | ArcadeDB's sparse arms return ids, as every comparator does: the embedded and served timed searches run `SELECT id, score FROM (SELECT expand(vector.sparseNeighbors(...)))` in place of `SELECT expand(vector.sparseNeighbors(...))`, and `resolve()` becomes the identity, as it is for every comparator (Qdrant `with_payload=False`, Elasticsearch `_source=False`, Milvus no output fields, pgvector `SELECT id`; the dense lane already projects ArcadeDB to its id). The change is in `l3_sparse.py`'s two ArcadeDB adapters, which the sparse multipass driver's warm passes also use. Every ArcadeDB sparse row, the lane's and the multipass driver's, records `sparse_result` (`id`), which `export_web._sparse_whole_record_note` and the page-level list of changes key on | ArcadeDB only (the embedded and served arms on the sparse table, l3s, and their multipass warm passes) | **decided 2026-10-04 (the user, DECISIONS #153 item 1, adopted as parity on my recommendation)**: build before the re-pin; **done on `repin-prep-2`, 2026-10-06** (`test_sparse_rows.py`: both adapters run the projection, `resolve()` is the identity on every arm, every ArcadeDB row and pass records `sparse_result`); until then the October page says each ArcadeDB search returns every hit's whole record while every other engine returns ids, and that the next measurement has ArcadeDB return ids (own branch `page-lsqb-and-repin-rows`, 2026-10-04) | `.notes/bench/repros/perf-sparse-20261004` (draft 3): the timed search returns each hit's whole record twice, about 46 KB of JSON per query; the projection returns the same ids in the same order on 50 of 50 queries on the pin and on main 98a1; 1.16x p50 on main, 1.22x at the pin (laptop, relative only; the served arm was not measured) | -| 63 | The documents lane loads ArcadeDB embedded through the bindings' columnar insert (own issue #150: each batch crosses JPype as whole columns, `long[]` and `double[]` from the numpy buffer and `String[]` from a list, and the documents are built in Java) instead of `insert_many`'s one JSON payload per 10,000-row batch (`ArcadeTPC.build` in `l1_tpc.py`); ArcadeDB served and every comparator load as before. The documents tables' ingest-path sentence (`export_web.OCT_PROSE["docs_oltp"]["ingest"]`, which docs_olap shares) changes with it and says which engines load whole frames (DuckDB, ArcadeDB embedded) and which are fed row by row. Every ArcadeDB embedded documents row records `columnar_insert` (the bindings call it loaded through), which the page-level list of changes keys on (`export_web._docs_row_load_tables`) | ArcadeDB embedded only (the ingest columns of both documents tables, docs_oltp and docs_olap; no timed query changes) | **decided 2026-10-04 (the user, DECISIONS #153 item 2), the user's choice over my recommendation**, recorded as the user's decision so a later reader does not mistake it for the knob-free rule's default. Precondition: the bindings ship the columnar insert (own issue #150) with tests and docs that name it the recommended path for column data, before the re-pin; until it ships the lane keeps `insert_many` and this row waits. Until the re-pin the October page's list of changes says ArcadeDB embedded loads through the columnar insert at the next measurement (own branch `page-lsqb-and-repin-rows`, 2026-10-04). **Built on `repin-prep-2`, 2026-10-06**: `Database.insert_columns` is in the bindings (commit `7a36dc650e`, `test_insert_columns.py`, 19 tests, run against the official 26.10.1 jars; the docs name it the recommended path for column data), and `ArcadeTPC.build` calls it with `parallel=True` for each parquet batch, records `columnar_insert` only when the call was made (a wheel without it keeps `insert_many` and the row says nothing, so a measurement meant to be columnar cannot silently be the other one), and the documents ingest sentence is chosen from the rows (`export_web._docs_ingest_note`): the columnar one when every ArcadeDB embedded row behind the table carries the field, the October `insert_many` one while any does not; `test_docs_columnar_load.py` holds it. **The measured wheel must contain the bindings commit**: the first official-engine wheel built for the re-pin did not, so the stages bake a wheel built from this tree (a laptop build of the same official jars plus this tree's bindings, which row 74 describes: its sha256 and the commit it was built from; see REPIN-REHEARSAL, round 5, for the wheel-identity question) | `.notes/bench/repros/perf-docs-ingest-20261004` (draft 1): the first 2,000,000 SF1 line items load in 8.21 s through the columnar prototype against 18.42 s through `insert_many` (2.24x, p50 of 3, laptop, relative only, with an engine that has upstream `62124466c9`); every arm stored the same sums and count. Moves only ArcadeDB's ingest numbers | -| 64 | The sparse multipass driver keeps each comparator's version, as the dense driver does: `sparse_multipass_driver.py` stores the adapter's own string as `lib_version` and restores `engine_version` after `run_conditions()` for the ArcadeDB arms (mirroring `dense_multipass_driver.py`), with a unit check that a comparator pass's `lib_version` starts with its engine's name. Every comparator pass records `lib_version`; no page sentence keys on it, because the page's sparse entries already take the comparator versions from the pins | every comparator's sparse multipass warm passes (`l3s_mp`); no timing moves | **decided 2026-10-04 (mine, DECISIONS #150)**: lands with the re-pin (own issue #158); **done on `repin-prep-2`, 2026-10-06** (`test_sparse_mp_versions.py`: `lib_version` starts with the engine's name on every comparator pass, `engine_version` is restored for the ArcadeDB arms; Elasticsearch's adapter reports the bare number, so the driver prefixes it); the October files stay as recorded | BUGS F173: every October comparator pass at the pin reads `engine_version: "26.10.1.dev0"` (`results/sparse_mp_417314c18/sp_{milvus,qdrant,elastic,pgvector}_tiny.json`), and a laptop Milvus pass read `"unknown (PackageNotFoundError)"`. No effect on the page, but the artifact a reader downloads misnames each comparator pass's engine (PAGE-SPEC rule 4) | -| 65 | The documents OLTP and durability operations get a disjoint untimed warm-up for every engine, as row 55 gives the graph reads: before the timed new-orders, `l1_tpc.py` runs untimed new-orders and payments, and the four single-record operations, on keys disjoint from and below the timed ones (order keys from a range below the timed set, so the timed inserts still land at the end of the order index as they do today; part keys from the same distribution), the same count for every engine and both durability classes. The documents OLTP table prints the warm median and keeps the first operation as the cold column (`cold_first_query_ms`, already recorded); the durability table's document rows print the warm median. Every documents OLTP row records `oltp_warmup` (the warm-up count), which `export_web._docs_warmup_note`, page_check's warm-up disclosure check, and the page-level list of changes key on. `NA_COLD_WARM_TXN` is retired for l1tpc: a row that records `oltp_warmup` stops stamping `cold_warm_na`, or `export_web._cold_note` falls back to the "already-warm by construction" sentence once `_docs_warmup_note` steps aside. Not for e2: `e2_hybrid.py` (line 1545) stamps the same sentence, but its transaction runs after 400 timed reads over the same paths and leaves out its first 20, and its window was not measured, so retiring it there needs its own measurement and decision | every engine on the documents OLTP table (docs_oltp) and the document rows of the durability table (durability); goes with row 55 | **decided 2026-10-04 (the user, DECISIONS #157)**: build in `l1_tpc.py` before the re-pin, with row 55; **built on `repin-prep-2`, 2026-10-06** (`OLTP_WARMUP` = 2,000, the laptop's measured count and a lane constant, `BENCH_DOCS_OLTP_WARMUP` for a smoke: untimed new-orders, payments, and the four single-record operations on keys 0..W-1, the timed keys starting at W so the timed inserts still land at the end of the index; the warm-up deletes its own single-record rows; the first operation of the session is the cold number; the row records `oltp_warmup` and `oltp_warmup_s` and no longer stamps `cold_warm_na`; `test_docs_oltp_warmup.py` runs the whole lane on the real SQLite adapter; the count to use on mini is still the bench host's to choose); the warm-up count is chosen on the bench host (2,000 cut the window 2.6x on the laptop, 10,000 cut it 3.0x). Until then the October page says these operations are timed from the first one after the load, so their columns include each engine's warm-up, which on the JVM engines includes compiling the statement path, and that the next measurement gives every engine an untimed warm-up on separate keys (own branch `page-arango-and-docs-warmup`, 2026-10-04) | `.notes/bench/repros/perf-commit-20261004` (draft 1): on the October rows ArcadeDB embedded's new-order is 2.34x (tpch1) and 2.56x (tpch10) its own payment, timed right after it, while every engine not on a JVM is 0.94x to 1.30x; on the laptop the lane's window is 2.90x to 3.79x the warm median (6 runs), and an untimed warm-up of 2,000 operations on disjoint keys cuts it to 0.38x (bootstrap 95% 0.31 to 0.47, Mann-Whitney p=0.024; laptop, relative only). It applies the existing warm/cold rule (DECISIONS #89, PROTOCOL) to every engine; it moves mainly the JVM engines, and the cold column stays published | -| 66 | `arango_common.calibrate_nprobe` raises `IndexNotAnswering` when recall at nProbe = nLists, read on the first 20 held-out queries, is below a fixed floor of 0.5 (not the calibration target: a healthy exhaustive probe reads 0.995 against a 0.9886 target), before the search and before any timed pass, so a broken index ends the cell in minutes instead of probing every list for 1.5 h per repetition, with a unit check that a search answering every query from one fixed list raises; and ArangoDB moves to its latest stable release (3.12.12 or newer) with the other comparators (DECISIONS #129), one image digest (`arangodb@sha256:4bc086d5050ca7ea11c6d00a36d8b910c838bb54ad553f8c1b715769d3499bcf`, 3.12.12, also `latest` on 2026-10-06) on the six ArangoDB arms in `runner.py` and in COMPARATORS.md (`build_images.sh` pins no ArangoDB server image) | ArangoDB: the calibration on the dense table (l3d) at both sizes and its multipass overlay; the release move reaches every ArangoDB arm, since one image serves documents, graph, dense, cross-model, and time series | **decided 2026-10-04 (the user, DECISIONS #156 item 3)**: build before the re-pin; the fail-fast stands whatever the release. **Done on `repin-prep-2`, 2026-10-06**: `test_arango_calibration.py` (a fixed-list search raises after 20 calls at nProbe = nLists, before the search; a healthy index calibrates; the floor is strictly below 0.5; the six arms share one digest that is not 3.12.11's). No re-run at the October pin with a capped list count (rejected in #156: a configuration unlike ArangoDB's other sizes) | BUGS F175 (`.notes/bench/repros/arango-recall-20261004`): on 3.12.11, once an IVF index has 10,000 or more lists, every multithreaded query probes the same single list (9,999 lists pass, 10,000 fail, independent of the data; after a restart with `OMP_NUM_THREADS=1` the same stored index answers with recall 1.0). The October deep10m rep 1 finished `ok` with recall 0.0 at nLists = nProbe = 12,643, the calibration having escalated to an exhaustive probe; 3.12.12 passes the minimal repro (`repro.py`, 50k vectors). Also the cause of September's F55 | -| 67 | The page's ArangoDB not-comparable declaration keys on the row fields, so it retires on its own: the freeze records `ivf_nlists` and `engine_version` beside each row it withholds for recall (`make_paper_tables.WITHHELD_RECALL`, the `withheld_recall.json` sidecar), and `export_web._one_list_groups` marks an ArangoDB dense cell `n/c`, with the sentence that names the defect in place of the unexplained withheld one, only where a withheld row's recall is below 0.01 (`_ONE_LIST_RECALL_BELOW`), its `ivf_nlists` is 10,000 or more (`_ONE_LIST_MIN_NLISTS`), and its release is 3.12.11 (`_ONE_LIST_RELEASES`); page_check's not-comparable check holds the payload to the same rule. Rows from row 66's release, or with fewer lists, do not match, so the cell returns as a measurement, or under the unexplained withheld sentence if the index still fails | ArangoDB at 10M on the dense table (l3d); no timing moves | **decided 2026-10-04 (the user, DECISIONS #156 item 2)**: done on own branch `page-arango-and-docs-warmup`, 2026-10-04. The dense table is pending, so it renders when the table lands; checked on stand-in rows built from the real October rep-1 row. Nothing to build at the re-pin beyond row 66 | BUGS F175; the October rep-1 row (`runs_page_417314c18.jsonl` on the bench host): `recall_at_10` 0.0, `ivf_nlists` 12,643, `engine_version` `arangodb:3.12.11`. The 1M rows (4,000 lists) answer at recall 0.98 to 0.99 and stay on the table | -| 68 | ArcadeDB's id indexes that the lane uses only for equality become `UNIQUE_HASH` at the 26.10.1 re-pin, as the maintainers answered on ArcadeData/arcadedb#9169 (PR #9222): an id that is only read, updated and deleted by equality gets a hash index, and an index whose key is also ranged over or ordered stays `UNIQUE`. The lines to change: documents OLTP `Part(p_partkey)`, `OrderNew(okey)` and `Crud(ckey)` (`l1_tpc.py` 988, 991 and 995 embedded, 1151, 1154 and 1158 served); tabular `orders(id)` only (`l1_tabular.py` 552 and 638); cross-model `Product(pid)` (`e2_hybrid.py` 232 and 422, and the declaration string in `fairness_check.py` 888 to 889); the graph message-half ids (`l2_graph.py` 238; its ingest is not yet measured). What stays `UNIQUE` or LSM, and why: `Person(id)` (`l2_graph.py` 167 and 370), because the untimed `person_scan` ranges over it (`q.id >= ...`) and a hash index turns that into a full label scan; the time-series `Point(host, ts)`; `l_shipdate` (`NOTUNIQUE`); and the lifecycle and recovery indexes. The page's condition sentence names the index kind and the ingest cost | ArcadeDB only (embedded and served documents OLTP, tabular, cross-model, and the graph message half); every other engine keeps its primary-key or default index, so no comparator moves | **decided 2026-10-05 (the user): adopt at the re-pin; built on the local `repin-prep-2` branch 2026-10-06, not yet in a PR.** The DDL text in the eight places the row lists is `UNIQUE_HASH` (the three documents ids, `orders(id)`, `Product(pid)`, and the message half's seven labels through the one statement both graph arms share), `Person(id)` stays `UNIQUE`; `fairness_check.ARCADEDB_HASH_ID_INDEXES` is the one registry, `test_index_kinds.py` pins every ArcadeDB `CREATE INDEX` statement in the four lane files to the row and fails when one moves back, every ArcadeDB arm of those lanes records `index_kinds` on its row, the engine's own `SELECT FROM schema:indexes` answer read after the schema is built and outside every timer (`fairness_check` F14d: a STAMPED row whose kind disagrees with the registry is a failure from the first day, because only a row this harness wrote can carry the stamp; a row with NO stamp is report-only until the re-pin campaign starts, because October's own rows predate the stamp and must not block landing an October stage or the freeze's consistency pass if this branch's gates are run over them. The flip for the unstamped case is ONE constant, `fairness_check.INDEX_KINDS_GATE_FAILS`, set to True in the commit that starts the re-pin campaign), and a generated sentence says the kind and the ingest cost under every table that shows an ArcadeDB arm and an ingest column, built ONLY from rows that carry the stamp and say HASH (a row without it asserts nothing, so regenerating the October preview over pre-re-pin rows does not claim a hash index). Rehearsed on the laptop against the CI engine (upstream d36b4ca3ae): the engine's own `schema:indexes` reports HASH for each of them on both deployments and LSM_TREE for `Person(id)`, `l_shipdate`, and `customer_id`. The October campaign keeps its pin and its indexes; the harness change is made in the re-pin PR, not before. The mini build and ingest times and the lookup effect the page publishes are re-measured at the re-pin, because the laptop cannot time at that scale; the graph message half's ingest is measured then too | ArcadeData/arcadedb#9169, answered by the maintainers; PR #9222 (26.10.1) makes the hash buckets keep their entries unordered. Measured on the laptop at the Java level, in the lane's own query patterns (ratio of medians, `UNIQUE` over `UNIQUE_HASH`, above 1 means the hash index is faster; 10 runs per arm on each of Temurin 21 and 25, 26.10.1 main of 2026-10-04, answers identical): the OLTP columns move for ArcadeDB, documents (2M parts) new-order 1.43x to 1.48x, `crud_insert` 1.44x to 1.83x, `crud_delete` 1.27x to 1.41x, tabular (2M) read 1.25x to 1.42x, cross-model (500k) documents 1.24x to 1.27x; the ingest columns move against it, because the harness's loaders insert ascending ids, which is LSM's best case for the insert: tabular `load_orders` 0.80x to 0.86x, documents `load_part` 0.92x to 0.95x at 2M. The page prints both. `Person(id)` stays because the `person_scan` range read goes from 16.3 to 28.4 ms (Temurin 25) and from 22.5 to 31.3 ms (Temurin 21) under a hash index, while the timed graph reads gain nothing. Laptop, relative only **F14d flip DONE 2026-10-06 (DECISIONS #161 D3): `fairness_check.INDEX_KINDS_GATE_FAILS = True` in the commit that starts the campaign; an unstamped row now fails, and the tests that assumed report-only set it explicitly.** | -| 69 | One served lane gets a sensitivity arm for ArcadeDB at the image's own JVM defaults, printed beside the main arm: documents OLTP, served, the 2M-part cell, 3 repetitions, with the container started without `ARCADEDB_OPTS_MEMORY` and `ARCADEDB_OPTS_GC` set (the harness sets `-Xms{heap} -Xmx{heap}` and clears the GC variable on every served ArcadeDB arm, `runner.py` 435 to 437 and 674 to 676), so the image applies its 75% of the container limit as heap, generational ZGC, and no `-Xms`, as the maintainers answered on ArcadeData/arcadedb#9167. The main arms keep one JVM configuration on both deployments (G1, heap 50% with `-Xms=-Xmx`), because the embedded-versus-served comparison isolates transport only when the JVM is the same; the page's condition text says the stock image differs and by what, and the memory column stays the committed heap. This is one arm with the vendor's documented defaults on one lane, not a search over heap sizes | ArcadeDB served only, one lane; no comparator moves and no embedded arm changes | **decided 2026-10-06 (the user): adopt at the re-pin; built on the local `repin-prep-2` branch 2026-10-06, not yet in a PR.** The arm is `runner.BACKENDS["arcadedb_imgdefaults_server"]`, derived from `arcadedb_server` with the two JVM variables dropped (so every other launch setting is the main arm's), registered for the documents lane, run on the transaction workload only, in a stage of its own (`qRO`: tpch10, 3 repetitions, relaxed class, last in the chain), all declared once in `runner.ARM_RUNS` and derived from there by the stage generator, its coverage check, and the page's roster gate; confirmed by the user 2026-10-06. Every served ArcadeDB cell now records `server_jvm_*` read from the running process (`/proc/1/cmdline`, `jcmd VM.flags`); the arm refuses to run unless the JVM reports 75% of the limit, generational ZGC, and no `-Xms`, and `fairness_check` F3b declares it and holds its rows to the JVM's own report. Not in the live campaign: a mid-run change splits the rows. The arm is a new `server_env` entry in `runner.BACKENDS` (a new backend name, so no existing row changes), added in the re-pin PR; the page prints it with its own label and a condition sentence generated from the row's recorded JVM flags **Gate found wrong 2026-10-06**: `fairness_check` F10 asked the arm for a strict cell it is declared never to run, which would have failed the landing of `qRO`; F10 now takes the required classes from `runner.ARM_RUNS` (`test_f10_restricted_arm.py`) | ArcadeData/arcadedb#9167, answered by the maintainers in a new arcadedb-docs section on JVM settings in a fixed memory budget: image defaults in Docker or Kubernetes; embedded has no default, start at 50% of the budget and go up to 75% for a dedicated process; pinning `-Xms` to `-Xmx` makes peak memory reflect the setting rather than the working set | -| 70 | The TimescaleDB time-series arm runs `ANALYZE p` after its `(host, ts DESC)` index, inside its ingest timer, exactly as the plain PostgreSQL arm does (`l4_tsbs.py` `TimescaleTS.ingest`; `PostgresTS.ingest` line 1084): the two PostgreSQL-family arms are then timed on the same steps, and the page's l4 ingest sentence says each is one COPY stream followed by an index and an ANALYZE, all inside the ingest timer | TimescaleDB, the time-series lane only; no other engine moves | **owed at the re-pin, done on repin-prep-2 (2026-10-06, the user's decision)**: the re-pin re-runs every arm, so no published row is split. October's TimescaleDB l4 ingest rows (417314c18) were measured WITHOUT the ANALYZE and stay as measured; the plain PostgreSQL arm is new at the re-pin (row 41), so it never ran without it | the plain PostgreSQL arm already paid for one, TimescaleDB's did not (found writing the ingest sentences for the re-pin); `test_ingest_prose.py` fails if either arm's `ingest` lacks the ANALYZE or places it before the index or inside the index timer | -| 72 | Every ArcadeDB SERVED arm that talks HTTP uses one persistent `http.client` connection per host, `benchmarks/experiments/lean_http.py`, in place of `requests.Session` (levers audit 2026-10-06: about three quarters of a served read was the Python client, not the engine; one bound one-row read costs 468 to 628 us through `requests` and 119 to 150 us through a persistent `http.client` connection against the same 26.10.1 server, and every comparator is driven by its own vendor driver, so the served ArcadeDB column measured the client library as much as the engine). The lanes change by one import (`lean_http.Session()`); the headers, Basic auth, JSON bodies (`json.dumps(obj, allow_nan=False)`), `Accept-Encoding: gzip, deflate` decoding, chunked iterator bodies, per-call timeouts and the text of a non-2xx (`raise_for_status()`) are requests' own, TCP_NODELAY is set, a connection the server closed is replaced, a read that died on a reused connection is retried once and a write never is. `BENCH_ARCADEDB_HTTP_CLIENT=requests` restores the October client (default `lean`; anything else is refused). Every served row records `arcadedb_http_client` (read from the session that ran, through `row_extra`, and in the l5 lifecycle row), the manifest records the resolved choice and, when set, the variable in `runner_env`, and `runner.run_cell` forwards the variable to the lane container | the 46 served cells measured behind the median (L2-SRV, DO-OLTP-SRV, DUR-NW-SRV, E2-SRV) and every other ArcadeDB served arm that posts to the server: documents tabular and TPC, sparse and dense vector, time-series document and native, hybrid E2, lifecycle server, and the restart lane's graph re-attach (`l1_tabular`, `l1_tpc`, `l2_graph`, `l3_sparse`, `l3d_dense`, `l4_tsbs` x2, `e2_hybrid`, `l5_lifecycle_server`, `l6_restart`); **no comparator client moves**, and the restart lane's one-shot `requests.post/get` probes, the e4 decomposition probe's own sessions (it measures with the standard client) and the runner's readiness probe stay as they are | **done on `levers-lean-http` (2026-10-06, the maintainer's decision; based on `repin-prep-2`, not pushed), owed at the re-pin**: lands with it, so every re-pin ArcadeDB served row runs lean. October's served rows (417314c18) were measured with `requests` and stay as measured; they carry no `arcadedb_http_client`, which reads as requests. Laptop A/B on cores 6-7, interleaved, round 0 dropped, 10 kept runs per arm, two independent runs, relative ratios only: requests/lean 3.05 to 3.07x on a bound one-row read through the lane's adapter, 2.5 to 2.6x on a one-statement write, 1.3 to 1.4x on the E2 session transaction, 1.06 to 1.07x on an 8,000-row result, 1.01x on a 2.5 MB JSON body, 0.99 to 1.03x on a chunked 30,000-vertex `/batch` | the lean client changes what an HTTP round trip costs, not what the engine does: a small served read or write gets 3 to 4x cheaper on the client side, a large body or a large result is engine-bound and does not move, so the served-versus-embedded gap shrinks on the small-operation cells and nowhere else. `test_lean_http.py` (70 tests: response parsing and `raise_for_status` text equal to requests' for 200, 4xx, 5xx, empty and large bodies, the request each client sends, reconnect after the server closes, retry only for reads, timeout class, the switch, the row and manifest records, the forwarding allowlist, the page gate) and `test_lean_http_live.py` (10 tests against `arcadedb-c25:26.10.1`: the lanes' own adapters, identical answer digests from both clients) **Addendum, the e4 deployment decomposition (2026-10-07):** the table that says what the client/server split costs was timed through ONE client, and the e4 cell's image (`dbbench:arcadedb`) has no `requests`, so that client was `e4_decomp.py`'s urllib shim, which opens a new connection for every call; October's e4 rows (417314c18) were measured with the shim and name no client. `deployment_decomp_probe.py` now measures every HTTP arm under BOTH clients in one rotating interleave (`inproc_http` and `docker_http` through the probe's own session, `inproc_http_lean` and `docker_http_lean` through `lean_http.LeanSession`, which gained the per-call `auth=` the probe passes), the artifact names each arm's client (`meta.arm_clients`, `meta.client_names`) and the measurement records it (`arcadedb_http_client`), `e4_decomp.py` stamps `arcadedb_http_clients` on the row (the switch does not apply to this lane: it measures both), and `export_web._e4_arms` prints each HTTP column once per client ("..., HTTP ms, urllib client" and "..., lean client"; an artifact without `arm_clients` keeps its three columns, repetitions that disagree about the clients refuse to build). **Owed at the re-pin:** the e4 stage re-runs and the page prose around the table (its caption and the "wire format" paragraph, which now covers client and wire) is rewritten from the new columns. Laptop A/B of the decomposition itself (cores 6-7, 10 runs per legacy client, 15 interleaved reps per arm per size inside each, legacy/lean ratio of the median p50s): the client is 0.4 to 2.0 ms of an HTTP call at 1 to 10,000 rows (legacy/lean 1.04 to 1.5x in-process, 1.04 to 1.3x against the container) and not separable at 100,000 rows, where the result's JSON dominates; the 2-core pin puts the in-process JVM and the client on the same cores, so only the legacy-versus-lean difference carries over, not the absolute columns. Tests: `test_e4_client_axis.py` (19) and two live tests in `test_lean_http_live.py` | -| 73 | JPype is pinned to 1.7.1 in the benchmark images and recorded per row; reason and the measurement: `.notes/bench/repros/jpype-master-check-20261006/`. The wheel declares `jpype1>=1.5.0` with no upper bound and `Dockerfile.bench` resolved the newest JPype at image build time, so an image rebuilt on the day of the next JPype release would change the instrument in the middle of a run and split its rows. `Dockerfile.bench` takes `ARG JPYPE_VERSION=1.7.1` and installs `jpype1==${JPYPE_VERSION}` in the same pip command as the local wheel (and whenever `PIP_PACKAGES` names `arcadedb-embedded`), then imports it and fails the build if the version is another; `build_images.sh` reads that default and passes it down, so a variable in the caller's environment cannot move it. `bench_common.JPYPE_PIN` is the same number: `make_2610_stages.py --check` (and every emit) refuses a Dockerfile whose default differs or that installs `jpype1` any way but a hard `==`. Every generated stage reads JPype back out of `dbbench:arcadedb` after the build and aborts, naming both versions, if the image runs another; the host-side Python-cost stage does the same for the repo venv (`uv pip install ... jpype1==1.7.1` in its message), and `verify_pair_c25.sh` has the same check on the embedded image as its fifth. Every result row records `jpype_version`, read from `jpype.__version__` of the process that ran the cell (`bench_common.run_conditions`, and the five lane writers that do not call it); an arm that never imports JPype (every comparator, every served ArcadeDB client) records an empty value, never a guess. The run manifest records the version the ArcadeDB bench image holds when the batch has an embedded ArcadeDB arm, and `page_check.NOT_PRINTED` declares the field as a recorded condition that is never a column | every embedded ArcadeDB arm (the image they run in) and the host-side Python-cost stage (the repo venv); the field is on every row of every lane. The wheel's own `jpype1>=1.5.0` requirement is the bindings' contract and does not change, and `benchmarks/python-bindings/docker/Dockerfile` (the SciPy paper's images) is not touched | **done on `repin-jpype-pin` (2026-10-06, based on `repin-prep-2` `a67c497c90`, not pushed), owed at the re-pin**: lands with it. October's rows (417314c18) carry no `jpype_version`; JPype's newest release was 1.7.1 (2026-05-06) throughout, so an image built then resolved it | laptop only, 2026-10-06: JPype master (240 commits ahead of 1.7.1, unreleased) against 1.7.1 under the 26.10.1 wheel. Raw JPype calls cost 15 to 25 percent less on master (static call 0.83x, boxed return 0.76x), package-level calls move within noise (0.88x to 1.1x), and 1.7.1 leaks one Python object per boxed number returned from Java (about 32 bytes, jpype#1379) where master leaks none. The next JPype release therefore changes per-call costs and memory behaviour of every embedded arm, which is why the version is a condition of the measurement and not a resolution of the day. Moving the pin is a deliberate edit of `JPYPE_PIN` and the Dockerfile default between campaigns, never an image rebuild. Tests: `test_jpype_pin.py` | -| 74 | The wheel the 26.10.1 measurement runs on carries the call-overhead change of bindings PR #214 (`fabbda7ab7`): `RowAccess.nextRows` and `RowBatcher.nextJsonBatch` close the result set they drain and Python stops after a short batch (one JPype crossing fewer per short read, and one fewer for the close), and scalar parameters are bound without the per-value conversion walk (`Database._java_parameters`; numpy scalars, Decimal, dates, lists, and `int` subclasses keep the general path). The wheel the stages bake is built from THIS tree (the merge of `origin/main` `abd73192e1`, commit `8989cbb0e8`, whose parent chain holds the merge `4a0d136bb1`) over the official 26.10.1 jars byte for byte (63 of the 65 jars in it; the other two are this tree's bridge and the bundled JRE's `jrt-fs.jar`), so it carries #214 and the columnar insert (row 63); it is NOT the file PyPI holds for the bindings 26.10.1 release (that release was cut from `629b242ccd`, before both). Version `26.10.1`, 72,766,626 bytes, sha256 `83a2a40369a411622696aed77361fd3b1c3d23b675d0a784b38b0b022c13c41b`, recorded in every cell's manifest and baked into every stage | ArcadeDB embedded arms only (the served arms do not cross the bridge, and no comparator uses this package); every ArcadeDB embedded read and write that goes through `to_list`, `first`, `to_json_list`, or a bound statement | **done on `repin-prep-2`, 2026-10-06 (round 6 of the re-pin rehearsal)**: the wheel was rebuilt after the merge and the pair re-verified (`PAIR VERIFIED d36b4ca3a`, JVM 25, JPype 1.7.1); the three bindings test files #214 touched pass on it (74 tests), and so does the full bindings suite (see REPIN-REHEARSAL, round 6); the first build of this wheel failed the bindings' own `test_jvm_payload.py` guard on the columnar insert's string path, fixed in `8989cbb0e8` and the wheel rebuilt. October's rows (417314c18) were measured on a bridge without it, so the re-pin's ArcadeDB embedded rows are not comparable with October's at the call-overhead level (the re-pin re-runs every arm, so nothing is split inside a table); the page's note on what the Python package costs should name the change when it lands | #214's own A/B (laptop, interleaved, relative only): the documents lane's SELECT by key through `to_list()` 0.83x, the same read through `first().get()` 0.90x, the bound Cypher read through `to_json_list()` 0.92x (not significant at 13 runs), `_java_parameters` 0.79x and 0.78x; a JPype call is 3 to 4 microseconds, a fifth of a one-row read | - -**Retired: settled with no change, listed so nobody re-opens them.** Raising ArcadeDB's statement-cache settings (a knob only ArcadeDB would get; item 2 removes the literal path instead). The sparse ingest path (already the faster one). Durable commit cost (one sync per commit, equal to SQLite's; not a defect). `expected_edge_count` as a speed lever (no measurable effect; item 1 adopts it only because it is the documented setting). **Done, and off the list:** the dense table's phases partition their total (`setup_s`, since 2026-09-22, BUGS F101), every stage estimates its whole cell against its cap before running (`cell_cost_check.py`, BUGS F74), one engine wears one version per page (`version_consistency_check`, a publish gate, BUGS F61), and `queue_lint` refuses the process-name wait that only holds for one exact launch line (BUGS F59; every October stage waits on its predecessor's ALL-DONE marker). - -The detail behind items 2 and 8, written when they were found (2026-09-23): - -* **Every lane must ask with PARAMETERS wherever a value varies per iteration.** Swept on 2026-09-23: the GRAPH lane interpolates `{id}` into one shared template, so every Cypher engine on it reuses no plan; the DOCUMENTS lane's CRUD half is worse because it is ASYMMETRIC (F114 -- embedded parameterises, served interpolates, inside a table that exists to measure the network hop); the CROSS-MODEL transaction interpolates its RELATED hop and its view update on BOTH ArcadeDB arms, so it is symmetric between them but still parse-bound against comparators whose driver APIs carry no query text at all. Clean: the documents lane's ANALYTICS half (`ARCADE_OLAP` is five constant texts), which matters because those cells dominate the remaining schedule. `graph_common.py` builds every read from one template with `{id}` in the text, so no Cypher engine on that table ever reuses a parsed plan and the point-lookup column is substantially a comparison of PARSERS. Measured at the pin: Cypher with an embedded literal 0.8702 ms against 0.0907 ms with `$id`, and SQL 0.4810 against 0.0910 -- with parameters the two languages are indistinguishable (1.00x) and both are 5-10x faster. Upstream told us this before we found it (ArcadeData/arcadedb#8132), and it applies to every engine, not only ours. Prepared on `repin-prep` and smoked per engine (item 2). -* **The documents lane's served arm must parameterise its CRUD statements** (BUGS F114; `qOE` measured the lane without it and landed 2026-09-27; the fix is prepared on `repin-prep`, item 2). ArcadeDB's embedded arm sends `ckey=:c` with a parameter map and reuses one plan; its served arm sends `f"ckey={i}"` and reuses nothing, inside a table whose whole point is the embedded-versus-served comparison. SurrealDB's arm interpolates too. The OLAP half of that lane is already clean -- `ARCADE_OLAP` is five constant query texts, so the analytics cells are not parse-bound. +| 37 | Elasticsearch joins the dense tables: an fp32 arm (`index_options.type: hnsw`) and an int8 arm (`int8_hnsw`, its default), `m` and `ef_construction` at the matched operating point, both sizes | Elasticsearch, the dense lane | **built and smoked (2026-10-02)**: `elasticsearch_dense` and `elasticsearch_dense_int8` in `l3d_dense.py`, registered in `runner.py`; no October rows, so tier 3 of the 26.10.1 measurement | DECISIONS #131 item 3; QUANTIZATION.md | +| 38 | Memgraph, FalkorDB, and LadybugDB join the dense tables: Memgraph's vector index (f32), FalkorDB's (6.0.1: 4.x segfaults on a write inside a vector statement), LadybugDB's official `vector` extension downloaded at connect | Memgraph, FalkorDB, LadybugDB, the dense lane | **built and smoked (2026-10-02)**: `memgraph_dense`, `falkordb_dense`, `ladybug_dense` in `l3d_dense.py`; micro and tiny smoke recall 0.98 to 0.9997 beside Qdrant's 0.9999; tier 3 | DECISIONS #131 item 3; QUANTIZATION.md | +| 39 | Memgraph and LadybugDB join the cross-model table: vector top-k, hop, and update in one transaction, atomicity by rollback; DuckDB (VSS and DuckPGQ in one engine) joins too | Memgraph, LadybugDB, DuckDB, the cross-model lane | **built and smoked (2026-10-02)**: `memgraph_e2`, LadybugDB, and DuckDB arms in `e2_hybrid.py`; all four graph-capable arms (these three and `pg_age_e2`) agree on every digest, atomicity 0 torn of 40 each; tier 3 | DECISIONS #131 item 4 | +| 40 | Each embedded engine on the page gets the lifecycle workloads its model supports (SQLite, DuckDB, LadybugDB, Chroma, LanceDB, sqlite-vec); SurrealDB served is not added (DECISIONS #135) | the lifecycle lane | **built (2026-10-02), then taken off the lane's roster (DECISIONS #139)**: the lifecycle table compares the embeddable multi-model engines, and the served engines are measured on the restart table instead; `l5_lifecycle_sql.py` and `l5_lifecycle_embedded.py` stay runnable by name, each situation an engine cannot express declared with the engine's own answer; the laptop smoke's documents, ten-index, graph, and time-series digests equal ArcadeDB's | DECISIONS #131 item 5, #133, #139 | +| 41 | Time series gains a plain-table PostgreSQL arm, as SQLite, DuckDB, SurrealDB, and ArangoDB run on plain tables | PostgreSQL, the time-series lane | **built and smoked (2026-10-02)**: `postgres_ts` in `l4_tsbs.py`; its digests equal TimescaleDB's and SQLite's on all six queries; tier 3 | DECISIONS #131 item 6 | +| 42 | A quantization survey: for every dense and sparse comparator, what its pinned image ships, recorded per engine against its DDL, with an int8 arm wherever the engine ships one (#53's rule extended to the sparse table) | every dense and sparse comparator | **built (2026-10-02)**: Neo4j `SCALAR`, MongoDB `scalar`, Memgraph `i8`, LanceDB's unquantized `IVF_HNSW_FLAT` beside its SQ arm, ArangoDB SQ8, and Qdrant sparse `uint8`, each set in its definition and read back onto the row, the cell refused on a mismatch; smoked at 5k and 20k SIFT and Big-ANN 100k; the survey is QUANTIZATION.md | DECISIONS #131 item 8; BUGS F164 | +| 43 | PostgreSQL + AGE joins the multi-model capability table | PG+AGE, the `multimodel` table | **built**: `export_web.MULTIMODEL_ENGINES` and `MULTIMODEL_ALIASES` | DECISIONS #131 item 9 | +| 44 | DuckDB VSS and ArangoDB split ingest from index on the dense lane (both built their index after the load inside one timer) | DuckDB VSS, ArangoDB, the dense lane | **built (DECISIONS #132)**: both now time their load and their index as two timers, so `fairness_check.PHASE_SPLIT_DISCLOSED` is empty and neither needs a disclosure | DECISIONS #132 | +| 45 | The 26.10.1 measurement runs in priority order, in three tiers and two repetition passes. **Tier 1:** every ArcadeDB arm, embedded and served, on the paper's lanes (graph transactional and analytics, cross-model, documents, dense, sparse, time series, e4, the Python-cost table). **Tier 2:** the other engines already in the harness on the same lanes and the new instrument, those whose version moved since October (ArangoDB 3.12.12, FalkorDB 6.0.1, LadybugDB 0.21.2) included; then ArcadeDB's lifecycle, restart, and image-defaults arms; then the lifecycle and restart comparators. **Tier 3:** the arms with no October rows (Elasticsearch dense, Memgraph dense and cross-model, the int8 variants, PG+AGE graph, and the rest), which may run in November or December. **Repetitions:** reps 1 to 3 of every cell first, reps 4 and 5 later, one whole lane at a time | every arm | **decided (the user, DECISIONS #133, #163, #164, #165); built**: `make_2610_stages.py --order tiers` (the default; `--order paper` regenerates the first 11:22Z chain, `qRA`..`qRO`) with `--rep-pass 3` (ids `qP01`..`qP34`, 34 stages), the generator changes in own issues #219, #221, #223, and #225. A table landed between the passes states n=3 in its caption and in `EXPECTED` (PROTOCOL section 1). The generator does not yet emit the second pass (reps 4 and 5, one lane at a time, its roster derived from the rows, DECISIONS #164) | DECISIONS #133, #134, #163 to #165; `.notes` projection-2026-10-02 | +| 46 | Both laptop smoke scripts ask LSQB: their analytics stage ran at `--scale sf1`, where the message half never loads (`ldbc_snb.loads_messages` is true only at sf1full), so every smoke since 2026-09-19 recorded `lsqb_na` and none asked LSQB's nine; both now run it at `sf1full` with the same caps | `oct_repin_smoke.sh` stage 27, `oct_skeleton_arcadedb.sh` stage 07 | **fixed** (`9f741d51e2`) | the PG+AGE smoke, 2026-10-02 | +| 47 | SurrealDB served runs both durability classes: `sync=never` on its storage path (relaxed) and `sync=every` (strict), the mode read back from the server's startup log before the client starts and the cell refused on a mismatch (`surreal_sync_mode` on the row); October ran its default, a sync at every commit, in both classes under a "not verified" label | the five SurrealDB served arms | **fixed**: `runner.py`, `surreal_common.py`; the October page marks its write cells `re-run` | BUGS F165; laptop strace, one fdatasync per commit at the default and none at `sync=never` | +| 48 | The client image moves LadybugDB to 0.21.2 (0.20.4 crashes the dense arm's mutation phase) and pins the harness's own libraries (numpy 2.5.3, pandas 3.0.6, pyarrow 25.0.1, requests 2.34.2, psycopg 3.3.6), which were bare in every image; the FalkorDB graph arm moves to the dense arm's 6.0.1 digest, whose pool follows the cpuset | every client-image arm; `falkordb_graph` | **done**: `build_images.sh`, `runner.py`; the laptop smoke at micro had 13 answer groups agreeing with Neo4j, and FalkorDB 6.0.1 answers LSQB q1's anonymous chain correctly where 4.20.6 lost multiplicity | DECISIONS #129; `build_images.sh`; `.notes` repros/falkordb-601-durability | +| 49 | The Qdrant sparse build waits for the exact point count it sent, not only for a green status, inside the build timer (`BENCH_QDRANT_SETTLE_S`, a short collection refuses the cell); rows record `qdrant_points_at_first_green` and `qdrant_settle_after_green_s` | `qdrant_sparse`, `qdrant_sparse_uint8` | **fixed** (`l3_sparse.py`); the October rows are unaffected (recall 1.0 at 100k and 1M, 0.9999 at 8.84M) | BUGS F166 | +| 50 | Neo4j's vector search expansion is set to 1.0 and read back on every Neo4j vector arm: each quantization type brings its own `vector.default_search_expansion_factor` (NONE 1.0, SCALAR 1.5, BINARY 3.0), and the lane matches search effort at ef 100 | `neo4j_dense`, `neo4j_dense_int8`, `neo4j_e2` | **done**: `l3d_dense.py`, `e2_hybrid.py`; smoke recall 0.9986 / 0.9908 at micro | QUANTIZATION.md; `.notes` repros/neo4j-202609-quant | +| 51 | The lifecycle answer gate compares embedded arms with embedded arms and served with served: its "not comparable" declaration, keyed on the query, had exempted every embedded comparator from the check | the lifecycle table's answer gate | **done** (`equivalence_check.py`, `850f607644`); October's SurrealDB lifecycle answers were checked by hand and all agree | BUGS F167 | +| 52 | The publish is rehearsed before the data lands: export_web and the six gates on a payload from the October skeleton rows plus a smoke of every new arm; 21 defects fixed (19 undeclared numeric row fields, the sparse table's cold column, e4 reported lost although declared, raw engine errors with unsourced digits in a note) | export_web, page_check | **done** (`1a82302f3b`, `ee058b55f9`, `850f607644`); the remaining refusals came only from stale laptop rows or the laptop host. The rule stays: rehearse the whole publish before the data lands | the rehearsal's own record is `.notes` git history, `bench/repin-round5/` | +| 53 | The 26.10.1 measurement's queue scripts come from a generator, `make_2610_stages.py`, a profile of `make_october_stages.py` so every guard of the October template reaches them: the release pin required (wheel name and sha256, one server image, one engine commit), comparator digests frozen at generation, coverage derived from the runner's lanes and the page's tables (every registered arm of every page lane in exactly one stage per workload), the e4 decomposition included, the Python-cost table host-side | every arm | **built (DECISIONS #133, order and passes #163 to #165)**: `make_2610_stages.py --check`, `queue_lint.py`, `test_stage_tiers.py`, `test_stage_rep_pass.py`, `test_stage_generator_after.py`; the order and the passes are row 45 | DECISIONS #133 | +| 54 | ArcadeDB's TIMESERIES types declare `COMPACTION_INTERVAL 1 HOURS` (the hourly aggregate's bucket), so compaction cuts sealed blocks at bucket boundaries | ArcadeDB only (embedded and served time-series arms) | **built (the user, DECISIONS #147, conditional on upstream confirming; confirmed 2026-10-05 on ArcadeData/arcadedb#9166, closed with #9246 in 26.10.1)**: both native arms create the type with it (`SHARDS 4` stays, recorded as `ts_shards`; upstream names `SHARDS` at its default, which the maintainer may choose to follow), the row stamps `ts_compaction_interval_ms` as `schema:types` reports it, and `overrides.py` registers `arcadedb_ts_compaction_interval` (the sentence under the time-series table, the F15 stamp rule, the PROTOCOL section 7 row), in force only for rows not measured at the October pin (`PRE_COMPACTION_PINS`); `test_ts_compaction.py` and `test_overrides.py`; the bindings' docs and examples teach it (#178). Only ArcadeDB has the knob, hence the user's call and the disclosure | `.notes/bench/repros/perf-timeseries-20261004` (laptop, relative only) | +| 55 | The graph interactive table's reads get an untimed warm-up on ids disjoint from the timed set, for every engine; the table prints the warm median and keeps the first query of a session as the cold column; the rows record `read_warmup` | every engine on the graph interactive table (l2) | **built (the user, DECISIONS #148)**: `l2_graph.py`, `graph_common.READ_WARMUP_IDS` (1,000 untimed reads per operation, `BENCH_GRAPH_READ_WARMUP`; a lane constant to tune on mini if the second timed pass still moves), `test_graph_read_warmup.py`; the l2 table prints a `cold first query ms` column for rows that record `read_warmup`, and those rows no longer stamp `cold_warm_na` (`bench_common.NA_COLD_WARM_TXN` is retired for this lane). A read that starts before a warm-up cap finishes after it, so a slow engine's warm-up can overshoot its cap by up to one read per operation (PostgreSQL + AGE: 171 s on the capped SF1 slice) | `.notes/bench/repros/perf-graph-point-20261004`: ArcadeDB embedded's October first pass was 3.59x its second, the other engines 0.89x to 1.46x, so the published gap was mostly the JVM compiling the query path | +| 56 | Every graph question is asked undirected, on every engine: each friendship stays one stored KNOWS edge, and each dialect uses its undirected or ANY form (`-[:KNOWS]-` in Cypher); every graph row records `knows_direction` (`undirected`) | every engine on the graph OLTP (l2) and graph analytics (l2olap) tables; LSQB already asks undirected | **built and verified (the user, DECISIONS #151)**: `graph_common.py`, `l2_graph.py`, `test_graph_undirected.py`. Verified 2026-10-06 against a DuckDB reference computed outside the engines on the capped SF1 slice: the five analytics questions and LSQB's seven agree on all twelve engines and the five interactive reads on eleven (PostgreSQL + AGE's filtered 3-hop is censored at its budget, so its digest is unverified); the reference found four defects in the first spellings (SurrealQL's integer division and closure scoping, FalkorDB's and LadybugDB's reuse of a relationship in a chain, MongoDB's `$lookup` `localField`), all fixed with tests; its record is the rehearsal's, in `.notes` git history under `bench/repin-round5/`. The synthetic smoke scales store friendships twice and are not a cross-dialect gate | BUGS F169 (`.notes/bench/repros/validity-20261004`): in all 180,623 SF1 KNOWS rows `Person1.id < Person2.id`, so every directed question answered a question about ids. Rejected: storing both directions (it doubles the edges and is not how LDBC defines knows) | +| 57 | The documents lane loads `l_tax` and runs the full TPC-H Q1 on every engine; every documents analytics row records `tpch_q1` (`full`) | every engine on the documents analytics table (docs_olap); the load gains one column on both documents tables | **built (the user, DECISIONS #151 item 3)**: `l1_tpc.py` (`l_tax` is LAST in `LI_COLS`, so every positional table keeps its order), `test_tpch_q1_full.py` (the real SQLite and DuckDB adapters against the specification in pandas and DuckDB's bundled official answer; the DuckDB adapter matched the official SF1 answer, 32 of 32 cells) | BUGS F170; rejected: keeping the partial Q1 (7 of its 10 columns) permanently | +| 58 | Every write the documents and graph tables time is inside an answer check: the new-order stock and the payment count, and the KNOWS edges into the written persons after the insert and after the delete | every engine on the documents OLTP (docs_oltp) and graph OLTP (l2) tables | **built (DECISIONS #151, F172, within #150's remit)**: `l1_tpc.py`, `l2_graph.py` (`edge_scan` on every adapter, `res_graph_insert_edges`, `res_graph_delete_edges`), `test_docs_post_state.py`, `test_graph_post_state.py`; `equivalence_check` names AGE's missing first edge as the same open defect as its missing first person (apache/age#2587) | BUGS F172 (`.notes/bench/repros/validity-20261004`): an engine that skipped the stock decrement or the edge write would have printed a faster write and passed every gate | +| 59 | The answer profile (`res__profile`) and the degenerate-answer check (`degenerate_check.py`: ZERO, CONSTANT, SCALE, and EMPTY answers) | every lane (one field added to every row) | **built (DECISIONS #151, within #150's remit)**: `bench_common.py`, `degenerate_check.py`, `test_degenerate_check.py`, `test_result_digest.py` | `.notes/bench/repros/validity-20261004/SUMMARY.md`: agreeing digests hid both F146 and F169 | +| 60 | LSQB's q5, q6, q8, and q9 run LSQB's own text on every engine (the node inequality, not the id comparison); every graph analytics row records `lsqb_text` (`lsqb`) | every engine on the graph analytics table (l2olap) | **built (the user, DECISIONS #154 items 1 and 2), except the `sf1full` exclusion**: `graph_common.py`, `l2_graph.py`, `test_lsqb_text.py` (the four texts against LSQB's published ones; LadybugDB runs LSQB's own Ladybug text, `id(tag1) <> id(tag2)`). **The exclusion of q6 and q9 at `sf1full` (`graph_common.TIER_EXCLUDED`, DECISIONS #109) is NOT lifted: it needs a one-cell first-touch probe on mini and a maintainer decision (DECISIONS #161 D2: lift it after the probe, at the end of the campaign).** The four queries' timings are re-baselined; their answers do not change | BUGS F174 (`.notes/bench/repros/perf-lsqb-q9-20261004`): ArcadeDB's planner recognises only the inequality between two node variables for its count operators, so the id form sent all four to the row pipeline | +| 61 | ArcadeDB's Graph Analytical View covers every vertex and edge type of the loaded graph, as ArcadeDB's documentation and upstream's LSQB runner build it; the no-view arm (`BENCH_GAV=0`, `backend_arm` `nogav`) stays as the like-for-like row; every with-view row records `gav_types` | ArcadeDB only (the embedded and served with-view arms on l2olap) | **built (the user, DECISIONS #154 item 3; an ArcadeDB-only setting, the user's call)**: both adapters in `l2_graph.py` create the view over Person plus `MSG_VERTEX_LABELS` and KNOWS plus `MSG_EDGE_TYPES` at the full-network tier (`Message`, the abstract supertype, is not listed, as upstream's runner does not list it), over Person and KNOWS where no message half is loaded; the engine's own report of the covered types is kept as `gav_types`, and a narrower cover refuses the cell; `test_gav_scope.py`; the page discloses the setting beside the table | `.notes/bench/repros/perf-lsqb-q9-20261004`: the old view served no LSQB query | +| 62 | ArcadeDB's sparse arms return ids, as every comparator does (`SELECT id, score FROM (...)`); every ArcadeDB sparse row records `sparse_result` (`id`) | ArcadeDB only (the embedded and served arms on the sparse table, l3s, and their multipass warm passes) | **built (the user, DECISIONS #153 item 1, adopted as parity)**: `l3_sparse.py`'s two ArcadeDB adapters, `sparse_multipass_driver.py`, `test_sparse_rows.py` | `.notes/bench/repros/perf-sparse-20261004`: the old search returned each hit's whole record twice, about 46 KB of JSON a query, and the ids and their order are identical on 50 of 50 queries | +| 63 | The documents lane loads ArcadeDB embedded through the bindings' columnar insert (own issue #150) instead of `insert_many`'s one JSON payload per batch; ArcadeDB served and every comparator load as before | ArcadeDB embedded only (the ingest columns of both documents tables; no timed query changes) | **built (the user's choice over my recommendation, DECISIONS #153 item 2)**: `Database.insert_columns` in the bindings (`7a36dc650e`, `test_insert_columns.py`), called by `ArcadeTPC.build` in `l1_tpc.py` with `parallel=True` per parquet batch; the row records `columnar_insert` only when the call was made (a wheel without it keeps `insert_many` and says nothing, so a measurement meant to be columnar cannot silently be the other one), and the ingest sentence is chosen from the rows (`export_web._docs_ingest_note`); `test_docs_columnar_load.py`. **The measured wheel must contain the bindings commit:** the stages bake a wheel built from this tree over the official jars (row 74), not the file PyPI holds for 26.10.1 | `.notes/bench/repros/perf-docs-ingest-20261004` (2.24x on the laptop, relative only) | +| 64 | The sparse multipass driver keeps each comparator's version, as the dense driver does: `sparse_multipass_driver.py` stores the adapter's own string as `lib_version` and restores `engine_version` for the ArcadeDB arms | every comparator's sparse multipass warm passes (`l3s_mp`); no timing moves | **built (DECISIONS #150's remit, own issue #158)**: `test_sparse_mp_versions.py`; the October files stay as recorded | BUGS F173 | +| 65 | The documents OLTP and durability operations get a disjoint untimed warm-up for every engine, as row 55 gives the graph reads; the table prints the warm median and keeps the first operation as the cold column; every documents OLTP row records `oltp_warmup` | every engine on the documents OLTP table (docs_oltp) and the document rows of the durability table; goes with row 55 | **built (the user, DECISIONS #157)**: `l1_tpc.py` (`OLTP_WARMUP` = 2,000, `BENCH_DOCS_OLTP_WARMUP`; the timed keys start at the warm-up count so the timed inserts still land at the end of the order index; the warm-up deletes its own single-record rows; rows record `oltp_warmup` and `oltp_warmup_s` and no longer stamp `cold_warm_na`), `test_docs_oltp_warmup.py` (the real SQLite adapter); the count is the laptop's measured one and the bench host's to choose. Not for e2: its transaction runs after 400 timed reads over the same paths and its window was not measured | `.notes/bench/repros/perf-commit-20261004` | +| 66 | `arango_common.calibrate_nprobe` fails fast when the index does not answer (recall at nProbe = nLists below a floor of 0.5), before any timed pass; ArangoDB moves to 3.12.12 with the other comparators (DECISIONS #129), one image digest (`arangodb@sha256:4bc086d5050ca7ea11c6d00a36d8b910c838bb54ad553f8c1b715769d3499bcf`) on the six ArangoDB arms in `runner.py` and in COMPARATORS.md | ArangoDB: the calibration on the dense table (l3d) at both sizes and its multipass overlay; the release move reaches every ArangoDB arm | **built (the user, DECISIONS #156 item 3)**: `test_arango_calibration.py`. No re-run at the October pin with a capped list count (rejected in #156) | BUGS F175 (`.notes/bench/repros/arango-recall-20261004`): on 3.12.11, once an IVF index has 10,000 or more lists, every multithreaded query probes the same single list; 3.12.12 passes the minimal repro | +| 67 | The page's ArangoDB not-comparable declaration keys on the row fields (`ivf_nlists`, `engine_version`, and the recall of a withheld row), so it retires on its own | ArangoDB at 10M on the dense table (l3d); no timing moves | **built (the user, DECISIONS #156 item 2)**: `page_check`'s not-comparable check holds the payload to the same rule, so a row from row 66's release, or with fewer lists, returns as a measurement | BUGS F175 | +| 68 | ArcadeDB's id indexes that a lane uses only for equality become `UNIQUE_HASH` (the maintainers' answer on ArcadeData/arcadedb#9169, PR #9222): `Part(p_partkey)`, `OrderNew(okey)`, `Crud(ckey)`, `orders(id)`, `Product(pid)`, and the graph message-half ids; `Person(id)` (the untimed `person_scan` ranges over it), `Point(host, ts)`, `l_shipdate`, and the lifecycle and recovery indexes stay `UNIQUE` or LSM | ArcadeDB only (embedded and served documents OLTP, tabular, cross-model, and the graph message half); every other engine keeps its primary-key or default index, so no comparator moves | **built (the user, DECISIONS #158); the F14d flip is done (DECISIONS #161 D3): `fairness_check.INDEX_KINDS_GATE_FAILS = True`, so an unstamped row now fails, and a stamped wrong kind always failed.** `fairness_check.ARCADEDB_HASH_ID_INDEXES` is the one registry, `test_index_kinds.py` pins every ArcadeDB `CREATE INDEX` statement in the four lane files to it, every ArcadeDB arm of those lanes records `index_kinds` (the engine's own `SELECT FROM schema:indexes` answer, read after the schema is built and outside every timer), and a generated sentence says the kind and the ingest cost under every table that shows an ArcadeDB arm and an ingest column, built only from rows that carry the stamp. The hash index moves both ways (faster id reads and writes, slower ingest because the loaders insert ascending ids), and the page prints both; the mini build and ingest times are measured in the 26.10.1 measurement | PR #9222 (26.10.1); the laptop measurements are DECISIONS #158's | +| 69 | One served lane gets a sensitivity arm for ArcadeDB at the image's own JVM defaults (75% heap, generational ZGC, no `-Xms`; ArcadeData/arcadedb#9167), printed beside the main arm: documents OLTP, served, the 2M-part cell, 3 repetitions. The main arms keep one JVM configuration on both deployments (G1, heap 50% with `-Xms=-Xmx`) because embedded-versus-served isolates transport only when the JVM is the same | ArcadeDB served only, one lane; no comparator moves and no embedded arm changes | **built (the user, DECISIONS #159)**: `runner.BACKENDS["arcadedb_imgdefaults_server"]`, derived from `arcadedb_server` with the two JVM variables dropped and declared once in `runner.ARM_RUNS` (documents lane, transaction workload only, tpch10, 3 repetitions, relaxed class), from which the stage generator, its coverage check, and the page's roster gate derive; every served ArcadeDB cell records `server_jvm_*` read from the running process, and the arm refuses to run unless the JVM reports 75% of the limit, generational ZGC, and no `-Xms`; `fairness_check` F3b declares it (`JVM_DEFAULTS_ARMS`) and F10 takes its required classes from `runner.ARM_RUNS` (`test_f10_restricted_arm.py`); `test_jvm_defaults_arm.py`; the page prints it with its own label and a sentence generated from the recorded flags | ArcadeData/arcadedb#9167, answered in a new arcadedb-docs section on JVM settings in a fixed memory budget | +| 70 | The TimescaleDB time-series arm runs `ANALYZE p` after its `(host, ts DESC)` index, inside its ingest timer, exactly as the plain PostgreSQL arm does (`l4_tsbs.py` `TimescaleTS.ingest` and `PostgresTS.ingest`), so the two PostgreSQL-family arms are timed on the same steps and the page's l4 ingest sentence says each is one COPY stream followed by an index and an ANALYZE | TimescaleDB, the time-series lane only; no other engine moves | **owed at the re-pin, done on repin-prep-2 (2026-10-06, the user's decision), now on main**: the re-pin re-runs every arm, so no published row is split; October's TimescaleDB l4 ingest rows (417314c18) were measured WITHOUT the ANALYZE and stay as measured | `test_ingest_prose.py` fails if either arm's `ingest` lacks the ANALYZE or places it before the index or inside the index timer; `test_l4_analyze.py` | +| 71 | The served graph arm over Bolt in place of HTTP | ArcadeDB served, the graph arm only | **decided against (DECISIONS #162 item 2)**: with the lean HTTP client (row 72) one bound read costs 121 us over HTTP against 171 to 194 us over Bolt, and upstream's question on which protocol a Python client should use (ArcadeData/arcadedb#9165) is unanswered; the branch `levers-served-bolt` (`21a6ffd47e`) stays unmerged and off by default | `.notes` repros/levers-audit-20261006/LEVERS.md | +| 72 | Every ArcadeDB SERVED arm that talks HTTP uses one persistent `http.client` connection per host (`lean_http.py`) in place of `requests.Session`: about three quarters of a served read was the Python client, not the engine | the 46 served cells measured behind the median (L2-SRV, DO-OLTP-SRV, DUR-NW-SRV, E2-SRV) and every other ArcadeDB served arm that posts to the server (`l1_tabular`, `l1_tpc`, `l2_graph`, `l3_sparse`, `l3d_dense`, `l4_tsbs` x2, `e2_hybrid`, `l5_lifecycle_server`, `l6_restart`); **no comparator client moves**, and the restart lane's one-shot probes, the e4 decomposition probe's own sessions, and the runner's readiness probe stay as they are | **built (the maintainer's decision, DECISIONS #162 item 1), in the 26.10.1 measurement**: the lanes change by one import (`lean_http.Session()`); headers, Basic auth, JSON bodies, gzip and deflate decoding, chunked bodies, timeouts, and the text of a non-2xx are requests' own, a read that died on a reused connection is retried once and a write never; `BENCH_ARCADEDB_HTTP_CLIENT=requests` restores the October client (default `lean`); every served row records `arcadedb_http_client` and the manifest the choice; `test_lean_http.py` and `test_lean_http_live.py` (against `arcadedb-c25:26.10.1`: identical answer digests through both clients). **The e4 decomposition** measures every HTTP arm under both clients in one rotating interleave (`deployment_decomp_probe.py`; `e4_decomp.py` stamps `arcadedb_http_clients`; `export_web._e4_arms` prints each HTTP column once per client and refuses a pin directory whose repetitions disagree; `test_e4_client_axis.py`). **Owed at the re-pin:** the e4 stage re-runs, and the page prose around the table (its caption and the "wire format" paragraph) is rewritten from the new columns. October's served rows (417314c18) were measured with `requests` and carry no `arcadedb_http_client` | `.notes` repros/levers-audit-20261006/LEVERS.md: a small served read or write gets 3 to 4x cheaper on the client side, and a large body or result is engine-bound and does not move, so the served-versus-embedded gap shrinks on the small-operation cells and nowhere else | +| 73 | JPype is pinned to 1.7.1 in the benchmark images and recorded per row; reason and the measurement: `.notes/bench/repros/jpype-master-check-20261006/`. The wheel declares `jpype1>=1.5.0` with no upper bound, so an image rebuilt on the day of a JPype release would change the instrument in the middle of a run and split its rows | every embedded ArcadeDB arm (the image they run in) and the host-side Python-cost stage (the repo venv); the field is on every row of every lane. The wheel's own requirement and `benchmarks/python-bindings/docker/Dockerfile` (the SciPy paper's images) do not change | **built (on main), owed at the re-pin**: `bench_common.JPYPE_PIN` and `Dockerfile.bench`'s `ARG JPYPE_VERSION=1.7.1` (`jpype1==${JPYPE_VERSION}` in the same pip command as the local wheel; the build fails if the imported version is another), `build_images.sh` passes the default down, and `make_2610_stages.py --check` refuses a Dockerfile whose default differs; every generated stage reads JPype back out of `dbbench:arcadedb` after the build (the host-side stage out of the repo venv, and `verify_pair_c25.sh` as its fifth check) and aborts, naming both versions, if the image runs another; every result row records `jpype_version` from the process that ran the cell (empty for an arm that never imports JPype), the manifest records the image's, and `page_check.NOT_PRINTED` declares the field; `test_jpype_pin.py`. October's rows (417314c18) carry no `jpype_version` | JPype's unreleased master is 15 to 25 percent cheaper per raw call than 1.7.1 and does not leak the one Python object per boxed number that 1.7.1 does (jpype#1379), so the next JPype release changes per-call costs and memory behaviour of every embedded arm: the version is a condition of the measurement and not a resolution of the day. Moving the pin is a deliberate edit of `JPYPE_PIN` and the Dockerfile default between campaigns, never an image rebuild | +| 74 | The wheel the 26.10.1 measurement runs on carries the call-overhead change of bindings PR #214 (`fabbda7ab7`) and the columnar insert (row 63) | ArcadeDB embedded arms only (the served arms do not cross the bridge, and no comparator uses this package) | **built and verified (2026-10-06)**: the stages bake a wheel built from this tree (`origin/main` `abd73192e1` merged, commit `8989cbb0e8`) over the official 26.10.1 jars byte for byte (63 of the 65 jars in it; the other two are this tree's bridge and the bundled JRE's `jrt-fs.jar`), so it carries #214 and the columnar insert (row 63); it is NOT the file PyPI holds for the bindings 26.10.1 release (cut from `629b242ccd`, before both). Version `26.10.1`, 72,766,626 bytes, sha256 `83a2a40369a411622696aed77361fd3b1c3d23b675d0a784b38b0b022c13c41b`, recorded in every cell's manifest and baked into every stage; the pair verifies (`PAIR VERIFIED d36b4ca3a`, JVM 25, JPype 1.7.1) and the bindings suite passes on it. October's rows were measured on a bridge without it, so the ArcadeDB embedded rows are not comparable with October's at the call-overhead level, and the page's note on what the Python package costs should name the change when it lands | #214's own A/B (laptop, relative only): a SELECT by key through `to_list()` 0.83x, scalar parameters 0.79x | + +**Retired: settled with no change, listed so nobody re-opens them.** Raising ArcadeDB's statement-cache settings (a knob only ArcadeDB would get; item 2 removes the literal path instead). The sparse ingest path (already the faster one). Durable commit cost (one sync per commit, equal to SQLite's; not a defect). `expected_edge_count` as a speed lever (no measurable effect; item 1 adopts it only because it is the documented setting). **Done, and off the list:** the dense table's phases partition their total (`setup_s`, since 2026-09-22, BUGS F101), every stage estimates its whole cell against its cap before running (`cell_cost_check.py`, BUGS F74), one engine wears one version per page (`version_consistency_check`, a publish gate, BUGS F61), and `queue_lint` refuses the process-name wait that only holds for one exact launch line (BUGS F59; every stage waits on its predecessor's ALL-DONE marker). + +The detail behind item 8, written when it was found (2026-09-23): * **The sparse corpus generator should yield NumPy** (measured 2026-09-23, NOT changed). `bigann_sparse._stream_csr` calls `.tolist()` on every document, so ArcadeDB's sparse arm converts NumPy to a list inside the timer and then hands JPype a list, which crosses element by element. Handing it the array instead measured **3.7% of the whole embedded build** (24.93 s to 24.01 s, 100,000 documents at 120 non-zeros, four alternating reps within 1%). The conversion step alone is 2.4x; against the build it is under 4%, and it moves no comparison -- at 1M documents ArcadeDB already builds in 56.7 s against Milvus 99.2 s and Qdrant 92.3 s. Not worth changing a generator shared by six adapters and three other scripts in front of a multi-day stage. At the re-pin: yield NumPy, have each comparator call `.tolist()` in its own adapter (the same work, still inside its own timer), and leave `sparse_common`'s synthetic generator alone -- it builds Python lists natively, so NumPy there would ADD work. -**October's pin, as run.** The plan of 2026-09-14 (DECISIONS #84) was to run the comparator stages first and queue every ArcadeDB arm behind them once 26.10.1 shipped. October instead measured every ArcadeDB arm at the 26.10.1 snapshot `417314c18` inside the chain, because the page pins an upstream commit, not a release. The official 26.10.1 build is held until the last stage's ALL-DONE and becomes the next pin (item 9); whether it extends October or starts the next measurement is decided when it lands, never by re-measuring over October's rows. - -**ONE LINE OF DEVELOPMENT, and this is why.** The instrument is developed on `main`. Between 2026-09-14 and 2026-09-19 it was developed on two branches at once, and on 2026-09-19 neither could run the campaign: `october-instrument` had the per-query budgets, the abandonment rule and the expanded query sets but none of the raised tiers in `TIMEOUT_BY_SCALE`; `main` had the raised tiers, the publish verification and the coverage rules but no budget mechanism at all. Each half looked finished from inside itself. Assembling them took a 23-file, ~75-hunk merge that had to be redone three times as `main` moved underneath it, and the merge itself introduced two duplicate dict keys and a duplicate module-level binding that silently killed four declarations (BUGS F65). - -A long-lived branch is what made "the instrument is ready" true of two trees and false of both. `october-instrument` was fully merged as of `631450fb3f` (its tip was `3bc6a353c5`) and deleted. One exception, bounded: a change that must NOT reach the running campaign (mini pulls `main` at every stage start) goes on the local `repin-prep` branch in `~/wt-repin`, never pushed, rebased onto `main` and merged at the re-pin; section 7 names each such commit. Anything else lands on `main`, behind a flag if it must, or waits. +**ONE LINE OF DEVELOPMENT, and this is why.** The instrument is developed on `main`. Between 2026-09-14 and 2026-09-19 it was developed on two branches at once, and on 2026-09-19 neither could run the campaign: one had the per-query budgets, the abandonment rule and the expanded query sets but not the raised tiers, the other the raised tiers, the publish verification and the coverage rules but no budget mechanism. Each half looked finished from inside itself. Assembling them took a 23-file, ~75-hunk merge that had to be redone three times as `main` moved, and the merge itself introduced two duplicate dict keys and a duplicate module-level binding that silently killed four declarations (BUGS F65). A long-lived branch is what made "the instrument is ready" true of two trees and false of both. The rule for a running campaign follows: mini pulls `main` at every stage start, so the instrument is landed on `main` before the first cell and never during the run, and a change that must not reach a running campaign waits for the next one (the held changes merged with the 26.10.1 re-pin PR, #212, before its first cell). Anything else lands on `main`, behind a flag if it must, or waits. **What a stage costs is projected from its components, never from an uplift.** Multiplying the previous campaign's cells by an assumed per-lane uplift goes wrong in both directions: a larger query set lengthens a stage several times over, and a stage that runs fewer workloads than the cells it is priced from runs shorter. Project each cell from its measured ingest, first touches, and budget ceiling, as `cell_cost_check` does, and replace a stage's projection with its measured cost once it runs. The current projection lives in HANDOFF. -**The chain is ordered by uncertainty, not by size.** A stage whose figure is the only projection on the board, or whose size an open diagnosis can block, runs early, so a wrong figure is found on day three rather than day thirteen. The well-characterised long stages run late, and the largest runs after the others, so every other table lands before it and an error in it costs no other table. +**The chain is ordered by priority across tiers and by uncertainty inside one.** The 26.10.1 order is the user's (DECISIONS #163 to #165, row 45). Inside a tier, a stage whose figure is the only projection on the board, or whose size an open diagnosis can block, runs early, so a wrong figure is found on day three rather than day thirteen; the well-characterised long stages run late, and the largest runs after the others, so every other table lands before it and an error in it costs no other table. **Before launching, check that every `budgets.py` key names a (lane, tier, query) a lane asks for**, since a name mismatch silently falls back to the lane's constant on every query. A new instrument's queries mostly run on the constant by design: the row records which of the two applied, and step 7 of the per-campaign list regenerates the table from the campaign's own rows. -**One switch at the end.** The live page stays on the September freeze at pin `8d6af9475`, untouched, for the whole campaign; October rows accumulate in their own per-pin file and land table by table on the preview route through `land_stage.py --preview`; when the freeze is complete and every gate is green, one commit copies the preview payload, images, and prose over the live ones and deletes the route (DECISIONS #83). The new columns have no September counterpart, so a partial landing on the live page would seat them beside old rows and the gates would refuse it. The route and the flag are exercised on real October stages, not written on switch day. +**One switch at the end.** The live page stays on the previous freeze (September's, pin `8d6af9475`) untouched for the whole campaign; the campaign's rows accumulate in their own per-pin file (`runs_page_.jsonl`) and land table by table on the preview route through `land_stage.py --preview`; when the freeze is complete and every gate is green, one commit copies the preview payload, images, and prose over the live ones and deletes the route (DECISIONS #83). The new columns have no September counterpart, so a partial landing on the live page would seat them beside old rows and the gates would refuse it. The route and the flag are exercised on real stages, not written on switch day. ## 8. Filing an engine weakness upstream, as a routine @@ -351,7 +313,7 @@ A long-lived branch is what made "the instrument is ready" true of two trees and **Step 8: record it in `.notes/bench/issues-trail.md`** with the ledger count, the arc that found it, and the repro's path. Keep the repro under `.notes/bench/repros/`, not in a scratch directory. -**Step 9: whenever the laptop is idle, hunt** (the user, 2026-10-03: *"hunting using laptop is a good idea"*, and again once it had delivered: *"this hunting on idle laptop is a very good idea. lets keep grinding"*). Mini runs only the measured cell (DECISIONS #141), so everything else runs on the laptop. A hunt asks one question two ways and drafts what disagrees; the agent that runs it never files. Each kind has its own oracle: another engine (openCypher against Neo4j, SQL aggregates against DuckDB), the same engine's other paths (an index against a scan, embedded against HTTP, Postgres wire, and Bolt, every write path against every other), a metamorphic check that needs no second engine (a predicate, its negation, and its `IS NULL` partition must rebuild the table; the same rows in every index layout), fault injection (SIGKILL a writer and damage its files, with the commits the writer acknowledged as the oracle), property-based tests of our own bindings, and fuzzing the parsers for crashes and hangs. Rotate the areas. Done by 2026-10-03: graph, time series, vectors, SQL aggregates, transactions and concurrency, openCypher against Neo4j, the HTTP, Postgres, and Bolt protocols, and schema coercion and constraints. Next: SQL metamorphic, crash recovery, and the bindings' round trips. Not yet tried: the Redis, MongoDB, gRPC, Gremlin, and GraphQL plugins, access control across protocols, parser fuzzing, the documentation's examples run as tests, and the Raft HA cluster. +**Step 9: whenever the laptop is idle, hunt** (the user, 2026-10-03: *"hunting using laptop is a good idea"*, and again once it had delivered: *"this hunting on idle laptop is a very good idea. lets keep grinding"*). Mini runs only the measured cell (DECISIONS #141), so everything else runs on the laptop. A hunt asks one question two ways and drafts what disagrees; the agent that runs it never files. Each kind has its own oracle: another engine (openCypher against Neo4j, SQL aggregates against DuckDB), the same engine's other paths (an index against a scan, embedded against HTTP, Postgres wire, and Bolt, every write path against every other), a metamorphic check that needs no second engine (a predicate, its negation, and its `IS NULL` partition must rebuild the table; the same rows in every index layout), fault injection (SIGKILL a writer and damage its files, with the commits the writer acknowledged as the oracle), property-based tests of our own bindings, and fuzzing the parsers for crashes and hangs. Rotate the areas. Which areas are done, and which come next, is state: it lives in `.notes/bench/HUNT-PLAN.md`, never here. **The review is what makes a draft fileable, and it is always mine.** Re-run every repro on the newest main on BOTH JDKs (Temurin 25 with compact object headers, and 21) and compare with the agent's output; check that each code block is byte-identical to the file that ran; tie every sentence of fact to a saved output (agents have written "no differences" over a matrix that showed some, cited an issue about something else, and described code they had not read at the commit they named); bisect every claim that something is new, on the first-parent history with the engine built at each step, and keep the first bad commit and its good predecessor in the log; read each permalink's lines at the full SHA; search a quoted docs sentence verbatim in the docs repository, and say whether it is about writes or only about lookups; check every cited issue number against its title; write the "Ours:" line (does any harness query take this path); and cut the draft down to what was measured. Search the tracker first, file one topic per issue, cross-reference sibling issues by number, and read the rendered body back from GitHub (one cross-reference went out as `#0` because an f-string evaluated `{0}`, and was patched within minutes). diff --git a/benchmarks/experiments/COMPARATORS.md b/benchmarks/experiments/COMPARATORS.md index 769204a8f90..19ddfd76b91 100644 --- a/benchmarks/experiments/COMPARATORS.md +++ b/benchmarks/experiments/COMPARATORS.md @@ -9,7 +9,7 @@ Which arms are on the October page, with their rows and columns, is PAGE-SPEC.md | Engine | Pin | Version | Lanes | Deployment | Ingest path | |---|---|---|---|---|---| -| ArcadeDB | wheel + `arcadedb-c25:` built from one commit | October's pin is commit `417314c18` (a 26.10.1 snapshot): the matched pair `arcadedb-c25:417314c18` and a 26.10.1.dev0 wheel built from the same commit, held for the whole campaign (CAMPAIGN.md section 7; the page pins a commit, not a release). September's was `26.9.1-dev · 8d6af9475`; the 26.10.1 release is the next campaign's pin. The laptop skeleton runs the stock published image `arcadedata/arcadedb:26.8.1@sha256:49036720b167…`, which its rows name. That stock digest is also the `server_image` BACKENDS carries for every served ArcadeDB arm, and it is what runs if `ARCADEDB_SERVER_IMAGE` is not exported -- which `runner._require_local_server_image()` refuses when a local pin is claimed, after 21 rows were once stamped with a commit they had not run (2026-08-27). Stated here because the code pins it and a pin no document names is a pin nobody checks | all | embedded and served | Python package / Java API; sqlscript over HTTP when served | +| ArcadeDB | wheel + `arcadedb-c25:` built from one commit | The 26.10.1 measurement's pin is the official release's own jar commit `d36b4ca3a` (DECISIONS #129): the matched pair `arcadedb-c25:26.10.1` (also tagged `arcadedb-c25:d36b4ca3a`: the release's 86 jars, unmodified, on Corretto 25) and a 26.10.1 wheel of the same jars plus this tree's bridge and package (CAMPAIGN.md section 7, row 74), held for the whole campaign (the page pins a commit, not a release). The old October chain ran `417314c18` (a 26.10.1 snapshot, `arcadedb-c25:417314c18`), and September `26.9.1-dev · 8d6af9475`. The laptop skeleton runs the stock published image `arcadedata/arcadedb:26.8.1@sha256:49036720b167…`, which its rows name. That stock digest is also the `server_image` BACKENDS carries for every served ArcadeDB arm, and it is what runs if `ARCADEDB_SERVER_IMAGE` is not exported -- which `runner._require_local_server_image()` refuses when a local pin is claimed, after 21 rows were once stamped with a commit they had not run (2026-08-27). Stated here because the code pins it and a pin no document names is a pin nobody checks | all | embedded and served | Python package / Java API; sqlscript over HTTP when served | | PostgreSQL | `postgres@sha256:7341002d…` | 18.6 | documents (TPC); time series (`postgres_ts`, a plain table, from the 26.10.1 measurement, DECISIONS #131 item 6); lifecycle is not PostgreSQL's (no embedded mode) | served | COPY FROM STDIN. A second arm, `postgres_tuned`, derives shared_buffers and friends from the container cap and leaves durability alone; it is an ablation of one image's defaults, so it runs on the lane and is kept off the document tables and the durability table, where it would read as a second engine (DECISIONS #76). Every PostgreSQL arm but `postgres` (the defaults arm by design) has its query pools fitted to the cell since 2026-10-02 (`runner.PG_FIT_CMD`: parallel workers per query = cpuset - 1, `work_mem` from the cap over the cpuset; FAIRNESS F6), read back onto the row. | | DuckDB | `dbbench:duckdb` (duckdb==1.5.4, moved down from 1.5.5 on 2026-09-17: 1.5.4 is the newest DuckDB the community-extensions registry has a DuckPGQ build for, a campaign pins each engine to the latest version at which it can be measured across every table it belongs on, and one engine wears one version per page, so every DuckDB arm moved together; DECISIONS #103d, and #103e for September) | 1.5.4 | documents, time series, dense (VSS), graph (DuckPGQ), cross-model (VSS and DuckPGQ in one engine, from the 26.10.1 measurement, DECISIONS #131 item 4); lifecycle (`duckdb_lifecycle`, from the 26.10.1 measurement, DECISIONS #131 item 5, l5_lifecycle_sql.py: empty, documents, ten indexes, time series as a plain table, dense vectors through VSS, and graph through a DuckPGQ property graph; the analytical view and sparse vectors declared with the engine's answer) | embedded | DataFrame / Arrow INSERT SELECT. The cross-model arm keeps the HNSW index under DuckDB's experimental persistence flag, as the dense arm does, and its hop is a DuckPGQ GRAPH_TABLE MATCH with the pid list written into the text (DuckPGQ binds no parameter) | | DuckPGQ | `dbbench:duckdb` (duckdb==1.5.4) + `INSTALL duckpgq FROM community` at connect (extension version recorded on the row; `f386a6c` on 2026-09-17) | duckdb 1.5.4 + duckpgq | graph | embedded | Arrow INSERT SELECT into the persons and knows tables; a `CREATE PROPERTY GRAPH` over them answers every read in SQL/PGQ (`GRAPH_TABLE ... MATCH`), writes are plain SQL in one transaction (DECISIONS #103d, #92). The community DuckPGQ build exists for 1.5.4 and 404s for 1.5.5 and 1.6.0, which is why every DuckDB arm pins to 1.5.4. Every graph question, interactive and analytics, LSQB's nine included, is expressible in SQL/PGQ and agrees with the other engines on the slice (DECISIONS #104b); the constructs are in `.notes/bench/COMPARATOR-DIALECTS.md`. | @@ -22,7 +22,7 @@ Which arms are on the October page, with their rows and columns, is PAGE-SPEC.md | QuestDB | `questdb/questdb@sha256:931af415…` | 10.0.1 | time series | served | InfluxDB line protocol over TCP on 9009. Re-pinned 2026-09-19 across a MAJOR. QuestDB 10 adds QWP, a binary columnar protocol over WebSocket on the HTTP port, beside the line protocol; `line.tcp.enabled` still defaults to true on 0.0.0.0:9009 in 10.x and only the UDP receiver is deprecated, so this lane keeps the path it has always measured. Moving to QWP would be a new measurement, not a version bump. | | SurrealDB (embedded) | `surrealdb==2.0.0` (Python SDK, in-process) | core 2.3.10, SDK 2.0.0 | cross-model, documents (TPC), graph, dense, time series (plain table), lifecycle (2026-09-16, DECISIONS #95b: its own SurrealKV store on disk, every situation in SurrealQL, the HNSW index for the vector situation; sparse vectors and the analytical view declared with the parse error captured verbatim, and no JVM start on its rows) | embedded, SurrealKV on disk | SDK `insert()` batches, bulk relation insert for edges. Time series (2026-09-15): no time-series type, so a SCHEMALESS table with a datetime `ts` and `DEFINE INDEX p_host_ts ON p FIELDS host, ts` defined before the load, SQLite's footing; buckets by `time::floor(ts, 1m)` / `time::floor(ts, 1h)`; the grouped-ordered-limited query is a subquery because core 2.3.10 sorts by the group key ascending after `GROUP BY` (BUGS F31). | | SurrealDB (served) | `surrealdb/surrealdb@sha256:6a500236…` (`v3.2.4`) | 3.2.4, RocksDB storage | documents (TPC), graph, dense, cross-model, time series (plain table) | served | SDK `insert()` and bulk relation insert over ws. Time series (2026-09-15): the same table, index and SurrealQL as the embedded arm, over ws through the reconnecting client (DECISIONS #91). Dense and cross-model: the HNSW index is defined before the load, which the 3.2.4 server fills in the background, so the build waits inside its index timer until a KNN probe's latency stops falling (`surreal_common.await_hnsw_settled`, recorded as `settle_s`; BUGS F134, DECISIONS #117). | -| MongoDB | `mongo@sha256:41afd6e1…` (`mongo:8.2.12`) | 8.2.12 | documents (TPC), time series (native time-series collection), graph | served | insert_many batches; single-node replica set, because TPC-C new-order needs a multi-document transaction. Client `pymongo==4.18.1` (unmoved, already the latest). Did NOT move in the October re-pin: 8.3.11 cannot boot on the bench kernel (SERVER-121912), so 8.2.12 is the latest measurable MongoDB; see the Unmoved list. The graph arm runs the SAME digest. From the 26.10.1 measurement its two- and three-hop reads use `$graphLookup` (DECISIONS #131 item 2), seeded where the stage's shortest-depth walk equals the exact-length paths the lane counts (two hops at the person, three at each first edge's end), which agreed with the chained form on 200 of 200 read-set ids at micro and on the LDBC slice; the one-hop read, the writes, and the analytics stay `$lookup` joins, which `$graphLookup` cannot state. The loader refuses a self-loop, on which the equivalence depends. The write looks up its anchor person first and writes nothing when it is absent, as the Cypher's MATCH does (until 2026-10-02 it inserted unconditionally, which only a capped slice could show). Every graph question, LSQB's nine included, is expressed as an aggregation pipeline and agrees on the slice (DECISIONS #104b). | +| MongoDB | `mongo@sha256:41afd6e1…` (`mongo:8.2.12`) | 8.2.12 | documents (TPC), time series (native time-series collection), graph | served | insert_many batches; single-node replica set, because TPC-C new-order needs a multi-document transaction. Client `pymongo==4.18.1` (unmoved, already the latest). Did NOT move in the October re-pin: 8.3.11 cannot boot on the bench kernel (SERVER-121912), so 8.2.12 is the latest measurable MongoDB; see the Unmoved list. The graph arm runs the SAME digest. From the 26.10.1 measurement every graph question is asked undirected over one stored edge per friendship, which `$graphLookup` states exactly only over a second, symmetric copy of the edges, so every hop and every analytics query is a chained `$lookup` (CAMPAIGN.md section 7, rows 27 and 56; PROTOCOL.md section 7). The loader refuses a self-loop, on which the undirected reads depend. The write looks up its anchor person first and writes nothing when it is absent, as the Cypher's MATCH does (until 2026-10-02 it inserted unconditionally, which only a capped slice could show). Every graph question, LSQB's nine included, is expressed as an aggregation pipeline and agrees on the slice (DECISIONS #104b). | | MongoDB + MongoDB Search | built image `dbbench:mongo-search` (Dockerfile.mongosearch) FROM `mongo@sha256:41afd6e1…` + `mongodb/mongodb-community-search@sha256:24c0cd30…`; both digests pinned in the Dockerfile | mongod 8.2.12 + mongot-community 1.70.4 | dense vectors, cross-model | served, one container holding both processes | `$vectorSearch` over a `vectorSearch` index at `hnswOptions.maxEdges`=16, `numEdgeCandidates`=100, stage `numCandidates`=100, i.e. the lane's matched point. The vector field's `quantization` is `none` on `mongodb_dense` and `scalar` on `mongodb_dense_int8` (DECISIONS #135, from the 26.10.1 measurement), read back from mongot's index definition onto the row (`mongot_vector_quantization`). mongod runs WITH access control, because mongot authenticates to its sync source with SCRAM and mongod authorizes it through the `searchCoordinator` role; the document arm above does not. Not `mongodb/mongodb-atlas-local`: MongoDB documents that image as "For evaluation only" and "not suitable for production use", and its vector-capable variant is a `:preview` tag. | | pgvector | `pgvector/pgvector@sha256:1d50c689…` (`0.8.6-pg18`) | PostgreSQL 18 + pgvector 0.8.6 | dense (`vector`, HNSW m=16, ef_construction=100, ef_search=100), sparse (`sparsevec`, HNSW, inner product) | served | COPY, then HNSW build. `maintenance_work_mem` sized to the cell's cap for the build, and parallel workers and `work_mem` fitted to the cell (resource fitting, disclosed; FAIRNESS F6). | | TimescaleDB | `timescale/timescaledb@sha256:f7036933…` (`2.30.1-pg18`) | 2.30.1 on PostgreSQL 18 | time series (hypertable, `time_bucket`) | served | COPY into a hypertable, index on (host, ts). One of TSBS's home engines. Pinned at 2.30.1 and never 2.30.0: 2.30.0 introduced the `DeferredChunkAppend` planner node and 2.30.1 fixes that node producing DUPLICATE ROWS, which is an answer bug on this lane's own query shapes. Plain PostgreSQL runs beside it on the same image family (`postgres_ts`: COPY into a plain table, the same (host, ts DESC) index, `date_trunc` buckets), so the table shows what the extension buys. Both have their pools fitted to the cell (FAIRNESS F6). | @@ -33,9 +33,9 @@ Which arms are on the October page, with their rows and columns, is PAGE-SPEC.md | Memgraph | `memgraph/memgraph@sha256:4710bee1…` (`memgraph:3.13.1`, the latest self-hosted stable on 2026-09-17; `latest` resolves to the same digest). License BSL 1.1 for the engine (`memgraph/memgraph` LICENSE), with Memgraph's own Enterprise features gated by a license key that the arm never has; not open source | 3.13.1, IN_MEMORY_TRANSACTIONAL (the documented default: the graph in RAM, WAL and periodic snapshots on disk) | graph; dense (#131, from the 26.10.1 measurement); cross-model (its vector index, from the 26.10.1 measurement, DECISIONS #131 item 4: the index takes no HNSW parameters, so its degree and breadth stay at the engine's defaults, disclosed; candidates ranked by `vector_search.cosine_similarity`, which is L2 order on the lane's unit vectors) | served, Bolt through the same `neo4j==6.3.1` driver the Neo4j arm uses (2026-09-17; moved from 6.2.0 in the October re-pin below, and this row moves with it because the two arms share one driver) | UNWIND batches over bolt, the Neo4j arm's statements verbatim; every query on the lane runs unchanged and every digest matched Neo4j's at micro. Four host-sized or unbounded defaults are set from the cell and read back onto the row: Bolt workers and snapshot threads from the cpuset (FAIRNESS F6), `--memory-limit` at 90% of the container cap (its own rule, which it otherwise applies to the host's physical memory), `--query-execution-timeout-sec=0` in place of the 600 s default so the lane's budget is the only censor, telemetry off. Durability from `SHOW CONFIG`: WAL fsynced every 100,000 transactions by default; the strict class sets 1 (both straced). Memory at SF10: about 1.1 KiB per stored object measured at 10k persons / 207k edges, so 2.3 GiB for the graph inside the tier's 24g. Dense (`memgraph_dense`, 2026-10-02): the same image and flags; `CREATE VECTOR INDEX` (l2sq, f32) after an UNWIND load. Its HNSW parameters are the engine's and fixed: 3.13.1 builds every index from a default USearch 2.21.4 config (connectivity 16, doubled at the base, so hnswlib's M=16; expansion_add 128; expansion_search 64) and ignores any other config key, so ef_construction runs at 128 against the lane's 100 (disclosed on the row) and ef_search is matched by asking the index for 100 results and keeping 10, as the Neo4j arm does. An int8 arm, `memgraph_dense_int8` (DECISIONS #135, from the 26.10.1 measurement), sets `scalar_kind` `i8` in place of `f32`; both read the kind back from `SHOW VECTOR INDEX INFO` onto the row (`memgraph_vector_scalar_kind`). | | FalkorDB | `falkordb/falkordb@sha256:2756fdea…` (6.0.1, the amd64 digest, every arm from 2026-10-02, #129; `falkordb/falkordb@sha256:0a9fe4d1…`, `v4.20.6`, before). License SSPL 1.0 (`FalkorDB/FalkorDB` LICENSE.txt) on Redis 8.6.3 (RSALv2 / SSPLv1 / AGPLv3); not open source under the page's tests | 6.0.1 on Redis 8.10.2 (4.20.6 on Redis 8.6.3 until 2026-10-02) | graph; dense (#131, from the 26.10.1 measurement) | served, the Redis protocol through `falkordb==1.7.1` (`redis==8.1.0`) (2026-09-17) | UNWIND batches as `GRAPH.QUERY` calls, the Neo4j arm's statements verbatim (the client carries `$rows` as a CYPHER parameter prefix); every query on the lane runs unchanged and every digest matched Neo4j's at micro. The image's own `FALKORDB_ARGS` ("MAX_QUEUED_QUERIES 25 TIMEOUT 1000 RESULTSET_SIZE 10000") is replaced: the 1 s timeout would abort the triangle count at micro (1.2 s) and the 10,000-row cap is the ArcadeDB HTTP trap in another engine; `THREAD_COUNT` is passed from the cpuset because 4.20.6 sized it from the host (16 threads under a 12-CPU cpuset, FAIRNESS F6; 6.0.1 follows the cpuset); `BROWSER=0` stops the image's Next.js process. Durability from `CONFIG GET`: RDB snapshots only by default (nothing synced at commit); the strict class runs AOF with `appendfsync always` through `REDIS_ARGS` (both straced). Dense (`falkordb_dense`, 2026-10-02) runs `falkordb/falkordb@sha256:2756fdea…` (6.0.1), because 4.20.6 and 4.22.0 crash on a write inside a vector statement (`repros/falkordb-vector-write/`); the graph arm moved to the same digest on 2026-10-02, smoked at micro in both durability classes and on the analytics workload with every answer matching Neo4j's. `vecf32` properties, `CREATE VECTOR INDEX` at M = the lane's M (hnswlib's), efConstruction and efRuntime 100, euclidean; the index builds in the background, so the build waits for `db.indexes()` to report OPERATIONAL (queried early, recall was 0.05). | -The composed cross-model stack (Qdrant + Neo4j) is not an engine of its own: it is two of the rows above, at the same digests, wired together in one container (its row above). Its vector half was the Qdrant client's pure-Python in-memory local mode until 2026-09-26, so its rows up to then name `qdrant-local`: the September page's read `qdrant-local:1.19.0+neo4j:2026.07.1` (qDT re-ran it on 2026-09-16, 1.19.0 being the client package), and October's rows before `qOM` read `qdrant-local:1.19.1+neo4j:2026.08.1` and are marked `re-run` on the page until `qOM`'s rows against the Qdrant server supersede them. +The composed cross-model stack (Qdrant + Neo4j) is not an engine of its own: it is two of the rows above, at the same digests, wired together in one container (its row above). Its vector half was the Qdrant client's pure-Python in-memory local mode until 2026-09-26, so its rows up to then name `qdrant-local`: the September page's read `qdrant-local:1.19.0+neo4j:2026.07.1` (1.19.0 being the client package), and the old October chain's rows read `qdrant-local:1.19.1+neo4j:2026.08.1` and are marked `re-run` on the page until the 26.10.1 measurement's rows against the Qdrant server supersede them. -**September carries two DuckDB versions, and that is a known defect of the stopped extension.** #103d and #103e say one engine wears one version per page, and the re-measure that would have made that true (stage qEE) was cancelled. DuckPGQ landed on the graph table at DuckDB 1.5.4 while the documents, time-series and dense-VSS rows still read 1.5.5, so the live page shows both until October re-measures every DuckDB arm at one version. Do not quote a DuckDB comparison across those tables without naming the version each row ran. +**September carries two DuckDB versions, and that is a known defect of the stopped extension.** #103d and #103e say one engine wears one version per page, and the re-measure that would have made that true was cancelled. DuckPGQ landed on the graph table at DuckDB 1.5.4 while the documents, time-series and dense-VSS rows still read 1.5.5, so the live page shows both until October re-measures every DuckDB arm at one version. Do not quote a DuckDB comparison across those tables without naming the version each row ran. From 2026-10 every engine above that has a durability setting runs the relaxed commit class, read out of the engine and recorded on the row; FAIRNESS.md F10 holds the mapping and names the engines that cannot be relaxed. @@ -63,7 +63,7 @@ differ). | Milvus | v2.6.13 | v3.0.1 | `sha256:2b2fc2cf499a…` | | Elasticsearch | 9.4.1 | 9.5.4 | `sha256:33178ff49e06…` | | QuestDB | 9.1.1 | 10.0.1 | `sha256:931af4156771…` | -| LadybugDB | `ladybug==0.19.1` | `ladybug==0.20.4` | client package | +| LadybugDB | `ladybug==0.19.1` | `ladybug==0.20.4` (0.21.2 from 2026-10-02, its row above) | client package | | LanceDB | `lancedb==0.37.1` | `lancedb==0.39.0` | client package | | Qdrant client | `qdrant-client==1.19.0` | `1.19.1` | client package | | Elasticsearch client | `elasticsearch==9.5.0` | `9.5.1` | client package | @@ -114,8 +114,9 @@ on the same day and was then at its latest stable release, unless a reason is gi October pin was not chased; the move now is a release change, not a rebuilt layer. The calibration also stops a cell whose index does not answer: it reads recall at nProbe = nLists first, on 20 held-out queries, and raises below a fixed floor of 0.5 (a healthy exhaustive probe reads about 0.995, so the floor is not the calibration target). -- **Memgraph stays at 3.13.1** and **FalkorDB at v4.20.6**; on 2026-09-19 both were the latest - releases and both tags resolved to the pinned digests, unchanged since the September survey. +- **Memgraph stays at 3.13.1**; on 2026-09-19 it was the latest release and its tag resolved to the pinned + digest, unchanged since the September survey. FalkorDB stayed at v4.20.6 that day and moved to 6.0.1 on + 2026-10-02 (its row above). - **Chroma stays at 1.5.9** and **sqlite-vec at 0.1.9**; both are the newest stable releases (sqlite-vec's only higher number, `0.1.10a4`, is a pre-release). - **pymongo 4.18.1, python-arango 8.3.5, falkordb 1.7.1, redis 8.1.0** are all already at the @@ -127,8 +128,8 @@ on the same day and was then at its latest stable release, unless a reason is gi the `python:3.12-slim` base carries, read back at connect. Verified by running after the rebuild, not assumed. - **ArcadeDB is not a comparator** and is untouched here: it is pinned to our own wheel and the - matching `arcadedb-c25:417314c18` server image at commit 417314c18d (DECISIONS #74), which - the stage scripts `make_october_stages.py` writes export as `ARCADEDB_SERVER_IMAGE`. + matching `arcadedb-c25` server image (its row above), which the stage scripts that + `make_october_stages.py` and `make_2610_stages.py` write export as `ARCADEDB_SERVER_IMAGE`. **The jumps that needed their own smoke, and why.** diff --git a/benchmarks/experiments/FAIRNESS.md b/benchmarks/experiments/FAIRNESS.md index ad288162684..355ea0dfe60 100644 --- a/benchmarks/experiments/FAIRNESS.md +++ b/benchmarks/experiments/FAIRNESS.md @@ -16,7 +16,7 @@ The failure mode this contract exists to close is **a correct number measured un A served backend gets the **full tier cap** and the client its own `BENCH_CLIENT_MEM` budget on top, stamped `mem_split="full+client"`, so a served engine sees exactly the cap an embedded engine of the same tier sees. Every frozen served row carries that stamp. `BENCH_SERVER_MEM_FRACTION` restores the older `server = 0.75 * total` split for a reproduction. Compare a served topology by `srv_cap / mem_split`, never by adding client and server: addition reads 1.75x the envelope. -**The edge this rule has on a lane whose DRIVER is memory-hungry, found 2026-09-21.** The rule gives the engine the full cap on both deployments, which is right. But a served cell runs the driver in its OWN container, while an EMBEDDED cell is one process, so on that side the driver competes with the engine for the one cap. It is invisible on most lanes because the driver is small. On `l4` it is not: the lane parses line protocol in Python inside every cell, and at `ts1000` that parser peaked at 7.6 GiB in the served arm's client container. So the embedded engine had roughly 24 GiB of headroom against the served engine's 32, under one stamped `32g` cap. SurrealDB embedded was OOM-killed there and its served arm completed at 27.5 GiB total (BUGS F66). The kill is not explained by this -- that arm was past 24 GiB long before the end -- but any sentence comparing the two deployments' MEMORY on this lane must say which side was carrying the parser, and a claim that one deployment accumulates and the other does not needs more than the two peaks to stand up. **Re-pin (2026-10-03, CAMPAIGN section 7 row 18):** the parsed corpus is now held at about a third of that (89 bytes a point against 281) and freed after ingest, so the parser's share at ts1000 is about 2.2 GiB held until ingest ends, not 7.6 GiB for the whole cell; October's 7.6 GiB stands for the rows measured then, and the `ts1000` rows of the next campaign carry the smaller figure. +**The edge this rule has on a lane whose DRIVER is memory-hungry, found 2026-09-21.** The rule gives the engine the full cap on both deployments, which is right. But a served cell runs the driver in its OWN container, while an EMBEDDED cell is one process, so on that side the driver competes with the engine for the one cap. It is invisible on most lanes because the driver is small. On `l4` it is not: the lane parses line protocol in Python inside every cell, and at `ts1000` that parser peaked at 7.6 GiB in the served arm's client container. So the embedded engine had roughly 24 GiB of headroom against the served engine's 32, under one stamped `32g` cap. SurrealDB embedded was OOM-killed there and its served arm completed at 27.5 GiB total (BUGS F66). The kill is not explained by this -- that arm was past 24 GiB long before the end -- but any sentence comparing the two deployments' MEMORY on this lane must say which side was carrying the parser, and a claim that one deployment accumulates and the other does not needs more than the two peaks to stand up. **Re-pin (2026-10-03, CAMPAIGN section 7 row 18):** the parsed corpus is now held at about a third of that (89 bytes a point against 281) and freed after ingest, so the parser's share at ts1000 is about 2.2 GiB held until ingest ends, not 7.6 GiB for the whole cell; October's 7.6 GiB stands for the rows measured then, and the 26.10.1 measurement's `ts1000` rows carry the smaller figure. **"Same heap" means the heap the engine RAN, not the heap the cell asked for.** A row stamps `heap` from the request, so a hardcoded server heap is invisible in the artifact; Elasticsearch ran 4g at three tiers while its comparators scaled 4g, 8g, 16g, stamped `heap=16g` throughout. `observe_server()` closes it by reading the container's real `-Xmx` back out of `docker inspect` into `server_heap` and failing any cell where the two disagree. Rows without that witness are dropped from published tables by `load_canonical`, so a tier shows a gap rather than an unfair number. @@ -51,7 +51,7 @@ Fitting the pool is resource fitting, the first of the four sanctioned override | Milvus | `go_sched_gomaxprocs_threads 12`; Go sizes from `sched_getaffinity` | cpuset | | Memgraph 3.13.1 (2026-09-17) | `SHOW CONFIG` under `--cpuset-cpus 0-11` on a 16-CPU laptop: `bolt_num_workers` 16 and `storage_snapshot_thread_count` 16, both documented as "the number of processing units available on the machine"; 51 tasks in `/proc/1/task` at idle. Its memory limit is host-sized the same way (`memory_limit` 0 reported as 30.35 GiB inside an 8g container) | **host**; fitted: runner passes `--bolt-num-workers={ncpu}`, `--storage-snapshot-thread-count={ncpu}` and `--memory-limit` at 90% of the cap, and the adapter reads all three back onto the row (`memgraph_bolt_workers`, `memgraph_snapshot_threads`, `memgraph_memory_limit_mib`; 12, 12, 7372 on the laptop smoke at an 8g cap) The cross-model arm (`memgraph_e2`, from the 26.10.1 measurement) gets the same flags and reads the same settings back. | | FalkorDB 4.20.6 (2026-09-17; 6.0.1 from 2026-10-02 sizes the query pool from the cpuset, `THREAD_COUNT` 4 under a 4-CPU cpuset, and keeps the explicit setting) | startup log under the same cpuset: "Thread pool created, using 16 threads" and "Maximum number of OpenMP threads set to 12"; `GRAPH.CONFIG GET THREAD_COUNT` 16, `OMP_THREAD_COUNT` 12. The query pool reads the host's logical cores, the GraphBLAS pool reads the affinity mask | **host** for the query pool, cpuset for OpenMP; fitted: runner passes `THREAD_COUNT {ncpu}` in `FALKORDB_ARGS`, the log then reads "using 12 threads" and the adapter records `falkordb_thread_count` and `falkordb_omp_threads` from `GRAPH.CONFIG GET` (12 and 12 on the laptop smoke) | -| LadybugDB 0.20.4 (2026-09-29, BUGS F160) | not audited on 2026-08-01 nor listed as owed; found from LadybugDB/ladybug#1070: `CALL current_setting("threads")` reads the host's CPU count under a cpuset (16 under `--cpuset-cpus 4-7` on the 16-CPU laptop, affinity 4), which is 20 on mini's 12-CPU cells, and the buffer pool defaults to 0.8 of physical RAM, about 49 GB against the 24 GB cell cap. A laptop A/B on the lane's own adapter at 20 against 12 threads put its reads 12-28% slow and its writes 12% fast | **host**; fitted 2026-09-29: the adapter passes `max_num_threads` from `sched_getaffinity` and `buffer_pool_size` at 0.8 of the cgroup `memory.max`, reads the thread count back, and records `ladybug_threads` and `ladybug_buffer_pool_mib` (4 and 3276 in a 4-CPU, 4 GB container); stage `qOA5` re-runs its OLTP rows, and `qOB2` pulls the fitted arm The cross-model arm (`ladybug_e2`, from the 26.10.1 measurement) opens through the same fit and records `ladybug_threads` and `ladybug_buffer_pool_mib`. | +| LadybugDB 0.20.4 (2026-09-29, BUGS F160) | not audited on 2026-08-01 nor listed as owed; found from LadybugDB/ladybug#1070: `CALL current_setting("threads")` reads the host's CPU count under a cpuset (16 under `--cpuset-cpus 4-7` on the 16-CPU laptop, affinity 4), which is 20 on mini's 12-CPU cells, and the buffer pool defaults to 0.8 of physical RAM, about 49 GB against the 24 GB cell cap. A laptop A/B on the lane's own adapter at 20 against 12 threads put its reads 12-28% slow and its writes 12% fast | **host**; fitted 2026-09-29: the adapter passes `max_num_threads` from `sched_getaffinity` and `buffer_pool_size` at 0.8 of the cgroup `memory.max`, reads the thread count back, and records `ladybug_threads` and `ladybug_buffer_pool_mib` (4 and 3276 in a 4-CPU, 4 GB container); its rows are re-run in the 26.10.1 measurement. The cross-model arm (`ladybug_e2`, from the 26.10.1 measurement) opens through the same fit and records `ladybug_threads` and `ladybug_buffer_pool_mib`. | | PostgreSQL + AGE 1.8.0 on PostgreSQL 18.6 (2026-10-02, both AGE arms) | `SHOW` under an 8-CPU cpuset and a 16 GiB cap: `max_parallel_workers_per_gather` 2 and `work_mem` 4MB, PostgreSQL's fixed defaults rather than host-sized ones, so not F6's bug as stated; but every other engine's pools reach the cell's cores and cap, and these reach 3 of 12 cores and spill rather than use the cap. Measured on the full SF1 network (`repros/age-dialect/resource_fit_probe.py`): workers fitted to the cpuset made LSQB q2 1.8x faster and q4, q5, q7 1.1-1.25x; `work_mem` fitted from the cap cut temp-file spill from 42 GB to 0.7 GB per analytics pass at no net time cost, with no OOM kill; four times that bought nothing | **fixed default**; fitted: the runner passes `max_parallel_workers_per_gather` and `max_parallel_maintenance_workers` = cpuset - 1, `max_parallel_workers` = cpuset, and `work_mem` = (cap x 0.75) / (cpuset x 16), and the graph adapter reads every one back onto the row (`pg_*`); the cross-model rows carry them in `server_cmd` | | PostgreSQL 18.6 arms: `postgres_tuned`, `timescaledb`, `pgvector_dense`, `pgvector_sparse`, and the new `postgres_ts` (2026-10-02) | the same fixed defaults as AGE's (2 workers per query, a pool of 8, `work_mem` 4MB; `postgres_tuned` carried a constant 64MB). Same-run A/B on the laptop, defaults against fitted (`BENCH_PG_FIT=off`), answers identical in every pair, no OOM kill: TPC-H Q1 at SF1 on `postgres_tuned` 1.47x faster fitted, Q6 1.13x; the plain-table time series' per-host hourly aggregate 1.35x, its other queries unchanged, and q_last on the same plan either way (EXPLAIN, interleaved in one session); TimescaleDB unchanged within run noise; pgvector dense and sparse search unchanged and recall equal (0.997 and 0.94 both ways), the HNSW build a little faster with more maintenance workers | **fixed default**; fitted with AGE's rule through `runner.PG_FIT_CMD` on every PostgreSQL arm except `postgres`, the defaults arm by design; the time-series arms read every setting back onto the row (`pg_*`), the others carry them in `server_cmd`. `BENCH_PG_FIT=off` exists for the A/B only | @@ -304,7 +304,7 @@ against an already-warm database by construction; the lifecycle lane is itself the cold measurement; the two vector lanes warm on a held-out slice before they time anything, and their cold and warm columns come from the multipass driver. -**F13. Close cost is an invariant, not a column.** Close should be O(what was written), not O(what is stored), and on the order of 100 ms; the reasoning, the situations, and the numbers are PAGE-SPEC.md section 4 (DECISIONS #50). Checked by `fairness_check.check_close_cost`, which prints under this number. Renumbered from F11 here because F11 and F12 are the October instrument's two rules above; `fairness_check.check_close_cost` still prints its section header as F11, and that mismatch is recorded rather than silently renamed in a gate's output. +**F13. Close cost is an invariant, not a column.** Close should be O(what was written), not O(what is stored), and on the order of 100 ms; the reasoning, the situations, and the numbers are PAGE-SPEC.md section 4 (DECISIONS #50). Checked by `fairness_check.check_close_cost`, which prints under this number. Renumbered from F11 here because F11 and F12 are the October instrument's two rules above; `fairness_check.check_close_cost` prints F13 too (it printed F11 until CAMPAIGN.md section 7, row 31, and `test_fairness_headers.py` holds the numbering against this file). **F14. Equivalent queries get equivalent index support, and the support is chosen by MEASUREMENT (DECISIONS #112, BUGS F98).** Every engine reaches a query through some access path, and which path it gets is a choice this harness makes on its behalf. Until 2026-09-22 that choice was made by reasoning about what ought to help, and it was wrong in both directions at once: the document lane gave ArcadeDB an `l_shipdate` index worth **6.1x** on Q6 and gave PostgreSQL, which gains **2.5x** from the same index, none; it gave SQLite one that costs it **20%**; and the time-series lane gave every engine a `(host, ts)` path except DuckDB, which wants one. @@ -315,7 +315,7 @@ Three things follow, and they are what make this checkable rather than a good in * **A new arm must declare its index decision.** `fairness_check` fails a lane backend that declares nothing, the way the capability table refuses a kind its legend cannot define. Silence is the state this invariant exists to remove. * **And the declaration is checked against the rows, not trusted (F14b).** A map saying an arm builds an index is a claim; `index_s` on that arm's rows is the evidence. The gate fails an arm whose declaration cites a measured with/without ratio and whose every row records `index_s=0`, and fails one that declares `NONE` and records a build -- an index removed for costing that engine coming back. Only those two directions are asserted: a declaration like "record id carries pid" or "native TIMESERIES type" describes an index with no build step to time, and those arms are reported as not asserted rather than pattern-matched into an expectation. The count of arms checked, arms not asserted, and arms whose rows predate the ingest/index split is printed every run, so the coverage is visible rather than implied. * **And where one arm times the two phases, every arm does (F14c).** The ingest/index split turns "this engine took 40 s to build the corpus" into "12 s writing, 28 s indexing", and it arrives per adapter -- so it is exactly the kind of thing that stops arriving when someone adds the next engine to a lane and writes a `build()` without knowing its neighbours time two phases inside theirs. The cost is not a missing number but an unfair TABLE: the column exists because the other arms fill it, so the new arm prints a blank where everyone else prints a figure, and a blank in a benchmark reads as a result. Three states pass -- the arm splits, or it declares `index_before_load` (the index is defined before the first row lands and its work is spread through the load, as SurrealDB does on the document and time-series lanes), or no arm on that lane splits at all (on the sparse lane every engine but pgvector builds its index as it ingests, and pgvector's index build follows its load inside the same single timer; on the graph lane every engine creates its index in the schema before loading). The finding is the fourth: a lane where some arms split, one does not, and it says nothing. On the dense lane the two phases are independent timers inside `build_s` rather than a division of it, so those rows also carry `setup_s` for the remainder, and the three add up (BUGS F101). -* **ArcadeDB's id indexes follow the vendor's rule for the index kind, and the page says which are which.** ArcadeDB's maintainers answered (ArcadeData/arcadedb#9169) that an id a workload only reads, updates, and deletes by equality takes `UNIQUE_HASH`, and that a key also ranged over or ordered stays `UNIQUE` (a sorted LSM tree). The documents lane's `Part(p_partkey)`, `OrderNew(okey)`, and `Crud(ckey)`, the tabular lane's `orders(id)`, the cross-model lane's `Product(pid)`, and the graph lane's LDBC message-half ids are hash indexes from the 26.10.1 re-pin; `Person(id)` stays sorted because the graph lane's untimed `person_scan` ranges over it, and `l_shipdate`, `customer_id`, the time-series `Point(host, ts)`, and the lifecycle and recovery indexes are not equality-only ids. Each comparator keeps the primary-key or default index its own maintainers give it, so this is the same footing and not a knob only ArcadeDB gets. It moves both ways: hash lookups are faster and the hash build is slower than the sorted build when ids arrive in ascending order, as every loader here sends them, so the ingest and index columns carry a cost and the transaction columns a gain, and the table says so in a generated sentence. `fairness_check.ARCADEDB_HASH_ID_INDEXES` is the one registry of which ids those are, and `test_index_kinds.py` holds it equal to the `CREATE INDEX` text in each lane. The registry is the claim and the row is the evidence (F14d): each ArcadeDB arm on those lanes records `index_kinds`, the engine's own `SELECT FROM schema:indexes` answer after the schema is built, a stamped row with another kind is a failure from the first day, and a row without the stamp is a warning until the re-pin campaign starts and a failure after (one constant, `INDEX_KINDS_GATE_FAILS`, flipped in the commit that starts it), and the page sentence is built only from rows that carry the stamp, so a page regenerated over rows measured before the stamp existed claims nothing. +* **ArcadeDB's id indexes follow the vendor's rule for the index kind, and the page says which are which.** ArcadeDB's maintainers answered (ArcadeData/arcadedb#9169) that an id a workload only reads, updates, and deletes by equality takes `UNIQUE_HASH`, and that a key also ranged over or ordered stays `UNIQUE` (a sorted LSM tree). The documents lane's `Part(p_partkey)`, `OrderNew(okey)`, and `Crud(ckey)`, the tabular lane's `orders(id)`, the cross-model lane's `Product(pid)`, and the graph lane's LDBC message-half ids are hash indexes from the 26.10.1 re-pin; `Person(id)` stays sorted because the graph lane's untimed `person_scan` ranges over it, and `l_shipdate`, `customer_id`, the time-series `Point(host, ts)`, and the lifecycle and recovery indexes are not equality-only ids. Each comparator keeps the primary-key or default index its own maintainers give it, so this is the same footing and not a knob only ArcadeDB gets. It moves both ways: hash lookups are faster and the hash build is slower than the sorted build when ids arrive in ascending order, as every loader here sends them, so the ingest and index columns carry a cost and the transaction columns a gain, and the table says so in a generated sentence. `fairness_check.ARCADEDB_HASH_ID_INDEXES` is the one registry of which ids those are, and `test_index_kinds.py` holds it equal to the `CREATE INDEX` text in each lane. The registry is the claim and the row is the evidence (F14d): each ArcadeDB arm on those lanes records `index_kinds`, the engine's own `SELECT FROM schema:indexes` answer after the schema is built, a stamped row with another kind is a failure from the first day, and a row without the stamp is a failure since the 26.10.1 measurement started (one constant, `INDEX_KINDS_GATE_FAILS`, flipped in the commit that started it and now `True`), and the page sentence is built only from rows that carry the stamp, so a page regenerated over rows measured before the stamp existed claims nothing. * **The decision names its evidence.** "No index" is a finding when it is measured (ArangoDB, DuckDB on the document lane) and a defect when it is an omission (PostgreSQL on the document lane, DuckDB on the time-series lane, both until 2026-09-22). * **Index build time is its own column** wherever the engine has a boundary to time. The dense vector table has always separated `ingest s` from `index s`; the rest folded index build into ingest, which hides both the cost and the asymmetry -- SurrealDB's document arm must build its index BEFORE the load, so about 5.3 s of index work sits inside an ingest number every other engine pays without any. @@ -351,7 +351,7 @@ Two consequences: - **Vendor settle steps** that have no equivalent elsewhere (Elasticsearch forcemerge, Milvus flush+load, Qdrant green-wait, ArcadeDB `COMPACT INDEX`). Each engine gets *its own*; none goes unmatched by the others having theirs. The one settle step this harness defines rather than a vendor is the SurrealDB server's HNSW wait on the dense and cross-model lanes, a latency probe recorded as `settle_s` (BUGS F134, DECISIONS #117), and PROTOCOL.md section 7 lists it as such. - **Operating points deliberately not matched**, such as the dense fp32 arms with the build cache pinned to the corpus against INT8 at the engine default (DECISIONS #56), stated in the l3d condition. - **Quality and precision differences** (int8 against fp32 postings, ES pruning). Report recall next to latency, always. -- **Intra-query parallelism at each engine's default.** ArcadeDB's SQL scans a type's buckets in parallel (`arcadedb.queryParallelScan`, on by default) only when the type has at least two buckets, and `arcadedb.typeDefaultBuckets` is 1. Every ArcadeDB type in this benchmark is created with the default, so its full scans run on one thread, as SQLite's and MongoDB's do, while DuckDB uses the whole cpuset (true at the October pin; 26.10.1 splits one bucket into page ranges for a parallel scan, PR #8524, CAMPAIGN.md section 7, item 10). Not tuned: it is a knob that moves only ArcadeDB, so the default stands until a campaign decides otherwise (decided 2026-09-23; CAMPAIGN.md section 7, item 10). A served ArcadeDB read is sequential for a second reason: `POST /api/v1/query` and `POST /api/v1/command` run it inside an auto-commit transaction, where a scan never runs in parallel (ArcadeData/arcadedb#8775). Harmless at the October pin, where no type scan ran in parallel in either deployment; the served reads move to a path with no transaction at the re-pin (CAMPAIGN.md section 7, row 25, BUGS F163). The per-record cost that makes that one thread slow is engine code, filed as ArcadeData/arcadedb#8260. +- **Intra-query parallelism at each engine's default.** ArcadeDB's SQL scans a type's buckets in parallel (`arcadedb.queryParallelScan`, on by default) only when the type has at least two buckets, and `arcadedb.typeDefaultBuckets` is 1. Every ArcadeDB type in this benchmark is created with the default. At the October pin that meant its full scans ran on one thread, as SQLite's and MongoDB's do, while DuckDB uses the whole cpuset; from 26.10.1 (PR #8524) one bucket is split into page ranges across the workers, so a scan outside a transaction runs in parallel at the engine default. Not tuned: the bucket count is a knob that moves only ArcadeDB, so the default stands until a campaign decides otherwise (decided 2026-09-23; CAMPAIGN.md section 7, item 10). A served ArcadeDB read through `POST /api/v1/command` runs inside an auto-commit transaction, where a scan never runs in parallel (ArcadeData/arcadedb#8775); the served reads go through `POST /api/v1/query`, which runs an idempotent statement without one (CAMPAIGN.md section 7, row 25, BUGS F163). The per-record cost that makes a single-threaded scan slow is engine code, filed as ArcadeData/arcadedb#8260. Anything else that differs is a defect, not an override. diff --git a/benchmarks/experiments/PAGE-SPEC.md b/benchmarks/experiments/PAGE-SPEC.md index f6b5789ccc6..ca947a2bf5c 100644 --- a/benchmarks/experiments/PAGE-SPEC.md +++ b/benchmarks/experiments/PAGE-SPEC.md @@ -1,6 +1,6 @@ # Project page spec: what the page contains and what each cell must satisfy -The page at `humem.ai/projects/arcadedb` is the PRIMARY artifact. There is no paper; "page material" is the only kind. This file is the contract: what tables exist, what rows and columns they carry, which figures are published, and what a cell must satisfy to be printed. +The page at `humem.ai/projects/arcadedb` is the PRIMARY artifact. A paper, when there is one, is written from the same frozen rows at a stable release (DECISIONS #42); this file governs what the page shows. This file is the contract: what tables exist, what rows and columns they carry, which figures are published, and what a cell must satisfy to be printed. `PROTOCOL.md` says how a number is produced. `FAIRNESS.md` says when two numbers may be compared. This file says what gets shown. If they disagree, PROTOCOL wins on production and FAIRNESS wins on comparison; this file never licenses a cell those two would refuse. @@ -48,10 +48,10 @@ The page's setup section is this block, read from `lscpu` and `lsblk`, not deriv ## 1. Engine identity: commit, not version -ArcadeDB is pinned by **upstream commit SHA**, not by a release number. The build line reads `.dev0` for every commit of a release line (`26.9.1.dev0` in September, `26.10.1.dev0` in October), so a version string cannot identify what ran, and printing one while claiming the page ships the upstream engine unmodified is a contradiction. +ArcadeDB is pinned by **upstream commit SHA**, not by a release number. The build line reads `.dev0` for every commit of a release line (`26.9.1.dev0` in September, `26.10.1.dev0` in October), so a version string cannot identify what ran, and printing one while claiming the page ships the upstream engine unmodified is a contradiction. A stable release pin reads its release number (`26.10.1`), and `engine_commit` is still the identity: the jar's own build number. - Every row carries `engine_commit` (short SHA of the upstream commit the wheel and server image were both built from). -- The page prints it as the engine identity, linked to the commit on GitHub. Every ArcadeDB row is named from its own engine string and commit (`arcadedb 26.9.1-dev · 8d6af9475` on the September page, `417314c18` in October), never from an image tag; served rows carry the row's `server_image_ref` (BUGS.md F16, F32). The page header lists the identities actually present. +- The page prints it as the engine identity, linked to the commit on GitHub. Every ArcadeDB row is named from its own engine string and commit (`arcadedb 26.9.1-dev · 8d6af9475` on the September page, `d36b4ca3a` at the 26.10.1 release), never from an image tag; served rows carry the row's `server_image_ref` (BUGS.md F16, F32). The page header lists the identities actually present. - Embedded and server arms in one table are built from the SAME commit, paired, and run the same JVM (DECISIONS #54). - A campaign FREEZES one commit and holds it start to finish. Upstream landing a fix mid-run does not restart the campaign; it becomes a dated changelog entry and a candidate for the next re-pin. - Re-pinning is a deliberate, dated act recorded in the changelog with what moved. No table falls back to an older pin, anywhere (DECISIONS #62). @@ -72,7 +72,7 @@ The generated block "Published tables" at the end of this file is the list of ta The columns below are the 2026-10 instrument's, which carries the operations of PROTOCOL.md section 2 and the one measurement set of DECISIONS #89 as amended: one cold column per CELL (`cold_first_query_ms`) and warm latency per query at p50 and p99, throughput where the operation has a rate, recall where the index is approximate, peak memory, on-disk size after the workload, and ingest and index build as separate columns on the vector tables. A table that omits one of those states why in its conditions, which `page_check` enforces. An operation an engine cannot express is a declared absence (DECISIONS #88): the cell is a dash and the condition names the engine and the operation, so it cannot be read as a slow engine or as a gap still to be filled. **A dash means that and only that (DECISIONS #111).** An engine that RAN and did not finish keeps its row and prints the outcome in the cell -- `>2h` for the tier's cap (derived from `runner.TIMEOUT_BY_SCALE`, so `>15m` through `>48h`), `OOM` for an envelope failure, `lost` for a dropped connection, `err` for a failure inside budget whose cause will not classify, `wd` for a withheld wrong answer, `re-run` for a cell measured before a fix to our own harness and being measured again -- because a vanished row reads as an untested engine, which flatters whichever engines finished. The legend is generated from the marks present and refuses one it cannot define. A harness failure never gets an engine mark, and a cell skipped because we expected it to be slow is published as its bound or not at all, never as a blank. The queue chain is CAMPAIGN.md section 6. -**`re-run` marks our own mistake, and clears itself (DECISIONS #117).** A cell or a whole row measured before a fix to this harness stays on the table with `re-run` where its number would be, and a generated note under the table says what was wrong, so a reader sees the engine is on the table and why its number is not; a marked cell carries text rather than a median (and a marked row an `outcome`), so nothing that ranks or averages reads it as a measurement. `export_web.STALE_UNTIL_RERUN` lists each case as a table, an engine, a column (or none, for the whole row), and a test that recognises a row measured BEFORE the fix, and a cell is marked only while the rows behind it pass that test. It is keyed on row fields, never on a date: the re-measured rows supersede the old ones under the canonical-row rule and fail the test, so the numbers return at the landing that carries them, with nothing to remember and no list to edit. Today it covers SurrealDB's graph-filtered search on `e2` (BUGS F132), the SurrealDB server's `e2` retrieval at 500k (F134), and the whole Qdrant + Neo4j row on `e2` (F133; its `e2atom` row stays), all three re-measured by `qOM`. Since 2026-10-02 it also covers Neo4j's whole row on `l3d` and on `e2` (F164: its vector index ran Neo4j's default binary quantization, which no definition overrode, while the rows recorded fp32; its `e2atom` row stays), recognised by the missing `neo4j_vector_quantization` read-back and re-measured in the 26.10.1 measurement. `_rows_behind` reads a table whose lane has one workload (mapped to `None`) as any workload, which the dense, sparse, time-series, and lifecycle tables need for an entry to fire. The lifecycle table has its own map, `export_web.LIFECYCLE_STALE`, because its entries fold by situation and size: it marks SurrealDB's peak memory and its 10M dense-vector row (BUGS F161, DECISIONS #134: its build held the whole input as one Python list, and the 10M vector kill was that list's), recognised by the October pin and re-measured in the 26.10.1 measurement; a row it withdraws is also kept out of the censored-cell notes, so our kill never prints as the engine's. +**`re-run` marks our own mistake, and clears itself (DECISIONS #117).** A cell or a whole row measured before a fix to this harness stays on the table with `re-run` where its number would be, and a generated note under the table says what was wrong, so a reader sees the engine is on the table and why its number is not; a marked cell carries text rather than a median (and a marked row an `outcome`), so nothing that ranks or averages reads it as a measurement. `export_web.STALE_UNTIL_RERUN` lists each case as a table, an engine, a column (or none, for the whole row), and a test that recognises a row measured BEFORE the fix, and a cell is marked only while the rows behind it pass that test. It is keyed on row fields, never on a date: the re-measured rows supersede the old ones under the canonical-row rule and fail the test, so the numbers return at the landing that carries them, with nothing to remember and no list to edit. Each case names the BUGS entry for its fix, and every one is re-measured in the 26.10.1 measurement. `_rows_behind` reads a table whose lane has one workload (mapped to `None`) as any workload, which the dense, sparse, time-series, and lifecycle tables need for an entry to fire. The lifecycle table has its own map, `export_web.LIFECYCLE_STALE`, because its entries fold by situation and size: it marks SurrealDB's peak memory and its 10M dense-vector row (BUGS F161, DECISIONS #134: its build held the whole input as one Python list, and the 10M vector kill was that list's), recognised by the October pin and re-measured in the 26.10.1 measurement; a row it withdraws is also kept out of the censored-cell notes, so our kill never prints as the engine's. **Sizes (DECISIONS #108, amending #103b and #103c).** Each table carries TWO sizes wherever a second one is affordable and informative, because one point cannot tell a query that moved elevenfold from one that did not move at all. #103c's "one size by default" and #103b's replacement rule are both retired: the large size is a ROW GROUP beside the small one rather than a switch, and an engine with no row at it is a declared outcome under #103g (censored, envelope failure, or failed inside budget) rather than a reason to withhold every other engine's rows. Graph analytics is the single one-size exception, the full SF1 network, because the full network at SF10 is not feasible. No table accumulates two generations of ONE size and no new table is made for a size. The skeleton runs every lane at its smallest laptop corpus with the size column naming that corpus (`export_web.SKELETON_SCALE_LABELS`), so the sizes named below are the campaign's, and the skeleton's labels say "skeleton" so the two cannot be confused. @@ -84,7 +84,7 @@ The columns below are the 2026-10 instrument's, which carries the operations of | id | title | rows | columns | |---|---|---|---| | `l3s` | Sparse vector search | ArcadeDB embedded and server, each int8 and fp32; Elasticsearch, Milvus, pgvector, Qdrant. ArangoDB, MongoDB, and SurrealDB have no row and are declared under the table (cannot express: none has a sparse index type, each with its own documentation cited, DECISIONS #95b), which is what the coverage table reads | p50 ms, p99 ms, recall@10, ingest+index vectors/s, ingest+index total s, peak memory GiB, disk GiB. The cold and warm columns with their gain come from the sparse multipass driver on the bench host (DECISIONS #89) and are absent from the skeleton | -| `l3d` | Dense vector search | ArcadeDB embedded and server, each int8 and fp32; ArangoDB (fp32), Chroma (fp32), DuckDB VSS (fp32, DuckDB 1.5.4), LanceDB (int8), Milvus (int8 and fp32), MongoDB (fp32, through MongoDB Search in the same container), Neo4j (fp32), pgvector (fp32), Qdrant (int8 and fp32), sqlite-vec (int8 and fp32), SurrealDB embedded and server (fp32). The int8 arms of Milvus, Qdrant, and sqlite-vec print at 9.99M only (`make_paper_tables.MP_ARMS_SMALL`). | cold p50 ms, cold p99 ms, after insert p50 ms, after delete p50 ms, insert into index ms/vector, recall@10 after insert, delete from index ms/vector, recall@10 after delete, index s, recall@10, ingest+index vectors/s, ingest s, ingest+index total s, peak memory GiB, disk GiB. Warm columns come from the dense multipass driver on the bench host and are absent from the skeleton | +| `l3d` | Dense vector search | ArcadeDB embedded and server, each int8 and fp32; ArangoDB (fp32), Chroma (fp32), DuckDB VSS (fp32, DuckDB 1.5.4), LanceDB (int8), Milvus (int8 and fp32), MongoDB (fp32, through MongoDB Search in the same container), Neo4j (fp32), pgvector (fp32), Qdrant (int8 and fp32), sqlite-vec (int8 and fp32), SurrealDB embedded and server (fp32). The int8 arms of Milvus, Qdrant, and sqlite-vec print at 9.99M only (`make_paper_tables.MP_ARMS_SMALL`). The arms of CAMPAIGN.md section 7 rows 37, 38, and 42 (Elasticsearch, Memgraph, FalkorDB, LadybugDB, and the int8 variants QUANTIZATION.md lists) have no October rows and run in tier 3 of the 26.10.1 measurement. | cold p50 ms, cold p99 ms, after insert p50 ms, after delete p50 ms, insert into index ms/vector, recall@10 after insert, delete from index ms/vector, recall@10 after delete, index s, recall@10, ingest+index vectors/s, ingest s, ingest+index total s, peak memory GiB, disk GiB. Warm columns come from the dense multipass driver on the bench host and are absent from the skeleton | Sizes: `l3s` three, 100k / 1M / 8.84M, the whole Big-ANN sparse track; `l3d` two, 1M and 10M (SIFT1M and DEEP-10M), both kept because the vector tables exist to show scaling (#108). The sparse table has one ingest timer: every engine on it builds the index as it ingests except pgvector, whose index build follows its load and sits inside the same single timer on this table; the dense table records `ingest s` and `index s` separately where the engine has the boundary, beside the single pair (#74). The two dense mutation operations, insert into a built index and delete from one, each followed by a search and its recall, run at the 1M tier only (#82d) and are forced on at the skeleton's micro size. @@ -98,10 +98,10 @@ Both dense sizes read one protocol: `_dense_overlay_entries(scale)` reads `dense | id | title | rows | columns | |---|---|---|---| -| `l2` | Graph OLTP | ArcadeDB embedded and server; ArangoDB, DuckPGQ (DuckDB 1.5.4 with the community DuckPGQ extension, SQL/PGQ), FalkorDB, LadybugDB, Memgraph, MongoDB, Neo4j, SurrealDB embedded and server | point p50 ms, point p99 ms, 1-hop p50 ms, 2-hop p50 ms, 3-hop filtered p50 ms, insert p50 ms, update p50 ms, delete p50 ms, ingest vertices+edges/s, ingest total s, peak memory GiB, disk GiB | -| `l2olap` | Graph OLAP | ArcadeDB embedded and server with the view built (labelled GAV), beside the view-off arms of section 4b in the campaign; ArangoDB, DuckPGQ, FalkorDB, LadybugDB, Memgraph, MongoDB, Neo4j, SurrealDB embedded and server. | average friend age p50 ms, average friend age p99 ms, friends in same city p50 ms, most friends p50 ms, degree distribution p50 ms, triangle count p50 ms, LSQB Q1 p50 ms through LSQB Q9 p50 ms, cold first query ms, view build s, ingest vertices+edges/s, ingest total s, peak memory GiB, disk GiB | +| `l2` | Graph OLTP | ArcadeDB embedded and server; ArangoDB, DuckPGQ (DuckDB 1.5.4 with the community DuckPGQ extension, SQL/PGQ), FalkorDB, LadybugDB, Memgraph, MongoDB, Neo4j, PostgreSQL + AGE (DECISIONS #128), SurrealDB embedded and server | point p50 ms, point p99 ms, 1-hop p50 ms, 2-hop p50 ms, 3-hop filtered p50 ms, insert p50 ms, update p50 ms, delete p50 ms, ingest vertices+edges/s, ingest total s, peak memory GiB, disk GiB | +| `l2olap` | Graph OLAP | ArcadeDB embedded and server with the view built (labelled GAV), beside the view-off arms of section 4b in the campaign; ArangoDB, DuckPGQ, FalkorDB, LadybugDB, Memgraph, MongoDB, Neo4j, PostgreSQL + AGE, SurrealDB embedded and server. | average friend age p50 ms, average friend age p99 ms, friends in same city p50 ms, most friends p50 ms, degree distribution p50 ms, triangle count p50 ms, LSQB Q1 p50 ms through LSQB Q9 p50 ms, cold first query ms, view build s, ingest vertices+edges/s, ingest total s, peak memory GiB, disk GiB | -Interactive (`l2`): seven operations, point lookup, one hop, two hops, three hops filtered on the far end, insert, update, and delete, every read against a fresh seed set per repetition and the three writes 1,000 each per repetition. The writes run in both durability classes (#90). The table declares no cold/warm split: every operation runs against a built, warm database by construction. Every engine traverses the same persons-and-KNOWS projection with edges stored in both directions. +Interactive (`l2`): seven operations, point lookup, one hop, two hops, three hops filtered on the far end, insert, update, and delete, every read against a fresh seed set per repetition and the three writes 1,000 each per repetition. The writes run in both durability classes (#90). The table declares no cold/warm split: every operation runs against a built, warm database by construction. Every engine traverses the same persons-and-KNOWS projection, each friendship stored as one KNOWS edge and every question asked undirected (CAMPAIGN.md section 7, row 56). Analytics (`l2olap`): fourteen query columns (DECISIONS #104), each asked of the whole graph and run 100 times per repetition under the graph lane's per-query budget. Five are hand-written: average friend age by city, friendships within a city, most friends, degree distribution, and the triangle count; two of them aggregate a property along the edges, which LSQB never does, so they stay. Nine are LSQB's pattern-matching counts, taken nearly verbatim from the published Cypher (undirected KNOWS, exactly as LSQB writes it) and translated into AQL, MongoDB aggregation, SurrealQL, and SQL/PGQ, with every digest agreeing on the slice across the graph engines and nothing declared unexpressible (#104a, #104b). What each LSQB query counts, in the words the page uses: @@ -119,15 +119,15 @@ The friends-of-friends pair (LSQB Q6 and Q9) is excluded at the full network (DE The triangle count is the cell most likely to exceed its budget at the full network. A named censored cell with its budget stated is a result and not a gap, printed the way the embedded SurrealDB cells already are (DECISIONS #82b). Both new analytics queries return a small deterministic answer, a histogram and a single integer, so they are the strongest equivalence checks on the page: two engines that disagree about how many triangles one graph holds are caught before anybody reads a latency. -Sizes: `l2` two, SF1 and SF10 of the persons-and-KNOWS projection, since per-seed reads never touch the message half and the full network at SF10 is not feasible. `l2olap` is the ONE table #108 leaves at a single size, the full SF1 network (`sf1full`), which LSQB's nine need and which SF10 cannot carry. The September page still prints the SF1 and SF10 projections there: qEB measured nine engines at `sf1full` and SurrealDB embedded was killed at the tier's memory envelope, so under #103b the switch was blocked and `export_web` `l2olap` `only_scales` still reads `{"sf1", "sf10"}`. Under #108 an engine absent from a size is a declared outcome rather than a block, and those rows publish in October at October's pin. The skeleton's analytics rows read a capped LDBC SF1 slice and are labelled as such. +Sizes: `l2` two, SF1 and SF10 of the persons-and-KNOWS projection, since per-seed reads never touch the message half and the full network at SF10 is not feasible. `l2olap` is the ONE table #108 leaves at a single size, the full SF1 network (`sf1full`), which LSQB's nine need and which SF10 cannot carry. The September page prints the SF1 and SF10 projections there; the October payload prints the full network (`export_web` `l2olap` `only_scales` is `{"sf1full"}` under the October switch), and under #108 an engine absent from a size is a declared outcome rather than a block. The skeleton's analytics rows read a capped LDBC SF1 slice and are labelled as such. ### Documents and time series -The page's document tables are TPC only (DECISIONS #67): `docs_oltp` and `docs_olap`, with PostgreSQL beside the default arm. September prints TPC-H SF1 on both. October carries TPC-H SF1 AND SF10 on both (#108), the transactional table's SF10 being the first to drop if the matrix does not fit. September's attempt to move the analytics table to SF10 alone did not switch: qEA measured DuckDB and both PostgreSQL arms there, three engines were censored at the tier's cap, SurrealDB served lost its connection and SurrealDB embedded was killed at the tier's memory envelope. Under #103b that blocked the raise and the SF10 outcomes appear as notes under SF1 rows; under #108 those three measured arms publish as an SF10 row group with the other engines' absences declared by outcome. +The page's document tables are TPC only (DECISIONS #67): `docs_oltp` and `docs_olap`, with PostgreSQL beside the default arm. September prints TPC-H SF1 on both. October carries TPC-H SF1 AND SF10 on both (#108), the transactional table's SF10 being the first to drop if the matrix does not fit. Under #108 an engine with no row at SF10 is a declared outcome (censored, envelope failure, or failed inside budget), and the engines that did measure there publish as an SF10 row group. Both are split by `_restructure_tables` from one lane. The tuned PostgreSQL arm (`postgres_tuned`) runs on the lane as an ablation of one image's defaults and is kept off both document tables and the durability table, because it reads as a second engine (DECISIONS #76). -On the live September page ArcadeDB has no row on `docs_olap` since 2026-09-14 (BUGS F42 and F43): answer checking found that its two queries were not the questions the comparators answered, our Q1 text computed four of the five aggregates, and its Q6 lost the boundary discount to the engine's decimal-literal comparison. Both errors made its numbers faster than the truth, so the cells came down. A table may lose its ArcadeDB row only by declaring it in a condition a reader sees, naming ArcadeDB and giving the reason; `page_check` passes a declared withdrawal and still fails a silent one. October's `/next` table carries ArcadeDB's embedded and served rows again, re-measured with every engine's answer compared. +On the live September page ArcadeDB has no row on `docs_olap` since 2026-09-14 (BUGS F42 and F43): answer checking found that its two queries were not the questions the comparators answered, our Q1 text computed four of the five aggregates, and its Q6 lost the boundary discount to the engine's decimal-literal comparison. Both errors made its numbers faster than the truth, so the cells came down. A table may lose its ArcadeDB row only by declaring it in a condition a reader sees, naming ArcadeDB and giving the reason; `page_check` passes a declared withdrawal and still fails a silent one. The October table carries ArcadeDB's embedded and served rows again, measured with every engine's answer compared. The decimal-literal defect is upstream #7609, filed 2026-09-15, and the same day's #7611 is the one that decides how this table is read at scale: an indexed lower bound loses part of a run of equal entries, so the revenue query disagreed only at the size where the loss landed on a qualifying row. That is why the revenue total and the revenue by month now count the rows they aggregate and compare the count exactly (DECISIONS #94): the count is part of the answer and not a page column, and without it a lost row sits inside the sums' rounding at the campaign's largest tier. @@ -135,11 +135,11 @@ The decimal-literal defect is upstream #7609, filed 2026-09-15, and the same day |---|---|---|---| | `docs_oltp` | Document OLTP | ArcadeDB embedded and server; ArangoDB, DuckDB (1.5.4), MongoDB, PostgreSQL, SQLite, SurrealDB embedded and server | new-order p50 ms, new-order p99 ms, payment p50 ms, insert p50 ms, read p50 ms, update p50 ms, delete p50 ms, OLTP ops/s, ingest documents/s, ingest total s, ingest s, index s, peak memory GiB, disk GiB | | `docs_olap` | Document OLAP | same rows as `docs_oltp` | Q1 p50 ms, Q1 p99 ms, Q6 p50 ms, top parts p50 ms, ship mode p50 ms, by month p50 ms, cold first query ms, ingest documents/s, ingest total s, ingest s, index s, peak memory GiB, disk GiB | -| `l4` | Time series | ArcadeDB embedded and server, each native time series and document path; ArangoDB, DuckDB (1.5.4), MongoDB, QuestDB, SQLite, SurrealDB embedded and server, TimescaleDB | newest reading p50 ms, newest reading p99 ms, 12h aggregate p50 ms, per-host hourly p50 ms, high-usage p50 ms, grouped, ordered, limited p50 ms, cold first query ms, ingest points/s, ingest total s, peak memory GiB, disk GiB | +| `l4` | Time series | ArcadeDB embedded and server, each native time series and document path; ArangoDB, DuckDB (1.5.4), MongoDB, PostgreSQL (a plain table), QuestDB, SQLite, SurrealDB embedded and server, TimescaleDB | newest reading p50 ms, newest reading p99 ms, 12h aggregate p50 ms, per-host hourly p50 ms, high-usage p50 ms, grouped, ordered, limited p50 ms, cold first query ms, ingest points/s, ingest total s, peak memory GiB, disk GiB | The ingest column becomes THREE columns wherever the rows carry the 2026-09-22 index work (FAIRNESS F14): `ingest s` and `index s` after `ingest total s`, which keeps its name and its meaning, the whole of `build()`. `docs_oltp` and `docs_olap` carry it from October's rows, as listed above (until 2026-09-29 the documents builder's keep-list dropped both columns although every row recorded them, humemai/arcadedb-embedded-python#8); `l4` gains it when its October rows land, since a column no row fills is dropped. Every table with the split carries a generated sentence saying what the two phases are and that they do NOT sum to the total, because each adapter starts its ingest clock after its own setup (BUGS F101). -**`e2` records the split too, and its columns fill as its relaxed rows are re-measured.** `e2_hybrid.py` stamps `ingest_s` and `index_s` on every row through `bench_common.index_split`, and `export_web.OCT_TABLE_METRICS` carries `ingest s` and `index s` for the table. Its arms were measured by `qOD`/`qOD2` before the split existed; `qOD3`, the STRICT durability pass, does carry it, but every table but the durability one publishes the relaxed class (BUGS F52), so those rows do not fill it. The relaxed re-runs that do are `qOL` (both ArcadeDB arms) and `qOM` (SurrealDB embedded and server, and the composed stack); ArangoDB, MongoDB, Neo4j, and PostgreSQL + pgvector + AGE keep relaxed rows without the split until e2 is re-measured in full, and a column no row fills is dropped, as on the tables above. +**`e2` records the split too.** `e2_hybrid.py` stamps `ingest_s` and `index_s` on every row through `bench_common.index_split`, and `export_web.OCT_TABLE_METRICS` carries `ingest s` and `index s` for the table; a column no row fills is dropped, as on the tables above. Document OLTP: six operations, TPC-C new-order and payment and the four single-record operations (insert, read, update, delete by key), 1,000 of each per repetition, with OLTP ops/s the rate of the two transactions together; the p99 is on new-order only; every write runs in both durability classes; no cold/warm split, declared on the rows. Document OLAP: five analytical queries over the whole line-item table, TPC-H Q1 and Q6 plus the top ten parts by revenue, the count by ship mode, and revenue by month, 100 iterations each under the document analytics budget (#100a; the censored cells are counted in a generated note); the cold column is Q1, the first query of the session. @@ -147,7 +147,7 @@ Time series: six TSBS queries, all 100 iterations per repetition under the time- `l4`'s double group-by was withheld on the served native arm while the HTTP serializer printed the time bucket as a date, collapsing every bucket inside one calendar day (upstream #7610, BUGS.md F44). The fix is in the October pin and the cell is published; the mechanism that withheld it is described once, in PUBLISHING.md. -Scales (#108): `docs_oltp` and `docs_olap` TPC-H SF1 + SF10, `l4` 100 + 1,000 hosts. September prints one size on each of the three; October carries both, and `docs_oltp`'s SF10 is the first size to drop if the matrix does not fit. The time-series lane parses the line protocol in Python inside every cell, which is about 29 minutes of the 1,000-host tier before any engine work, so that size is priced as parse plus engine and not as engine alone. **And 1,000 hosts is near the envelope, not comfortably inside it:** measured 2026-09-20, the ArcadeDB document path held 24.4 GiB of its 32 GiB cap at that size. Any proposal for a larger time-series tier starts from that number, because an engine at three quarters of its allowance at one size is an ENVELOPE failure at the next -- which is exactly how SurrealDB embedded died twice in September's extension (#103g), and an envelope failure is evidence for nothing about latency in either direction. +Scales (#108): `docs_oltp` and `docs_olap` TPC-H SF1 + SF10, `l4` 100 + 1,000 hosts. September prints one size on each of the three; October carries both, and `docs_oltp`'s SF10 is the first size to drop if the matrix does not fit. The time-series lane parses the line protocol in Python inside every cell, which is about 29 minutes of the 1,000-host tier before any engine work, so that size is priced as parse plus engine and not as engine alone. **And 1,000 hosts is near the envelope, not comfortably inside it:** measured 2026-09-20, the ArcadeDB document path held 24.4 GiB of its 32 GiB cap at that size. Any proposal for a larger time-series tier starts from that number, because an engine at three quarters of its allowance at one size is an ENVELOPE failure at the next -- which is exactly how SurrealDB embedded died twice in September (#103g), and an envelope failure is evidence for nothing about latency in either direction. Every document table publishes ingest rate beside OLTP ops/s: publishing the win without the loss is selective. @@ -157,7 +157,7 @@ Both transactional tables carry the four single-record operations, because nothi | id | title | rows | columns | |---|---|---|---| -| `e2` | Cross-model transaction | ArcadeDB (one transaction), embedded and server; ArangoDB, MongoDB (its vector hit runs outside the transaction because the engine refuses it inside one, recorded as `txn_scope`, #98), Neo4j (vector index), PostgreSQL + pgvector + AGE, Qdrant + Neo4j (no shared transaction; the Qdrant server beside Neo4j in one container, #117), SurrealDB embedded and server. What the timed transaction holds is stated per engine (#118, below) | transaction p50 ms, transaction p99 ms, retrieval p50 ms, graph-filtered search p50 ms, retrieval recall@10, filtered recall@10, ingest+index vertices+edges/s, ingest+index total s, peak memory GiB, disk GiB | +| `e2` | Cross-model transaction | ArcadeDB (one transaction), embedded and server; ArangoDB, MongoDB (its vector hit runs outside the transaction because the engine refuses it inside one, recorded as `txn_scope`, #98), Neo4j (vector index), PostgreSQL + pgvector + AGE, Memgraph, LadybugDB, DuckDB (VSS and DuckPGQ in one engine; DECISIONS #131 item 4), Qdrant + Neo4j (no shared transaction; the Qdrant server beside Neo4j in one container, #117), SurrealDB embedded and server. What the timed transaction holds is stated per engine (#118, below) | transaction p50 ms, transaction p99 ms, retrieval p50 ms, graph-filtered search p50 ms, retrieval recall@10, filtered recall@10, ingest+index vertices+edges/s, ingest+index total s, peak memory GiB, disk GiB | | `e2atom` | Cross-model transaction: what survives a crash | same as `e2` | trials, torn results | | `e4` | What the client/server split costs | 1 to 100,000 documents | in-process, in-process HTTP, separate container | | `pycost` | What Python costs | Java, Python, to_columns, to_json_list, to_list | time ms (p50, column 7 of `mini_results.csv`; column 6 is the mean and is never printed), vs Java | @@ -167,7 +167,7 @@ Four cross-model operations (#82c): the composed transaction (a vector hit, a on **What the transaction holds is declared per engine, not equalised (DECISIONS #118).** The operation is defined by its update being atomic; the two reads that choose the products may run outside the transaction, because that is what every engine on the table can do. Each adapter declares a `TXN_SCOPE`: `whole`, the reads inside the transaction with the update (ArcadeDB embedded and server, Neo4j, and PostgreSQL + pgvector + AGE); `update`, the reads first and only the update in a transaction (ArangoDB, MongoDB, and SurrealDB embedded and server; MongoDB's is asked of its server at build time and recorded as `txn_scope`, since mongod refuses `$vectorSearch` inside a multi-document transaction); or `none`, the composed stack, whose two systems share no transaction. The table carries a generated sentence naming the engines on each side (`export_web._txn_scope_note`), read from `txn_scope` where a row recorded it and from the adapter otherwise. ArcadeDB keeps its reads inside, the setting that does not flatter it. -Scales (#108): `e2` 50k + 500k products. September prints 50k alone; the 500k raise was queued as qED and cancelled with the rest of the extension, so it is October's, as a second row group rather than a replacement. +Scales (#108): `e2` 50k + 500k products. September prints 50k alone; October carries both, the larger as a second row group rather than a replacement. The two read paths keep `e2` from being only a write argument (DECISIONS #82c). The retrieval path is a vector top-k, a one-hop expansion, and a projection of the documents found, which is the multi-model read a reader recognises from their own application; the graph-filtered search is a top-k restricted to the neighbourhood of a start node, which is where one engine pushes the filter into the index while a composed stack moves a candidate set between two systems and has to choose between filtering before or after the search. Both carry recall, measured against a brute-force answer over the same filtered candidate set, because an engine that filters after the search shows it there and not in latency. diff --git a/benchmarks/experiments/PROTOCOL.md b/benchmarks/experiments/PROTOCOL.md index bde3a13b831..6e78fac2368 100644 --- a/benchmarks/experiments/PROTOCOL.md +++ b/benchmarks/experiments/PROTOCOL.md @@ -6,7 +6,7 @@ How this benchmark harness must be operated. Every rule carries a tag saying wha - **HAND-RUN** `claims_check`: checked only when someone runs `claims_check.py`. It is a helper library that `page_check` imports, not a gate. - **REMEMBERED**: nothing will catch the next violation. -Companion documents: FAIRNESS.md (F1 to F15, when two numbers may be compared), PAGE-SPEC.md (what the page contains), PUBLISHING.md (how it is published), READING-RESULTS.md (how to read `results/`), CAMPAIGN.md (how a campaign is run), COMPARATORS.md (what each comparator is pinned to). The canonical-row definition is section 1 below. There is no paper; the page is the only publication target, and the generated LaTeX tables T2 to T5 survive, and `page_check` pins the page's 9.99M dense cells and the prose numbers that quote them against T5. +Companion documents: FAIRNESS.md (F1 to F15, when two numbers may be compared), PAGE-SPEC.md (what the page contains), PUBLISHING.md (how it is published), READING-RESULTS.md (how to read `results/`), CAMPAIGN.md (how a campaign is run), COMPARATORS.md (what each comparator is pinned to). The canonical-row definition is section 1 below. The page is the living artifact. A paper written from the same frozen rows pins a stable release where the page may pin a commit (DECISIONS #42), so the generated LaTeX tables T2 to T5 survive, and `page_check` pins the page's 9.99M dense cells and the prose numbers that quote them against T5. --- @@ -42,14 +42,14 @@ The admission filter is `make_paper_tables.load_canonical()`. Every test below r - A summed or averaged measure is declared numeric in its query's digest spec and compared as a number whatever type the driver returns, while counts and identifiers stay exact. ArangoDB's AQL `SUM` over integer-stored values returns an integer where every other engine returns a double, which split the pricing summary seven engines to one at SF1 on an answer nobody had wrong, and a count rounded to six significant digits would let 1,234,567 and 1,234,568 agree. The declaration is per query and never per engine, so it cannot be used to make one engine's answer match another's. **GATE** `equivalence_check` over `bench_common.canonical_rows` - The revenue total and the revenue by month count the rows they aggregate, and that count is compared exactly, because at the campaign's largest tier one lost row moves each sum by less than the digest's rounding and was detected only 10.8 and 36.5 per cent of the time (DECISIONS #94). **GATE** `equivalence_check` - ArcadeDB SQL in this repository compares a numeric column through `BETWEEN` or a bound parameter, never against a bare decimal literal, which the engine parses at single precision so that the comparison loses the boundary value (upstream #7609, BUGS.md F42). **GATE** `equivalence_check` -- A served query that projects a temporal value is checked against its embedded twin, because the HTTP serializer prints a time bucket as a date and every bucket inside one calendar day collapses to one value (upstream #7610, BUGS.md F44); nothing is withheld today: the served time-bucket cell was, until #7610 reached the pin on 2026-09-18, and the mechanism that held it (`equivalence_check.KNOWN_DISAGREEMENTS` and `export_web.WITHHELD_CELLS`, moved together, both empty now) is described once, in PUBLISHING.md. **GATE** `equivalence_check` +- A served query that projects a temporal value is checked against its embedded twin, because the HTTP serializer prints a time bucket as a date and every bucket inside one calendar day collapses to one value (upstream #7610, BUGS.md F44); nothing is withheld today (`export_web.WITHHELD_CELLS` is empty): the served time-bucket cell was, until #7610 reached the pin on 2026-09-18, and the mechanism that held it (`equivalence_check.KNOWN_DISAGREEMENTS` and `export_web.WITHHELD_CELLS`, moved together) is described once, in PUBLISHING.md. `KNOWN_DISAGREEMENTS` holds PostgreSQL + AGE's CRUD read-backs (apache/age#2587), whose cells publish with a note and are not withheld (DECISIONS #134). **GATE** `equivalence_check` - **Three different things stop work, and they are not interchangeable.** A query **budget** stops one query; a whole-cell **cap** stops a cell; **abandonment** stops a query early when its budget cannot buy a useful measurement. Each is a property of the work, never of the engine (#85a), so no engine is ever given more room than the engine beside it. - **Every cell has a time cap, and a cell that exceeds it is censored with the cap named.** The cap is a property of the tier, the same for every engine on it (`runner.TIMEOUT_BY_SCALE`), sized from the measured build at the tier below with headroom and then bounded so the matrix fits: the campaign runs about nine engines at five repetitions across several lanes, and an uncapped slow engine costs days to produce a row that says no more than a censored cell does. **The value is taken from the FASTEST engine's measured cell and the MEDIAN engine's projected one, never from the slowest**, for the same reason a budget uses the median: the slowest engine is the one the cap exists to bound, so sizing the cap to it would bound nothing. When the first repetition leaves no clean row the stage skips the rest, because repeating a timeout measures nothing; the table then shows one censored cell, not five. **GATE** `runner` (the cap), **REMEMBERED** the stage script (the skip). DECISIONS #105 - **A query's budget is derived per lane, per tier and per query, from the bench host's own frozen rows.** It is three times the MEDIAN engine's measured p50 for that query at that tier, times the lane's iteration count: the median because the slowest engine is exactly what a budget exists to bound, and the factor of three because it is the one judgement in the formula and it is the same everywhere. `derive_budgets.py` writes the table into `budgets.py` from the campaign's frozen rows (`--rows results/runs_paper_oct.csv` for October; its default is September's `results/runs_paper.csv`) and never from a laptop, whose loaded and throttled timings would hand out budgets nobody needs; `budget_lookup.budget_for()` reads it. Where a tier has no measurement the lookup falls back to the lane's old flat constant and stamps `_budget_source` on the row saying so, so a projection can never be mistaken on the page for a measurement. A budget may vary by lane, tier and query, because those are properties of the work; it may never vary by engine. **A budget whose own roster is the survivors of a looser bound is withheld rather than published** (#106a): if the engines that left a row for that query and tier are fewer than half the widest roster answering it anywhere, the median is measuring its own selection, and the generator names the entry with its counts and falls back to the lane's constant. The case that forced this: at TPC-H SF10 three engines of ten survived, and their median would have given a ten-times-larger corpus a four-times-smaller budget than SF1's. A query that exceeds it stops at the iteration it reached, its numbers are over those iterations, and the cell's other queries keep theirs. **GATE** `budget_lookup` (October instrument; September's lanes carry the flat constants). DECISIONS #106 - **A query whose budget cannot buy five iterations is abandoned after its cold pass.** The cold pass is already paid for, so it is kept and reported; `budget_lookup.abandon(cold_s, budget_s, iterations)` then asks whether the budget buys at least five iterations at the cold pass's own rate, five because a p50 over fewer is not a median of anything. When it does not, the row carries `_abandoned` with the reason in seconds rather than the cell burning hours to prove what the first touch already showed. The test uses only the engine's own cold pass, so it is a property of the work rather than of who is running it. The other queries in the cell are unaffected. **GATE** `budget_lookup.abandon`, called by `l1_tpc`, `l2_graph` and `l4_tsbs` (October instrument); `page_check` allows `\w+_abandoned` as a recorded field. DECISIONS #107 - An indexed lower bound is not assumed to return a whole run of equal entries: an ascending range seek starts where the binary search landed inside the run and loses `floor((k-1)/2)` of the k entries at the bound (upstream #7611, BUGS.md F46), which is why the revenue query disagreed only at the size where the loss landed on a qualifying row. **GATE** `equivalence_check` - What answer checking cannot prove: DuckDB, PostgreSQL, and SQLite share a SQL dialect closely enough that a defect in a shape all three spell the same way would agree with itself, so the independent evidence comes from the engines with their own languages, MongoDB, SurrealDB, and ArangoDB on the document lane and SurrealDB (SurrealQL), ArangoDB (AQL), MongoDB (the aggregation pipeline), and DuckPGQ (SQL/PGQ) on the graph lane; the Cypher engines there share one query text with ArcadeDB. ArcadeDB collects the most findings partly because it is the engine examined hardest here, with its source, two deployments, and hand-written SQL. **REMEMBERED** -- Every timed WRITE runs at both durability settings (DECISIONS #90): the six document operations, the three graph writes, and the cross-model transaction, once relaxed and once strict. The class is a property of the cell, not a second measurement inside one, because ArcadeDB's `txWalFlush` is per database and SurrealDB's and QuestDB's are server flags; `runner.py --durability strict` is the axis, `durability_class` is on the row and in the canonical key, and the four engines with no knob (Neo4j and the arms built on it, DuckDB including its DuckPGQ arm, LadybugDB, the SurrealDB 3.2.4 server) run once and declare it. Bulk ingest and every read path stay at the relaxed setting. **GATE** `fairness_check.check_durability` (F10b) +- Every timed WRITE runs at both durability settings (DECISIONS #90): the six document operations, the three graph writes, and the cross-model transaction, once relaxed and once strict. The class is a property of the cell, not a second measurement inside one, because ArcadeDB's `txWalFlush` is per database and SurrealDB's and QuestDB's are server flags; `runner.py --durability strict` is the axis, `durability_class` is on the row and in the canonical key, and the three engines with no knob (Neo4j and the arms built on it, DuckDB including its DuckPGQ arm, LadybugDB) run once and declare it (the SurrealDB 3.2.4 server stood here as a fourth until BUGS F165 found its `sync` setting). Bulk ingest and every read path stay at the relaxed setting. **GATE** `fairness_check.check_durability` (F10b) - Bulk ingest runs at ONE durability setting on every lane, and the reason is measured rather than assumed (DECISIONS #90a, proposed, narrowed and withdrawn on 2026-09-15 before any machine time was spent on it). A bulk load commits in batches of thousands, so the flushes over a whole ingest are seconds and a second setting buys a second build rather than a second number. The check that this is true of OUR loaders and not merely of the theory ran once, on the laptop, at micro scale on 2026-09-15: the document, graph, and cross-model lanes each load the corpus inside the write cells that already run in both classes, so twenty-one engine-lane pairs carry a relaxed and a strict build with nothing else changed, and every ratio landed between 0.92 and 1.14. No loader commits per row. A ratio far from one on any future lane is an artifact of the adapter and not a property of the engine, and that is the only thing the pair was ever going to catch. **REMEMBERED** - A served SurrealDB cell that loses its WebSocket rebuilds the connection, re-authenticates, re-selects the namespace and database, and lets the call through once more; the sample taken across the drop is DISCARDED rather than published, and every SurrealDB row records `reconnects`, zero when nothing happened. The client's keepalive is the thing that dropped it (the SDK leaves the websockets library's 20 s ping and 20 s pong deadline in place, and a server busy answering a long query does not pong in time), so the options are set explicitly and named on the row in `surreal_ws_options`. **GATE** `test_surreal_reconnect.py`; **REMEMBERED** for the field (DECISIONS #91) - The two dense mutation operations run at the one-million tier only, and in the order delete-then-reinsert, which is what keeps their recall exact: the ground truth after deleting a set is the original one minus that set, and re-inserting exactly that set restores it, so nothing is brute-force recomputed on any arm. The victims are drawn from the ground truth so the deletions bite, and the row records how many deleted ids the index still returned, which must be zero. **REMEMBERED** `l3d_dense.mutation_victims` @@ -59,7 +59,7 @@ The admission filter is `make_paper_tables.load_canonical()`. Every test below r - Warm queries must be disjoint from cold ones. Replaying one set cannot separate residency from recall of the same queries. **GATE** `sparse_multipass_driver` protocol for l3s; **REMEMBERED** and knowingly violated for l3d, whose driver replays the `test` set on every pass after 20 untimed warmups drawn from it - Each engine gets its own vendor-documented settle step, timed inside build so the build is timed to a queryable index, and none is skipped. **REMEMBERED** - Ingest timers must stop on the same completion semantics across a lane, or the rates price different work. **REMEMBERED**: a new time-series adapter must stop its timer at the same semantics its lane siblings do -- The dense and sparse lanes record `ingest_s` and `index_s` separately wherever the engine has the boundary (on the dense lane ArcadeDB, pgvector, Neo4j, Milvus, LanceDB, the SurrealDB server, and MongoDB), additive, with `build_s` unchanged (DECISIONS #74 item 2). The document (`l1tpc`), time-series (`l4`), and cross-model (`e2`) lanes record the same pair through `bench_common.index_split` (FAIRNESS F14), and an arm whose index is defined before the load declares `index_before_load` instead (SurrealDB on the document and time-series lanes). Engines that index while ingesting record neither and carry the total with that as the stated reason. The October dense table prints `ingest s` and `index s` beside the ingest+index pair; the sparse table prints the pair alone and records no split: pgvector's index build follows its load (COPY, then CREATE INDEX) and sits inside the same single timer on this table, and the other sparse engines index as they ingest; the September page keeps one ingest+index pair. **REMEMBERED** +- The dense and sparse lanes record `ingest_s` and `index_s` separately wherever the engine has the boundary (on the dense lane every arm except those `fairness_check.PHASE_SPLIT_DECLARED` names, which index while they load), additive, with `build_s` unchanged (DECISIONS #74 item 2). The document (`l1tpc`), time-series (`l4`), and cross-model (`e2`) lanes record the same pair through `bench_common.index_split` (FAIRNESS F14), and an arm whose index is defined before the load declares `index_before_load` instead (SurrealDB on the document and time-series lanes). Engines that index while ingesting record neither and carry the total with that as the stated reason. The October dense table prints `ingest s` and `index s` beside the ingest+index pair; the sparse table prints the pair alone and records no split: pgvector's index build follows its load (COPY, then CREATE INDEX) and sits inside the same single timer on this table, and the other sparse engines index as they ingest; the September page keeps one ingest+index pair. **REMEMBERED** - Observe conditions from the daemon and the engine; never record what the launcher meant to do. A heap that disagrees with the request fails the cell. **GATE** `runner.observe_server` - Overlay drivers sample their own peak memory from inside the container, started before the build. `run_conditions` records the ceiling, not the use. **GATE** `bench_common.SelfMemorySampler` - The cgroup memory reading is valid only inside a bench container; record `host` so a reader knows which they have. **REMEMBERED** @@ -126,7 +126,7 @@ The admission filter is `make_paper_tables.load_canonical()`. Every test below r - Enforce a comparator-selection rule in code, not on the axis: the real rule is "fastest engine at recall at least ours". **GATE** `_check_f4_comparators` - Read labels back out of the rendered PDF. A figure can be wrong while every data check is right. **GATE** `_check_labels_intact` + `EXPECT_IN_PDF` - A figure label must carry its own valence and must not borrow another experiment's vocabulary. **REMEMBERED** -- A caption's asserted engine version must appear in the versions its feeding overlays record; distinguish BAD (data contradicts) from UNVERIFIABLE (data cannot say). **REMEMBERED**: `provenance_check.caption_versions` needs `paper.tex`, which no longer exists (the paper was dropped 2026-09-11) +- A caption's asserted engine version must appear in the versions its feeding overlays record; distinguish BAD (data contradicts) from UNVERIFIABLE (data cannot say). **REMEMBERED**: `provenance_check.caption_versions` needs a `paper.tex` next to the tables, and this tree has none - Flag a table whose rows come from more than one engine version, and name every driver that writes a hardcoded version literal about itself. **GATE (report only)** `provenance_check` - The page agrees with its own sources: its 9.99M dense cells with the generated T5 table, the atomicity counts with the artifact, no table losing its ArcadeDB row against the live page, every operation of the query set a column with every engine answering or declared (`OPERATION_MANIFEST`), every backend a lane registers on its tables, the capability table re-derived from the tables, every number under a table traced to a source, no sentence naming an arm the page does not print, the zero-age disclosure wherever rows measured it, the setup paragraph naming the CPU and the cpuset the rows ran on, and every disk column in GiB. A pin for a claim the page dropped is deleted rather than left failing. **GATE** `page_check`; the deletion is **REMEMBERED** - Pin every hand-typed number in the page's prose to a named table column, keep cold and warm entries listed apart, and treat an unmatched regex or an unparseable capture as a failure. **GATE** `page_check.PROSE` @@ -150,7 +150,7 @@ The admission filter is `make_paper_tables.load_canonical()`. Every test below r ## 6. How work is queued and stored -- Every campaign dataset switch lives in `campaign_env.sh` in the repo. No queue script sources it: `make_october_stages.py` writes each October stage's switches inline, and the September scripts set the same variables inline. A launcher in a home directory is not where configuration lives. **REMEMBERED**: nothing sources the file, and the generated October scripts and the September ones hardcode their own switches +- Every campaign dataset switch lives in `campaign_env.sh` in the repo. No queue script sources it: `make_october_stages.py` and `make_2610_stages.py` write each stage's switches inline, and the September scripts set the same variables inline. A launcher in a home directory is not where configuration lives. **REMEMBERED**: nothing sources the file, and the generated scripts and the September ones hardcode their own switches - Check each corpus is present, not merely that its path variable is set; a set path with no corpus is how the synthetic sparse rows got in. **REMEMBERED**: `campaign_env_check()` exists, but no campaign script sources `campaign_env.sh`, so nothing calls it - Run a smoke stage before any long matrix and let it abort the campaign. **REMEMBERED** - One runner per bench host. `sweep_orphans()` destroys a live campaign's in-flight cells, so the runner takes an exclusive flock; never rebuild a bench image and never co-run a smoke while a campaign is live. **GATE** the exclusive flock on `$TMPDIR/dbbench-runner.lock` (`runner.acquire_host_lock`), one file for every checkout on the host, so a second runner is refused at startup (BUGS F58) @@ -177,7 +177,7 @@ We run every engine as shipped, and override a default only where leaving it wou What we never do: per-system expert tuning beyond vendor guidance, or tuning ArcadeDB with insider knowledge not applied to comparators. -An override nobody can find is a defect, whichever way it moves the number. Every override is stamped on the row where a field exists (`es_prune`, `degree_param`, `degree_family`, `heap`, `server_heap`) **and** named in reader-facing text (table caption or page condition). `DISCLOSED WHERE = NOWHERE` marks an override not yet disclosed; closing them is CAMPAIGN.md section 7, item 21. Every override a table shows is registered in `overrides.py`, which holds the row field the adapter reads back from the engine, the generated sentence, and the checks (`page_check` for the sentence, `fairness_check` F15 for the stamp, and `test_overrides.py`, which reads this table); a row's last column cites its registration as `override: `. +An override nobody can find is a defect, whichever way it moves the number. Every override is stamped on the row where a field exists (`es_prune`, `degree_param`, `degree_family`, `heap`, `server_heap`) **and** named in reader-facing text (table caption or page condition). `DISCLOSED WHERE = NOWHERE` marks an override not yet disclosed (the column has none left; closing them was CAMPAIGN.md section 7, item 21, and `test_overrides.py` fails on a new one). Every override a table shows is registered in `overrides.py`, which holds the row field the adapter reads back from the engine, the generated sentence, and the checks (`page_check` for the sentence, `fairness_check` F15 for the stamp, and `test_overrides.py`, which reads this table); a row's last column cites its registration as `override: `. | Engine | Setting | Category | Why | Disclosed where | |---|---|---|---|---| @@ -186,7 +186,7 @@ An override nobody can find is a defect, whichever way it moves the number. Ever | QuestDB (L4) | WAL-apply poll runs as a settle step after the timer | settle step | No engine settles inside the timer. | l4 page condition | | ArcadeDB native TS (embedded) | ingest timer stops at `wait_completion()` (accepted, sealing outstanding) | operating-point | The rate is timed to acceptance; sealing continues afterwards, during the lane's settle and outside the timer. | row field `ts_mutable_at_ingest_end` (the engine's own count of unsealed samples when the timer stopped) and a generated sentence under the time-series table; `override: arcadedb_ts_acceptance` | | ArcadeDB native TS (embedded + served) | `COMPACTION_INTERVAL 1 HOURS` (default none) | operating-point | Declares the bucket of the table's hourly aggregate, so compaction cuts sealed blocks at its boundaries and the aggregate can answer from block statistics. Upstream's answer to our question (ArcadeData/arcadedb#9166, 2026-10-05) is that this is the declaration for a type whose main query is an hourly aggregate. ArcadeDB's own knob: no comparator has an equivalent. | row field `ts_compaction_interval_ms` (read back from the engine, `compactionBucketIntervalMs`) and a generated sentence under the time-series table; `override: arcadedb_ts_compaction_interval` | -| ArcadeDB native TS | `SHARDS 4` (default 0 = 11 under this cpuset) | resource fitting | Fixes the shard count instead of deriving it from the cpuset. | raw JSON only | +| ArcadeDB native TS | `SHARDS 4` (default 0 = 11 under this cpuset) | resource fitting | Fixes the shard count instead of deriving it from the cpuset. | row field `ts_shards` | | ArcadeDB native TS | `TS_PRIMITIVE=1` (lane default 1); `TS_NUMPY` already defaults to 1 | escape hatch | One opt-in fast path under the native time-series headline; comparators use ordinary bulk paths. `TS_NUMPY=0` exists only to reproduce the pre-fix number. | row field `ts_path`, page row label 'native time series' | | DuckDB (every lane) | `PRAGMA threads = len(sched_getaffinity(0))` | resource fitting | The F6 fix. Applied on the tabular lane first and on l1_tpc, l3d, and l4 since DECISIONS #101 (2026-10 instrument); September's DuckDB rows on those three lanes were measured oversubscribed and stay as measured. | FAIRNESS.md F6; row field `duckdb_threads` (`duckpgq_threads` on the graph arm), read back from the engine and held to the size of the cell's cpuset by `fairness_check` F15; a generated sentence under every table that shows DuckDB (the single-model embedded engines are not on the lifecycle table by decision, so its DuckDB arm carries no stamp); `override: duckdb_threads` | | DuckDB VSS (L3d) | `hnsw_enable_experimental_persistence=true` | escape hatch | Required to persist an HNSW index at all. The row is measured on a vendor-labeled experimental configuration. | row field `duckdb_hnsw_persistence`, read back from the engine, and a generated sentence under every table that shows the arm; `override: duckdb_vss` | @@ -195,7 +195,7 @@ An override nobody can find is a defect, whichever way it moves the number. Ever | ArcadeDB (documents, cross-model, graph message half) | `CREATE INDEX ... UNIQUE_HASH` on the ids a lane only looks up by equality: `Part(p_partkey)`, `OrderNew(okey)`, `Crud(ckey)`, `Product(pid)`, and the LDBC message half's vertex ids; `UNIQUE` (sorted) stays on `Person(id)` and wherever a key is ranged over or ordered | operating-point (the vendor's recommendation for the index kind, not a tuning knob) | ArcadeDB's maintainers answered on ArcadeData/arcadedb#9169 that an id read, updated, and deleted only by equality takes a hash index and a key that is also ranged or ordered stays sorted. Every other engine keeps its primary-key or default index, so no comparator moves. A hash index keeps no key order, so it can cost more to build when the loader sends ids in ascending order, as ours do; the ingest and index columns carry that cost and the transaction columns carry the gain | FAIRNESS F14; `fairness_check.ARCADEDB_HASH_ID_INDEXES`, held equal to the DDL by `test_index_kinds.py`; row field `index_kinds`, the engine's own `schema:indexes` answer read after the schema is built, held to the registry by `fairness_check` F14d; a generated sentence under every table that shows an ArcadeDB arm of those lanes, built only from rows that carry the stamp | | Elasticsearch (L3s) | `xpack.security.enabled=false` | escape hatch | Harness plumbing; also removes TLS and auth from every ES latency measurement. | row field `es_security_enabled` (from the engine's `_xpack`), and a generated sentence under every table that shows Elasticsearch; `override: es_security` | | Elasticsearch (L3s) | `number_of_replicas=0` | resource fitting | A single-node cluster cannot allocate a replica; the index would sit yellow forever. | row field `es_replicas` (from the index settings), and a generated sentence under every table that shows Elasticsearch; `override: es_replicas` | -| composed Qdrant + Neo4j (E2) | the Qdrant server beside Neo4j in ONE container, `dbbench:composed` from `Dockerfile.composed` (the dense lane's pinned Qdrant v1.19.1 binary, Neo4j at the digest every Neo4j arm runs), reached over REST on 6333, with Qdrant's HNSW defaults (m=16, ef_construct=100) and a payload index on `pid` for the candidate filter (DECISIONS #117, BUGS F133) | resource fitting (one server cgroup for both halves) + operating-point (Qdrant's documented configuration for a filtered field) | Until 2026-09-26 the vector half was `QdrantClient(location=":memory:")`, the client package's pure-Python local mode, which the comparator review had rejected and whose graph-filtered search was a Python loop over every point. The runner caps, pins, and measures one server container per arm, so both processes share it, as MongoDB Search does. Rows measured before the fix carry `qdrant-local:` in `engine_version` and the page marks them `re-run` until `qOM`'s rows supersede them. | row field `engine_version` (`qdrant:+neo4j:`), COMPARATORS.md, page `re-run` note | +| composed Qdrant + Neo4j (E2) | the Qdrant server beside Neo4j in ONE container, `dbbench:composed` from `Dockerfile.composed` (the dense lane's pinned Qdrant v1.19.1 binary, Neo4j at the digest every Neo4j arm runs), reached over REST on 6333, with Qdrant's HNSW defaults (m=16, ef_construct=100) and a payload index on `pid` for the candidate filter (DECISIONS #117, BUGS F133) | resource fitting (one server cgroup for both halves) + operating-point (Qdrant's documented configuration for a filtered field) | Until 2026-09-26 the vector half was `QdrantClient(location=":memory:")`, the client package's pure-Python local mode, which the comparator review had rejected and whose graph-filtered search was a Python loop over every point. The runner caps, pins, and measures one server container per arm, so both processes share it, as MongoDB Search does. Rows measured before the fix carry `qdrant-local:` in `engine_version` and the page marks them `re-run` until the 26.10.1 measurement's rows supersede them. | row field `engine_version` (`qdrant:+neo4j:`), COMPARATORS.md, page `re-run` note | | Neo4j (L2, E2) | heap pinned per tier; `server_memory_pagecache_size` = container mem less heap less a 1 GiB reserve, floored at 512m | resource fitting | Heap parity, with the rest of the cap fitted for Neo4j rather than left at its default. The composed E2 container (`dbbench:composed`) takes the same formula although it also runs the Qdrant server, so Qdrant lives in the 1 GiB reserve beside Neo4j's page cache, which Neo4j fills lazily as pages are read; the formula was not re-derived for a second process. A disclosure, not a change. | row fields `server_pagecache` (the container's env) and `neo4j_pagecache` (the engine's `SHOW SETTINGS`), checked against each other by `fairness_check` F15, and a generated sentence under every table that shows Neo4j; `override: neo4j_pagecache` | | Neo4j (graph) and the composed cross-model stack | `NEO4J_db_checkpoint_interval_time=5s` (default 15 minutes) | measurement (disk reading) | The store is checkpointed before the disk reading, which read 0.9 MB at SF1 under the default (BUGS F27); a laptop A/B on the pinned image put single-edge writes at 7.67 against 7.67 ms p50. Preallocation stays at its default. | row fields `neo4j_checkpoint_interval` and `neo4j_checkpoint_interval_default` (the engine's `SHOW SETTINGS`), and a generated sentence under every table that shows an arm that sets it; `override: neo4j_checkpoint` | | Memgraph (graph) | `--bolt-num-workers` and `--storage-snapshot-thread-count` from the cpuset, `--memory-limit` at 90% of the server cap, `--query-execution-timeout-sec=0`, `--telemetry-enabled=false` | resource fitting (workers, threads, memory) + escape hatch (timeout) | The three pools default to the host (16 under a 12-CPU cpuset, and 90% of physical RAM); the 600 s default timeout would abort a whole-graph query that the lane's own budget is there to censor | row fields `memgraph_bolt_workers`, `memgraph_snapshot_threads`, `memgraph_memory_limit_mib`; FAIRNESS F6; COMPARATORS.md | @@ -204,15 +204,15 @@ An override nobody can find is a defect, whichever way it moves the number. Ever | pgvector (dense, sparse), TimescaleDB, PG+AGE | `shared_buffers` a quarter of the server cap, `effective_cache_size` three quarters, `maintenance_work_mem` half, `max_wal_size` 4GB (8GB on pgvector) | resource fitting | Fitted to the container by PostgreSQL's own guidance rather than left at the image's 128 MB `shared_buffers`; the plain `postgres` arm keeps the image's memory defaults, and `postgres_tuned` is its ablation (below) | row field `server_cmd`; COMPARATORS.md (pgvector) | | ArangoDB (every lane) | `arangod --vector-index true` | escape hatch (forced) | The 3.12 opt-in for the vector (FAISS IVF) index the dense and cross-model lanes use, set on every ArangoDB arm | COMPARATORS.md | | MongoDB (documents, time series, graph) | `--replSet rs0`, a single-node replica set; the MongoDB Search image's entrypoint initiates its own | escape hatch (forced) | TPC-C new-order needs a multi-document transaction, which MongoDB allows only on a replica set, and mongot cannot sync from a standalone | COMPARATORS.md | -| every time-series arm (L4) | `BENCH_TS_SETTLE_S=90`, set by the October l4 stages: a 90 s wait after the ingest timer and before the first timed query | settle step, harness-defined | The served ArcadeDB time-series aggregate answers from sealed data only and misses whole hours for about a minute after an ingest returns (BUGS F68). The same wait for every arm, outside the timer; the embedded native arm's adapter reads the same variable and waits it a second time (CAMPAIGN.md section 7, row 15) | page condition generated from `settle_s`; row fields `settle_s`, `settle_s_lane`, `settle_s_adapter` | +| every time-series arm (L4) | `BENCH_TS_SETTLE_S=90`, set by the l4 stages of both generators: a 90 s wait after the ingest timer and before the first timed query | settle step, harness-defined | The served ArcadeDB time-series aggregate answers from sealed data only and misses whole hours for about a minute after an ingest returns (BUGS F68). The same wait for every arm, outside the timer; the embedded native arm's adapter reads the same variable and waits it a second time (CAMPAIGN.md section 7, row 15) | page condition generated from `settle_s`; row fields `settle_s`, `settle_s_lane`, `settle_s_adapter` | | Every PostgreSQL-family server (postgres, postgres_tuned, TimescaleDB, pgvector, PG+AGE) | `synchronous_commit=off` in `server_cmd` (2026-10, DECISIONS #81); the adapter reads `SHOW synchronous_commit` and records the answer | operating-point (durability class) | A commit returns without waiting for the disk, the class ArcadeDB's default runs in | row field `durability`, FAIRNESS F10, page condition | | MongoDB (documents, time series) | timed writes at write concern `w=1, j=false` (2026-10) | operating-point (durability class) | A replica set's default is `w:majority` with journaling, which waits for the disk | row field `durability`, FAIRNESS F10 | | MongoDB Search (dense, cross-model) | the server is `dbbench:mongo-search`, mongod 8.2.12 plus mongot-community 1.70.4 in ONE container, and mongod runs WITH access control while the document arm runs without it (2026-10) | escape hatch (forced) | mongot authenticates to its sync source with SCRAM and mongod authorizes it through the `searchCoordinator` role; there is no way to run mongot against an open mongod. Both processes in one container so the whole engine lands in the one server cgroup the cell caps and samples | row field `mongo_access_control`, COMPARATORS.md | | MongoDB Search (dense) | `hnswOptions.maxEdges`=16, `numEdgeCandidates`=100, stage `numCandidates`=100, passed explicitly although they are also mongot's defaults | operating-point | The lane's matched point, set rather than inherited, so a future default change cannot move this arm without anyone noticing; the created index is read back onto the row | row fields `degree_param`, `degree_family`, `mongot_hnsw_options` | | MongoDB Search (dense) | `index_s` carries a fixed 12-30 s floor that is mongot noticing the index, not building it (measured: a one-document index becomes queryable in 12.51/29.63/30.12/30.10 s, and 1,000 and 20,000 documents take the same 28-30 s) | **disclosure, not an override** | At micro and tiny the floor IS the number, so this arm's small-tier build timers are not comparable with another engine's. Not subtracted: an adapter that removed a constant from its own engine's timer would be doing the comparator's arithmetic for it | adapter docstring, this table | | MongoDB (graph) | multi-hop reads use chained `$lookup` (two indexed lookups per hop, on `src` and on `dst`) rather than `$graphLookup`, although $graphLookup was 30.01 ms against 38.27 ms per three-hop operation on the directed question at micro | operating-point, runs AGAINST the comparator | DECISIONS #131 item 2 says to use `$graphLookup` wherever it expresses the question and to keep and disclose `$lookup` where it cannot. From the 26.10.1 measurement every graph question is asked undirected over one stored edge per friendship (CAMPAIGN section 7 row 56), and `$graphLookup` walks ONE direction of `knows` (`connectFromField` to `connectToField`), so it states the directed question exactly and the undirected one only over a second, symmetric copy of the edges, which would change this arm's storage while every other engine keeps one edge per friendship. Every hop and every analytics query is therefore chained `$lookup`, no timed query uses `$graphLookup`, and the arm's `query_language` says so | adapter docstring, COMPARATOR-DIALECTS.md | -| MongoDB (graph) | every undirected hop after the first drops the edges already on the path by their `_id` (the second hop one, the third two) | correctness | Cypher's MATCH and ArangoDB's traversal both forbid reusing a relationship within one path; neither $lookup nor $graphLookup has such a rule, and undirected a friendship can be walked back along itself at every hop, so without the exclusion this arm would silently over-count against every engine that enforces it. Held by `test_graph_undirected.py`; the comparison of every engine's digests with a DuckDB reference at LDBC SF1 is recorded in REPIN-REHEARSAL. Until the re-pin the clause was on the third hop only (0 of 50 ids at micro) | adapter docstring | -| FalkorDB and LadybugDB (graph) | the 2-hop, filtered 3-hop, and visited reads name the exclusions on the vertices (`fof <> p`; `m2 <> p AND x <> m1`) that the other Cypher engines get from their MATCH | correctness | Cypher's MATCH forbids reusing a relationship within one pattern, which is what makes these reads walks of DISTINCT friendships. FalkorDB and LadybugDB do not apply the rule to a chain of relationships: on the SF1 slice both answered these three reads differently from a DuckDB reference, because the walk along a friendship and straight back along it was counted (the start person was its own friend of a friend; a four-person graph shows it on LadybugDB in the unit test). In a simple graph, which LDBC's knows is (one row per friendship), the walks the exclusions remove are exactly the walks that reuse a relationship, so the two spellings answer alike. Held by `test_graph_undirected.py`, which shows the defect on LadybugDB itself; every other Cypher engine runs the shared text, and the comparison of each engine's digests with a DuckDB reference on the SF1 slice is recorded in REPIN-REHEARSAL | adapter attribute `REPEATS_RELATIONSHIPS`, `graph_common.OLTP_READS_BY_VERTEX` | +| MongoDB (graph) | every undirected hop after the first drops the edges already on the path by their `_id` (the second hop one, the third two) | correctness | Cypher's MATCH and ArangoDB's traversal both forbid reusing a relationship within one path; neither $lookup nor $graphLookup has such a rule, and undirected a friendship can be walked back along itself at every hop, so without the exclusion this arm would silently over-count against every engine that enforces it. Held by `test_graph_undirected.py`; the comparison of every engine's digests with a DuckDB reference at LDBC SF1 is recorded in the re-pin rehearsal (`.notes` git history, `bench/repin-round5/`). Until the re-pin the clause was on the third hop only (0 of 50 ids at micro) | adapter docstring | +| FalkorDB and LadybugDB (graph) | the 2-hop, filtered 3-hop, and visited reads name the exclusions on the vertices (`fof <> p`; `m2 <> p AND x <> m1`) that the other Cypher engines get from their MATCH | correctness | Cypher's MATCH forbids reusing a relationship within one pattern, which is what makes these reads walks of DISTINCT friendships. FalkorDB and LadybugDB do not apply the rule to a chain of relationships: on the SF1 slice both answered these three reads differently from a DuckDB reference, because the walk along a friendship and straight back along it was counted (the start person was its own friend of a friend; a four-person graph shows it on LadybugDB in the unit test). In a simple graph, which LDBC's knows is (one row per friendship), the walks the exclusions remove are exactly the walks that reuse a relationship, so the two spellings answer alike. Held by `test_graph_undirected.py`, which shows the defect on LadybugDB itself; every other Cypher engine runs the shared text, and the comparison of each engine's digests with a DuckDB reference on the SF1 slice is recorded in the re-pin rehearsal (`.notes` git history, `bench/repin-round5/`) | adapter attribute `REPEATS_RELATIONSHIPS`, `graph_common.OLTP_READS_BY_VERTEX` | | MongoDB (cross-model) | the vector hit runs OUTSIDE the composed transaction | **forced by the engine** | mongod refuses `$vectorSearch` inside a multi-document transaction ("Aggregation stage $vectorSearch cannot run within a multi-document transaction", probed on this pin at build time). $lookup and $graphLookup ARE permitted, and the document update is inside. So MongoDB's torn_count of zero demonstrates document atomicity, not three-model atomicity | row field `txn_scope` | | SurrealDB server (all five arms) | `sync=never` on the storage path (relaxed), `sync=every` (strict), read back from the startup log (BUGS F165, 2026-10-02) | relaxed / strict (durability class) | The engine default is `sync=every`, one fdatasync per commit, which the October campaign ran in both classes under a "not verified" label (the setting is a path parameter, not a `SURREAL_*` variable, which is where the September search looked). The embedded 2.x core is in the relaxed class by default (`SURREAL_SYNC_DATA` unset), verified by an strace A/B | row fields `durability`, `surreal_sync_mode`, FAIRNESS F10 | | ArcadeDB (every write path) | `txWalFlush=0`, the engine default, kept (DECISIONS #81 withdrew the fsync plan of #65) | default | The comparators are matched to it, not it to them | row field `durability`, FAIRNESS F10 | diff --git a/benchmarks/experiments/PUBLISHING.md b/benchmarks/experiments/PUBLISHING.md index ce97cb6f074..6f8cd5a0d77 100644 --- a/benchmarks/experiments/PUBLISHING.md +++ b/benchmarks/experiments/PUBLISHING.md @@ -57,7 +57,7 @@ The gate scripts (`refresh_web_page.GATES`), then three structural checks. All o `PROSE` is its own section because the page's two surfaces fail differently. Cells are written by the exporter straight from the frozen results, so a wrong one is nearly impossible. Prose is typed by hand, so a wrong one is nearly inevitable: a caption once gave one engine's dense latency from the canonical CSV and the other's from the matched overlay, inside one sentence, with no published cell wrong and nothing invented. -A disagreement that has been reproduced, understood, and filed upstream prints as KNOWN rather than FAIL and its cell is withheld from the page instead of being published beside an answer it does not match (`equivalence_check.KNOWN_DISAGREEMENTS`, `export_web.WITHHELD_CELLS`); nothing is withheld today (the last entry, the served native time-series group-by, upstream #7610, was released on 2026-09-18 when the fix reached the pin). The entry is deleted at the re-pin that carries the fix, which re-arms the gate for that query, because a gate that cannot pass until an upstream release lands is a gate somebody deletes. +A disagreement that has been reproduced, understood, and filed upstream prints as KNOWN rather than FAIL and its cell is withheld from the page instead of being published beside an answer it does not match (`equivalence_check.KNOWN_DISAGREEMENTS`, `export_web.WITHHELD_CELLS`); nothing is withheld today (`WITHHELD_CELLS` is empty; the last entry, the served native time-series group-by, upstream #7610, was released on 2026-09-18 when the fix reached the pin). `KNOWN_DISAGREEMENTS` holds PostgreSQL + AGE's CRUD read-backs (apache/age#2587), whose cells publish with a note and are not withheld (DECISIONS #134). An entry is deleted at the re-pin that carries the fix, which re-arms the gate for that query, because a gate that cannot pass until an upstream release lands is a gate somebody deletes. `WEB_ONLY_FIGURES` is the escape hatch for a figure the page shows and nothing else cites. It is deliberately empty; earn a line in it, do not assume one. diff --git a/benchmarks/experiments/QUANTIZATION.md b/benchmarks/experiments/QUANTIZATION.md index d169707a4bd..b262e82f250 100644 --- a/benchmarks/experiments/QUANTIZATION.md +++ b/benchmarks/experiments/QUANTIZATION.md @@ -21,7 +21,7 @@ The quantization survey of DECISIONS #131 item 8 (CAMPAIGN.md section 7 row 42), | Neo4j 2026.08.1 | `vector.quantization.type` `NONE`, `SCALAR`, `BINARY`; **unset means `BINARY`**, with `vector.default_search_expansion_factor` 3.0 | until 2026-10-02 **recorded fp32, ran BINARY**: `neo4j_dense` and `neo4j_e2` created their index without the key (BUGS F164). Now `neo4j_dense` and `neo4j_e2` set `NONE`, and both read the applied type back (`neo4j_vector_quantization`) | built: `neo4j_dense_int8` sets `SCALAR` (repin-prep `32c0a011da`). Each type brings its own `vector.default_search_expansion_factor` (NONE 1.0, SCALAR 1.5, BINARY 3.0 on 2026.08.1 and 2026.09.0, the candidate multiple searched before re-scoring); the lane matches search effort at ef 100 (Neo4j asks for 100 candidates and keeps 10), so both arms set 1.0 and read it back (`neo4j_vector_search_expansion`), as the Elasticsearch int8 arm sets the minimum oversample. 2026.09.0 (the re-pin candidate, `neo4j@sha256:eaab5879…`) keeps the BINARY default (`.notes` repros/neo4j-202609-quant) | image: `SHOW VECTOR INDEXES` on an index created with exactly our options reports provider `vector-2026.08` and `vector.quantization.type: "BINARY"`; `INT8` and `FLOAT32` are syntax errors | | MongoDB 8.2.12 + mongot-community 1.70.4 | `vectorSearch` index `quantization`: `none`, `scalar`, `binary` | `mongodb_dense`: `none` | built: `mongodb_dense_int8` sets `scalar`. Both read the field back from mongot's index definition (`mongot_vector_quantization`, `none` / `scalar`). Recall fp32 / int8: 0.9996 / 0.969 at 5,000, 0.9977 / 0.9699 at 20,000 | adapter; smoke: micro, tiny (`scalar` accepted, built, queryable); docs for `binary` | | SurrealDB (core 2.3.10 embedded, 3.2.4 served) | HNSW `TYPE` F64, F32, I64, I32, I16: element types of the stored array, not quantization of float vectors | `TYPE F32` | none shipped | adapter; docs | -| ArangoDB 3.12.11 | FAISS IVF vector index; a custom FAISS `factory` string (SQ8, SQ4, PQ), forwarded to FAISS and validated at creation only from 3.12.12 | `arangodb_dense`: IVF flat, fp32 (`nLists`, `nProbe` calibrated) | built: `arangodb_dense_int8`, factory `IVF,SQ8` at the fp32 arm's nLists, nProbe calibrated to the same recall target; the server's factory string is read back (`ivf_factory`, None on the fp32 arm) beside its training state. Recall fp32 / int8, each calibrated on the held-out slice to the laptop's fallback target 0.95: 0.9402 / 0.9465 at 5,000, 0.9486 / 0.9447 at 20,000, at nProbe 17 / 19 and 23 / 24 | image: 20,000 SIFT vectors at nLists 566, both indexes `ready`, recall@10 at nProbe 1, 4, 16, 64: plain 0.345, 0.666, 0.932, 0.9985; SQ8 0.345, 0.6645, 0.928, 0.9905; docs (factory text, the 3.12.12 validation); smoke: micro, tiny; 3.12.12 (the re-pin candidate, `arangodb@sha256:5b734c1a…`): `IVF70,SQ8` trains to ready and reads back unchanged at the same recall as 3.12.11, and an invalid factory string is now refused at creation, where 3.12.11 built an `unusable` index that answered by an exact scan (recall 1.0); the arm refuses any index that is not `ready` either way (`.notes` repros/arangodb-31212-sq8) | +| ArangoDB 3.12.12 (3.12.11 until the 26.10.1 measurement) | FAISS IVF vector index; a custom FAISS `factory` string (SQ8, SQ4, PQ), forwarded to FAISS and validated at creation only from 3.12.12 | `arangodb_dense`: IVF flat, fp32 (`nLists`, `nProbe` calibrated) | built: `arangodb_dense_int8`, factory `IVF,SQ8` at the fp32 arm's nLists, nProbe calibrated to the same recall target; the server's factory string is read back (`ivf_factory`, None on the fp32 arm) beside its training state. Recall fp32 / int8, each calibrated on the held-out slice to the laptop's fallback target 0.95: 0.9402 / 0.9465 at 5,000, 0.9486 / 0.9447 at 20,000, at nProbe 17 / 19 and 23 / 24 | image: 20,000 SIFT vectors at nLists 566, both indexes `ready`, recall@10 at nProbe 1, 4, 16, 64: plain 0.345, 0.666, 0.932, 0.9985; SQ8 0.345, 0.6645, 0.928, 0.9905; docs (factory text, the 3.12.12 validation); smoke: micro, tiny; 3.12.12 (the pin since the 26.10.1 measurement; its digest is in COMPARATORS.md): `IVF70,SQ8` trains to ready and reads back unchanged at the same recall as 3.12.11, and an invalid factory string is now refused at creation, where 3.12.11 built an `unusable` index that answered by an exact scan (recall 1.0); the arm refuses any index that is not `ready` either way (`.notes` repros/arangodb-31212-sq8) | | Elasticsearch 9.5.4 (joins dense, row 37) | `hnsw`, `int8_hnsw`, `int4_hnsw`, `bbq_hnsw`, `bbq_disk`, `int8_flat` (all accepted); **default `int8_hnsw` at 96 and 128 dimensions** (m 16, ef_construction 100), `bbq_hnsw` with 3x oversampling from 384 | `elasticsearch_dense`: `hnsw`, set explicitly | built: `elasticsearch_dense_int8`, `int8_hnsw` with `rescore_vector.oversample` 1.0; both read the mapping's `index_options` back (`es_index_options`) | image: dense_vector mappings on the pinned server, defaults read through `_mapping/field?include_defaults=true` | | Memgraph 3.13.1 (joins dense, row 38) | `scalar_kind` f64, f32, f16, bf16, i8 (`b1` refused) | `memgraph_dense`: f32 | built: `memgraph_dense_int8`, `i8`; both read `scalar_kind` back from `SHOW VECTOR INDEX INFO` (`memgraph_vector_scalar_kind`). The i8 distances `vector_search.search` yields are integers, and the arm orders by them, so nothing re-ranks against the floats. Recall fp32 / int8: 0.9997 / 0.9423 at 5,000, 0.9992 / 0.9361 at 20,000 | image: `CREATE VECTOR INDEX ... scalar_kind` on the pinned server, `SHOW VECTOR INDEX INFO`; smoke: micro, tiny | | FalkorDB (6.0.1 at the re-pin, row 38) | vector index over float32 | `falkordb_dense`: fp32 | none documented | docs | diff --git a/benchmarks/experiments/READING-RESULTS.md b/benchmarks/experiments/READING-RESULTS.md index 483282a5ef7..e4fadfd9c36 100644 --- a/benchmarks/experiments/READING-RESULTS.md +++ b/benchmarks/experiments/READING-RESULTS.md @@ -50,7 +50,7 @@ The single rule, from which most of the rest follows: **`instrument` names the query set, timers, and durability rule a row ran under.** Rows before 2026-10 carry none and are the September instrument; `load_canonical` and `export_web` refuse two values in one table. Read a 2026-10 document OLTP row's `oltp_ops_per_s` as new-order and payment together (`oltp_ops` says how many), where a September row's is new-order alone. -**`durability` is what the engine ran at commit, read from the engine where it can be read.** PostgreSQL-family rows carry the server's own `SHOW synchronous_commit` answer; a value ending "(NOT the #81 setting)" means the server was not started with the flag and the row fails F10. Strings starting "fsync at commit" are the named exceptions (Neo4j, LadybugDB, DuckDB), and "unverified" is SurrealDB served, which has no setting to read. +**`durability` is what the engine ran at commit, read from the engine where it can be read.** PostgreSQL-family rows carry the server's own `SHOW synchronous_commit` answer; a value ending "(NOT the #81 setting)" means the server was not started with the flag and the row fails F10. Strings starting "fsync at commit" are the named exceptions (Neo4j, LadybugDB, DuckDB), and "unverified" is a class no engine is in now (`fairness_check.UNVERIFIED_ALLOWED` is empty; SurrealDB served was the last, until BUGS F165 found its `sync` setting). **A digest is of the canonical answer, not of the raw rows.** The digest is taken after the answer is put in canonical form: sorted unless the query defines an order, floats rounded to a fixed precision, engine-specific row wrappers and column ordering gone. Two engines whose digests match did not return identical result objects, and hashing what a driver handed back instead will disagree on every engine pair for reasons that are not about the answer. @@ -58,13 +58,13 @@ The single rule, from which most of the rest follows: **The revenue total and the revenue by month carry a row count in the answer.** It is compared exactly and is not a page column (DECISIONS #94): at the campaign's largest tier a single lost row moves either sum by less than the digest's rounding, detected 10.8 and 36.5 per cent of the time, and F46 is an engine that loses exactly one row to an index bound. -**A withheld cell is not a missing measurement.** A disagreement that has been reproduced and filed upstream prints as KNOWN and its cell is withheld (`equivalence_check.KNOWN_DISAGREEMENTS`, `export_web.WITHHELD_CELLS`); nothing is withheld today; the last entry (the served native time-series group-by, upstream #7610) was released on 2026-09-18. +**A withheld cell is not a missing measurement.** A disagreement that has been reproduced and filed upstream prints as KNOWN and its cell is withheld (`equivalence_check.KNOWN_DISAGREEMENTS`, `export_web.WITHHELD_CELLS`); nothing is withheld today (`WITHHELD_CELLS` is empty; the last entry, the served native time-series group-by, upstream #7610, was released on 2026-09-18). `KNOWN_DISAGREEMENTS` holds PostgreSQL + AGE's CRUD read-backs (apache/age#2587), whose cells publish with a note. **Every row since 2026-09-14 says how hot the machine was.** `host_temp_c_start` and `_end`, `host_throttle_count_start` and `_end`, and `host_throttled_ms` (BUGS.md F45). mini throttles under sustained load while the busy cores hold 4.3 GHz, the power mode is unchanged by decision, and rows from different stages are compared with those fields read rather than assumed equal. **"Unexpressible" is a declaration, not a failure.** An engine whose adapter declares an operation absent has said so deliberately (DECISIONS #88); `equivalence_check` names it and the table prints a dash with the reason in its condition. It is not a crashed cell, a timeout, or a gap to be filled, and it is not evidence that the engine is slow. A silently missing answer is the failure, and that is what the gate exists to tell apart from this. -**`ingest_s` and `index_s` are present only where the engine has the boundary** (on the dense lane ArcadeDB, pgvector, Neo4j, Milvus, LanceDB, the SurrealDB server, and MongoDB; on the document, time-series, and cross-model lanes every arm, through `bench_common.index_split`, with `index_before_load` true where the index was defined before the load and its work sits inside `ingest_s`); `build_s` is still the whole timer and the two do not sum to it exactly (schema creation and Milvus's compaction sit outside them). Milvus's `index_s` is its post-build wait, flush through load. +**`ingest_s` and `index_s` are present only where the engine has the boundary** (on the dense lane every arm except those `fairness_check.PHASE_SPLIT_DECLARED` names, which index while they load; on the document, time-series, and cross-model lanes every arm, through `bench_common.index_split`, with `index_before_load` true where the index was defined before the load and its work sits inside `ingest_s`); `build_s` is still the whole timer and the two do not sum to it exactly (schema creation and Milvus's compaction sit outside them). Milvus's `index_s` is its post-build wait, flush through load. **The graph write pass now has a partner.** `delete_p50_ms` and `delete_p99_ms` come from deleting, in order, the persons the write pass created (with their edge), one transaction each; `hop3f_*` is the 3-hop read filtered on the far end. Neither exists on September rows. @@ -76,9 +76,8 @@ is what the ENGINE reported, read back out of it wherever it can be asked; healthy row. A row where they differ is a flag that did not take -- an environment variable the server ignores, a JVM property that never reached the JVM -- and `fairness_check` fails it rather than publishing the asserted value. -`durability_no_setting` marks the four engines with no knob (Neo4j, DuckDB, -LadybugDB, the SurrealDB 3.2.4 server); they run once and the page prints that -one number in both columns. `durability_server_flags` says what the runner +`durability_no_setting` marks the three engines with no knob (Neo4j, DuckDB, +LadybugDB); they run once and the page prints that one number in both columns. `durability_server_flags` says what the runner changed on a server container for the class, and is blank for an embedded arm. **A strict cell is a different cell, not a different column.** The strict arm of diff --git a/benchmarks/experiments/RESULTS-MAP.md b/benchmarks/experiments/RESULTS-MAP.md index 92a3b410bf2..be07057d68d 100644 --- a/benchmarks/experiments/RESULTS-MAP.md +++ b/benchmarks/experiments/RESULTS-MAP.md @@ -78,6 +78,8 @@ A campaign file reports "no reader" when you grep the publishing scripts. That i | `pagecache.py` | evicts one database's files from the page cache and proves they left, for a cold open | `l5_lifecycle.py`, `l5_lifecycle_surreal.py` | | `l5_lifecycle_server.py` | the lifecycle lane's served ArcadeDB arm | `l5_lifecycle.py` | | `l5_lifecycle_surreal.py` | the lifecycle lane's SurrealDB embedded arm | `l5_lifecycle.py` | +| `lean_http.py` | the persistent `http.client` session every ArcadeDB served arm talks HTTP through (CAMPAIGN.md section 7, row 72) | `l1_tabular.py`, `l1_tpc.py`, `l2_graph.py`, `l3_sparse.py`, `l3d_dense.py`, `l4_tsbs.py`, `e2_hybrid.py`, `l5_lifecycle_server.py`, `l6_restart.py`, `deployment_decomp_probe.py` | +| `overrides.py` | the registry of every override a table shows: the row field read back from the engine, the generated sentence, and the checks (PROTOCOL.md section 7) | the lanes, `export_web.py`, `fairness_check.py`, `page_check.py`, `test_overrides.py` | The answer digest and its readable sample, the `durability` string, and the thermal fields (BUGS.md F45) are row fields rather than files, so nothing under `results/` holds them separately and a question about any of the three is answered by printing the row. @@ -121,12 +123,10 @@ Add it to `FEEDS`/`FEEDS_FILES` in `provenance_check.py` **or** to this file. Th ## Queue scripts -The live chain and where finished scripts go: CAMPAIGN.md section 6; what each stage runs: `make_october_stages.STAGES`. Every script gates on `verify_pair_c25.sh`, which is in this directory alongside `build_matched_pair.sh` and `build_c25_wheel.sh`, the pair recipe. Retired `qB*` scripts pinned to `b7c6c800d` live in `queue-archive-20260830/` on the bench host; they verify a locally compiled pair, which is the wrong claim for a pair assembled from upstream's published jars. +The live chain and where finished scripts go: CAMPAIGN.md section 6; what each stage runs: `make_october_stages.STAGES` for the October chain and the generator `make_2610_stages.py` for the 26.10.1 measurement. Every script gates on `verify_pair_c25.sh`, which is in this directory alongside `build_matched_pair.sh` and `build_c25_wheel.sh`, the pair recipe. Retired `qB*` scripts pinned to `b7c6c800d` live in `queue-archive-20260830/` on the bench host; they verify a locally compiled pair, which is the wrong claim for a pair assembled from upstream's published jars. -## The 8d6af9475 pin +## Pins and the files they name -The engine is pinned to upstream's published snapshot `8d6af9475`, and the served arm runs Corretto 25 / glibc / G1 with compact object headers, matched to the embedded arm (DECISIONS #54). +Three pins have written rows. `8d6af9475` (upstream's published snapshot; the served arm runs Corretto 25 / glibc / G1 with compact object headers, matched to the embedded arm, DECISIONS #54) is the September campaign and the LIVE page. `417314c18` (a 26.10.1 snapshot) is the old October chain, and `d36b4ca3a` (the official 26.10.1 release) is the 26.10.1 measurement that replaces it table by table. Every ArcadeDB row measured under another pin is superseded, embedded as well as served; no ArcadeDB row is carried forward, and a comparator row carried forward from a previous pin is stamped `carried_forward_from` and `carried_forward_reason`. -Every ArcadeDB row measured before that re-pin is superseded, embedded as well as served: the jars moved 84 commits. The campaign started by carrying clean COMPARATOR rows forward from the previous pin's file, each stamped `carried_forward_from` and `carried_forward_reason`; no ArcadeDB row was carried. - -The live page stays on this pin for the whole October campaign (DECISIONS #83). October rows accumulate in their own `runs_*_.jsonl` under the new pin, are frozen and exported separately, and reach the site only through the preview target, whose payload is `arcadedb-benchmarks-next.json` and whose table inventory is `results/generated/preview-tables.md`. Its tables, figures, `GATE_STATUS.txt` and `withheld_recall.json` are written to `results/generated_oct/`, not to `results/generated/`, which is September's published set and what `page_check` pins the live page against; `BENCH_SKELETON=1` writes to `results/generated_skeleton/`, which is untracked. A row of one instrument is never merged into the freeze of the other: `instrument` on the row says which it is, and the freeze and the exporter refuse a lane that holds both. +The live page stays on the September pin for the whole campaign (DECISIONS #83). A campaign's rows accumulate in their own `runs_*_.jsonl`, are frozen and exported separately, and reach the site only through the preview target, whose payload is `arcadedb-benchmarks-next.json` and whose table inventory is `results/generated/preview-tables.md`. Its tables, figures, `GATE_STATUS.txt` and `withheld_recall.json` are written to `results/generated_oct/`, not to `results/generated/`, which is September's published set and what `page_check` pins the live page against; `BENCH_SKELETON=1` writes to `results/generated_skeleton/`, which is untracked. A row of one instrument is never merged into the freeze of the other: `instrument` on the row says which it is, and the freeze and the exporter refuse a lane that holds both. diff --git a/benchmarks/python-bindings/README.md b/benchmarks/python-bindings/README.md index 057be240517..a6f318bf2f9 100644 --- a/benchmarks/python-bindings/README.md +++ b/benchmarks/python-bindings/README.md @@ -16,7 +16,7 @@ Comparators (`python:3.12-slim`, x86_64): | arcadedb-embedded | `arcadedb-embedded` | 26.8.1 | documents + graph + vectors | | SQLite | stdlib `sqlite3` | 3.46.1 | documents, transactional | | DuckDB | `duckdb` | 1.5.4 | documents, analytical | -| LadybugDB | `ladybug` | 0.18.1 | graph (see `docs/ladybug-package.md`) | +| LadybugDB | `ladybug` | 0.18.1 | graph (the official package from LadybugDB/ladybug-python; the earlier `real_ladybug` 0.15.3, published from lbugdb/lbug, is frozen and not used) | | Chroma | `chromadb` | 1.5.9 | vectors (HNSW) | | Faiss | `faiss-cpu` | 1.14.3 | exact recall baseline | @@ -100,9 +100,6 @@ benchmarks/python-bindings/ │ ├── mem/ per-run memory time series (untracked) │ ├── overhead_2681/ an earlier JPype-overhead run (RESULT lines; results.csv tracked) │ └── results_abl_*/ ablation campaigns (GAV off, strict durability) -└── docs/ - ├── dev-log-2026-06.md development log, June to August 2026 - └── ladybug-package.md LadybugDB package and version note ``` ## What is and is not committed diff --git a/benchmarks/python-bindings/docs/dev-log-2026-06.md b/benchmarks/python-bindings/docs/dev-log-2026-06.md deleted file mode 100644 index 1dccab0ab5b..00000000000 --- a/benchmarks/python-bindings/docs/dev-log-2026-06.md +++ /dev/null @@ -1,141 +0,0 @@ -# Development log, June to August 2026 - -Historical notes from building and validating the harness, moved out of the README on -2026-09-07. Numbers here are laptop validation runs on the tiny tier, not paper numbers; -"tabular" is the name the paper later replaced with "documents"; the "planned layout" and -"remaining work" sections describe the state before the harness was complete. The paper's -numbers come from `results/runs_paper.csv` (ArcadeDB 26.8.1), see the README. - -Cross-check: ArcadeDB and Chroma returned identical cosine distances on seeded data → recall@k -methodology is sound. Faiss smoke used L2; the real exact baseline must use the **cosine** space -(normalize + `IndexFlatIP`). - -### Dataset prep — working - -`datasets/prepare.py ` downloads a Stack Exchange dump -(`https://archive.org/download/stackexchange/.7z`, CC BY-SA) and writes one normalized -multi-model dataset to `datasets/prepared//`: `posts`, `users`, `edges_posted`, -`edges_answers` (Parquet). `posts.text` = title + de-HTML'd body (for embeddings). -Validated on `coffee.stackexchange.com` (4.5k posts, 11.6k users) in ~6s. - -- **Site choice for the paper:** pick a thematically-fitting *small* site (candidates: - `cs.stackexchange.com` — the established choice; or `datascience`/`ai.stackexchange.com` - for ML relevance). `coffee` is only the pipeline smoke test. -- Snapshot note: archive.org serves the latest dump; record access date for repro. - -### Reuse from existing work (do NOT redo) - -The package repo already contains most of what we need under `bindings/python/examples/`: - -- **Data downloaded:** `examples/data/stackoverflow-tiny/` (~100K records, cs.stackexchange-derived, - full XML) and `stackoverflow-small/` (cs.stackexchange, 1.41M). No re-download. -- **Embeddings + recall ground-truth precomputed:** `stackoverflow-tiny/vectors/` — - `all-MiniLM-L6-v2`, **384-d**, 19,591 vectors (Q/A/comments) + `*.gt.jsonl` - (**1,000 queries × top-50 exact NN**). Reuse as the vector data AND the recall@k baseline. -- **Workload logic + SQLite/DuckDB/Faiss adapters:** examples 07–12. Harness/adapt. -- **Docker matrix orchestration:** `examples/scripts/run_*_matrix.sh`, `summarize_*`. Reuse pattern. - -**Chosen small dataset = `stackoverflow-tiny`** (real, laptop-scale, already has vectors + gt). -`prepare.py`'s download path is redundant; keep it only as an optional local-XML normalizer. - -### Vector lane — validated (unlimited, tiny) - -`vector_bench.py --backend {arcadedb,chroma}` loads the precomputed shards, builds the index, -runs the 1000 gt queries, computes **recall@k vs the exact cosine gt**. Validated on full `tiny` -(19,591 vec, 1000 q): ArcadeDB build 1.76s / q 2.68ms / recall@10 0.995; Chroma build 1.56s / -q 0.95ms / recall@10 0.998. (Unlimited resources — validation, not official numbers.) -IDs are `vector_id` (post_id is NOT unique in the 'all' corpus); gt is in vector_id space. - -### Orchestrator `run.py` — validated (laptop) - -`run.py --datasets tiny,small,medium --backends arcadedb,chroma --reps 5` runs each backend -container **one-at-a-time** under identical caps (**`--cpuset-cpus=0-7` + `--memory` 8g/16g**), -with a host-side **memory sidecar** (samples cgroup `memory.current`/`anon` every 0.25s → -time series; reads exact kernel `memory.peak`). Outputs: `results/runs.csv` (per-rep: timings, -recall@k, peak+anon MiB, limits, image digest), `results/mem/*.csv` (memory-over-time), -`results/ENV.md` (host + image digests). Validated: vector lane, tiny, 2 reps, both backends. - -Standard caps: **cpuset 0-7 (8 P-cores), mem 8g (16g medium)** — same on the laptop and the benchmark host. - -### Tabular lane — validated (tiny) - -`tabular_bench.py --backend {sqlite,duckdb,arcadedb} --workload {oltp,olap}` on normalized -`posts.parquet`. Idiomatic bulk load per engine (DuckDB native ingest, not row-by-row). -tiny result: ArcadeDB OLTP 4402 ops/s (vs sqlite 297, duckdb 375); OLAP duckdb 2.7ms / -sqlite 4.1ms / arcadedb 81ms. On-thesis: OLTP-first vs analytics-oriented. -`datasets/prepare.py ` normalizes local XML → parquet (no re-download). - -### Graph lane — validated (tiny) - -`graph_bench.py --backend {ladybug,arcadedb} --workload {oltp,olap}`. Graph: -(User)-[:POSTED]->(Post), (Post)-[:ANSWERS]->(Post); edges filtered to existing endpoints. -Shared Cypher (ints embedded). LadybugDB loads via COPY (idiomatic bulk); ArcadeDB via -transactional CREATE. tiny (validated on the pre-fork 0.11.3 release; lane unchanged under -LadybugDB): ArcadeDB OLTP 3827 ops/s (vs 1006), write 0.29ms vs 5.3ms; graph OLAP 8.5ms -vs arcadedb 59ms + faster bulk load. On-thesis (OLTP-first vs OLAP-optimized). - -### All three lanes validated (laptop, tiny) — coherent story - -ArcadeDB: competitive vector recall; **wins OLTP** (tabular ~12×, graph ~4×); loses OLAP to the -specialists (DuckDB/SQLite ~25×, LadybugDB 7×). Honest, no cherry-picking. - -### Full suite orchestrated — validated (laptop, tiny, 2 reps) - -`run.py --datasets tiny --reps 2 --lanes vector,tabular,graph` ran all 12 cells × 2 reps = 24 -runs clean, capped (cpuset 0-7, 8g) with memory tracking → `runs.csv` (24 rows), `manifest.json`, -`ENV.md`, 24 `mem/*.csv`. Memory tradeoff quantified: ArcadeDB (JVM) ~440-700 MiB vs -SQLite ~63 / DuckDB ~107 / LadybugDB ~210 / Chroma ~560 MiB. - -**Harness complete.** Remaining = (1) `prepare.py` for small + medium (tabular/graph need their -`prepared/` parquet; vector already has all three), (2) move to the benchmark host: clone repo, build images, -rsync data, run official `--reps 5 --datasets tiny,small,medium`. - -### Real remaining work (everything else is reuse) - -- **Chroma** vector adapter (load the existing `.f32` vectors; recall@k vs `gt.jsonl`). -- **LadybugDB** graph adapter + graph workload. -- **Clean, trimmed runner** for our 4 lanes × small scale × 5 reps (drop server backends + the - sprawling matrix) — this is the "fit SciPy" repackaging. -- Wire recall@k to the existing `gt.jsonl`; quiet JVM INFO logs. - -## What it does - -DB-to-DB comparison, all **embedded / in-process from Python**, full OLTP×OLAP crossing per -data-model lane, plus vector build/search with honest recall@k. - -| Lane | Systems | Workloads | -|---|---|---| -| Tabular | SQLite, DuckDB, ArcadeDB | OLTP × OLAP | -| Graph | LadybugDB, ArcadeDB | OLTP × OLAP | -| Vector (HNSW) | Chroma, ArcadeDB (+ Faiss/bruteforce exact baseline) | build + search, recall@k | - -## Protocol - -- Latest versions, **pinned** (record version + Docker image digest). -- **Docker, one experiment at a time**, identical CPU/memory limits across containers. -- Each comparator runs as an **embedded library inside one Python process** (not a server). -- **5 repetitions** per cell → report **mean ± std** (min/max ok). -- Discard a warm-up for query/throughput; measure **cold JVM startup** separately. -- Idiomatic index per system; **vector recall@k vs exact** (Faiss/bruteforce) ground truth. -- **Scaled down:** small/medium dataset only (no LARGE tier); one representative size in-paper. -- Dataset: Stack Overflow (same data for all systems); cite the dump source. - -## Planned layout (to build) - -``` -experiments/ - README.md # this file - datasets/ # download + prep scripts (small/medium) - backends/ # one adapter per system: arcadedb, sqlite, duckdb, ladybug, chroma, exact - workloads/ # tabular_oltp, tabular_olap, graph_oltp, graph_olap, vector - docker/ # per-backend Dockerfiles / pinned images - run.py # orchestrator: 5 reps, isolation, env capture → results/*.csv - results/ # output CSVs (mean/std) — feed the paper's plotting - ENV.md # captured environment (CPU/RAM/OS/Docker/versions/digests) -``` - -## Running (target: the benchmark host) - -- Sync this repo to the benchmark host, run the orchestrator in the background (tmux/nohup) over SSH. -- Collect `results/*.csv`; copy the light plotting + CSVs into the PR as supplementary. -- Record environment to `ENV.md` for the paper's reproducibility section. diff --git a/benchmarks/python-bindings/docs/ladybug-package.md b/benchmarks/python-bindings/docs/ladybug-package.md deleted file mode 100644 index 4f62f974802..00000000000 --- a/benchmarks/python-bindings/docs/ladybug-package.md +++ /dev/null @@ -1,16 +0,0 @@ -# LadybugDB package + versions (updated 2026-08-28) - -This suite (the SciPy paper's) uses the OFFICIAL LadybugDB package **`ladybug`** -(0.18.1, from LadybugDB/ladybug-python); the live harness under `experiments/` -pins its own version (COMPARATORS.md). Earlier they pinned `real_ladybug` (0.15.3), which is -published from a different repo (lbugdb/lbug) and is frozen; we switched after -LadybugDB shipped 0.18.1. - -Versions pinned (all on PyPI): -- ArcadeDB: `arcadedb-embedded==26.8.1` (release; every paper row was re-measured on - the PyPI 26.8.1 wheel on 2026-08-04, image sha256:d0dbe7c653c1). -- LadybugDB: `ladybug==0.18.1`. -- DuckDB 1.5.4, SQLite 3.46.1, Chroma 1.5.9 (unchanged). - -The paper and poster were re-measured on the benchmark host and updated to these versions and -numbers. Prior campaigns are not kept in the repository. diff --git a/benchmarks/python-bindings/jpype_overhead/REPORT.md b/benchmarks/python-bindings/jpype_overhead/REPORT.md index afd66729bfc..f11a0a88798 100644 --- a/benchmarks/python-bindings/jpype_overhead/REPORT.md +++ b/benchmarks/python-bindings/jpype_overhead/REPORT.md @@ -80,7 +80,7 @@ workaround was removed and the exit-hang regression test now guards the engine f ## What changed in the bindings -New/changed public API (all with fallbacks and regression tests; suite 352 passed after the embedded-only cut): +New/changed public API (all with fallbacks and regression tests; the suite passed, 352 tests, when this report was written): - `ResultSet.to_json_list(batch_size=)` / `iter_json_batches()` — bulk materialization via batched Java-side JSON serialization: one JPype crossing per @@ -121,7 +121,7 @@ isinstance chain runs once per *type*, not per value); cached JClass/java.time handles; Java-RID fast path in vector search (no string round-trip + re-fetch per hit); per-wrapper index/PQ-check caching; `export_to_csv` streams JSON batches. -**The bridge jar.** `RowBatcher`/`EdgeBatcher`/`VertexBatcher`/`ColumnBatcher` (~450 lines, +**The bridge jar.** `RowBatcher`/`EdgeBatcher`/`VertexBatcher`/`ColumnBatcher` (~450 lines when this report was written; the bridge has grown since, `bindings/python/src/java/com/arcadedb/python/`) compile into `arcadedb-python-bridge.jar` during both wheel builds (`Dockerfile.build` and `build-native.sh`) and ship inside the wheel next to the engine JARs. They are @@ -167,35 +167,6 @@ Python side falls back to pure-JPype paths if the jar is absent. `probe_round3.py`); two plausible designs measured slower and were rejected (`Result.toMap()`, string-array marshaling). -## Wheel-size audit (2026-07-05) - -Same quarantine-and-gate method applied to the wheel contents (82MB → **70.8MB**): - -- **Cut (~11MB, feature-preserving; verified by full suite incl. server tests + - examples run)**: `arcadedb-tracing` (4.4MB), `snappy-java` (2.4MB, engine uses - LZ4 not snappy), `jackson-*` (2.1MB), `commons-math3` (2.0MB, zero users), - `jline` (0.4MB, console-only). `micrometer` must stay — server startup - requires it (found by bisection). -- **Kept deliberately**: Lucene stack (~7.5MB — FULL_TEXT tokenization + - geospatial genuinely use it), JTS (geospatial). -- **Server mode removed (follow-up decision, 2026-07-05)**: the - server/studio/undertow/xnio/micrometer set (~7.7MB uncompressed) was - initially kept under the "small fraction" bar, then cut when the maintainer - opted for an embedded-only package: wheel **70.8MB → 63.5MB**, 51 JARs. - The official ArcadeDB server distribution is the supported client-server - path. (micrometer's only consumer was server startup, so it went too.) -- **JRE lever**: initially judged dead — with the server jars present, the - jdeps-detected module set measured within 1MB of the `java.se` umbrella. - The audit also found the jdeps detection had been **silently broken** — - `lucene-spatial-extras` declares a missing module and aborted the whole scan, - so builds always used the fallback. Fixed in both build scripts (excluded from - the scan, like jboss/wildfly). After the server removal the detected set - shrank well below `java.se`, so the umbrella was dropped (2026-07-05): - jlink now builds from the detected 16 modules — wheel **63.5MB → 61.6MB**, - gated by the full suite + examples. Floor reached: the remaining candidates - (java.desktop is statically required by jts-core/commons-lang3; native-symbol - stripping; Lucene sub-jar bisection) are crumbs or rule-violations. - ## Paper-grade verification on the benchmark host (2026-07-05) Independent machine (20 cores, idle), 5 runs/layer, same wheel: