Skip to content
This repository was archived by the owner on Nov 21, 2026. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,10 @@ in scope and in where they publish, not in the kind of thing they measure.
against specialist and multi-model engines on documents,
graph, dense and sparse vectors, time series, and cross-model queries. It
is what every campaign runs, and its results are the project page at
<https://humem.ai/projects/arcadedb>. There is no paper; the page is the
only artifact, and every number on it is generated from the frozen rows by
the exporter and checked by gates before it can be published.
<https://humem.ai/projects/arcadedb>. The page is the living artifact, and
every number on it is generated from the frozen rows by the exporter and
checked by gates before it can be published; a paper written from the same
rows pins a stable release where the page may pin a commit (DECISIONS #42).
- `python-bindings/` is the **SciPy 2026 paper's artifact**, frozen with the
paper: the package as published on PyPI against four embedded Python
specialists on one corpus, plus `jpype_overhead/`, which times the binding
Expand All @@ -22,8 +23,8 @@ in scope and in where they publish, not in the kind of thing they measure.
extended; new work goes in `experiments/`. The one exception is
`jpype_overhead/results/mini_results_<pin>.csv`, the bench-host run that feeds the
page's Python-cost table: `export_web.py` reads the file named by the pin it
publishes, only the September pin's exists, and the October payload omits the
table until it is re-measured. Its own README carries its versions and layout.
publishes, only the September pin's exists until the 26.10.1 measurement's
Python-cost stage lands, and the October payload omits the table until then. Its own README carries its versions and layout.

## Where to start in `experiments/`

Expand All @@ -38,6 +39,7 @@ its detail. Read it first; the rest are the references it points to.
| `PROTOCOL.md` | How a row is produced, and what enforces each rule (a gate, a check, or a person) |
| `FAIRNESS.md` | When two numbers may be compared: the invariants and the disclosed overrides |
| `COMPARATORS.md` | What every comparator is pinned to, and why it runs the way it does |
| `QUANTIZATION.md` | What each dense and sparse engine ships, and which quantized arm the page runs |
| `PAGE-SPEC.md` | What the page contains: tables, rows, columns, and what each cell must satisfy |
| `PUBLISHING.md` | How the numbers get from the frozen rows onto the page, the preview route, and the results asset |
| `READING-RESULTS.md` | How to read what came out without misreading it |
Expand Down
216 changes: 89 additions & 127 deletions benchmarks/experiments/CAMPAIGN.md

Large diffs are not rendered by default.

19 changes: 10 additions & 9 deletions benchmarks/experiments/COMPARATORS.md

Large diffs are not rendered by default.

10 changes: 5 additions & 5 deletions benchmarks/experiments/FAIRNESS.md

Large diffs are not rendered by default.

32 changes: 16 additions & 16 deletions benchmarks/experiments/PAGE-SPEC.md

Large diffs are not rendered by default.

24 changes: 12 additions & 12 deletions benchmarks/experiments/PROTOCOL.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion benchmarks/experiments/PUBLISHING.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ The gate scripts (`refresh_web_page.GATES`), then three structural checks. All o

`PROSE` is its own section because the page's two surfaces fail differently. Cells are written by the exporter straight from the frozen results, so a wrong one is nearly impossible. Prose is typed by hand, so a wrong one is nearly inevitable: a caption once gave one engine's dense latency from the canonical CSV and the other's from the matched overlay, inside one sentence, with no published cell wrong and nothing invented.

A disagreement that has been reproduced, understood, and filed upstream prints as KNOWN rather than FAIL and its cell is withheld from the page instead of being published beside an answer it does not match (`equivalence_check.KNOWN_DISAGREEMENTS`, `export_web.WITHHELD_CELLS`); nothing is withheld today (the last entry, the served native time-series group-by, upstream #7610, was released on 2026-09-18 when the fix reached the pin). The entry is deleted at the re-pin that carries the fix, which re-arms the gate for that query, because a gate that cannot pass until an upstream release lands is a gate somebody deletes.
A disagreement that has been reproduced, understood, and filed upstream prints as KNOWN rather than FAIL and its cell is withheld from the page instead of being published beside an answer it does not match (`equivalence_check.KNOWN_DISAGREEMENTS`, `export_web.WITHHELD_CELLS`); nothing is withheld today (`WITHHELD_CELLS` is empty; the last entry, the served native time-series group-by, upstream #7610, was released on 2026-09-18 when the fix reached the pin). `KNOWN_DISAGREEMENTS` holds PostgreSQL + AGE's CRUD read-backs (apache/age#2587), whose cells publish with a note and are not withheld (DECISIONS #134). An entry is deleted at the re-pin that carries the fix, which re-arms the gate for that query, because a gate that cannot pass until an upstream release lands is a gate somebody deletes.

`WEB_ONLY_FIGURES` is the escape hatch for a figure the page shows and nothing else cites. It is deliberately empty; earn a line in it, do not assume one.

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/experiments/QUANTIZATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ The quantization survey of DECISIONS #131 item 8 (CAMPAIGN.md section 7 row 42),
| Neo4j 2026.08.1 | `vector.quantization.type` `NONE`, `SCALAR`, `BINARY`; **unset means `BINARY`**, with `vector.default_search_expansion_factor` 3.0 | until 2026-10-02 **recorded fp32, ran BINARY**: `neo4j_dense` and `neo4j_e2` created their index without the key (BUGS F164). Now `neo4j_dense` and `neo4j_e2` set `NONE`, and both read the applied type back (`neo4j_vector_quantization`) | built: `neo4j_dense_int8` sets `SCALAR` (repin-prep `32c0a011da`). Each type brings its own `vector.default_search_expansion_factor` (NONE 1.0, SCALAR 1.5, BINARY 3.0 on 2026.08.1 and 2026.09.0, the candidate multiple searched before re-scoring); the lane matches search effort at ef 100 (Neo4j asks for 100 candidates and keeps 10), so both arms set 1.0 and read it back (`neo4j_vector_search_expansion`), as the Elasticsearch int8 arm sets the minimum oversample. 2026.09.0 (the re-pin candidate, `neo4j@sha256:eaab5879…`) keeps the BINARY default (`.notes` repros/neo4j-202609-quant) | image: `SHOW VECTOR INDEXES` on an index created with exactly our options reports provider `vector-2026.08` and `vector.quantization.type: "BINARY"`; `INT8` and `FLOAT32` are syntax errors |
| MongoDB 8.2.12 + mongot-community 1.70.4 | `vectorSearch` index `quantization`: `none`, `scalar`, `binary` | `mongodb_dense`: `none` | built: `mongodb_dense_int8` sets `scalar`. Both read the field back from mongot's index definition (`mongot_vector_quantization`, `none` / `scalar`). Recall fp32 / int8: 0.9996 / 0.969 at 5,000, 0.9977 / 0.9699 at 20,000 | adapter; smoke: micro, tiny (`scalar` accepted, built, queryable); docs for `binary` |
| SurrealDB (core 2.3.10 embedded, 3.2.4 served) | HNSW `TYPE` F64, F32, I64, I32, I16: element types of the stored array, not quantization of float vectors | `TYPE F32` | none shipped | adapter; docs |
| ArangoDB 3.12.11 | FAISS IVF vector index; a custom FAISS `factory` string (SQ8, SQ4, PQ), forwarded to FAISS and validated at creation only from 3.12.12 | `arangodb_dense`: IVF flat, fp32 (`nLists`, `nProbe` calibrated) | built: `arangodb_dense_int8`, factory `IVF<nLists>,SQ8` at the fp32 arm's nLists, nProbe calibrated to the same recall target; the server's factory string is read back (`ivf_factory`, None on the fp32 arm) beside its training state. Recall fp32 / int8, each calibrated on the held-out slice to the laptop's fallback target 0.95: 0.9402 / 0.9465 at 5,000, 0.9486 / 0.9447 at 20,000, at nProbe 17 / 19 and 23 / 24 | image: 20,000 SIFT vectors at nLists 566, both indexes `ready`, recall@10 at nProbe 1, 4, 16, 64: plain 0.345, 0.666, 0.932, 0.9985; SQ8 0.345, 0.6645, 0.928, 0.9905; docs (factory text, the 3.12.12 validation); smoke: micro, tiny; 3.12.12 (the re-pin candidate, `arangodb@sha256:5b734c1a…`): `IVF70,SQ8` trains to ready and reads back unchanged at the same recall as 3.12.11, and an invalid factory string is now refused at creation, where 3.12.11 built an `unusable` index that answered by an exact scan (recall 1.0); the arm refuses any index that is not `ready` either way (`.notes` repros/arangodb-31212-sq8) |
| ArangoDB 3.12.12 (3.12.11 until the 26.10.1 measurement) | FAISS IVF vector index; a custom FAISS `factory` string (SQ8, SQ4, PQ), forwarded to FAISS and validated at creation only from 3.12.12 | `arangodb_dense`: IVF flat, fp32 (`nLists`, `nProbe` calibrated) | built: `arangodb_dense_int8`, factory `IVF<nLists>,SQ8` at the fp32 arm's nLists, nProbe calibrated to the same recall target; the server's factory string is read back (`ivf_factory`, None on the fp32 arm) beside its training state. Recall fp32 / int8, each calibrated on the held-out slice to the laptop's fallback target 0.95: 0.9402 / 0.9465 at 5,000, 0.9486 / 0.9447 at 20,000, at nProbe 17 / 19 and 23 / 24 | image: 20,000 SIFT vectors at nLists 566, both indexes `ready`, recall@10 at nProbe 1, 4, 16, 64: plain 0.345, 0.666, 0.932, 0.9985; SQ8 0.345, 0.6645, 0.928, 0.9905; docs (factory text, the 3.12.12 validation); smoke: micro, tiny; 3.12.12 (the pin since the 26.10.1 measurement; its digest is in COMPARATORS.md): `IVF70,SQ8` trains to ready and reads back unchanged at the same recall as 3.12.11, and an invalid factory string is now refused at creation, where 3.12.11 built an `unusable` index that answered by an exact scan (recall 1.0); the arm refuses any index that is not `ready` either way (`.notes` repros/arangodb-31212-sq8) |
| Elasticsearch 9.5.4 (joins dense, row 37) | `hnsw`, `int8_hnsw`, `int4_hnsw`, `bbq_hnsw`, `bbq_disk`, `int8_flat` (all accepted); **default `int8_hnsw` at 96 and 128 dimensions** (m 16, ef_construction 100), `bbq_hnsw` with 3x oversampling from 384 | `elasticsearch_dense`: `hnsw`, set explicitly | built: `elasticsearch_dense_int8`, `int8_hnsw` with `rescore_vector.oversample` 1.0; both read the mapping's `index_options` back (`es_index_options`) | image: dense_vector mappings on the pinned server, defaults read through `_mapping/field?include_defaults=true` |
| Memgraph 3.13.1 (joins dense, row 38) | `scalar_kind` f64, f32, f16, bf16, i8 (`b1` refused) | `memgraph_dense`: f32 | built: `memgraph_dense_int8`, `i8`; both read `scalar_kind` back from `SHOW VECTOR INDEX INFO` (`memgraph_vector_scalar_kind`). The i8 distances `vector_search.search` yields are integers, and the arm orders by them, so nothing re-ranks against the floats. Recall fp32 / int8: 0.9997 / 0.9423 at 5,000, 0.9992 / 0.9361 at 20,000 | image: `CREATE VECTOR INDEX ... scalar_kind` on the pinned server, `SHOW VECTOR INDEX INFO`; smoke: micro, tiny |
| FalkorDB (6.0.1 at the re-pin, row 38) | vector index over float32 | `falkordb_dense`: fp32 | none documented | docs |
Expand Down
11 changes: 5 additions & 6 deletions benchmarks/experiments/READING-RESULTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,21 +50,21 @@ The single rule, from which most of the rest follows:

**`instrument` names the query set, timers, and durability rule a row ran under.** Rows before 2026-10 carry none and are the September instrument; `load_canonical` and `export_web` refuse two values in one table. Read a 2026-10 document OLTP row's `oltp_ops_per_s` as new-order and payment together (`oltp_ops` says how many), where a September row's is new-order alone.

**`durability` is what the engine ran at commit, read from the engine where it can be read.** PostgreSQL-family rows carry the server's own `SHOW synchronous_commit` answer; a value ending "(NOT the #81 setting)" means the server was not started with the flag and the row fails F10. Strings starting "fsync at commit" are the named exceptions (Neo4j, LadybugDB, DuckDB), and "unverified" is SurrealDB served, which has no setting to read.
**`durability` is what the engine ran at commit, read from the engine where it can be read.** PostgreSQL-family rows carry the server's own `SHOW synchronous_commit` answer; a value ending "(NOT the #81 setting)" means the server was not started with the flag and the row fails F10. Strings starting "fsync at commit" are the named exceptions (Neo4j, LadybugDB, DuckDB), and "unverified" is a class no engine is in now (`fairness_check.UNVERIFIED_ALLOWED` is empty; SurrealDB served was the last, until BUGS F165 found its `sync` setting).

**A digest is of the canonical answer, not of the raw rows.** The digest is taken after the answer is put in canonical form: sorted unless the query defines an order, floats rounded to a fixed precision, engine-specific row wrappers and column ordering gone. Two engines whose digests match did not return identical result objects, and hashing what a driver handed back instead will disagree on every engine pair for reasons that are not about the answer.

**A measure is compared as a number; a count and an identifier are compared exactly.** A summed or averaged column is declared numeric once per query, so an engine returning an integer and one returning a double agree: ArangoDB's `SUM` returns 37,734,107 where the others return 37,734,107.0, which split the pricing summary seven engines to one at SF1 while both sides held the same number. A count is never declared that way, because the float side rounds to six significant digits and 1,234,567 would agree with 1,234,568.

**The revenue total and the revenue by month carry a row count in the answer.** It is compared exactly and is not a page column (DECISIONS #94): at the campaign's largest tier a single lost row moves either sum by less than the digest's rounding, detected 10.8 and 36.5 per cent of the time, and F46 is an engine that loses exactly one row to an index bound.

**A withheld cell is not a missing measurement.** A disagreement that has been reproduced and filed upstream prints as KNOWN and its cell is withheld (`equivalence_check.KNOWN_DISAGREEMENTS`, `export_web.WITHHELD_CELLS`); nothing is withheld today; the last entry (the served native time-series group-by, upstream #7610) was released on 2026-09-18.
**A withheld cell is not a missing measurement.** A disagreement that has been reproduced and filed upstream prints as KNOWN and its cell is withheld (`equivalence_check.KNOWN_DISAGREEMENTS`, `export_web.WITHHELD_CELLS`); nothing is withheld today (`WITHHELD_CELLS` is empty; the last entry, the served native time-series group-by, upstream #7610, was released on 2026-09-18). `KNOWN_DISAGREEMENTS` holds PostgreSQL + AGE's CRUD read-backs (apache/age#2587), whose cells publish with a note.

**Every row since 2026-09-14 says how hot the machine was.** `host_temp_c_start` and `_end`, `host_throttle_count_start` and `_end`, and `host_throttled_ms` (BUGS.md F45). mini throttles under sustained load while the busy cores hold 4.3 GHz, the power mode is unchanged by decision, and rows from different stages are compared with those fields read rather than assumed equal.

**"Unexpressible" is a declaration, not a failure.** An engine whose adapter declares an operation absent has said so deliberately (DECISIONS #88); `equivalence_check` names it and the table prints a dash with the reason in its condition. It is not a crashed cell, a timeout, or a gap to be filled, and it is not evidence that the engine is slow. A silently missing answer is the failure, and that is what the gate exists to tell apart from this.

**`ingest_s` and `index_s` are present only where the engine has the boundary** (on the dense lane ArcadeDB, pgvector, Neo4j, Milvus, LanceDB, the SurrealDB server, and MongoDB; on the document, time-series, and cross-model lanes every arm, through `bench_common.index_split`, with `index_before_load` true where the index was defined before the load and its work sits inside `ingest_s`); `build_s` is still the whole timer and the two do not sum to it exactly (schema creation and Milvus's compaction sit outside them). Milvus's `index_s` is its post-build wait, flush through load.
**`ingest_s` and `index_s` are present only where the engine has the boundary** (on the dense lane every arm except those `fairness_check.PHASE_SPLIT_DECLARED` names, which index while they load; on the document, time-series, and cross-model lanes every arm, through `bench_common.index_split`, with `index_before_load` true where the index was defined before the load and its work sits inside `ingest_s`); `build_s` is still the whole timer and the two do not sum to it exactly (schema creation and Milvus's compaction sit outside them). Milvus's `index_s` is its post-build wait, flush through load.

**The graph write pass now has a partner.** `delete_p50_ms` and `delete_p99_ms` come from deleting, in order, the persons the write pass created (with their edge), one transaction each; `hop3f_*` is the 3-hop read filtered on the far end. Neither exists on September rows.

Expand All @@ -76,9 +76,8 @@ is what the ENGINE reported, read back out of it wherever it can be asked;
healthy row. A row where they differ is a flag that did not take -- an
environment variable the server ignores, a JVM property that never reached the
JVM -- and `fairness_check` fails it rather than publishing the asserted value.
`durability_no_setting` marks the four engines with no knob (Neo4j, DuckDB,
LadybugDB, the SurrealDB 3.2.4 server); they run once and the page prints that
one number in both columns. `durability_server_flags` says what the runner
`durability_no_setting` marks the three engines with no knob (Neo4j, DuckDB,
LadybugDB); they run once and the page prints that one number in both columns. `durability_server_flags` says what the runner
changed on a server container for the class, and is blank for an embedded arm.

**A strict cell is a different cell, not a different column.** The strict arm of
Expand Down
Loading
Loading