Skip to content
This repository was archived by the owner on Nov 21, 2026. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 8 additions & 10 deletions benchmarks/experiments/CAMPAIGN.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion benchmarks/experiments/COMPARATORS.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,7 @@ Pins follow DECISIONS #87 (the latest self-hosted stable release at campaign tim
| Neo4j driver (Neo4j AND Memgraph arms) | `neo4j==6.3.1` | client package |
| Neo4j driver Rust extension (the vendor's documented accelerator, DECISIONS #173) | `neo4j-rust-ext==6.3.1.0` | client package |
| pgvector Python adapter (binary COPY, DECISIONS #174) | `pgvector==0.5.0` | client package |
| ArcadeDB official Python driver (sync, httpx; the served ArcadeDB arms' headline client under BENCH_INSTRUMENT=2026-11, CAMPAIGN section 7 row 72) | `arcadedb-driver==0.2.0` | client package |
| ArcadeDB official Python driver (sync, httpx; the served ArcadeDB arms' headline client, CAMPAIGN section 7 row 72) | `arcadedb-driver==0.2.0` | client package |

**Held below the newest, and why.**

Expand Down
10 changes: 5 additions & 5 deletions benchmarks/experiments/FAIRNESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ The failure mode this contract exists to close is **a correct number measured un

**F2. Serial only.** Published cells run one at a time. `runner.py` forces `workers=1` on the paper tier and errors otherwise; queue scripts pass `--tier paper --workers 1` explicitly. The parallel sweep tier (disjoint cpuset shards, shuffled order) exists for exploration and must never reach a table. A published row from a sweep is detectable after the fact as a partial cpuset such as `0-5`, and `load_canonical` drops it.

**F3. Same memory envelope per (lane, scale).** Every backend at a tier gets the same `--memory`/`--memory-swap` cap and, for JVM engines, the same heap (the same `-Xmx`; under `BENCH_INSTRUMENT=2026-11` the embedded ArcadeDB arms leave `-Xms` unset, as the bindings document, while each server engine keeps the initial heap its own documentation recommends: PROTOCOL.md section 7, "JVM initial heap"). Two JVM processes are exceptions and run at their own default heap: QuestDB, whose rows record the tier heap although no heap reaches its server, and mongot inside the MongoDB Search image (CAMPAIGN.md section 7, row 30). One ArcadeDB arm is a declared exception on purpose (F3b): the served arm at the vendor image's own JVM settings, on the documents transaction workload only (CAMPAIGN.md section 7, row 69). It gets the same memory cap as every other arm and a different heap, collector, and initial heap size; its row records no tier heap, stamps `server_jvm_defaults`, and carries the JVM's own report of what it ran (`server_jvm_*`, read from the process). `fairness_check.JVM_DEFAULTS_ARMS` names it and what it differs in, and the gate fails a `jvm_defaults` arm that is not declared and a declared arm that is not marked, so the one arm that runs other settings than its siblings says so and shows it.
**F3. Same memory envelope per (lane, scale).** Every backend at a tier gets the same `--memory`/`--memory-swap` cap and, for JVM engines, the same heap (the same `-Xmx`; the embedded ArcadeDB arms leave `-Xms` unset, as the bindings document, while each server engine keeps the initial heap its own documentation recommends: PROTOCOL.md section 7, "JVM initial heap"). Two JVM processes are exceptions and run at their own default heap: QuestDB, whose rows record the tier heap although no heap reaches its server, and mongot inside the MongoDB Search image (CAMPAIGN.md section 7, row 30). One ArcadeDB arm is a declared exception on purpose (F3b): the served arm at the vendor image's own JVM settings, on the documents transaction workload only (CAMPAIGN.md section 7, row 69). It gets the same memory cap as every other arm and a different heap, collector, and initial heap size; its row records no tier heap, stamps `server_jvm_defaults`, and carries the JVM's own report of what it ran (`server_jvm_*`, read from the process). `fairness_check.JVM_DEFAULTS_ARMS` names it and what it differs in, and the gate fails a `jvm_defaults` arm that is not declared and a declared arm that is not marked, so the one arm that runs other settings than its siblings says so and shows it.

A served backend gets the **full tier cap** and the client its own `BENCH_CLIENT_MEM` budget on top, stamped `mem_split="full+client"`, so a served engine sees exactly the cap an embedded engine of the same tier sees. Every frozen served row carries that stamp. `BENCH_SERVER_MEM_FRACTION` restores the older `server = 0.75 * total` split for a reproduction. Compare a served topology by `srv_cap / mem_split`, never by adding client and server: addition reads 1.75x the envelope.

Expand Down Expand Up @@ -216,15 +216,15 @@ Two consequences:

**Sharding, when parallel is allowed.** Disjoint cpuset shards, never overlapping, never crossing an SMT sibling pair, and the same shard width for every arm of a comparison. Three 4-thread shards over `0-11`, not "whatever is free". An arm given a wider shard than its neighbour is F1 violated with extra steps.

## Memory and disk per phase are read the same way for every engine (instrument 2026-11)
## Memory and disk per phase are read the same way for every engine

The question a memory column answers is "what did this engine hold, at the end of this phase", and the answer has to come from something every engine has and none can change. That is the cgroup of the container the engine runs in: `anon` plus `shmem` at the end of the phase, summed over the cell's containers (the server's and the client's for a served engine, the client's alone for an embedded one), read by the runner from the host. The headline column is that number and nothing else. Page cache is context and stays out for every engine alike. Disk and IO follow the same rule: the headline is the bytes of the container that holds the engine's data, less its empty start, and the bytes that container's cgroup read and wrote since the previous phase, both from the host and both defined for every engine whatever its storage layout. The write-ahead log is found by file name and is context only, because naming it is a per-engine table.

What a runtime exposes beyond that is recorded and never ranked across runtimes. A JVM engine gets its heap at the end of each phase (used, committed, direct buffers where visible), after a forced full GC only at the end of the last timed phase and in idle, never right before a timed phase, so no JVM engine starts a timed phase on a freshly collected heap that a native engine does not get (the row's `jvm_forced_gc_phases` says which phases were collected, and a heap read without a GC is named `jvm_<phase>_used_mib`, not after-GC); an embedded engine gets its own process's anonymous pages and resident set. Nothing is done for a native engine that is not done for a JVM one before a timed phase. The GC is an extra reading about the JVM, taken after the timed work so that its working set can be told from the heap it was given; the headline reading of a JVM engine at `queries` follows it, as a native engine's follows its own allocator's state at the same moment. A native allocator that keeps freed pages (glibc arenas, jemalloc) stays as it is: the harness does not trim it, and that difference is a property of the engines.

A pinned heap (`-Xms` equal to `-Xmx`) is a resource decision that moves the kernel's reading to the cap, whatever the workload. It is why the 2026-10 memory column of a pinned JVM arm reads its pin. The 2026-11 instrument does not pin the embedded ArcadeDB arms, so their headline is what the engine grew to; served engines start with the heap their vendors' documentation gives, and the row says which. Any engine's row can be read without the JVM numbers, and no JVM-only number is ever the headline.
A pinned heap (`-Xms` equal to `-Xmx`) is a resource decision that moves the kernel's reading to the cap, whatever the workload. It is why the memory column of a pinned JVM arm reads its pin. The embedded ArcadeDB arms are not pinned, so their headline is what the engine grew to; served engines start with the heap their vendors' documentation gives, and the row says which. Any engine's row can be read without the JVM numbers, and no JVM-only number is ever the headline.

**GATE** `test_memory_probe.py` (parsers on synthetic cgroup, smaps and `jcmd` input; the lane and host handshake; a 2026-10 row has no new field).
**GATE** `test_memory_probe.py` (parsers on synthetic cgroup, smaps and `jcmd` input; the lane and host handshake; a row stamped by an earlier instrument has no new field).

## Allowed to differ (must be DISCLOSED, per PROTOCOL.md section 7)

Expand All @@ -233,7 +233,7 @@ A pinned heap (`-Xms` equal to `-Xmx`) is a resource decision that moves the ker
- **Quality and precision differences** (int8 against fp32 postings, ES pruning). Report recall next to latency, always.
- **Product quantization arms (2026-11 instrument, `BENCH_DENSE_PQ=1`).** One PQ setting per engine family (32 subspaces of 256 centroids on 128-d), matched as closely as each engine allows and stamped on the row (`pq_subspaces`, `pq_clusters`, `pq_rerank`); the re-rank is each engine's own (ArcadeDB's default, Qdrant `oversampling`, Milvus `refine_k`, none for ArangoDB's FAISS factory), so recall sits next to latency and size in their own row family. An ArcadeDB PQ arm runs only on a build whose SQL accepts `rerank`; on any other it refuses (QUANTIZATION.md).
- **Intra-query parallelism at each engine's default.** ArcadeDB's SQL scans a type's buckets in parallel (`arcadedb.queryParallelScan`, on by default) only when the type has at least two buckets, and `arcadedb.typeDefaultBuckets` is 1. Every ArcadeDB type in this benchmark is created with the default except the document lane's `LineItem`, which gets `cores - 1` buckets to match its async load workers (CAMPAIGN.md section 7, row 10; the value is on the rows as `lineitem_buckets`). From 26.10.1 (PR #8524) one bucket is split into page ranges across the workers, so a scan outside a transaction runs in parallel at the engine default. Not tuned: the bucket count is a knob that moves only ArcadeDB, so the default stands until a campaign decides otherwise (CAMPAIGN.md section 7, item 10). A served ArcadeDB read through `POST /api/v1/command` runs inside an auto-commit transaction, where a scan never runs in parallel (ArcadeData/arcadedb#8775); the served reads go through `POST /api/v1/query` (CAMPAIGN.md section 7, row 25, BUGS F163).
- **Matched recall is the fair dense headline (2026-11 instrument).** At one beam setting two engines sit at different recalls, and the cheaper answer reads as the faster engine; the beam is also the one knob every engine has under a different name and a different scale. ANN-benchmarks practice is to fix the recall and compare latency, so the dense table's headline is the lowest p50 among the sweep passes that reach recall@10 0.95 (0.90 and 0.99 beside it), with the beam that reached it printed, and the full recall-latency curve kept in the rows. Nothing is tuned per engine except that beam, over the same list for every HNSW engine; an engine that cannot reach the target is marked, not shown at a lower recall. Engines with no beam setting run once and say so (PROTOCOL.md section 2).
- **Matched recall is the fair dense headline.** At one beam setting two engines sit at different recalls, and the cheaper answer reads as the faster engine; the beam is also the one knob every engine has under a different name and a different scale. ANN-benchmarks practice is to fix the recall and compare latency, so the dense table's headline is the lowest p50 among the sweep passes that reach recall@10 0.95 (0.90 and 0.99 beside it), with the beam that reached it printed, and the full recall-latency curve kept in the rows. Nothing is tuned per engine except that beam, over the same list for every HNSW engine; an engine that cannot reach the target is marked, not shown at a lower recall. Engines with no beam setting run once and say so (PROTOCOL.md section 2).

Anything else that differs is a defect, not an override.

Expand Down
4 changes: 2 additions & 2 deletions benchmarks/experiments/PAGE-SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Four things about that table are load-bearing and must be said on the page, not

**Frequency is not pinned.** The governor is `powersave` and turbo is ENABLED (`intel_pstate/no_turbo` = 0), so a long build and a short query do not see the same clock. This is a mobile-class part in a small chassis, so sustained all-core work throttles in a way a server part would not. It is why every published number runs SERIAL: co-scheduling on this host does not merely add noise, it changes the clock the other cell sees.

**CPU idle states are not controlled.** The host runs with the `intel_idle` driver and every sleep state enabled, down to C10 (exit latencies of 170 to 230 microseconds for C6, C8, and C10). A single-client time under a millisecond can therefore include the cores' wake-up latency, and the page says so in one plain sentence in its setup block (`setup.cpu_idle_note`, DECISIONS #177). The runner can hold the idle states off for the length of a timed cell (`BENCH_CPU_DMA_LATENCY_US`, default off, PROTOCOL.md section 7); a row that was run that way records `cpu_dma_latency_us` and `cpu_dma_latency_held`.
**CPU idle states are not controlled.** The host runs with the `intel_idle` driver and every sleep state enabled, down to C10 (exit latencies of 170 to 230 microseconds for C6, C8, and C10). A single-client time under a millisecond can therefore include the cores' wake-up latency, and the page says so in one plain sentence in its setup block (`setup.cpu_idle_note`, DECISIONS #177). The runner can hold the idle states off for the length of a timed cell (`bench_common.CPU_DMA_LATENCY_DEFAULT_US`, 0 microseconds, PROTOCOL.md section 7); every row records `cpu_dma_latency_us` and `cpu_dma_latency_held`.

Under a long build the package bounces off its thermal ceiling (BUGS.md F45). The power mode stays as the machine ships, because pinning the clock would lower every absolute number on the page, and every row instead records `host_temp_c_start` and `_end`, `host_throttle_count_start` and `_end`, and `host_throttled_ms` as the evidence.

Expand Down Expand Up @@ -289,7 +289,7 @@ A condition sentence on an October payload that is neither generated nor registe

Once, at the end, linked from every table: the global conditions; the environment table (`cpuset`, `mem_cap`, `server_mem_cap`, `mem_split`, `heap`, `server_heap`); the machine block from 0a, stating that the cpuset is 12 threads on 6 physical cores, that the governor is `powersave` with turbo enabled so frequency is not pinned, that the power mode is left as it ships; the package DOES throttle under sustained load and every row records its temperature and the kernel's throttle counters (BUGS.md F45), but that evidence is INTERNAL and is not published in any phrasing (user decision 2026-09-21, closing BUGS.md F71) -- it is recorded so a question about the host can be answered off the page, and the serial-execution rule above is what the page relies on instead, and that the bench disk is NVMe while the same machine holds a rotational disk used only for backups; the overrides table from `PROTOCOL.md` section 7 with a column saying **which way each override moves the number**; the durability note (DECISIONS #70, #81, #90); engine identity per section 1; the query set by table (two operations excluded at the full graph network, #109), and which of them come from a published benchmark (those of DECISIONS #82d and LSQB's nine, #104); repetition counts per table (generated, not typed); how equivalence is checked, that every timed query's answer is digested and compared across the engines of its table, and which operations an engine declared it cannot express (DECISIONS #88); ground truth (how recall@10 is computed and against which truth); outcome accounting; a dated changelog; and "Reproducing this", linking PROTOCOL, FAIRNESS, CAMPAIGN, READING-RESULTS, PUBLISHING, and the frozen artifacts.

Beside every peak-memory column with a JVM engine on it, not only the ones that compare ArcadeDB variants: `-Xms=-Xmx` commits the heap, so the column measures reservation, not demand (under `BENCH_INSTRUMENT=2026-11` the embedded ArcadeDB arms no longer pin `-Xms`, so their column measures demand up to `-Xmx`; the sentence then names only the arms that still pin). The sentence is generated (`export_web._jvm_memory_note`) on September's and October's tables alike and names every JVM arm on the table whose rows record a heap (ArcadeDB, Neo4j, and the composed stack's Neo4j half among them). This never clears; it is stated, not fixed.
Beside every peak-memory column with a JVM engine on it, not only the ones that compare ArcadeDB variants: `-Xms=-Xmx` commits the heap, so the column measures reservation, not demand (the embedded ArcadeDB arms do not pin `-Xms`, so their column measures demand up to `-Xmx`; the sentence names only the arms that still pin). The sentence is generated (`export_web._jvm_memory_note`) on September's and October's tables alike and names every JVM arm on the table whose rows record a heap (ArcadeDB, Neo4j, and the composed stack's Neo4j half among them). This never clears; it is stated, not fixed.

And a closing list of what the page does NOT measure: concurrency and load (it needs a non-Python load generator; our own harness is censored above ~4 clients by GIL queueing, and a bad concurrency table is worse than a disclosed absence); import, export, and backup; replication beyond the failover trial; strict durability for bulk ingest and reads (the timed writes run at both classes and the durability table prints them, DECISIONS #90); updates against a live vector index, where the 2026-10 dense table times an insert and a delete into a built index at the 1M tier and nothing above it (DECISIONS #82d); dense dimensionality above 128; anything across a real network; k other than 10.

Expand Down
Loading
Loading