Skip to content
This repository was archived by the owner on Nov 21, 2026. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion benchmarks/experiments/CAMPAIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ Beyond the lane's own metrics:

**One cap per tier, identical for every engine.** A cap is a ceiling rather than a reservation, so an engine that needs less is unaffected by a tier whose cap is generous. A per-backend cap would be the unfairness the caps exist to prevent.

**One heap per tier, identical for every JVM engine.** ArcadeDB embedded, ArcadeDB server, Neo4j and Elasticsearch all receive the same `{heap}` from one table (`-Xmx`, `NEO4J_server_memory_heap_max__size`, `ES_JAVA_OPTS` respectively). Non-JVM comparators have no equivalent knob, which is why heap sizing is a documented resource-fitting override rather than a per-engine advantage. Two JVM processes are exceptions and run at their own default heap: QuestDB, which `runner.JVM_BACKENDS` lists, so its rows record the tier heap although no heap is passed to its server, and mongot, the search process inside the MongoDB Search image, whose heap nothing sets (section 7, row 30).
**One heap per tier for ArcadeDB, and each vendor's own rule for Neo4j and Elasticsearch (DECISIONS #202).** ArcadeDB embedded and ArcadeDB server receive the same `{heap}` from one table (`-Xmx`). Neo4j takes the heap and page cache its own `neo4j-admin server memory-recommendation` prints for the cell's cap, and Elasticsearch a heap of at most half the cap and under 26 GiB (`vendor_memory.py`; at the half-cap tiers the Elasticsearch heap is unchanged). Non-JVM comparators have no equivalent knob, which is why heap sizing is a documented resource-fitting override rather than a per-engine advantage. Two JVM processes are exceptions and run at their own default heap: QuestDB, which `runner.JVM_BACKENDS` lists, so its rows record the tier heap although no heap is passed to its server, and mongot, the search process inside the MongoDB Search image, whose heap nothing sets (section 7, row 30).

**heap = 0.50 x cap, with three exceptions** (`runner.HEAP_BY_SCALE` against `MEM_BY_SCALE`): deep10m at 0.67 (24g in 36g), and the lifecycle tiers lc1m at 0.67 (16g in 24g) and lc10m at 0.86 (24g in 28g). The runner prints this table at startup and marks any deviation.

Expand Down
2 changes: 2 additions & 0 deletions benchmarks/experiments/FAIRNESS.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ A served backend gets the **full tier cap** and the client its own `BENCH_CLIENT

**"Same heap" means the heap the engine RAN, not the heap the cell asked for.** A row stamps `heap` from the request, so a hardcoded server heap is invisible in the artifact (Elasticsearch once ran 4g at three tiers while stamped `heap=16g`). `observe_server()` closes it by reading the container's real `-Xmx` back out of `docker inspect` into `server_heap` and failing any cell where the two disagree. Rows without that witness are dropped from published tables by `load_canonical`, so a tier shows a gap rather than an unfair number.

**F3d. Neo4j and Elasticsearch run their vendors' memory rules, so their heap is not the tier heap (DECISIONS #202).** F3 says every JVM engine gets the same `-Xmx`; the harness runs every engine as its vendor recommends, and for these two the vendor gives a rule that differs from half the cap. Neo4j takes the heap and page cache that `neo4j-admin server memory-recommendation --memory=<cap>` prints, run from the pinned image for each tier cap (`vendor_memory.NEO4J_MEMORY`, held equal to the committed tool output `neo4j_memory_recommendation.json` by `test_vendor_memory.py`), with the initial heap equal to the maximum. Elasticsearch takes the tier heap capped at 50% of the cap and at 26 GiB (Elastic: "no more than 50% of the total memory available to each Elasticsearch node", "26GB is safe on most systems"), `-Xms` = `-Xmx`; only deep10m moves, from 24g to 18g. The memory cap stays the same for every engine, and `fairness_check.check_envelope` judges the cap with everyone else and these two arms' heap against the vendor table. The heap each engine ran is still read back from the container (`server_heap`) and the running JVM.

The check is honest about its own limit: it reads the container's ENV, which is what we passed in, not the JVM's live heap. It proves the plumbing, not the obedience. The stronger form is to ask the engine (Elasticsearch reports `jvm.mem.heap_max_in_bytes` from `/_nodes/jvm`); it is owed, and CAMPAIGN.md section 7, item 20 carries it.

**A resource raised for one engine obliges a re-measure of every engine at that tier.** This is the sharpest rule here, and the one that has been broken: the dense envelope went 28g/16g to 36g/24g for a legitimate reason and only ArcadeDB was re-measured, which turned a fix into a 29% memory advantage.
Expand Down
8 changes: 4 additions & 4 deletions benchmarks/experiments/PROTOCOL.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ Every row carries these fields. The code is `memory_probe.py`, and `bench_common
## 3. How a comparison is kept fair

- Every container in a published cell gets the full 0-11 cpuset; a client/server pair shares that one cpuset rather than each getting its own cores, so CPU competition stays inside the deployment under test. **GATE** `fairness_check.check_cpuset` (F1) + the loader filter, for the client or embedded container only; **REMEMBERED** for the server side of a pair, whose `server_cpuset` no consumer reads
- Every backend at a (lane, scale) gets the same `--memory`/`--memory-swap` and, for JVM engines, the same heap. Raising a resource for one engine obliges a re-measure of every engine at that tier. **GATE** `fairness_check.check_envelope` (F3)
- Every backend at a (lane, scale) gets the same `--memory`/`--memory-swap` and, for JVM engines, the same heap (Neo4j and Elasticsearch excepted: each runs its vendor's memory rule, `vendor_memory.py`, DECISIONS #202). Raising a resource for one engine obliges a re-measure of every engine at that tier. **GATE** `fairness_check.check_envelope` (F3)
- Normalize a served topology before comparing: total is `srv_cap / mem_split`, never client + server. **GATE** `_total_envelope` in `fairness_check`
- Do not compare a setting to its own absence: heaps are compared only among engines that have one, and a missing heap on an ArcadeDB row is reported as a recording gap. **GATE** `check_envelope`
- Every dense backend at a scale gets the same effective base-layer degree, read in each engine's own unit (ArcadeDB `maxConnections` as-is, hnswlib-family `M` doubled at layer 0). A row recording no degree FAILS; sqlite-vec is exempt as an exact scan, and an index with no degree at all is matched by effect instead (F7). Stamp the matched parameter from one shared function, called from both the lane and its bespoke driver. **GATE** `fairness_check.check_degree` (F7) + `l3d_dense.degree_stamp(backend)`
Expand Down Expand Up @@ -223,7 +223,7 @@ An override nobody can find is a defect, whichever way it moves the number. Ever
| Elasticsearch (L3s) | `xpack.security.enabled=false` | escape hatch | Harness plumbing; also removes TLS and auth from every ES latency measurement. | row field `es_security_enabled` (from the engine's `_xpack`), and a generated sentence; `override: es_security` |
| Elasticsearch (L3s) | `number_of_replicas=0` | resource fitting | A single-node cluster cannot allocate a replica; the index would sit yellow forever. | row field `es_replicas` (from the index settings), and a generated sentence; `override: es_replicas` |
| composed Qdrant + Neo4j (E2) | the Qdrant server beside Neo4j in ONE container, `dbbench:composed` from `Dockerfile.composed` (the dense lane's pinned Qdrant v1.19.1 binary, Neo4j at the digest every Neo4j arm runs), reached over REST on 6333, with Qdrant's HNSW defaults (m=16, ef_construct=100) and a payload index on `pid` for the candidate filter (DECISIONS #117, BUGS F133) | resource fitting (one server cgroup for both halves) + operating-point (Qdrant's documented configuration for a filtered field) | Until 2026-09-26 the vector half was the client package's pure-Python local mode, rejected by the comparator review. Rows measured before carry `qdrant-local:` in `engine_version` and the page marks them `re-run`. | row field `engine_version` (`qdrant:<server>+neo4j:<server>`), COMPARATORS.md, page `re-run` note |
| Neo4j (L2, E2) | heap pinned per tier; `server_memory_pagecache_size` = container mem less heap less a 1 GiB reserve, floored at 512m | resource fitting | Heap parity, with the rest of the cap fitted for Neo4j. The composed E2 container takes the same formula; Qdrant lives in the 1 GiB reserve. | row fields `server_pagecache` (the container's env) and `neo4j_pagecache` (the engine's `SHOW SETTINGS`), checked against each other by `fairness_check` F15, and a generated sentence; `override: neo4j_pagecache` |
| Neo4j (L2, L3d, E2) | the heap and page cache that `neo4j-admin server memory-recommendation --memory=<cap>` prints from the pinned image, one row per tier cap in `vendor_memory.NEO4J_MEMORY` (8g: heap 3600m, page cache 2g; 16g: 5g and 7g; 32g: 12g and 12g; 48g: 18g and 20g; the other caps in the same table), initial heap = max heap; was heap at half the cap and page cache = container memory less heap less 1 GiB, until DECISIONS #202 | vendor recommendation | The Neo4j manual: "Use the neo4j-admin server memory-recommendation command to get an initial recommendation", and "It is recommended to set these two parameters to the same value" for the initial and maximum heap. The tool is run for an empty data directory. The composed E2 container takes the same numbers; Qdrant lives in what they leave. The tool also prints `server.jvm.additional=-XX:+ExitOnOutOfMemoryError`, which the harness does not set. | row fields `server_pagecache` (the container's env) and `neo4j_pagecache` (the engine's `SHOW SETTINGS`), checked against each other by `fairness_check` F15, and a generated sentence; `override: neo4j_pagecache` |
| Neo4j (graph) and the composed cross-model stack | `NEO4J_db_checkpoint_interval_time=5s` (default 15 minutes) | measurement (disk reading) | The store is checkpointed before the disk reading, which read 0.9 MB at SF1 under the default (BUGS F27). Preallocation stays at its default. | row fields `neo4j_checkpoint_interval` and `neo4j_checkpoint_interval_default` (the engine's `SHOW SETTINGS`), and a generated sentence; `override: neo4j_checkpoint` |
| Memgraph (graph) | `--bolt-num-workers` and `--storage-snapshot-thread-count` from the cpuset, `--memory-limit` at 90% of the server cap, `--query-execution-timeout-sec=0`, `--telemetry-enabled=false` | resource fitting (workers, threads, memory) + escape hatch (timeout) | The three pools default to the host, and the 600 s default timeout would abort a whole-graph query that the lane's own budget is there to censor. | row fields `memgraph_bolt_workers`, `memgraph_snapshot_threads`, `memgraph_memory_limit_mib`; FAIRNESS F6; COMPARATORS.md |
| Memgraph (dense, both precisions) | the graph arm's flags, and `--storage-access-timeout-sec=172800` (`runner.MEMGRAPH_STORAGE_ACCESS_TIMEOUT_S`) | escape hatch (timeout) | The image's default is 1 s; the deep10m vector index build was refused in both arms after about 1,540 s (humemai/arcadedb-embedded-python#344). Rows before #344 ran at the default. | row field `memgraph_storage_access_timeout_s`, read back from `SHOW CONFIG`; COMPARATORS.md |
Expand Down Expand Up @@ -261,8 +261,8 @@ An override nobody can find is a defect, whichever way it moves the number. Ever
| l4 `q_high` and `q_groupby` | ArcadeDB embedded `to_arrow()`, DuckDB `to_arrow_table()`, SQLite `fetchall()` (no columnar path); `to_list()` / `fetchall()` timed beside it as `<query>_plain_ms` | each engine on its documented best path for a result of thousands of rows | The bindings' docs recommend a bulk read for large results; small results (1 to 100 rows) are unchanged. | row fields `read_path`, `read_path_queries`, `read_path_plain`, `read_path_sensitivity`, `<query>_plain_ms`, `<query>_plain_digest`, `<query>_plain_digest_match` (the cell ends if the typed and plain digests differ; DuckDB's Arrow rows are read by position, like its `fetchall()`) |
| JVM initial heap (`-Xms`) per engine | **ArcadeDB embedded: unset** (only `-Xmx` from the cell's memory budget). ArcadeDB served: `-Xms` = `-Xmx` through `ARCADEDB_OPTS_MEMORY`, unchanged. Elasticsearch: `ES_JAVA_OPTS=-Xms{heap} -Xmx{heap}`, unchanged. Neo4j: `server.memory.heap.initial_size` = `max_size`, unchanged. QuestDB: the harness sets no heap (image defaults), unchanged | each server engine as its own documentation recommends; the embedded engine as the bindings document | ArcadeDB's server script (`package/src/main/scripts/server.sh`) leaves the heap to the JVM unless `ARCADEDB_OPTS_MEMORY` is set and shows `-Xms2G -Xmx2G` as its example; Elasticsearch: set `Xms` and `Xmx` to the same value (https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-heap-size); Neo4j: set initial and maximum heap to the same value (https://neo4j.com/docs/operations-manual/current/performance/memory-configuration/); QuestDB: no setting by the harness (https://questdb.com/docs/configuration/). | `server_heap` / the cell's heap fields, as before |
| ArcadeDB server (all arms) | `ARCADEDB_OPTS_MEMORY=-Xms{heap} -Xmx{heap}` (image ships 2G) | resource fitting | Heap parity per tier. The 2 GB default cost the server 15% p50 and 35% build time at 10M. | T5 caption, FAIRNESS.md F3, `server_heap` read back |
| Elasticsearch (L3s) | `ES_JAVA_OPTS=-Xms{heap} -Xmx{heap}` | resource fitting | Envelope equality across the lane's JVM engines. | FAIRNESS.md F3, enforced by `observe_server` |
| all served backends | server takes the full tier cap, client a separate `BENCH_CLIENT_MEM` (default 8g), stamped `mem_split="full+client"`; heap 50% of cap except deep10m and lc1m (67%) and lc10m (86%), per `runner.HEAP_BY_SCALE` against `MEM_BY_SCALE` | resource fitting | A served engine sees exactly the cap an embedded engine of the same tier sees, with the driver on top. `BENCH_SERVER_MEM_FRACTION` restores the old 0.75/0.25 split for a reproduction. | FAIRNESS.md F3 |
| Elasticsearch (L3s, L3d) | `ES_JAVA_OPTS=-Xms{heap} -Xmx{heap}`, `{heap}` = the tier heap brought under Elastic's two ceilings: at most 50% of the cap and at most 26 GiB (`vendor_memory.elasticsearch_heap`); only deep10m moves (18g in 36g, was 24g), every other tier it runs is already at half the cap | vendor recommendation | Elastic: "Set `Xms` and `Xmx` to no more than 50% of the total memory available to each Elasticsearch node." and "The exact threshold varies but 26GB is safe on most systems" (https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html). | FAIRNESS.md F3, enforced by `observe_server` and `fairness_check.check_envelope` (heap judged against `vendor_memory`) |
| all served backends | server takes the full tier cap, client a separate `BENCH_CLIENT_MEM` (default 8g), stamped `mem_split="full+client"`; heap 50% of cap except deep10m and lc1m (67%) and lc10m (86%), per `runner.HEAP_BY_SCALE` against `MEM_BY_SCALE`; Neo4j and Elasticsearch take their own heap (the two rows above) | resource fitting | A served engine sees exactly the cap an embedded engine of the same tier sees, with the driver on top. `BENCH_SERVER_MEM_FRACTION` restores the old 0.75/0.25 split for a reproduction. | FAIRNESS.md F3 |
| ES, Milvus, Qdrant, ArcadeDB, Neo4j, LadybugDB | forcemerge / flush+load / green-wait / COMPACT INDEX / awaitIndexes / CHECKPOINT, inside build | settle step | Each engine's own documented preparation. ArcadeDB takes none on L2 OLTP; in the dense lane only Qdrant and Milvus define one (the SurrealDB server's wait, next row, is ours). | disclosed as a class: FAIRNESS.md, page conditions |
| SurrealDB server (L3d, E2) | the build waits, inside its index timer, until a KNN probe's latency stops falling (`surreal_common.await_hnsw_settled`), recorded on the row as `settle_s` with `settle_probes`, `settle_first_ms`, and `settle_last_ms` | settle step, **harness-defined** | The 3.2.4 server fills an HNSW index defined before the load in the background and reports it ready, so queries were timed inside the build; defining it after the load fails with a transaction conflict at 50k. The vendor documents no settle step, so this probe is ours (BUGS F134, DECISIONS #117). | row field `settle_s`, this table |
| Milvus dense (L3d) | `dataCoord.segment.sealProportion=0.5` (image default 0.12) via a mounted `milvus.yaml` that differs from the image's in that one line; then flush, wait for the HNSW index, a compaction pass that is a no-op by then, load, all inside the timed build | resource fitting + settle step | The image seals growing segments at 12% of 1024 MB, so a 10M ingest lands as 26-28 segments that Milvus's own compaction merges at an unpredictable moment (BUGS.md F8); sealing at 50% yields the merged layout directly. Fp32 and INT8; the sparse lane stays at the image default. | here and in the page condition; `milvus_segments_trace` on every row |
Expand Down
3 changes: 3 additions & 0 deletions benchmarks/experiments/export_web.py
Original file line number Diff line number Diff line change
Expand Up @@ -4366,6 +4366,9 @@ def _entry_heaps(e, lane, wl, exact_only=False):
column (DECISIONS #116 item 7), so the two cannot name different heaps."""
rs = _entry_rows(e, lane, wl, exact_only)
heaps = {str(r.get("heap") or r.get("server_heap") or _jvm_reported_heap(r) or "").strip() for r in rs}
# A heap in MiB (Neo4j's own tool recommends 3600m, DECISIONS #202) is read as the GiB it is.
heaps = {(f"{float(h[:-1]) / 1024:.2f}".rstrip("0").rstrip(".") + "g")
if h and h[-1].lower() == "m" and h[:-1].isdigit() else h for h in heaps}
heaps = {h for h in heaps if h and h[-1].lower() == "g" and h[:-1].replace(".", "").isdigit()}
return heaps

Expand Down
Loading
Loading