From 021043078778fedf594014021ebb4ffed9c90d59 Mon Sep 17 00:00:00 2001 From: Taewoon Kim Date: Thu, 22 Oct 2026 13:20:04 +0900 Subject: [PATCH] Benchmarks: Neo4j and Elasticsearch memory settings follow their vendors' guidance (#460) Neo4j takes the heap and page cache of neo4j-admin server memory-recommendation for each tier cap; Elasticsearch deep10m heap goes from 24g to 18g (50% of the cap, under 26 GiB). DECISIONS #202. Co-Authored-By: Claude Opus 5.5 --- benchmarks/experiments/CAMPAIGN.md | 2 +- benchmarks/experiments/FAIRNESS.md | 2 + benchmarks/experiments/PROTOCOL.md | 8 +- benchmarks/experiments/export_web.py | 3 + benchmarks/experiments/fairness_check.py | 26 ++++ .../neo4j_memory_recommendation.json | 54 ++++++++ benchmarks/experiments/runner.py | 36 +++++- benchmarks/experiments/test_vendor_memory.py | 118 ++++++++++++++++++ benchmarks/experiments/vendor_memory.py | 112 +++++++++++++++++ bindings/python/docs/benchmarks/protocol.md | 5 + 10 files changed, 357 insertions(+), 9 deletions(-) create mode 100644 benchmarks/experiments/neo4j_memory_recommendation.json create mode 100644 benchmarks/experiments/test_vendor_memory.py create mode 100644 benchmarks/experiments/vendor_memory.py diff --git a/benchmarks/experiments/CAMPAIGN.md b/benchmarks/experiments/CAMPAIGN.md index 62177d0affe..12ffa66ae44 100644 --- a/benchmarks/experiments/CAMPAIGN.md +++ b/benchmarks/experiments/CAMPAIGN.md @@ -69,7 +69,7 @@ Beyond the lane's own metrics: **One cap per tier, identical for every engine.** A cap is a ceiling rather than a reservation, so an engine that needs less is unaffected by a tier whose cap is generous. A per-backend cap would be the unfairness the caps exist to prevent. -**One heap per tier, identical for every JVM engine.** ArcadeDB embedded, ArcadeDB server, Neo4j and Elasticsearch all receive the same `{heap}` from one table (`-Xmx`, `NEO4J_server_memory_heap_max__size`, `ES_JAVA_OPTS` respectively). Non-JVM comparators have no equivalent knob, which is why heap sizing is a documented resource-fitting override rather than a per-engine advantage. Two JVM processes are exceptions and run at their own default heap: QuestDB, which `runner.JVM_BACKENDS` lists, so its rows record the tier heap although no heap is passed to its server, and mongot, the search process inside the MongoDB Search image, whose heap nothing sets (section 7, row 30). +**One heap per tier for ArcadeDB, and each vendor's own rule for Neo4j and Elasticsearch (DECISIONS #202).** ArcadeDB embedded and ArcadeDB server receive the same `{heap}` from one table (`-Xmx`). Neo4j takes the heap and page cache its own `neo4j-admin server memory-recommendation` prints for the cell's cap, and Elasticsearch a heap of at most half the cap and under 26 GiB (`vendor_memory.py`; at the half-cap tiers the Elasticsearch heap is unchanged). Non-JVM comparators have no equivalent knob, which is why heap sizing is a documented resource-fitting override rather than a per-engine advantage. Two JVM processes are exceptions and run at their own default heap: QuestDB, which `runner.JVM_BACKENDS` lists, so its rows record the tier heap although no heap is passed to its server, and mongot, the search process inside the MongoDB Search image, whose heap nothing sets (section 7, row 30). **heap = 0.50 x cap, with three exceptions** (`runner.HEAP_BY_SCALE` against `MEM_BY_SCALE`): deep10m at 0.67 (24g in 36g), and the lifecycle tiers lc1m at 0.67 (16g in 24g) and lc10m at 0.86 (24g in 28g). The runner prints this table at startup and marks any deviation. diff --git a/benchmarks/experiments/FAIRNESS.md b/benchmarks/experiments/FAIRNESS.md index b8edce4eaeb..0ed4bf9b97a 100644 --- a/benchmarks/experiments/FAIRNESS.md +++ b/benchmarks/experiments/FAIRNESS.md @@ -20,6 +20,8 @@ A served backend gets the **full tier cap** and the client its own `BENCH_CLIENT **"Same heap" means the heap the engine RAN, not the heap the cell asked for.** A row stamps `heap` from the request, so a hardcoded server heap is invisible in the artifact (Elasticsearch once ran 4g at three tiers while stamped `heap=16g`). `observe_server()` closes it by reading the container's real `-Xmx` back out of `docker inspect` into `server_heap` and failing any cell where the two disagree. Rows without that witness are dropped from published tables by `load_canonical`, so a tier shows a gap rather than an unfair number. +**F3d. Neo4j and Elasticsearch run their vendors' memory rules, so their heap is not the tier heap (DECISIONS #202).** F3 says every JVM engine gets the same `-Xmx`; the harness runs every engine as its vendor recommends, and for these two the vendor gives a rule that differs from half the cap. Neo4j takes the heap and page cache that `neo4j-admin server memory-recommendation --memory=` prints, run from the pinned image for each tier cap (`vendor_memory.NEO4J_MEMORY`, held equal to the committed tool output `neo4j_memory_recommendation.json` by `test_vendor_memory.py`), with the initial heap equal to the maximum. Elasticsearch takes the tier heap capped at 50% of the cap and at 26 GiB (Elastic: "no more than 50% of the total memory available to each Elasticsearch node", "26GB is safe on most systems"), `-Xms` = `-Xmx`; only deep10m moves, from 24g to 18g. The memory cap stays the same for every engine, and `fairness_check.check_envelope` judges the cap with everyone else and these two arms' heap against the vendor table. The heap each engine ran is still read back from the container (`server_heap`) and the running JVM. + The check is honest about its own limit: it reads the container's ENV, which is what we passed in, not the JVM's live heap. It proves the plumbing, not the obedience. The stronger form is to ask the engine (Elasticsearch reports `jvm.mem.heap_max_in_bytes` from `/_nodes/jvm`); it is owed, and CAMPAIGN.md section 7, item 20 carries it. **A resource raised for one engine obliges a re-measure of every engine at that tier.** This is the sharpest rule here, and the one that has been broken: the dense envelope went 28g/16g to 36g/24g for a legitimate reason and only ArcadeDB was re-measured, which turned a fix into a 29% memory advantage. diff --git a/benchmarks/experiments/PROTOCOL.md b/benchmarks/experiments/PROTOCOL.md index 1c3342c69fd..aeb8ddd8c5f 100644 --- a/benchmarks/experiments/PROTOCOL.md +++ b/benchmarks/experiments/PROTOCOL.md @@ -99,7 +99,7 @@ Every row carries these fields. The code is `memory_probe.py`, and `bench_common ## 3. How a comparison is kept fair - Every container in a published cell gets the full 0-11 cpuset; a client/server pair shares that one cpuset rather than each getting its own cores, so CPU competition stays inside the deployment under test. **GATE** `fairness_check.check_cpuset` (F1) + the loader filter, for the client or embedded container only; **REMEMBERED** for the server side of a pair, whose `server_cpuset` no consumer reads -- Every backend at a (lane, scale) gets the same `--memory`/`--memory-swap` and, for JVM engines, the same heap. Raising a resource for one engine obliges a re-measure of every engine at that tier. **GATE** `fairness_check.check_envelope` (F3) +- Every backend at a (lane, scale) gets the same `--memory`/`--memory-swap` and, for JVM engines, the same heap (Neo4j and Elasticsearch excepted: each runs its vendor's memory rule, `vendor_memory.py`, DECISIONS #202). Raising a resource for one engine obliges a re-measure of every engine at that tier. **GATE** `fairness_check.check_envelope` (F3) - Normalize a served topology before comparing: total is `srv_cap / mem_split`, never client + server. **GATE** `_total_envelope` in `fairness_check` - Do not compare a setting to its own absence: heaps are compared only among engines that have one, and a missing heap on an ArcadeDB row is reported as a recording gap. **GATE** `check_envelope` - Every dense backend at a scale gets the same effective base-layer degree, read in each engine's own unit (ArcadeDB `maxConnections` as-is, hnswlib-family `M` doubled at layer 0). A row recording no degree FAILS; sqlite-vec is exempt as an exact scan, and an index with no degree at all is matched by effect instead (F7). Stamp the matched parameter from one shared function, called from both the lane and its bespoke driver. **GATE** `fairness_check.check_degree` (F7) + `l3d_dense.degree_stamp(backend)` @@ -223,7 +223,7 @@ An override nobody can find is a defect, whichever way it moves the number. Ever | Elasticsearch (L3s) | `xpack.security.enabled=false` | escape hatch | Harness plumbing; also removes TLS and auth from every ES latency measurement. | row field `es_security_enabled` (from the engine's `_xpack`), and a generated sentence; `override: es_security` | | Elasticsearch (L3s) | `number_of_replicas=0` | resource fitting | A single-node cluster cannot allocate a replica; the index would sit yellow forever. | row field `es_replicas` (from the index settings), and a generated sentence; `override: es_replicas` | | composed Qdrant + Neo4j (E2) | the Qdrant server beside Neo4j in ONE container, `dbbench:composed` from `Dockerfile.composed` (the dense lane's pinned Qdrant v1.19.1 binary, Neo4j at the digest every Neo4j arm runs), reached over REST on 6333, with Qdrant's HNSW defaults (m=16, ef_construct=100) and a payload index on `pid` for the candidate filter (DECISIONS #117, BUGS F133) | resource fitting (one server cgroup for both halves) + operating-point (Qdrant's documented configuration for a filtered field) | Until 2026-09-26 the vector half was the client package's pure-Python local mode, rejected by the comparator review. Rows measured before carry `qdrant-local:` in `engine_version` and the page marks them `re-run`. | row field `engine_version` (`qdrant:+neo4j:`), COMPARATORS.md, page `re-run` note | -| Neo4j (L2, E2) | heap pinned per tier; `server_memory_pagecache_size` = container mem less heap less a 1 GiB reserve, floored at 512m | resource fitting | Heap parity, with the rest of the cap fitted for Neo4j. The composed E2 container takes the same formula; Qdrant lives in the 1 GiB reserve. | row fields `server_pagecache` (the container's env) and `neo4j_pagecache` (the engine's `SHOW SETTINGS`), checked against each other by `fairness_check` F15, and a generated sentence; `override: neo4j_pagecache` | +| Neo4j (L2, L3d, E2) | the heap and page cache that `neo4j-admin server memory-recommendation --memory=` prints from the pinned image, one row per tier cap in `vendor_memory.NEO4J_MEMORY` (8g: heap 3600m, page cache 2g; 16g: 5g and 7g; 32g: 12g and 12g; 48g: 18g and 20g; the other caps in the same table), initial heap = max heap; was heap at half the cap and page cache = container memory less heap less 1 GiB, until DECISIONS #202 | vendor recommendation | The Neo4j manual: "Use the neo4j-admin server memory-recommendation command to get an initial recommendation", and "It is recommended to set these two parameters to the same value" for the initial and maximum heap. The tool is run for an empty data directory. The composed E2 container takes the same numbers; Qdrant lives in what they leave. The tool also prints `server.jvm.additional=-XX:+ExitOnOutOfMemoryError`, which the harness does not set. | row fields `server_pagecache` (the container's env) and `neo4j_pagecache` (the engine's `SHOW SETTINGS`), checked against each other by `fairness_check` F15, and a generated sentence; `override: neo4j_pagecache` | | Neo4j (graph) and the composed cross-model stack | `NEO4J_db_checkpoint_interval_time=5s` (default 15 minutes) | measurement (disk reading) | The store is checkpointed before the disk reading, which read 0.9 MB at SF1 under the default (BUGS F27). Preallocation stays at its default. | row fields `neo4j_checkpoint_interval` and `neo4j_checkpoint_interval_default` (the engine's `SHOW SETTINGS`), and a generated sentence; `override: neo4j_checkpoint` | | Memgraph (graph) | `--bolt-num-workers` and `--storage-snapshot-thread-count` from the cpuset, `--memory-limit` at 90% of the server cap, `--query-execution-timeout-sec=0`, `--telemetry-enabled=false` | resource fitting (workers, threads, memory) + escape hatch (timeout) | The three pools default to the host, and the 600 s default timeout would abort a whole-graph query that the lane's own budget is there to censor. | row fields `memgraph_bolt_workers`, `memgraph_snapshot_threads`, `memgraph_memory_limit_mib`; FAIRNESS F6; COMPARATORS.md | | Memgraph (dense, both precisions) | the graph arm's flags, and `--storage-access-timeout-sec=172800` (`runner.MEMGRAPH_STORAGE_ACCESS_TIMEOUT_S`) | escape hatch (timeout) | The image's default is 1 s; the deep10m vector index build was refused in both arms after about 1,540 s (humemai/arcadedb-embedded-python#344). Rows before #344 ran at the default. | row field `memgraph_storage_access_timeout_s`, read back from `SHOW CONFIG`; COMPARATORS.md | @@ -261,8 +261,8 @@ An override nobody can find is a defect, whichever way it moves the number. Ever | l4 `q_high` and `q_groupby` | ArcadeDB embedded `to_arrow()`, DuckDB `to_arrow_table()`, SQLite `fetchall()` (no columnar path); `to_list()` / `fetchall()` timed beside it as `_plain_ms` | each engine on its documented best path for a result of thousands of rows | The bindings' docs recommend a bulk read for large results; small results (1 to 100 rows) are unchanged. | row fields `read_path`, `read_path_queries`, `read_path_plain`, `read_path_sensitivity`, `_plain_ms`, `_plain_digest`, `_plain_digest_match` (the cell ends if the typed and plain digests differ; DuckDB's Arrow rows are read by position, like its `fetchall()`) | | JVM initial heap (`-Xms`) per engine | **ArcadeDB embedded: unset** (only `-Xmx` from the cell's memory budget). ArcadeDB served: `-Xms` = `-Xmx` through `ARCADEDB_OPTS_MEMORY`, unchanged. Elasticsearch: `ES_JAVA_OPTS=-Xms{heap} -Xmx{heap}`, unchanged. Neo4j: `server.memory.heap.initial_size` = `max_size`, unchanged. QuestDB: the harness sets no heap (image defaults), unchanged | each server engine as its own documentation recommends; the embedded engine as the bindings document | ArcadeDB's server script (`package/src/main/scripts/server.sh`) leaves the heap to the JVM unless `ARCADEDB_OPTS_MEMORY` is set and shows `-Xms2G -Xmx2G` as its example; Elasticsearch: set `Xms` and `Xmx` to the same value (https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html#set-jvm-heap-size); Neo4j: set initial and maximum heap to the same value (https://neo4j.com/docs/operations-manual/current/performance/memory-configuration/); QuestDB: no setting by the harness (https://questdb.com/docs/configuration/). | `server_heap` / the cell's heap fields, as before | | ArcadeDB server (all arms) | `ARCADEDB_OPTS_MEMORY=-Xms{heap} -Xmx{heap}` (image ships 2G) | resource fitting | Heap parity per tier. The 2 GB default cost the server 15% p50 and 35% build time at 10M. | T5 caption, FAIRNESS.md F3, `server_heap` read back | -| Elasticsearch (L3s) | `ES_JAVA_OPTS=-Xms{heap} -Xmx{heap}` | resource fitting | Envelope equality across the lane's JVM engines. | FAIRNESS.md F3, enforced by `observe_server` | -| all served backends | server takes the full tier cap, client a separate `BENCH_CLIENT_MEM` (default 8g), stamped `mem_split="full+client"`; heap 50% of cap except deep10m and lc1m (67%) and lc10m (86%), per `runner.HEAP_BY_SCALE` against `MEM_BY_SCALE` | resource fitting | A served engine sees exactly the cap an embedded engine of the same tier sees, with the driver on top. `BENCH_SERVER_MEM_FRACTION` restores the old 0.75/0.25 split for a reproduction. | FAIRNESS.md F3 | +| Elasticsearch (L3s, L3d) | `ES_JAVA_OPTS=-Xms{heap} -Xmx{heap}`, `{heap}` = the tier heap brought under Elastic's two ceilings: at most 50% of the cap and at most 26 GiB (`vendor_memory.elasticsearch_heap`); only deep10m moves (18g in 36g, was 24g), every other tier it runs is already at half the cap | vendor recommendation | Elastic: "Set `Xms` and `Xmx` to no more than 50% of the total memory available to each Elasticsearch node." and "The exact threshold varies but 26GB is safe on most systems" (https://www.elastic.co/guide/en/elasticsearch/reference/current/advanced-configuration.html). | FAIRNESS.md F3, enforced by `observe_server` and `fairness_check.check_envelope` (heap judged against `vendor_memory`) | +| all served backends | server takes the full tier cap, client a separate `BENCH_CLIENT_MEM` (default 8g), stamped `mem_split="full+client"`; heap 50% of cap except deep10m and lc1m (67%) and lc10m (86%), per `runner.HEAP_BY_SCALE` against `MEM_BY_SCALE`; Neo4j and Elasticsearch take their own heap (the two rows above) | resource fitting | A served engine sees exactly the cap an embedded engine of the same tier sees, with the driver on top. `BENCH_SERVER_MEM_FRACTION` restores the old 0.75/0.25 split for a reproduction. | FAIRNESS.md F3 | | ES, Milvus, Qdrant, ArcadeDB, Neo4j, LadybugDB | forcemerge / flush+load / green-wait / COMPACT INDEX / awaitIndexes / CHECKPOINT, inside build | settle step | Each engine's own documented preparation. ArcadeDB takes none on L2 OLTP; in the dense lane only Qdrant and Milvus define one (the SurrealDB server's wait, next row, is ours). | disclosed as a class: FAIRNESS.md, page conditions | | SurrealDB server (L3d, E2) | the build waits, inside its index timer, until a KNN probe's latency stops falling (`surreal_common.await_hnsw_settled`), recorded on the row as `settle_s` with `settle_probes`, `settle_first_ms`, and `settle_last_ms` | settle step, **harness-defined** | The 3.2.4 server fills an HNSW index defined before the load in the background and reports it ready, so queries were timed inside the build; defining it after the load fails with a transaction conflict at 50k. The vendor documents no settle step, so this probe is ours (BUGS F134, DECISIONS #117). | row field `settle_s`, this table | | Milvus dense (L3d) | `dataCoord.segment.sealProportion=0.5` (image default 0.12) via a mounted `milvus.yaml` that differs from the image's in that one line; then flush, wait for the HNSW index, a compaction pass that is a no-op by then, load, all inside the timed build | resource fitting + settle step | The image seals growing segments at 12% of 1024 MB, so a 10M ingest lands as 26-28 segments that Milvus's own compaction merges at an unpredictable moment (BUGS.md F8); sealing at 50% yields the merged layout directly. Fp32 and INT8; the sparse lane stays at the image default. | here and in the page condition; `milvus_segments_trace` on every row | diff --git a/benchmarks/experiments/export_web.py b/benchmarks/experiments/export_web.py index 6969d026b12..5645ccd44dd 100644 --- a/benchmarks/experiments/export_web.py +++ b/benchmarks/experiments/export_web.py @@ -4366,6 +4366,9 @@ def _entry_heaps(e, lane, wl, exact_only=False): column (DECISIONS #116 item 7), so the two cannot name different heaps.""" rs = _entry_rows(e, lane, wl, exact_only) heaps = {str(r.get("heap") or r.get("server_heap") or _jvm_reported_heap(r) or "").strip() for r in rs} + # A heap in MiB (Neo4j's own tool recommends 3600m, DECISIONS #202) is read as the GiB it is. + heaps = {(f"{float(h[:-1]) / 1024:.2f}".rstrip("0").rstrip(".") + "g") + if h and h[-1].lower() == "m" and h[:-1].isdigit() else h for h in heaps} heaps = {h for h in heaps if h and h[-1].lower() == "g" and h[:-1].replace(".", "").isdigit()} return heaps diff --git a/benchmarks/experiments/fairness_check.py b/benchmarks/experiments/fairness_check.py index 059d1dd4c25..c303aa6bc1a 100644 --- a/benchmarks/experiments/fairness_check.py +++ b/benchmarks/experiments/fairness_check.py @@ -36,6 +36,7 @@ HERE = os.path.dirname(os.path.abspath(__file__)) sys.path.insert(0, HERE) import publish_instrument # noqa: E402 +import vendor_memory # noqa: E402 RESULTS = str(publish_instrument.results_dir()) FULL_CPUSET = "0-11" # Fallback only. Every served row records its own mem_split; this is for the @@ -431,6 +432,17 @@ def _total_envelope(r): return (str(r.get("heap")), f"{cap:g}g") +def _vendor_heap(r): + """The heap the vendor rule gives this Neo4j or Elasticsearch row's tier (vendor_memory.heap_for), or None when the + row's cap is not one the table covers.""" + import runner # noqa: E402 + cap = str(r.get("mem_cap")) + scale = r.get("scale") + if cap not in vendor_memory.NEO4J_MEMORY or scale not in runner.HEAP_BY_SCALE: + return None + return vendor_memory.heap_for(r.get("backend"), runner.HEAP_BY_SCALE[scale], cap) + + def check_envelope(rows): """F3: same memory/heap envelope across backends within a (lane, scale).""" print("\n=== F3: same memory envelope per (lane, scale) ===") @@ -448,6 +460,7 @@ def check_envelope(rows): rows = rows + _dense_rows() g = collections.defaultdict(lambda: collections.defaultdict(set)) noheap = collections.defaultdict(set) + vendor_bad = [] for r in rows: h, m = _total_envelope(r) # THE DECLARED IMAGE-DEFAULTS ARM (JVM_DEFAULTS_ARMS): the memory CAP is judged @@ -457,6 +470,16 @@ def check_envelope(rows): if str(r.get("backend")) in JVM_DEFAULTS_ARMS: g[(r["lane"], r["scale"])][r["backend"]].add(("n/a", m)) continue + # NEO4J AND ELASTICSEARCH RUN THEIR VENDORS' MEMORY RULE (DECISIONS #202, vendor_memory.py): their heap is not + # ArcadeDB's, so it is judged against the vendor table below instead of against the other engines' heap. The cap + # is judged with everyone else's. A row measured before the rule (no `vendor_memory` stamp) is judged as it always was. + # A row that carries no heap witness is left to the NO-WITNESS rule above. + if r.get("vendor_memory"): + want = _vendor_heap(r) + if h not in ("NO-WITNESS", "None", "", "n/a", None) and want is not None and _gib(h) != _gib(want): + vendor_bad.append(f"{r['lane']} {r['scale']} {r['backend']}: heap {h}, the vendor rule gives {want}") + g[(r["lane"], r["scale"])][r["backend"]].add(("n/a", m)) + continue # A JVM heap is only comparable between engines that HAVE one. Qdrant, # Milvus, LanceDB and sqlite-vec are Rust/Go/C and report heap=None; # calling that a mismatched envelope against ArcadeDB's 24g compares a @@ -480,6 +503,9 @@ def check_envelope(rows): h = "n/a" g[(r["lane"], r["scale"])][r["backend"]].add((h, m)) bad = 0 + for line in sorted(set(vendor_bad)): + bad += 1 + print(f" FAIL {line}") for key in sorted(g, key=str): per_be = g[key] envs = {e for s in per_be.values() for e in s} diff --git a/benchmarks/experiments/neo4j_memory_recommendation.json b/benchmarks/experiments/neo4j_memory_recommendation.json new file mode 100644 index 00000000000..6435a0d22e0 --- /dev/null +++ b/benchmarks/experiments/neo4j_memory_recommendation.json @@ -0,0 +1,54 @@ +{ + "image": "neo4j@sha256:e702d6b535d9d3ae01ee7b132ec87aa40e23d3f0ace82fbfc344e2048cb81960", + "command": "docker run --rm -m 2g --entrypoint neo4j-admin neo4j@sha256:e702d6b535d9d3ae01ee7b132ec87aa40e23d3f0ace82fbfc344e2048cb81960 server memory-recommendation --memory={cap} --docker", + "caps": { + "8g": { + "heap_initial": "3600m", + "heap_max": "3600m", + "pagecache": "2g", + "output": "# Memory settings recommendation:\n#\n# Assuming the system is dedicated to running Neo4j and has 8.000GiB of memory,\n# we recommend a heap size of around 3600m, and a page cache of around 2g,\n# and that about 2600m is left for the operating system, and the native memory\n# needed by Lucene and Netty.\n#\n# Tip: If the indexing storage use is high, e.g. there are many indexes or most\n# data indexed, then it might advantageous to leave more memory for the\n# operating system.\n#\n# Tip: The more concurrent transactions your workload has and the more updates\n# they do, the more heap memory you will need. However, don't allocate more\n# than 31g of heap, since this will disable pointer compression, also known as\n# \"compressed oops\", in the JVM and make less effective use of the heap.\n#\n# Tip: Setting the initial and the max heap size to the same value means the\n# JVM will never need to change the heap size. Changing the heap size otherwise\n# involves a full GC, which is desirable to avoid.\n#\n# Based on the above, the following memory settings are recommended:\nNEO4J_server_memory_heap_initial__size='3600m'\nNEO4J_server_memory_heap_max__size='3600m'\nNEO4J_server_memory_pagecache_size='2g'\n#\n# It is also recommended turning out-of-memory errors into full crashes,\n# instead of allowing a partially crashed database to continue running:\nNEO4J_server_jvm_additional='-XX:+ExitOnOutOfMemoryError'\n#\n# The numbers below have been derived based on your current databases located at: '/var/lib/neo4j/data/databases'.\n# They can be used as an input into more detailed memory analysis.\n# Total size of lucene indexes in all databases: 0k\n# Total size of data and native indexes in all databases: 0k\n" + }, + "12g": { + "heap_initial": "4600m", + "heap_max": "4600m", + "pagecache": "4g", + "output": "# Memory settings recommendation:\n#\n# Assuming the system is dedicated to running Neo4j and has 12.00GiB of memory,\n# we recommend a heap size of around 4600m, and a page cache of around 4g,\n# and that about 3600m is left for the operating system, and the native memory\n# needed by Lucene and Netty.\n#\n# Tip: If the indexing storage use is high, e.g. there are many indexes or most\n# data indexed, then it might advantageous to leave more memory for the\n# operating system.\n#\n# Tip: The more concurrent transactions your workload has and the more updates\n# they do, the more heap memory you will need. However, don't allocate more\n# than 31g of heap, since this will disable pointer compression, also known as\n# \"compressed oops\", in the JVM and make less effective use of the heap.\n#\n# Tip: Setting the initial and the max heap size to the same value means the\n# JVM will never need to change the heap size. Changing the heap size otherwise\n# involves a full GC, which is desirable to avoid.\n#\n# Based on the above, the following memory settings are recommended:\nNEO4J_server_memory_heap_initial__size='4600m'\nNEO4J_server_memory_heap_max__size='4600m'\nNEO4J_server_memory_pagecache_size='4g'\n#\n# It is also recommended turning out-of-memory errors into full crashes,\n# instead of allowing a partially crashed database to continue running:\nNEO4J_server_jvm_additional='-XX:+ExitOnOutOfMemoryError'\n#\n# The numbers below have been derived based on your current databases located at: '/var/lib/neo4j/data/databases'.\n# They can be used as an input into more detailed memory analysis.\n# Total size of lucene indexes in all databases: 0k\n# Total size of data and native indexes in all databases: 0k\n" + }, + "16g": { + "heap_initial": "5g", + "heap_max": "5g", + "pagecache": "7g", + "output": "# Memory settings recommendation:\n#\n# Assuming the system is dedicated to running Neo4j and has 16.00GiB of memory,\n# we recommend a heap size of around 5g, and a page cache of around 7g,\n# and that about 4g is left for the operating system, and the native memory\n# needed by Lucene and Netty.\n#\n# Tip: If the indexing storage use is high, e.g. there are many indexes or most\n# data indexed, then it might advantageous to leave more memory for the\n# operating system.\n#\n# Tip: The more concurrent transactions your workload has and the more updates\n# they do, the more heap memory you will need. However, don't allocate more\n# than 31g of heap, since this will disable pointer compression, also known as\n# \"compressed oops\", in the JVM and make less effective use of the heap.\n#\n# Tip: Setting the initial and the max heap size to the same value means the\n# JVM will never need to change the heap size. Changing the heap size otherwise\n# involves a full GC, which is desirable to avoid.\n#\n# Based on the above, the following memory settings are recommended:\nNEO4J_server_memory_heap_initial__size='5g'\nNEO4J_server_memory_heap_max__size='5g'\nNEO4J_server_memory_pagecache_size='7g'\n#\n# It is also recommended turning out-of-memory errors into full crashes,\n# instead of allowing a partially crashed database to continue running:\nNEO4J_server_jvm_additional='-XX:+ExitOnOutOfMemoryError'\n#\n# The numbers below have been derived based on your current databases located at: '/var/lib/neo4j/data/databases'.\n# They can be used as an input into more detailed memory analysis.\n# Total size of lucene indexes in all databases: 0k\n# Total size of data and native indexes in all databases: 0k\n" + }, + "24g": { + "heap_initial": "8g", + "heap_max": "8g", + "pagecache": "10g", + "output": "# Memory settings recommendation:\n#\n# Assuming the system is dedicated to running Neo4j and has 24.00GiB of memory,\n# we recommend a heap size of around 8g, and a page cache of around 10g,\n# and that about 6g is left for the operating system, and the native memory\n# needed by Lucene and Netty.\n#\n# Tip: If the indexing storage use is high, e.g. there are many indexes or most\n# data indexed, then it might advantageous to leave more memory for the\n# operating system.\n#\n# Tip: The more concurrent transactions your workload has and the more updates\n# they do, the more heap memory you will need. However, don't allocate more\n# than 31g of heap, since this will disable pointer compression, also known as\n# \"compressed oops\", in the JVM and make less effective use of the heap.\n#\n# Tip: Setting the initial and the max heap size to the same value means the\n# JVM will never need to change the heap size. Changing the heap size otherwise\n# involves a full GC, which is desirable to avoid.\n#\n# Based on the above, the following memory settings are recommended:\nNEO4J_server_memory_heap_initial__size='8g'\nNEO4J_server_memory_heap_max__size='8g'\nNEO4J_server_memory_pagecache_size='10g'\n#\n# It is also recommended turning out-of-memory errors into full crashes,\n# instead of allowing a partially crashed database to continue running:\nNEO4J_server_jvm_additional='-XX:+ExitOnOutOfMemoryError'\n#\n# The numbers below have been derived based on your current databases located at: '/var/lib/neo4j/data/databases'.\n# They can be used as an input into more detailed memory analysis.\n# Total size of lucene indexes in all databases: 0k\n# Total size of data and native indexes in all databases: 0k\n" + }, + "28g": { + "heap_initial": "10g", + "heap_max": "10g", + "pagecache": "11g", + "output": "# Memory settings recommendation:\n#\n# Assuming the system is dedicated to running Neo4j and has 28.00GiB of memory,\n# we recommend a heap size of around 10g, and a page cache of around 11g,\n# and that about 7g is left for the operating system, and the native memory\n# needed by Lucene and Netty.\n#\n# Tip: If the indexing storage use is high, e.g. there are many indexes or most\n# data indexed, then it might advantageous to leave more memory for the\n# operating system.\n#\n# Tip: The more concurrent transactions your workload has and the more updates\n# they do, the more heap memory you will need. However, don't allocate more\n# than 31g of heap, since this will disable pointer compression, also known as\n# \"compressed oops\", in the JVM and make less effective use of the heap.\n#\n# Tip: Setting the initial and the max heap size to the same value means the\n# JVM will never need to change the heap size. Changing the heap size otherwise\n# involves a full GC, which is desirable to avoid.\n#\n# Based on the above, the following memory settings are recommended:\nNEO4J_server_memory_heap_initial__size='10g'\nNEO4J_server_memory_heap_max__size='10g'\nNEO4J_server_memory_pagecache_size='11g'\n#\n# It is also recommended turning out-of-memory errors into full crashes,\n# instead of allowing a partially crashed database to continue running:\nNEO4J_server_jvm_additional='-XX:+ExitOnOutOfMemoryError'\n#\n# The numbers below have been derived based on your current databases located at: '/var/lib/neo4j/data/databases'.\n# They can be used as an input into more detailed memory analysis.\n# Total size of lucene indexes in all databases: 0k\n# Total size of data and native indexes in all databases: 0k\n" + }, + "32g": { + "heap_initial": "12g", + "heap_max": "12g", + "pagecache": "12g", + "output": "# Memory settings recommendation:\n#\n# Assuming the system is dedicated to running Neo4j and has 32.00GiB of memory,\n# we recommend a heap size of around 12g, and a page cache of around 12g,\n# and that about 8g is left for the operating system, and the native memory\n# needed by Lucene and Netty.\n#\n# Tip: If the indexing storage use is high, e.g. there are many indexes or most\n# data indexed, then it might advantageous to leave more memory for the\n# operating system.\n#\n# Tip: The more concurrent transactions your workload has and the more updates\n# they do, the more heap memory you will need. However, don't allocate more\n# than 31g of heap, since this will disable pointer compression, also known as\n# \"compressed oops\", in the JVM and make less effective use of the heap.\n#\n# Tip: Setting the initial and the max heap size to the same value means the\n# JVM will never need to change the heap size. Changing the heap size otherwise\n# involves a full GC, which is desirable to avoid.\n#\n# Based on the above, the following memory settings are recommended:\nNEO4J_server_memory_heap_initial__size='12g'\nNEO4J_server_memory_heap_max__size='12g'\nNEO4J_server_memory_pagecache_size='12g'\n#\n# It is also recommended turning out-of-memory errors into full crashes,\n# instead of allowing a partially crashed database to continue running:\nNEO4J_server_jvm_additional='-XX:+ExitOnOutOfMemoryError'\n#\n# The numbers below have been derived based on your current databases located at: '/var/lib/neo4j/data/databases'.\n# They can be used as an input into more detailed memory analysis.\n# Total size of lucene indexes in all databases: 0k\n# Total size of data and native indexes in all databases: 0k\n" + }, + "36g": { + "heap_initial": "13800m", + "heap_max": "13800m", + "pagecache": "14g", + "output": "# Memory settings recommendation:\n#\n# Assuming the system is dedicated to running Neo4j and has 36.00GiB of memory,\n# we recommend a heap size of around 13800m, and a page cache of around 14g,\n# and that about 8700m is left for the operating system, and the native memory\n# needed by Lucene and Netty.\n#\n# Tip: If the indexing storage use is high, e.g. there are many indexes or most\n# data indexed, then it might advantageous to leave more memory for the\n# operating system.\n#\n# Tip: The more concurrent transactions your workload has and the more updates\n# they do, the more heap memory you will need. However, don't allocate more\n# than 31g of heap, since this will disable pointer compression, also known as\n# \"compressed oops\", in the JVM and make less effective use of the heap.\n#\n# Tip: Setting the initial and the max heap size to the same value means the\n# JVM will never need to change the heap size. Changing the heap size otherwise\n# involves a full GC, which is desirable to avoid.\n#\n# Based on the above, the following memory settings are recommended:\nNEO4J_server_memory_heap_initial__size='13800m'\nNEO4J_server_memory_heap_max__size='13800m'\nNEO4J_server_memory_pagecache_size='14g'\n#\n# It is also recommended turning out-of-memory errors into full crashes,\n# instead of allowing a partially crashed database to continue running:\nNEO4J_server_jvm_additional='-XX:+ExitOnOutOfMemoryError'\n#\n# The numbers below have been derived based on your current databases located at: '/var/lib/neo4j/data/databases'.\n# They can be used as an input into more detailed memory analysis.\n# Total size of lucene indexes in all databases: 0k\n# Total size of data and native indexes in all databases: 0k\n" + }, + "48g": { + "heap_initial": "18g", + "heap_max": "18g", + "pagecache": "20g", + "output": "# Memory settings recommendation:\n#\n# Assuming the system is dedicated to running Neo4j and has 48.00GiB of memory,\n# we recommend a heap size of around 18g, and a page cache of around 20g,\n# and that about 10g is left for the operating system, and the native memory\n# needed by Lucene and Netty.\n#\n# Tip: If the indexing storage use is high, e.g. there are many indexes or most\n# data indexed, then it might advantageous to leave more memory for the\n# operating system.\n#\n# Tip: The more concurrent transactions your workload has and the more updates\n# they do, the more heap memory you will need. However, don't allocate more\n# than 31g of heap, since this will disable pointer compression, also known as\n# \"compressed oops\", in the JVM and make less effective use of the heap.\n#\n# Tip: Setting the initial and the max heap size to the same value means the\n# JVM will never need to change the heap size. Changing the heap size otherwise\n# involves a full GC, which is desirable to avoid.\n#\n# Based on the above, the following memory settings are recommended:\nNEO4J_server_memory_heap_initial__size='18g'\nNEO4J_server_memory_heap_max__size='18g'\nNEO4J_server_memory_pagecache_size='20g'\n#\n# It is also recommended turning out-of-memory errors into full crashes,\n# instead of allowing a partially crashed database to continue running:\nNEO4J_server_jvm_additional='-XX:+ExitOnOutOfMemoryError'\n#\n# The numbers below have been derived based on your current databases located at: '/var/lib/neo4j/data/databases'.\n# They can be used as an input into more detailed memory analysis.\n# Total size of lucene indexes in all databases: 0k\n# Total size of data and native indexes in all databases: 0k\n" + } + } +} diff --git a/benchmarks/experiments/runner.py b/benchmarks/experiments/runner.py index 50d94119bf6..11a26f992e5 100644 --- a/benchmarks/experiments/runner.py +++ b/benchmarks/experiments/runner.py @@ -45,6 +45,7 @@ sys.path.insert(0, HERE) import bench_common # noqa: E402 (INSTRUMENT, stamped on every row) import memory_probe # noqa: E402 (memory per phase, instrument 2026-11 only) +import vendor_memory # noqa: E402 (Neo4j's and Elasticsearch's own memory rules, DECISIONS #202) DATA = os.path.abspath(os.environ.get("BENCH_DATA", os.path.join(HERE, "data"))) RESULTS = os.path.join(HERE, "results") RAW = os.path.join(RESULTS, "raw") @@ -2107,7 +2108,20 @@ def _batch_jpype_version(jobs): return jpype_version_in_image(ARCADEDB_BENCH_IMAGE) -def _pagecache_for(server_mem_bytes, heap): +# False under a diagnostic --heap or --mem override, which breaks the policy on purpose (see main()). +VENDOR_MEMORY = True + + +def heap_for(backend, scale): + """The heap one arm runs with at one tier: HEAP_BY_SCALE's, except Neo4j (the vendor tool's number for the cell's + cap) and Elasticsearch (at most half the cap and under the compressed-oops threshold), from vendor_memory.py.""" + tier = HEAP_BY_SCALE[scale] + if not VENDOR_MEMORY: + return tier + return vendor_memory.heap_for(backend, tier, MEM_BY_SCALE[scale]) + + +def _pagecache_for(server_mem_bytes, heap, backend=None, cap=None): """What is left for an off-heap page cache once the heap is taken out. Neo4j's own guidance is heap + page cache + about 1g of JVM overhead @@ -2127,6 +2141,10 @@ def _pagecache_for(server_mem_bytes, heap): forced = os.environ.get("BENCH_NEO4J_PAGECACHE") if forced: return forced + # Neo4j's own tool decides (vendor_memory.py): the page cache `neo4j-admin server memory-recommendation` prints for the + # cell's cap, beside the heap it prints. The 1 GiB reserve below is the formula for any other caller. + if VENDOR_MEMORY and backend is not None and vendor_memory.is_neo4j(backend) and cap is not None: + return vendor_memory.neo4j_pagecache(cap) reserve = 1 << 30 left = server_mem_bytes - mem_bytes(heap) - reserve gib = left / float(1 << 30) @@ -2542,8 +2560,9 @@ def observe_server(cid): hits = re.findall(r"heap[_.]max_?_?size=(\d+)([gGmM])", envs, re.I) if hits: val, unit = hits[-1] + # Exact, not rounded down to whole GiB: Neo4j's own tool recommends heaps such as 3600m (DECISIONS #202). out["server_heap"] = (f"{val}g" if unit in "gG" - else f"{int(val) // 1024}g") + else (f"{int(val) // 1024}g" if int(val) % 1024 == 0 else f"{int(val)}m")) # The other half of Neo4j's memory contract, and the setting the audit # found pinned at the image default of 512M on every tier. Recorded so a # future reader can see it was sized rather than inherited. @@ -2929,7 +2948,9 @@ def run_cell(job, rep, scale, cpuset, tier, net_name): if os.path.exists(stale): os.unlink(stale) # belt-and-braces vs stale out-file reads total_mem = mem_bytes(MEM_BY_SCALE[scale]) - heap = HEAP_BY_SCALE[scale] + # THE HEAP THIS ARM RUNS WITH: the tier's half-cap heap, except Neo4j and Elasticsearch, whose vendors give their own rule + # (vendor_memory.py, DECISIONS #202). A diagnostic --heap or --mem override turns that off (VENDOR_MEMORY). + heap = heap_for(job["backend"], scale) # ENGINE PROVENANCE. A locally built wheel reports version "26.9.1.dev0" no matter which commit it # came from, so two engines a day apart are indistinguishable in engine_version. Since the project page # now tracks main rather than a release, the commit is the only thing that identifies what was measured @@ -2968,6 +2989,10 @@ def run_cell(job, rep, scale, cpuset, tier, net_name): if any(t in job["backend"] for t in JVM_BACKENDS) else {}), "mem_cap": MEM_BY_SCALE[scale], "ts_utc": datetime.now(timezone.utc).isoformat()} + # WHICH RULE SET THIS ARM'S HEAP (DECISIONS #202): stamped on the Neo4j and Elasticsearch arms that took their vendor's memory rule, so + # a row measured before it (heap at half the cap) is told apart from one after, and fairness_check judges each by its own rule. + if VENDOR_MEMORY and (vendor_memory.is_neo4j(job["backend"]) or vendor_memory.is_elasticsearch(job["backend"])): + row["vendor_memory"] = "vendor_memory.py, DECISIONS #202" # cli_cid initialised HERE, not at its assignment 90 lines down: the # finally block reads it, and a server that fails to start returns @@ -3049,7 +3074,7 @@ def run_cell(job, rep, scale, cpuset, tier, net_name): "--cpuset-cpus", cpuset, "--memory", str(server_mem), "--memory-swap", str(server_mem), "--shm-size", str(server_mem)] - + [s.format(heap=heap, pagecache=_pagecache_for(server_mem, heap), + + [s.format(heap=heap, pagecache=_pagecache_for(server_mem, heap, job["backend"], MEM_BY_SCALE[scale]), **_fit) for s in be.get("server_env", [])] + be.get("server_volumes", []) @@ -3949,6 +3974,8 @@ def main(): # instead of measuring. Same diagnostic-only warning as --heap. for k in MEM_BY_SCALE: MEM_BY_SCALE[k] = args.mem + global VENDOR_MEMORY + VENDOR_MEMORY = False print(f"MEM CAP OVERRIDE: {args.mem} (diagnostic)") if args.heap: # Applied to every scale, because an invocation runs one scale and a @@ -3956,6 +3983,7 @@ def main(): # records it. for k in HEAP_BY_SCALE: HEAP_BY_SCALE[k] = args.heap + VENDOR_MEMORY = False print(f"HEAP OVERRIDE: {args.heap} (diagnostic; breaks the heap policy)") if args.timeout: for k in TIMEOUT_BY_SCALE: diff --git a/benchmarks/experiments/test_vendor_memory.py b/benchmarks/experiments/test_vendor_memory.py new file mode 100644 index 00000000000..c1a7111ec78 --- /dev/null +++ b/benchmarks/experiments/test_vendor_memory.py @@ -0,0 +1,118 @@ +"""Neo4j and Elasticsearch run their vendors' memory rules (DECISIONS #202; vendor_memory.py). + +Run with `uv run python -m pytest test_vendor_memory.py -q` from this directory. Held, each by the failure it would cause: + + * the Neo4j table in code equals the tool output committed in `neo4j_memory_recommendation.json` (generated by + `python3 vendor_memory.py --regenerate`, never typed), re-parsed here from the raw output, for every cap in the tier table; + * the file names the image digest every Neo4j arm pins and the command the code states, and the tool's initial heap equals its max; + * the runner passes Neo4j the table's heap and page cache, and its arms keep initial heap = max heap; + * Elasticsearch's heap at every tier is at most half the cap and under the compressed-oops threshold, `-Xms` = `-Xmx`, + and deep10m moved from 24g to 18g; + * the heap observed in a container's environment is read back exactly (3600m is not 3g). +""" +import json +import os + +import pytest + +import runner +import vendor_memory as vm + +FILE = json.load(open(vm.NEO4J_RECOMMENDATION_FILE)) +NEO4J_ARMS = [b for b in runner.BACKENDS if vm.is_neo4j(b)] +ES_ARMS = [b for b in runner.BACKENDS if vm.is_elasticsearch(b)] +CAPS = sorted(set(runner.MEM_BY_SCALE.values()), key=vm._gib) + + +def test_table_covers_every_cap_in_the_tier_table(): + assert set(vm.NEO4J_MEMORY) == set(CAPS) == set(FILE["caps"]) + + +@pytest.mark.parametrize("cap", CAPS) +def test_table_equals_the_committed_tool_output(cap): + rec = FILE["caps"][cap] + hi, hm, pc = vm.parse_recommendation(rec["output"]) # the raw text, not the summary fields beside it + assert hi == hm, "the tool recommends initial heap = max heap" + assert (hm, pc) == (rec["heap_max"], rec["pagecache"]) == vm.NEO4J_MEMORY[cap] + + +def test_file_names_the_image_and_the_command(): + assert FILE["image"] == "neo4j@" + vm.NEO4J_IMAGE_DIGEST + assert FILE["command"] == vm.NEO4J_RECOMMENDATION_COMMAND + for b in NEO4J_ARMS: + img = runner.BACKENDS[b]["server_image"] + if img == "dbbench:composed": # the composed stack builds its Neo4j half FROM the same digest + assert vm.NEO4J_IMAGE_DIGEST in open(os.path.join(vm.HERE, "Dockerfile.composed")).read() + else: + assert img == FILE["image"], b + + +def test_neo4j_arms_keep_initial_heap_equal_to_max(): + assert NEO4J_ARMS + for b in NEO4J_ARMS: + env = " ".join(runner.BACKENDS[b]["server_env"]) + assert "NEO4J_server_memory_heap_initial__size={heap}" in env, b + assert "NEO4J_server_memory_heap_max__size={heap}" in env, b + assert "NEO4J_server_memory_pagecache_size={pagecache}" in env, b + + +@pytest.mark.parametrize("scale", sorted(runner.HEAP_BY_SCALE)) +def test_runner_gives_neo4j_the_tool_numbers(scale): + cap = runner.MEM_BY_SCALE[scale] + for b in NEO4J_ARMS: + assert runner.heap_for(b, scale) == vm.NEO4J_MEMORY[cap][0] + assert runner._pagecache_for(runner.mem_bytes(cap), runner.heap_for(b, scale), b, cap) == vm.NEO4J_MEMORY[cap][1] + + +def test_other_engines_keep_the_tier_heap_and_the_formula(): + for scale in runner.HEAP_BY_SCALE: + assert runner.heap_for("arcadedb_graph", scale) == runner.HEAP_BY_SCALE[scale] + assert runner.heap_for("qdrant_dense", scale) == runner.HEAP_BY_SCALE[scale] + # the formula (heap, 1 GiB reserve, floor 512m) still answers a caller that names no Neo4j arm + assert runner._pagecache_for(runner.mem_bytes("8g"), "4g") == "3.0g" + + +def test_override_switches_the_vendor_rule_off(monkeypatch): + monkeypatch.setattr(runner, "VENDOR_MEMORY", False) + assert runner.heap_for("neo4j_graph", "small") == runner.HEAP_BY_SCALE["small"] + assert runner._pagecache_for(runner.mem_bytes("16g"), "8g", "neo4j_graph", "16g") == "7.0g" + + +def test_elasticsearch_heap_rule(): + assert ES_ARMS + for scale, cap in runner.MEM_BY_SCALE.items(): + if scale not in runner.HEAP_BY_SCALE: + continue + h = vm._gib(runner.heap_for("elasticsearch_dense", scale)) + assert h <= 0.5 * vm._gib(cap), scale + assert h <= vm.ES_HEAP_OOPS_CEILING_GIB, scale + assert h <= vm._gib(runner.HEAP_BY_SCALE[scale]), scale # never above what it ran at + assert runner.heap_for("elasticsearch_dense", "deep10m") == "18g" # was 24g in 36g + for scale in ("micro", "tiny", "small", "medium", "large"): # the half-cap tiers do not move + assert runner.heap_for("elasticsearch_sparse", scale) == runner.HEAP_BY_SCALE[scale] + + +def test_elasticsearch_arms_keep_xms_equal_to_xmx(): + for b in ES_ARMS: + assert "ES_JAVA_OPTS=-Xms{heap} -Xmx{heap}" in " ".join(runner.BACKENDS[b]["server_env"]), b + + +@pytest.mark.parametrize("heap,expect", [("3600m", "3600m"), ("13800m", "13800m"), ("4096m", "4g"), ("12g", "12g")]) +def test_observed_heap_is_read_back_exactly(monkeypatch, heap, expect): + env = json.dumps(["NEO4J_server_memory_heap_initial__size=" + heap, "NEO4J_server_memory_heap_max__size=" + heap]) + monkeypatch.setattr(runner, "sh", lambda cmd, *a, **k: env if "{{json .Config.Env}}" in cmd else "") + assert runner.observe_server("cid").get("server_heap") == expect + + +def test_a_vendor_rule_row_is_stamped_and_judged_by_its_own_rule(capsys): + import fairness_check as fc + def row(be, heap, **kw): + return dict(lane="l2", scale="small", backend=be, mem_cap="16g", heap=heap, server_heap=heap, server_mem_cap="16g", + mem_split="full+client", client_mem_cap="8g", rep=1, **kw) + ok = [row("neo4j_graph", "5g", vendor_memory="x"), row("arcadedb_graph", "8g")] + assert fc.check_envelope(ok) == 0 + bad = [row("neo4j_graph", "8g", vendor_memory="x"), row("arcadedb_graph", "8g")] + assert fc.check_envelope(bad) == 1 + assert "the vendor rule gives 5g" in capsys.readouterr().out + old = [row("neo4j_graph", "8g"), row("arcadedb_graph", "8g")] # measured before the rule: judged as before + assert fc.check_envelope(old) == 0 diff --git a/benchmarks/experiments/vendor_memory.py b/benchmarks/experiments/vendor_memory.py new file mode 100644 index 00000000000..669a43459a1 --- /dev/null +++ b/benchmarks/experiments/vendor_memory.py @@ -0,0 +1,112 @@ +"""The memory settings of the two comparators whose vendors give a rule for them (DECISIONS #202). + +Every engine runs as its vendor recommends. For two JVM comparators the vendor's rule differs from the half-cap heap the +rest of the harness uses (`runner.HEAP_BY_SCALE`), so their arms take their heap, and Neo4j its page cache, from here: + + * Neo4j: the numbers `neo4j-admin server memory-recommendation` prints for the cell's cap, run from the pinned image + (`neo4j_memory_recommendation.json`, written by `python3 vendor_memory.py --regenerate`; the test compares the table + below with that file). The tool sets the initial heap equal to the maximum heap, as the Neo4j manual recommends. + * Elasticsearch: the heap is no more than 50% of the cell's cap and below the compressed-oops threshold ("26GB is safe + on most systems"), `-Xms` = `-Xmx`. + +Nothing here imports the runner, so the runner, the tests and the page generator all read one table. +""" +import json +import os +import re +import subprocess +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +NEO4J_RECOMMENDATION_FILE = os.path.join(HERE, "neo4j_memory_recommendation.json") + +# The image the tool ran from: the digest every Neo4j arm in runner.BACKENDS pins (2026.08.1-community). +NEO4J_IMAGE_DIGEST = "sha256:e702d6b535d9d3ae01ee7b132ec87aa40e23d3f0ace82fbfc344e2048cb81960" +# The command that produced the table, per cap (`{cap}` is the cell's memory cap). It runs against an empty data directory, so +# the page cache it recommends is the one for a server whose memory is the only input, as for the cells that build their store. +NEO4J_RECOMMENDATION_COMMAND = ("docker run --rm -m 2g --entrypoint neo4j-admin neo4j@" + NEO4J_IMAGE_DIGEST + + " server memory-recommendation --memory={cap} --docker") + +# cap -> (heap, page cache), both as the tool prints them. Initial heap = maximum heap. Generated from the file above. +NEO4J_MEMORY = { + "8g": ("3600m", "2g"), + "12g": ("4600m", "4g"), + "16g": ("5g", "7g"), + "24g": ("8g", "10g"), + "28g": ("10g", "11g"), + "32g": ("12g", "12g"), + "36g": ("13800m", "14g"), + "48g": ("18g", "20g"), +} + +# Elasticsearch: "Set Xms and Xmx to no more than 50% of the total memory available to each Elasticsearch node" and "to no more +# than the threshold for compressed ordinary object pointers ... 26GB is safe on most systems". +ES_HEAP_CAP_FRACTION = 0.5 +ES_HEAP_OOPS_CEILING_GIB = 26 + + +def is_neo4j(backend): + return "neo4j" in str(backend) + + +def is_elasticsearch(backend): + return str(backend).startswith("elasticsearch") + + +def _gib(spec): + m = re.fullmatch(r"(\d+)([gm])", str(spec)) + n = int(m.group(1)) + return n if m.group(2) == "g" else n / 1024.0 + + +def parse_recommendation(output): + """(heap_initial, heap_max, pagecache) as printed, from the tool's `--docker` output.""" + def get(name): + m = re.search(r"^NEO4J_server_memory_" + name + r"='([^']+)'", output, re.M) + return m.group(1) if m else None + return get("heap_initial__size"), get("heap_max__size"), get("pagecache_size") + + +def elasticsearch_heap(tier_heap, cap): + """The tier's heap, brought under Elastic's two ceilings (half the cap, the compressed-oops threshold), as `g`.""" + gib = min(_gib(tier_heap), _gib(cap) * ES_HEAP_CAP_FRACTION, ES_HEAP_OOPS_CEILING_GIB) + return f"{int(gib)}g" if gib == int(gib) else f"{int(gib * 1024)}m" + + +def heap_for(backend, tier_heap, cap): + """The heap this arm runs with at a tier whose half-cap heap is `tier_heap` and whose cap is `cap`.""" + if is_neo4j(backend): + return NEO4J_MEMORY[cap][0] + if is_elasticsearch(backend): + return elasticsearch_heap(tier_heap, cap) + return tier_heap + + +def neo4j_pagecache(cap): + return NEO4J_MEMORY[cap][1] + + +def regenerate(caps): + """Run the tool from the pinned image for each cap and write the file the test reads.""" + out = {"image": "neo4j@" + NEO4J_IMAGE_DIGEST, "command": NEO4J_RECOMMENDATION_COMMAND, "caps": {}} + for cap in caps: + cmd = NEO4J_RECOMMENDATION_COMMAND.format(cap=cap).split() + r = subprocess.run(cmd, capture_output=True, text=True, check=True) + hi, hm, pc = parse_recommendation(r.stdout) + assert hi == hm and hm and pc, (cap, hi, hm, pc) + out["caps"][cap] = {"heap_initial": hi, "heap_max": hm, "pagecache": pc, "output": r.stdout} + with open(NEO4J_RECOMMENDATION_FILE + ".new", "w") as f: + json.dump(out, f, indent=1, sort_keys=False) + f.write("\n") + os.replace(NEO4J_RECOMMENDATION_FILE + ".new", NEO4J_RECOMMENDATION_FILE) + print("wrote", NEO4J_RECOMMENDATION_FILE) + for cap, v in out["caps"].items(): + print(f' "{cap}": ("{v["heap_max"]}", "{v["pagecache"]}"),') + + +if __name__ == "__main__": + if "--regenerate" not in sys.argv: + sys.exit(__doc__) + sys.path.insert(0, HERE) + import runner + regenerate(sorted(set(runner.MEM_BY_SCALE.values()), key=_gib)) diff --git a/bindings/python/docs/benchmarks/protocol.md b/bindings/python/docs/benchmarks/protocol.md index eb8c9f29728..bae70014fe3 100644 --- a/bindings/python/docs/benchmarks/protocol.md +++ b/bindings/python/docs/benchmarks/protocol.md @@ -46,6 +46,11 @@ worker at the publishing tier. A cell measured two-at-once is exploration foreve engine's own documentation. Embedded ArcadeDB leaves `-Xms` unset, as the Python bindings document, and the server engines keep the initial heap their vendors recommend. - **Raising a resource for one engine obliges a re-measure of every engine at that tier.** +- **Neo4j and Elasticsearch run their vendors' memory rules.** Neo4j takes the heap and page cache + that its own `neo4j-admin server memory-recommendation` prints for the cell's memory cap, and + Elasticsearch a heap of at most half the cap and under the compressed-oops threshold (26 GiB). + Both keep the initial heap equal to the maximum, as their documentation recommends, and every + engine still gets the same memory cap. !!! note "The memory contract cuts against ArcadeDB, and is kept anyway" The JVM heap lives inside the container cap, so ArcadeDB's heap leaves the rest of the