Skip to content
This repository was archived by the owner on Nov 21, 2026. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 11 additions & 11 deletions benchmarks/experiments/COMPARATORS.md

Large diffs are not rendered by default.

8 changes: 4 additions & 4 deletions benchmarks/experiments/Dockerfile.composed
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The composed cross-model stack's server: Neo4j 2026.08.1-community + Qdrant
# v1.19.1 in one container (BUGS F133, DECISIONS #117).
# The composed cross-model stack's server: Neo4j 2026.10.0-community + Qdrant
# v1.20.0 in one container (BUGS F133, DECISIONS #117).
#
# WHY. The composed stack's vector half was QdrantClient(location=":memory:"),
# the qdrant-client package's pure-Python local mode, which the comparator
Expand All @@ -20,8 +20,8 @@
# are Debian 13), so nothing is fetched from a package mirror.
#
# Built as dbbench:composed by build_images.sh.
FROM qdrant/qdrant@sha256:0699e7733a6fa7fa7f6b95dcbed84ebb04584110da525cdfdef9f305c4f57738 AS qdrant
FROM neo4j@sha256:e702d6b535d9d3ae01ee7b132ec87aa40e23d3f0ace82fbfc344e2048cb81960
FROM qdrant/qdrant@sha256:2d1b0bfd0a961275481ea09400fc98008216ae54e1b52ed7d24520d63050c644 AS qdrant
FROM neo4j@sha256:c3903fa96db31479c2a28e5d00e05260b039ff856f8b2cca7280c78299877731
COPY --from=qdrant /qdrant /qdrant
COPY --from=qdrant /lib/x86_64-linux-gnu/libunwind-ptrace.so.0 /lib/x86_64-linux-gnu/libunwind-x86_64.so.8 /lib/x86_64-linux-gnu/libunwind.so.8 /lib/x86_64-linux-gnu/
COPY docker-conf/composed-entrypoint.sh /usr/local/bin/composed-entrypoint.sh
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/experiments/PROTOCOL.md
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@ An override nobody can find is a defect, whichever way it moves the number. Ever
| ArcadeDB (documents, cross-model, graph message half) | `CREATE INDEX ... UNIQUE_HASH` on the ids a lane only looks up by equality: `Part(p_partkey)`, `OrderNew(okey)`, `Crud(ckey)`, `Product(pid)`, and the LDBC message half's vertex ids; `UNIQUE` (sorted) stays on `Person(id)` and wherever a key is ranged over or ordered | operating-point (the vendor's recommendation for the index kind, not a tuning knob) | ArcadeDB's maintainers' recommendation (ArcadeData/arcadedb#9169): equality-only ids take a hash index, keys also ranged or ordered stay sorted. The ingest and index columns carry the build cost and the transaction columns the gain. | FAIRNESS F14; `fairness_check.ARCADEDB_HASH_ID_INDEXES`, held equal to the DDL by `test_index_kinds.py`; row field `index_kinds`, the engine's own `schema:indexes` answer read after the schema is built, held to the registry by `fairness_check` F14d; a generated sentence built only from rows that carry the stamp |
| Elasticsearch (L3s) | `xpack.security.enabled=false` | escape hatch | Harness plumbing; also removes TLS and auth from every ES latency measurement. | row field `es_security_enabled` (from the engine's `_xpack`), and a generated sentence; `override: es_security` |
| Elasticsearch (L3s) | `number_of_replicas=0` | resource fitting | A single-node cluster cannot allocate a replica; the index would sit yellow forever. | row field `es_replicas` (from the index settings), and a generated sentence; `override: es_replicas` |
| composed Qdrant + Neo4j (E2) | the Qdrant server beside Neo4j in ONE container, `dbbench:composed` from `Dockerfile.composed` (the dense lane's pinned Qdrant v1.19.1 binary, Neo4j at the digest every Neo4j arm runs), reached over REST on 6333, with Qdrant's HNSW defaults (m=16, ef_construct=100) and a payload index on `pid` for the candidate filter (DECISIONS #117, BUGS F133) | resource fitting (one server cgroup for both halves) + operating-point (Qdrant's documented configuration for a filtered field) | Until 2026-09-26 the vector half was the client package's pure-Python local mode, rejected by the comparator review. Rows measured before carry `qdrant-local:` in `engine_version` and the page marks them `re-run`. | row field `engine_version` (`qdrant:<server>+neo4j:<server>`), COMPARATORS.md, page `re-run` note |
| composed Qdrant + Neo4j (E2) | the Qdrant server beside Neo4j in ONE container, `dbbench:composed` from `Dockerfile.composed` (the dense lane's pinned Qdrant v1.20.0 binary, Neo4j at the digest every Neo4j arm runs), reached over REST on 6333, with Qdrant's HNSW defaults (m=16, ef_construct=100) and a payload index on `pid` for the candidate filter (DECISIONS #117, BUGS F133) | resource fitting (one server cgroup for both halves) + operating-point (Qdrant's documented configuration for a filtered field) | Until 2026-09-26 the vector half was the client package's pure-Python local mode, rejected by the comparator review. Rows measured before carry `qdrant-local:` in `engine_version` and the page marks them `re-run`. | row field `engine_version` (`qdrant:<server>+neo4j:<server>`), COMPARATORS.md, page `re-run` note |
| Neo4j (L2, L3d, E2) | the heap and page cache that `neo4j-admin server memory-recommendation --memory=<cap>` prints from the pinned image, one row per tier cap in `vendor_memory.NEO4J_MEMORY` (8g: heap 3600m, page cache 2g; 16g: 5g and 7g; 32g: 12g and 12g; 48g: 18g and 20g; the other caps in the same table), initial heap = max heap; was heap at half the cap and page cache = container memory less heap less 1 GiB, until DECISIONS #202 | vendor recommendation | The Neo4j manual: "Use the neo4j-admin server memory-recommendation command to get an initial recommendation", and "It is recommended to set these two parameters to the same value" for the initial and maximum heap. The tool is run for an empty data directory. The composed E2 container takes the same numbers; Qdrant lives in what they leave. The tool also prints `server.jvm.additional=-XX:+ExitOnOutOfMemoryError`, which the harness does not set. | row fields `server_pagecache` (the container's env) and `neo4j_pagecache` (the engine's `SHOW SETTINGS`), checked against each other by `fairness_check` F15, and a generated sentence; `override: neo4j_pagecache` |
| Neo4j (graph) and the composed cross-model stack | `NEO4J_db_checkpoint_interval_time=5s` (default 15 minutes) | measurement (disk reading) | The store is checkpointed before the disk reading, which read 0.9 MB at SF1 under the default (BUGS F27). Preallocation stays at its default. | row fields `neo4j_checkpoint_interval` and `neo4j_checkpoint_interval_default` (the engine's `SHOW SETTINGS`), and a generated sentence; `override: neo4j_checkpoint` |
| Memgraph (graph) | `--bolt-num-workers` and `--storage-snapshot-thread-count` from the cpuset, `--memory-limit` at 90% of the server cap, `--query-execution-timeout-sec=0`, `--telemetry-enabled=false` | resource fitting (workers, threads, memory) + escape hatch (timeout) | The three pools default to the host, and the 600 s default timeout would abort a whole-graph query that the lane's own budget is there to censor. | row fields `memgraph_bolt_workers`, `memgraph_snapshot_threads`, `memgraph_memory_limit_mib`; FAIRNESS F6; COMPARATORS.md |
Expand Down
8 changes: 4 additions & 4 deletions benchmarks/experiments/QUANTIZATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,19 +11,19 @@ The quantization survey of DECISIONS #131 item 8 (CAMPAIGN.md section 7 row 42),
| engine (pin) | ships | our arm today | int8 arm | PQ arm (gated) | source |
|---|---|---|---|---|---|
| ArcadeDB (26.10.1 at the re-pin) | `LSM_VECTOR` quantization `NONE`, `INT8`, `BINARY`, `PRODUCT` | fp32 and INT8, embedded and served (no `PRODUCT` arm through 26.10.x: the SQL search scored candidates with the full-precision vectors, so PQ saved no memory there, [ArcadeData/arcadedb#10514](https://github.com/ArcadeData/arcadedb/issues/10514), fixed for 26.11.1 by [#10545](https://github.com/ArcadeData/arcadedb/pull/10545)) | exists | built, gated: `arcadedb_dense_embedded_pq`, `arcadedb_dense_server_pq`; `quantization: 'PRODUCT'`, `pqSubspaces` 32, `pqClusters` 256; SQL `vectorNeighbors` at the engine's default re-rank (10 candidates per neighbour on the stored vectors); the arm refuses an engine whose SQL rejects `rerank` | adapter (`l3d_dense._QUANT_DDL`, the engine's own error text for anything else) |
| Qdrant v1.19.1 | scalar (int8), product, binary, turbo | fp32 and scalar int8 (quantile 0.99, always_ram, rescore at search) | exists | built, gated: `qdrant_dense_pq`; `ProductQuantization` x16 (512 to 32 bytes, 32 subspaces of 256 centroids, `always_ram`), search with `rescore` and `oversampling` 10; the collection's compression is read back (`qdrant_quantization`) | image: `qdrant_client.models` in `dbbench:client` lists `ScalarQuantization` (`int8`), `ProductQuantization`, `BinaryQuantization`, `TurboQuantization` |
| Qdrant v1.20.0 | scalar (int8), product, binary, turbo (v1.20.0 adds the `turbo8` and `turbo16` datatypes; no arm uses them) | fp32 and scalar int8 (quantile 0.99, always_ram, rescore at search) | exists | built, gated: `qdrant_dense_pq`; `ProductQuantization` x16 (512 to 32 bytes, 32 subspaces of 256 centroids, `always_ram`), search with `rescore` and `oversampling` 10; the collection's compression is read back (`qdrant_quantization`) | image: `qdrant_client.models` in `dbbench:client` lists `ScalarQuantization` (`int8`), `ProductQuantization`, `BinaryQuantization`, `TurboQuantization` |
| Milvus v3.0.1 | HNSW (fp32), `HNSW_SQ` (`sq_type` SQ8 and others), `HNSW_PQ`, `HNSW_PRQ`, IVF variants | HNSW fp32 and `HNSW_SQ` SQ8 | exists | built, gated: `milvus_dense_pq`; `HNSW_PQ` `m` 32, `nbits` 8, `refine` FP32, search `refine_k` 10; the index type and params are read back (`milvus_index_type`, `milvus_index_params`) | adapter (both arms build on the image); docs for the rest |
| sqlite-vec 0.1.9 | `float32`, `int8`, `bit` column types, `vec_quantize_int8()` | fp32 and int8 | exists | none shipped (an exact scan) | image: `dbbench:dense` |
| LanceDB 0.39.0 | `IVF_HNSW_FLAT` (unquantized HNSW in IVF partitions), `IVF_HNSW_SQ`, `IVF_HNSW_PQ`, `IVF_FLAT`, `IVF_SQ`, `IVF_PQ`, `IVF_RQ` | `lancedb_dense`: `IVF_HNSW_SQ`, INT8 (it kept its name, so its history stays one series); `lancedb_dense_fp32`: `IVF_HNSW_FLAT`, built 2026-10-02. The old premise (SQ as LanceDB's only HNSW offering) is gone from the adapter, runner, and paper-table comments | the SQ arm is the int8 arm; the fp32 arm now exists. Read back: `lancedb_index_type` (`index_stats`) and `lancedb_index_details` (`list_indices`: HNSW 16/100, `compression` sq 8 bits on the SQ arm only). Recall fp32 / int8: 0.9999 / 0.9902 at 5,000, 0.9994 / 0.9871 at 20,000 | no arm: `IVF_HNSW_PQ` trains its codebook per IVF partition, not over the corpus, so it is a different PQ setting; the fp32 and SQ arms stay its comparison | image: each type passed to `create_index` on a 3,000-row table in `dbbench:dense`; `HNSW_FLAT` and `HNSW_SQ` (without IVF) are refused; smoke: micro, tiny |
| Chroma 1.5.9 | hnswlib, fp32 only | fp32 | none shipped | none shipped | adapter (only `hnsw:*` collection metadata exists); docs |
| DuckDB VSS (duckdb 1.5.4) | HNSW over `FLOAT[]` arrays; no quantization setting | fp32 | none shipped | none shipped | image: `duckdb_settings()` in `dbbench:duckdb` lists only `hnsw_ef_search` and `hnsw_enable_experimental_persistence` |
| pgvector 0.8.6 | `vector` (fp32), `halfvec` (fp16), `bit` (binary, by an expression index over `binary_quantize`), `sparsevec` | fp32 | no int8 type; fp16 and binary exist but are not int8, so no arm under #53 | none shipped (`halfvec` and `bit` are not PQ) | docs |
| Neo4j 2026.08.1 | `vector.quantization.type` `NONE`, `SCALAR`, `BINARY`; **unset means `BINARY`**, with `vector.default_search_expansion_factor` 3.0 | until 2026-10-02 **recorded fp32, ran BINARY**: `neo4j_dense` and `neo4j_e2` created their index without the key (BUGS F164). Now `neo4j_dense` and `neo4j_e2` set `NONE`, and both read the applied type back (`neo4j_vector_quantization`) | built: `neo4j_dense_int8` sets `SCALAR` (repin-prep `32c0a011da`). Each type brings its own `vector.default_search_expansion_factor` (NONE 1.0, SCALAR 1.5, BINARY 3.0 on 2026.08.1 and 2026.09.0, the candidate multiple searched before re-scoring); the lane matches search effort at ef 100 (Neo4j asks for 100 candidates and keeps 10), so both arms set 1.0 and read it back (`neo4j_vector_search_expansion`), as the Elasticsearch int8 arm sets the minimum oversample. 2026.09.0 (the re-pin candidate, `neo4j@sha256:eaab5879…`) keeps the BINARY default (`.notes` repros/neo4j-202609-quant) | none shipped (`NONE`, `SCALAR`, `BINARY`) | image: `SHOW VECTOR INDEXES` on an index created with exactly our options reports provider `vector-2026.08` and `vector.quantization.type: "BINARY"`; `INT8` and `FLOAT32` are syntax errors |
| Neo4j 2026.10.0 | `vector.quantization.type` `NONE`, `SCALAR`, `BINARY`; **unset means `BINARY`**, with `vector.default_search_expansion_factor` 3.0 | until 2026-10-02 **recorded fp32, ran BINARY**: `neo4j_dense` and `neo4j_e2` created their index without the key (BUGS F164). Now `neo4j_dense` and `neo4j_e2` set `NONE`, and both read the applied type back (`neo4j_vector_quantization`) | built: `neo4j_dense_int8` sets `SCALAR` (repin-prep `32c0a011da`). Each type brings its own `vector.default_search_expansion_factor` (NONE 1.0, SCALAR 1.5, BINARY 3.0 on 2026.08.1, 2026.09.0 and 2026.10.0 (the last re-probed 2026-10-23: unset gives BINARY, 3.0), the candidate multiple searched before re-scoring); the lane matches search effort at ef 100 (Neo4j asks for 100 candidates and keeps 10), so both arms set 1.0 and read it back (`neo4j_vector_search_expansion`), as the Elasticsearch int8 arm sets the minimum oversample. 2026.09.0 (the re-pin candidate, `neo4j@sha256:eaab5879…`) keeps the BINARY default (`.notes` repros/neo4j-202609-quant) | none shipped (`NONE`, `SCALAR`, `BINARY`) | image: `SHOW VECTOR INDEXES` on an index created with exactly our options reports provider `vector-2026.08` and `vector.quantization.type: "BINARY"`; `INT8` and `FLOAT32` are syntax errors |
| MongoDB 9.0.4 + mongot-community 1.70.5 (pin since 2026-10-23; the recall figures below are history, measured on 8.2.12 + 1.70.4) | `vectorSearch` index `quantization`: `none`, `scalar`, `binary` | `mongodb_dense`: `none` | built: `mongodb_dense_int8` sets `scalar`. Both read the field back from mongot's index definition (`mongot_vector_quantization`, `none` / `scalar`). Recall fp32 / int8: 0.9996 / 0.969 at 5,000, 0.9977 / 0.9699 at 20,000 | none shipped (`none`, `scalar`, `binary`) | adapter; smoke: micro, tiny (`scalar` accepted, built, queryable); docs for `binary` |
| SurrealDB (core 2.3.10 embedded, 3.2.4 served) | HNSW `TYPE` F64, F32, I64, I32, I16: element types of the stored array, not quantization of float vectors | `TYPE F32` | none shipped | none shipped | adapter; docs |
| ArangoDB 3.12.12 (3.12.11 until the 26.10.1 measurement) | FAISS IVF vector index; a custom FAISS `factory` string (SQ8, SQ4, PQ), forwarded to FAISS and validated at creation only from 3.12.12 | `arangodb_dense`: IVF flat, fp32 (`nLists`, `nProbe` calibrated) | built: `arangodb_dense_int8`, factory `IVF<nLists>,SQ8` at the fp32 arm's nLists, nProbe calibrated to the same recall target; the server's factory string is read back (`ivf_factory`, None on the fp32 arm) beside its training state. Recall fp32 / int8, each calibrated on the held-out slice to the laptop's fallback target 0.95: 0.9402 / 0.9465 at 5,000, 0.9486 / 0.9447 at 20,000, at nProbe 17 / 19 and 23 / 24 | built, gated: `arangodb_dense_pq`; FAISS factory `IVF<nLists>,PQ32x8` at the fp32 arm's nLists, nProbe calibrated to the same recall target; the factory string is read back (`ivf_factory`); the factory has no re-rank stage, so the arm answers from the codes | image: 20,000 SIFT vectors at nLists 566, both indexes `ready`, recall@10 at nProbe 1, 4, 16, 64: plain 0.345, 0.666, 0.932, 0.9985; SQ8 0.345, 0.6645, 0.928, 0.9905; docs (factory text, the 3.12.12 validation); smoke: micro, tiny; 3.12.12 (the pin since the 26.10.1 measurement; its digest is in COMPARATORS.md): `IVF70,SQ8` trains to ready and reads back unchanged at the same recall as 3.12.11, and an invalid factory string is now refused at creation, where 3.12.11 built an `unusable` index that answered by an exact scan (recall 1.0); the arm refuses any index that is not `ready` either way (`.notes` repros/arangodb-31212-sq8) |
| Elasticsearch 9.5.4 (joins dense, row 37) | `hnsw`, `int8_hnsw`, `int4_hnsw`, `bbq_hnsw`, `bbq_disk`, `int8_flat` (all accepted); **default `int8_hnsw` at 96 and 128 dimensions** (m 16, ef_construction 100), `bbq_hnsw` with 3x oversampling from 384 | `elasticsearch_dense`: `hnsw`, set explicitly | built: `elasticsearch_dense_int8`, `int8_hnsw` with `rescore_vector.oversample` 1.0; both read the mapping's `index_options` back (`es_index_options`) | none shipped (`bbq` is binary) | image: dense_vector mappings on the pinned server, defaults read through `_mapping/field?include_defaults=true` |
| Memgraph 3.13.1 (joins dense, row 38) | `scalar_kind` f64, f32, f16, bf16, i8 (`b1` refused) | `memgraph_dense`: f32 | built: `memgraph_dense_int8`, `i8`; both read `scalar_kind` back from `SHOW VECTOR INDEX INFO` (`memgraph_vector_scalar_kind`). The i8 distances `vector_search.search` yields are integers, and the arm orders by them, so nothing re-ranks against the floats. Recall fp32 / int8: 0.9997 / 0.9423 at 5,000, 0.9992 / 0.9361 at 20,000 | none shipped | image: `CREATE VECTOR INDEX ... scalar_kind` on the pinned server, `SHOW VECTOR INDEX INFO`; smoke: micro, tiny |
| Memgraph 3.14.0 (joins dense, row 38) | `scalar_kind` f64, f32, f16, bf16, i8 (`b1` refused) | `memgraph_dense`: f32 | built: `memgraph_dense_int8`, `i8`; both read `scalar_kind` back from `SHOW VECTOR INDEX INFO` (`memgraph_vector_scalar_kind`). The i8 distances `vector_search.search` yields are integers, and the arm orders by them, so nothing re-ranks against the floats. Recall fp32 / int8: 0.9997 / 0.9423 at 5,000, 0.9992 / 0.9361 at 20,000 | none shipped | image: `CREATE VECTOR INDEX ... scalar_kind` on the pinned server, `SHOW VECTOR INDEX INFO`; smoke: micro, tiny |
| FalkorDB (6.0.1 at the re-pin, row 38) | vector index over float32 | `falkordb_dense`: fp32 | none documented | none documented | docs |
| LadybugDB 0.21.2 (row 38) | the official `vector` extension, HNSW over float arrays | `ladybug_dense`: fp32 | none documented | none documented | docs |

Expand All @@ -42,7 +42,7 @@ Smoke, 2026-10-21, laptop, the first 5,000 siftsmall vectors and 80 timed querie
| engine (pin) | ships | our arm today | int8 arm | source |
|---|---|---|---|---|
| ArcadeDB | `LSM_SPARSE_VECTOR` `weightQuantization` INT8 (the default), FP16, FP32 | INT8 (default) and FP32 | exists | adapter (`l3_sparse.ArcadeEmbedded.quant`, engine #5143) |
| Qdrant v1.19.1 | sparse `datatype` `float32` (default), `float16`, `uint8`, `turbo4` | `qdrant_sparse`: `float32`, now set in the definition | built: `qdrant_sparse_uint8`, `uint8`; both read the type back from the collection (`qdrant_sparse_datatype`). Recall fp32 / uint8: 1.0 / 1.0 on the 5,000-document synthetic corpus, 1.0 / 0.9952 on the 100,000-document Big-ANN SPLADE slice (once the arms' settle waits for every point, 2026-10-02: before that, both searched a collection still applying its last batches and read 0.93 and 0.88) | image: `qdrant_client.models.Datatype` in `dbbench:client`; smoke: synthetic micro, Big-ANN tiny |
| Qdrant v1.20.0 | sparse `datatype` `float32` (default), `float16`, `uint8`, `turbo4` | `qdrant_sparse`: `float32`, now set in the definition | built: `qdrant_sparse_uint8`, `uint8`; both read the type back from the collection (`qdrant_sparse_datatype`). Recall fp32 / uint8: 1.0 / 1.0 on the 5,000-document synthetic corpus, 1.0 / 0.9952 on the 100,000-document Big-ANN SPLADE slice (once the arms' settle waits for every point, 2026-10-02: before that, both searched a collection still applying its last batches and read 0.93 and 0.88) | image: `qdrant_client.models.Datatype` in `dbbench:client`; smoke: synthetic micro, Big-ANN tiny |
| Milvus v3.0.1 | `SPARSE_FLOAT_VECTOR` with `SPARSE_INVERTED_INDEX` | float32 weights | none documented | adapter; docs |
| Elasticsearch 9.5.4 | `sparse_vector`; weights kept at reduced precision by the engine, no setting | the default | none to choose | docs |
| pgvector 0.8.6 | `sparsevec`, fp32 | fp32 | none shipped | docs |
Expand Down
Loading
Loading