Skip to content
This repository was archived by the owner on Nov 21, 2026. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions benchmarks/experiments/COMPARATORS.md

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions benchmarks/experiments/Dockerfile.pgage
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# PostgreSQL 18 + pgvector 0.8.6 + Apache AGE 1.8.0: the "one engine" rival to
# PostgreSQL 18 + pgvector 0.8.7 + Apache AGE 1.8.0: the "one engine" rival to
# the cross-model transaction (pgvector hit, Cypher hop through AGE, row update,
# one transaction). Base image pinned by digest; AGE from the PGDG apt repo the
# postgres image already carries. Built as dbbench:pg-age by build_images.sh.
Expand All @@ -24,6 +24,6 @@
# measured on (`dpkg-query` on dbbench:pg-age: 1.8.0~rc0-2.pgdg12+1, which the
# rows record as age:1.8.0). If PGDG stops serving it, the build fails here
# rather than moving AGE silently.
FROM pgvector/pgvector@sha256:1d50c689b0a6511b9ea0a15615281c81a59fd04a08eb35057ec8646fb3a2118a
FROM pgvector/pgvector@sha256:ad249f9fb9572979643a47694e59b9a8cdcdf6abc6accebf49bb80ad5f20ff9b
RUN apt-get update && apt-get install -y --no-install-recommends postgresql-18-age=1.8.0~rc0-2.pgdg12+1 \
&& rm -rf /var/lib/apt/lists/*
4 changes: 2 additions & 2 deletions benchmarks/experiments/QUANTIZATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ The quantization survey of DECISIONS #131 item 8 (CAMPAIGN.md section 7 row 42),
| LanceDB 0.39.0 | `IVF_HNSW_FLAT` (unquantized HNSW in IVF partitions), `IVF_HNSW_SQ`, `IVF_HNSW_PQ`, `IVF_FLAT`, `IVF_SQ`, `IVF_PQ`, `IVF_RQ` | `lancedb_dense`: `IVF_HNSW_SQ`, INT8 (it kept its name, so its history stays one series); `lancedb_dense_fp32`: `IVF_HNSW_FLAT`, built 2026-10-02. The old premise (SQ as LanceDB's only HNSW offering) is gone from the adapter, runner, and paper-table comments | the SQ arm is the int8 arm; the fp32 arm now exists. Read back: `lancedb_index_type` (`index_stats`) and `lancedb_index_details` (`list_indices`: HNSW 16/100, `compression` sq 8 bits on the SQ arm only). Recall fp32 / int8: 0.9999 / 0.9902 at 5,000, 0.9994 / 0.9871 at 20,000 | no arm: `IVF_HNSW_PQ` trains its codebook per IVF partition, not over the corpus, so it is a different PQ setting; the fp32 and SQ arms stay its comparison | image: each type passed to `create_index` on a 3,000-row table in `dbbench:dense`; `HNSW_FLAT` and `HNSW_SQ` (without IVF) are refused; smoke: micro, tiny |
| Chroma 1.5.9 | hnswlib, fp32 only | fp32 | none shipped | none shipped | adapter (only `hnsw:*` collection metadata exists); docs |
| DuckDB VSS (duckdb 1.5.4) | HNSW over `FLOAT[]` arrays; no quantization setting | fp32 | none shipped | none shipped | image: `duckdb_settings()` in `dbbench:duckdb` lists only `hnsw_ef_search` and `hnsw_enable_experimental_persistence` |
| pgvector 0.8.6 | `vector` (fp32), `halfvec` (fp16), `bit` (binary, by an expression index over `binary_quantize`), `sparsevec` | fp32 | no int8 type; fp16 and binary exist but are not int8, so no arm under #53 | none shipped (`halfvec` and `bit` are not PQ) | docs |
| pgvector 0.8.7 | `vector` (fp32), `halfvec` (fp16), `bit` (binary, by an expression index over `binary_quantize`), `sparsevec` | fp32 | no int8 type; fp16 and binary exist but are not int8, so no arm under #53 | none shipped (`halfvec` and `bit` are not PQ) | docs |
| Neo4j 2026.10.0 | `vector.quantization.type` `NONE`, `SCALAR`, `BINARY`; **unset means `BINARY`**, with `vector.default_search_expansion_factor` 3.0 | until 2026-10-02 **recorded fp32, ran BINARY**: `neo4j_dense` and `neo4j_e2` created their index without the key (BUGS F164). Now `neo4j_dense` and `neo4j_e2` set `NONE`, and both read the applied type back (`neo4j_vector_quantization`) | built: `neo4j_dense_int8` sets `SCALAR` (repin-prep `32c0a011da`). Each type brings its own `vector.default_search_expansion_factor` (NONE 1.0, SCALAR 1.5, BINARY 3.0 on 2026.08.1, 2026.09.0 and 2026.10.0 (the last re-probed 2026-10-23: unset gives BINARY, 3.0), the candidate multiple searched before re-scoring); the lane matches search effort at ef 100 (Neo4j asks for 100 candidates and keeps 10), so both arms set 1.0 and read it back (`neo4j_vector_search_expansion`), as the Elasticsearch int8 arm sets the minimum oversample. 2026.09.0 (the re-pin candidate, `neo4j@sha256:eaab5879…`) keeps the BINARY default (`.notes` repros/neo4j-202609-quant) | none shipped (`NONE`, `SCALAR`, `BINARY`) | image: `SHOW VECTOR INDEXES` on an index created with exactly our options reports provider `vector-2026.08` and `vector.quantization.type: "BINARY"`; `INT8` and `FLOAT32` are syntax errors |
| MongoDB 9.0.4 + mongot-community 1.70.5 (pin since 2026-10-23; the recall figures below are history, measured on 8.2.12 + 1.70.4) | `vectorSearch` index `quantization`: `none`, `scalar`, `binary` | `mongodb_dense`: `none` | built: `mongodb_dense_int8` sets `scalar`. Both read the field back from mongot's index definition (`mongot_vector_quantization`, `none` / `scalar`). Recall fp32 / int8: 0.9996 / 0.969 at 5,000, 0.9977 / 0.9699 at 20,000 | none shipped (`none`, `scalar`, `binary`) | adapter; smoke: micro, tiny (`scalar` accepted, built, queryable); docs for `binary` |
| SurrealDB (core 2.3.10 embedded, 3.2.4 served) | HNSW `TYPE` F64, F32, I64, I32, I16: element types of the stored array, not quantization of float vectors | `TYPE F32` | none shipped | none shipped | adapter; docs |
Expand Down Expand Up @@ -45,7 +45,7 @@ Smoke, 2026-10-21, laptop, the first 5,000 siftsmall vectors and 80 timed querie
| Qdrant v1.20.0 | sparse `datatype` `float32` (default), `float16`, `uint8`, `turbo4` | `qdrant_sparse`: `float32`, now set in the definition | built: `qdrant_sparse_uint8`, `uint8`; both read the type back from the collection (`qdrant_sparse_datatype`). Recall fp32 / uint8: 1.0 / 1.0 on the 5,000-document synthetic corpus, 1.0 / 0.9952 on the 100,000-document Big-ANN SPLADE slice (once the arms' settle waits for every point, 2026-10-02: before that, both searched a collection still applying its last batches and read 0.93 and 0.88) | image: `qdrant_client.models.Datatype` in `dbbench:client`; smoke: synthetic micro, Big-ANN tiny |
| Milvus v3.0.1 | `SPARSE_FLOAT_VECTOR` with `SPARSE_INVERTED_INDEX` | float32 weights | none documented | adapter; docs |
| Elasticsearch 9.5.4 | `sparse_vector`; weights kept at reduced precision by the engine, no setting | the default | none to choose | docs |
| pgvector 0.8.6 | `sparsevec`, fp32 | fp32 | none shipped | docs |
| pgvector 0.8.7 | `sparsevec`, fp32 | fp32 | none shipped | docs |

## What follows

Expand Down
2 changes: 1 addition & 1 deletion benchmarks/experiments/build_images.sh
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ declare -A PKGS=(
# arm's mutation phase after a deleted vector key is re-inserted, and 0.21.1
# carries #1074 (COPY linear for compressible columns) and #1075 (the pool
# sized from the cgroup limit). 0.21.2 (2026-10-01) is the newest stable.
[client]="requests==2.34.2 psycopg[binary]==3.3.6 $HARNESS_PKGS surrealdb==2.0.0 qdrant-client==1.20.0 pymilvus[bulk_writer]==3.0.2 minio==7.2.20 azure-storage-blob==12.31.0 ml-dtypes==0.6.0 elasticsearch==9.5.1 neo4j==6.4.1 ladybug==0.21.2 pymongo==4.18.1 python-arango==8.3.5 falkordb==1.7.1 redis==8.1.0 neo4j-rust-ext==6.4.1.0 pgvector==0.5.0 arcadedb-driver==0.2.0"
[client]="requests==2.34.2 psycopg[binary]==3.3.6 $HARNESS_PKGS surrealdb==2.0.0 qdrant-client==1.20.0 pymilvus[bulk_writer]==3.0.2 minio==7.2.20 azure-storage-blob==12.31.0 ml-dtypes==0.6.0 elasticsearch==9.5.1 neo4j==6.4.1 ladybug==0.21.2 pymongo==4.18.1 python-arango==8.3.5 falkordb==1.7.1 redis==8.1.0 neo4j-rust-ext==6.4.1.0 pgvector==0.5.1 arcadedb-driver==0.2.0"
[dense]="chromadb==1.5.9 lancedb==0.39.0 sqlite-vec==0.1.9 duckdb==1.5.4 $HARNESS_PKGS"
)
# A GUARD, not a comment. The dev pin above survived because nothing checked
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/experiments/l2_graph.py
Original file line number Diff line number Diff line change
Expand Up @@ -1736,7 +1736,7 @@ class PgAgeGraph(Base):
"""PostgreSQL with Apache AGE, served (DECISIONS #128): every graph read
and write is the lane's own Cypher text, run through AGE's `cypher()`
table function, on the `dbbench:pg-age` image the cross-model lane already
pins (PostgreSQL 18 + pgvector 0.8.6 + AGE 1.8.0~rc0; the row records the
pins (PostgreSQL 18 + pgvector 0.8.7 + AGE 1.8.0~rc0; the row records the
versions the server reports).

THE TEXTS RUN AS WRITTEN (laptop probe 2026-10-02,
Expand Down
18 changes: 9 additions & 9 deletions benchmarks/experiments/runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -613,7 +613,7 @@ def _milvus_arm(dense):
"timescaledb": {
"topology": "client_server",
"image": "dbbench:client",
"server_image": "timescale/timescaledb@sha256:f7036933154c52dbc500f7b08ff8e28404a8ccabbf8ce3528142cde6ab253eef", # 2.30.1-pg18
"server_image": "timescale/timescaledb@sha256:26e20a5e5a7ab6bec7ff217a52ffe21c9f4799603816718b2a573e7be839d8b4", # 2.31.0-pg18
"server_env": ["-e", "POSTGRES_PASSWORD=dbbenchpass", "-e", "POSTGRES_DB=bench"],
"server_cmd": ["-c", "synchronous_commit=off", "-c", "shared_buffers={sb}", "-c", "effective_cache_size={ecs}",
"-c", "maintenance_work_mem={mwm}", "-c", "max_wal_size=4GB"] + PG_FIT_CMD,
Expand All @@ -626,7 +626,7 @@ def _milvus_arm(dense):
"postgres_ts": {
"topology": "client_server",
"image": "dbbench:client",
"server_image": "postgres@sha256:7341002d2b8c7c5bdd7542a671a95b36196c0b5b888daf454ae4fc33ba5346d7", # 18.6, the `postgres` arm's digest
"server_image": "postgres@sha256:9770878138a2e83abcb10856d30cc3f24dae0d317bb956db6ab9159de6c3c80d", # 18.6, the `postgres` arm's digest
"server_env": ["-e", "POSTGRES_PASSWORD=dbbenchpass", "-e", "POSTGRES_DB=bench"],
"server_cmd": ["-c", "synchronous_commit=off", "-c", "shared_buffers={sb}", "-c", "effective_cache_size={ecs}",
"-c", "maintenance_work_mem={mwm}", "-c", "max_wal_size=4GB"] + PG_FIT_CMD,
Expand Down Expand Up @@ -736,7 +736,7 @@ def _milvus_arm(dense):
"postgres": {
"topology": "client_server",
"image": "dbbench:client",
"server_image": "postgres@sha256:7341002d2b8c7c5bdd7542a671a95b36196c0b5b888daf454ae4fc33ba5346d7", # 18.6
"server_image": "postgres@sha256:9770878138a2e83abcb10856d30cc3f24dae0d317bb956db6ab9159de6c3c80d", # 18.6
"server_env": ["-e", "POSTGRES_PASSWORD=dbbenchpass", "-e", "POSTGRES_DB=bench"],
# DURABILITY MATCHED AT THE RELAXED END (DECISIONS #81): a commit
# returns when the WAL record is in the OS, the walwriter flushes it
Expand Down Expand Up @@ -782,7 +782,7 @@ def _milvus_arm(dense):
"postgres_tuned": {
"topology": "client_server",
"image": "dbbench:client",
"server_image": "postgres@sha256:7341002d2b8c7c5bdd7542a671a95b36196c0b5b888daf454ae4fc33ba5346d7", # 18.6, same digest as the default arm
"server_image": "postgres@sha256:9770878138a2e83abcb10856d30cc3f24dae0d317bb956db6ab9159de6c3c80d", # 18.6, same digest as the default arm
"server_env": ["-e", "POSTGRES_PASSWORD=dbbenchpass", "-e", "POSTGRES_DB=bench"],
# Sized from the container, not written as a constant. The two lanes
# that run PostgreSQL get different envelopes (24g at medium, 12g at
Expand Down Expand Up @@ -962,7 +962,7 @@ def _milvus_arm(dense):
"pgage_graph": {
"topology": "client_server",
"image": "dbbench:client",
"server_image": "dbbench:pg-age", # PostgreSQL 18 + pgvector 0.8.6 + AGE 1.8.0, built from Dockerfile.pgage
"server_image": "dbbench:pg-age", # PostgreSQL 18 + pgvector 0.8.7 + AGE 1.8.0, built from Dockerfile.pgage
"server_env": ["-e", "POSTGRES_PASSWORD=dbbenchpass", "-e", "POSTGRES_DB=bench"],
"server_cmd": ["-c", "synchronous_commit=off", "-c", "shared_buffers={sb}", "-c", "effective_cache_size={ecs}",
"-c", "maintenance_work_mem={mwm}", "-c", "max_wal_size=4GB"] + PG_FIT_CMD,
Expand All @@ -989,7 +989,7 @@ def _milvus_arm(dense):
"pg_age_e2": {
"topology": "client_server",
"image": "dbbench:client",
"server_image": "dbbench:pg-age", # PostgreSQL 18 + pgvector 0.8.6 + AGE 1.8.0, built from Dockerfile.pgage
"server_image": "dbbench:pg-age", # PostgreSQL 18 + pgvector 0.8.7 + AGE 1.8.0, built from Dockerfile.pgage
"server_env": ["-e", "POSTGRES_PASSWORD=dbbenchpass", "-e", "POSTGRES_DB=bench"],
"server_cmd": ["-c", "synchronous_commit=off", "-c", "shared_buffers={sb}", "-c", "effective_cache_size={ecs}",
"-c", "maintenance_work_mem={mwm}", "-c", "max_wal_size=4GB"] + PG_FIT_CMD,
Expand Down Expand Up @@ -1438,15 +1438,15 @@ def _milvus_arm(dense):
"server_port": 6333,
"ready_regex": r"Qdrant (HTTP|gRPC) listening|Actix runtime found",
},
# pgvector 0.8.6 on PostgreSQL 18. maintenance_work_mem at half the cap:
# pgvector 0.8.7 on PostgreSQL 18. maintenance_work_mem at half the cap:
# the HNSW build spills to a slow path when the graph outgrows it, and at
# the 64 MB default a 10M build does not finish inside the envelope.
# shared_buffers a quarter of the cap so the index can be resident, the
# way every other comparator holds its index in memory from the build.
"pgvector_dense": {
"topology": "client_server",
"image": "dbbench:client",
"server_image": "pgvector/pgvector@sha256:1d50c689b0a6511b9ea0a15615281c81a59fd04a08eb35057ec8646fb3a2118a", # 0.8.6-pg18
"server_image": "pgvector/pgvector@sha256:ad249f9fb9572979643a47694e59b9a8cdcdf6abc6accebf49bb80ad5f20ff9b", # 0.8.7-pg18
"server_env": ["-e", "POSTGRES_PASSWORD=dbbenchpass", "-e", "POSTGRES_DB=bench"],
"server_cmd": ["-c", "synchronous_commit=off", "-c", "shared_buffers={sb}", "-c", "effective_cache_size={ecs}",
"-c", "maintenance_work_mem={mwm}", "-c", "max_wal_size=8GB"] + PG_FIT_CMD,
Expand All @@ -1457,7 +1457,7 @@ def _milvus_arm(dense):
"pgvector_sparse": {
"topology": "client_server",
"image": "dbbench:client",
"server_image": "pgvector/pgvector@sha256:1d50c689b0a6511b9ea0a15615281c81a59fd04a08eb35057ec8646fb3a2118a", # 0.8.6-pg18
"server_image": "pgvector/pgvector@sha256:ad249f9fb9572979643a47694e59b9a8cdcdf6abc6accebf49bb80ad5f20ff9b", # 0.8.7-pg18
"server_env": ["-e", "POSTGRES_PASSWORD=dbbenchpass", "-e", "POSTGRES_DB=bench"],
"server_cmd": ["-c", "synchronous_commit=off", "-c", "shared_buffers={sb}", "-c", "effective_cache_size={ecs}",
"-c", "maintenance_work_mem={mwm}", "-c", "max_wal_size=8GB"] + PG_FIT_CMD,
Expand Down
Loading
Loading