From a67a53d7e4ec518814c5b24dba9d9b8356780ac2 Mon Sep 17 00:00:00 2001 From: Taewoon Kim Date: Fri, 23 Oct 2026 16:35:50 +0900 Subject: [PATCH] bench: re-pin pgvector 0.8.7, TimescaleDB 2.31.0, pgvector-python 0.5.1 pgvector/pgvector 0.8.6-pg18 -> 0.8.7-pg18 (amd64 manifest sha256:ad249f9fb957...), also the base of dbbench:pg-age. timescale/timescaledb 2.30.1-pg18 -> 2.31.0-pg18 (amd64 manifest sha256:26e20a5e5a7a...). pgvector (Python adapter) 0.5.0 -> 0.5.1 in the client image. PostgreSQL 18.6 and Apache AGE 1.8.0~rc0-2 (PGDG) are current and unchanged. COMPARATORS.md, QUANTIZATION.md, Dockerfile.pgage and the stage-draft hash list follow the pins. Co-Authored-By: Claude Sonnet 5.5 --- benchmarks/experiments/COMPARATORS.md | 12 ++++++------ benchmarks/experiments/Dockerfile.pgage | 4 ++-- benchmarks/experiments/QUANTIZATION.md | 4 ++-- benchmarks/experiments/build_images.sh | 2 +- benchmarks/experiments/l2_graph.py | 2 +- benchmarks/experiments/runner.py | 12 ++++++------ benchmarks/experiments/stage_drafts_2611.sha256 | 14 +++++++------- benchmarks/experiments/test_sparse_mp_versions.py | 2 +- 8 files changed, 26 insertions(+), 26 deletions(-) diff --git a/benchmarks/experiments/COMPARATORS.md b/benchmarks/experiments/COMPARATORS.md index 892c5489d3d..81aa0e5c495 100644 --- a/benchmarks/experiments/COMPARATORS.md +++ b/benchmarks/experiments/COMPARATORS.md @@ -24,10 +24,10 @@ Which arms are on the October page, with their rows and columns, is PAGE-SPEC.md | SurrealDB (served) | `surrealdb/surrealdb@sha256:6a500236…` (`v3.2.4`) | 3.2.4, RocksDB storage | documents (TPC), graph, dense, cross-model, time series (plain table) | served | SDK `insert()` and bulk relation insert over ws. Time series (2026-09-15): the same table, index and SurrealQL as the embedded arm, over ws through the reconnecting client (DECISIONS #91). Dense and cross-model: the HNSW index is defined before the load, which the 3.2.4 server fills in the background, so the build waits inside its index timer until a KNN probe's latency stops falling (`surreal_common.await_hnsw_settled`, recorded as `settle_s`; BUGS F134, DECISIONS #117). | | MongoDB | `mongo@sha256:41afd6e1…` (`mongo:8.2.12`) | 8.2.12 | documents (TPC), time series (native time-series collection), graph | served | insert_many batches; single-node replica set, because TPC-C new-order needs a multi-document transaction. Client `pymongo==4.18.1`. Held below 8.3.11, which cannot boot on the bench kernel (SERVER-121912); 8.2.12 is the latest measurable MongoDB (the held-below list). The graph arm runs the SAME digest. Every hop and every analytics query is a chained `$lookup`, never `$graphLookup` (PROTOCOL.md section 7). The loader refuses a self-loop, on which the undirected reads depend. The write looks up its anchor person first and writes nothing when it is absent, as the Cypher's MATCH does (until 2026-10-02 it inserted unconditionally, which only a capped slice could show). Every graph question, LSQB's nine included, is expressed as an aggregation pipeline and agrees on the slice (DECISIONS #104b). | | MongoDB + MongoDB Search | built image `dbbench:mongo-search` (Dockerfile.mongosearch) FROM `mongo@sha256:41afd6e1…` + `mongodb/mongodb-community-search@sha256:24c0cd30…`; both digests pinned in the Dockerfile | mongod 8.2.12 + mongot-community 1.70.4 | dense vectors, cross-model | served, one container holding both processes | `$vectorSearch` over a `vectorSearch` index at the lane's matched point (PROTOCOL.md section 7). The vector field's `quantization` is `none` on `mongodb_dense` and `scalar` on `mongodb_dense_int8` (DECISIONS #135, from the 26.10.1 measurement), read back from mongot's index definition onto the row (`mongot_vector_quantization`). mongod runs WITH access control, unlike the document arm above (PROTOCOL.md section 7). Not `mongodb/mongodb-atlas-local`: MongoDB documents that image as "For evaluation only" and "not suitable for production use", and its vector-capable variant is a `:preview` tag. | -| pgvector | `pgvector/pgvector@sha256:1d50c689…` (`0.8.6-pg18`) | PostgreSQL 18 + pgvector 0.8.6 | dense (`vector`, HNSW m=16, ef_construction=100, ef_search=100), sparse (`sparsevec`, HNSW, inner product) | served | COPY, then HNSW build. Memory and pools fitted to the cell (PROTOCOL.md section 7, FAIRNESS F6). | -| TimescaleDB | `timescale/timescaledb@sha256:f7036933…` (`2.30.1-pg18`) | 2.30.1 on PostgreSQL 18 | time series (hypertable, `time_bucket`) | served | COPY into a hypertable, index on (host, ts). One of TSBS's home engines. Pinned at 2.30.1 and never 2.30.0: 2.30.0 introduced the `DeferredChunkAppend` planner node and 2.30.1 fixes that node producing DUPLICATE ROWS, which is an answer bug on this lane's own query shapes. Plain PostgreSQL runs beside it on the same image family (`postgres_ts`: COPY into a plain table, the same (host, ts DESC) index, `date_trunc` buckets), so the table shows what the extension buys. | +| pgvector | `pgvector/pgvector@sha256:ad249f9f…` (`0.8.7-pg18`) | PostgreSQL 18 + pgvector 0.8.7 | dense (`vector`, HNSW m=16, ef_construction=100, ef_search=100), sparse (`sparsevec`, HNSW, inner product) | served | COPY, then HNSW build. Memory and pools fitted to the cell (PROTOCOL.md section 7, FAIRNESS F6). | +| TimescaleDB | `timescale/timescaledb@sha256:26e20a5e…` (`2.31.0-pg18`) | 2.31.0 on PostgreSQL 18 | time series (hypertable, `time_bucket`) | served | COPY into a hypertable, index on (host, ts). One of TSBS's home engines. Never 2.30.0: it introduced the `DeferredChunkAppend` planner node, and 2.30.1 fixed that node producing DUPLICATE ROWS, which is an answer bug on this lane's own query shapes; 2.31.0 (2026-10-20) carries the fix. Re-pinned from 2.30.1 on 2026-10-23: its release notes rename `timescaledb.enable_columnarscan` and hide the direct-compress sorting GUCs, and neither is set by this lane. Plain PostgreSQL runs beside it on the same image family (`postgres_ts`: COPY into a plain table, the same (host, ts DESC) index, `date_trunc` buckets), so the table shows what the extension buys. | | Neo4j | `neo4j@sha256:e702d6b5…` (`2026.08.1-community`) | 2026.08.1 | graph, dense (its vector index, HNSW), cross-model | served | UNWIND batches over bolt. Dense queries use the Cypher 25 `SEARCH` clause; it has no candidate-count option and `LIMIT k` alone gave recall 0.50 at micro scale, so the adapter asks for `ef_search` (100) candidates and keeps the best k, the operating point every other engine runs at. Its vector index's quantization is set, never left to the default: 2026.08.1 builds BINARY (with a 3x search expansion) when the definition names none, which ours did through October (BUGS F164, DECISIONS #135); from the 26.10.1 measurement the fp32 arms set `NONE`, a `neo4j_dense_int8` arm sets `SCALAR`, and both read the applied type back onto the row (`neo4j_vector_quantization`). | -| PostgreSQL + pgvector + Apache AGE | built image `dbbench:pg-age` (Dockerfile.pgage) FROM `pgvector/pgvector@sha256:1d50c689…` + `postgresql-18-age=1.8.0~rc0-2.pgdg12+1` from PGDG (the package pinned since 2026-10-02; unpinned, two builds could carry two AGE builds under one tag); both extension versions recorded on the row at connect | PostgreSQL 18 + pgvector 0.8.6 + AGE 1.8.0 | cross-model; graph, both workloads (DECISIONS #128, from the 26.10.1 measurement) | served | Cross-model: COPY plus Cypher CREATE through AGE; the strongest "one engine" rival to the page's central claim. Graph: the lane's own Cypher through `cypher()`, loaded by COPY into AGE's label tables with the graphids AGE's sequences assign, Message a PostgreSQL inheritance parent of Post and Comment, one btree on the Person `id` property (measured, #112). AGE's btree range scan from a `>=` bound skips the equal key (apache/age#2587), which costs the graph arm its two CRUD read-back digests, declared in `equivalence_check`. | +| PostgreSQL + pgvector + Apache AGE | built image `dbbench:pg-age` (Dockerfile.pgage) FROM `pgvector/pgvector@sha256:ad249f9f…` + `postgresql-18-age=1.8.0~rc0-2.pgdg12+1` from PGDG (the package pinned since 2026-10-02; unpinned, two builds could carry two AGE builds under one tag); both extension versions recorded on the row at connect | PostgreSQL 18 + pgvector 0.8.7 + AGE 1.8.0 | cross-model; graph, both workloads (DECISIONS #128, from the 26.10.1 measurement) | served | Cross-model: COPY plus Cypher CREATE through AGE; the strongest "one engine" rival to the page's central claim. Graph: the lane's own Cypher through `cypher()`, loaded by COPY into AGE's label tables with the graphids AGE's sequences assign, Message a PostgreSQL inheritance parent of Post and Comment, one btree on the Person `id` property (measured, #112). AGE's btree range scan from a `>=` bound skips the equal key (apache/age#2587), which costs the graph arm its two CRUD read-back digests, declared in `equivalence_check`. | | Qdrant + Neo4j (the composed cross-model stack) | built image `dbbench:composed` (Dockerfile.composed) FROM `neo4j@sha256:e702d6b5…` with the Qdrant binary copied from `qdrant/qdrant@sha256:0699e773…`; both digests pinned in the Dockerfile (DECISIONS #117, BUGS F133) | Qdrant v1.19.1 + Neo4j 2026.08.1, stamped on the row as `qdrant:+neo4j:`, each read from its own server at connect | cross-model | served, one container holding both processes, so the one server cgroup caps, pins and measures both halves | Qdrant upsert batches over REST on 6333, then a payload index on `pid` and a wait for green status inside the index timer; Neo4j UNWIND batches over bolt. No transaction spans the two systems, which is what the arm is on the table to show. Qdrant runs with `RUN_MODE=production`, as its own image does (HNSW settings: PROTOCOL.md section 7). | | ArangoDB | `arangodb@sha256:4bc086d5…` (`arangodb:3.12.12`, also `latest` on 2026-10-06, from the 26.10.1 measurement; `arangodb@sha256:563cb2c0…`, `3.12.11`, before; the image reports "enterprise", which since 3.12 is the one self-managed build). License BSL 1.1 per the repository's LICENSE at tag v3.12.12 (the same file as at v3.12.11), so not open source; the page's license table says so | 3.12.12, RocksDB storage (3.12.11 until the 26.10.1 measurement) | documents (TPC), graph, dense, cross-model, time series (plain table) | served only: python-arango 8.3.5 is an HTTP client and the engine has no in-process mode | bulk import API in batches (`arangod --vector-index true`, PROTOCOL.md section 7). 3.12.11 had a defect: with 10,000 or more IVF lists every multithreaded query probes the same single list, so the October deep10m repetition answered at recall 0.0 after the calibration escalated to an exhaustive probe; 3.12.12 answers the same 12,643-list index at recall@10 0.9827 at the calibrated nProbe 167. The `3.12.11` tag was REBUILT on 2026-09-18 (its amd64 digest moved to `sha256:460b77c5349f…` while the pin held `sha256:563cb2c07af0…`). The calibration reads recall at nProbe = nLists first, on 20 held-out queries, and raises below a floor of 0.5 (a healthy exhaustive probe reads about 0.995). Its vector index is FAISS IVF, matched by effect rather than by knob, and the `ivf_*` fields on the row make the operating point auditable (FAIRNESS.md F7). From the 26.10.1 measurement an int8 arm, `arangodb_dense_int8` (DECISIONS #135), builds the same IVF through the FAISS factory string `IVF,SQ8`, so the lists hold 8-bit codes; its nProbe is calibrated to the same target, and the server's factory string is read back onto the row (`ivf_factory`, None on the fp32 arm). Time series (2026-09-15): no time-series type, so one collection with `ts` as epoch milliseconds and a persistent index over `["host", "ts"]`, SQLite's footing; buckets by `DATE_TRUNC(d.ts, 'minute')` / `DATE_TRUNC(d.ts, 'hour')`, which return ISO strings the digest reads as instants. | | Memgraph | `memgraph/memgraph@sha256:4710bee1…` (`memgraph:3.13.1`, the latest self-hosted stable on 2026-09-17; `latest` resolves to the same digest). License BSL 1.1 for the engine (`memgraph/memgraph` LICENSE), with Memgraph's own Enterprise features gated by a license key that the arm never has; not open source | 3.13.1, IN_MEMORY_TRANSACTIONAL (the documented default: the graph in RAM, WAL and periodic snapshots on disk) | graph; dense (#131, from the 26.10.1 measurement); cross-model (its vector index, from the 26.10.1 measurement, DECISIONS #131 item 4: the index takes no HNSW parameters, so its degree and breadth stay at the engine's defaults, disclosed; candidates ranked by `vector_search.cosine_similarity`, which is L2 order on the lane's unit vectors) | served, Bolt through the same `neo4j==6.3.1` driver the Neo4j arm uses | UNWIND batches over bolt, the Neo4j arm's statements verbatim; every query on the lane runs unchanged and every digest matched Neo4j's at micro. Host-sized or unbounded defaults (Bolt workers, snapshot threads, `--memory-limit`, the query timeout, telemetry) are set from the cell and read back onto the row (PROTOCOL.md section 7, FAIRNESS F6). Durability from `SHOW CONFIG`: WAL fsynced every 100,000 transactions by default; the strict class sets 1 (both straced). Memory at SF10: about 1.1 KiB per stored object measured at 10k persons / 207k edges, so 2.3 GiB for the graph inside the tier's 24g. Dense (`memgraph_dense`, 2026-10-02): the same image and flags, plus `--storage-access-timeout-sec=172800` since #344 (PROTOCOL.md section 7); `CREATE VECTOR INDEX` (l2sq, f32) after an UNWIND load. Its HNSW parameters are the engine's and fixed: 3.13.1 builds every index from a default USearch 2.21.4 config (connectivity 16, doubled at the base, so hnswlib's M=16; expansion_add 128; expansion_search 64) and ignores any other config key, so ef_construction runs at 128 against the lane's 100 (disclosed on the row) and ef_search is matched by asking the index for 100 results and keeping 10, as the Neo4j arm does. An int8 arm, `memgraph_dense_int8` (DECISIONS #135, from the 26.10.1 measurement), sets `scalar_kind` `i8` in place of `f32`; both read the kind back from `SHOW VECTOR INDEX INFO` onto the row (`memgraph_vector_scalar_kind`). | @@ -48,9 +48,9 @@ Pins follow DECISIONS #87 (the latest self-hosted stable release at campaign tim | Engine | Pin | amd64 manifest digest | |---|---|---| | PostgreSQL (and `postgres_tuned`) | 18.6 | `sha256:7341002d2b8c…` | -| pgvector | 0.8.6-pg18 | `sha256:1d50c689b0a6…` | +| pgvector | 0.8.7-pg18 | `sha256:ad249f9fb957…` | | PostgreSQL + pgvector + AGE | PG18 + AGE 1.8.0 | built image, base as pgvector | -| TimescaleDB | 2.30.1-pg18 | `sha256:f7036933154c…` | +| TimescaleDB | 2.31.0-pg18 | `sha256:26e20a5e5a7a…` | | MongoDB Search (mongot) | 1.70.4 | `sha256:24c0cd30baf0…` | | Neo4j | 2026.08.1-community | `sha256:e702d6b535d9…` | | Qdrant | v1.19.1 | `sha256:0699e7733a6f…` | @@ -64,7 +64,7 @@ Pins follow DECISIONS #87 (the latest self-hosted stable release at campaign tim | Elasticsearch client | `elasticsearch==9.5.1` | client package | | Neo4j driver (Neo4j AND Memgraph arms) | `neo4j==6.3.1` | client package | | Neo4j driver Rust extension (the vendor's documented accelerator, DECISIONS #173) | `neo4j-rust-ext==6.3.1.0` | client package | -| pgvector Python adapter (binary COPY, DECISIONS #174) | `pgvector==0.5.0` | client package | +| pgvector Python adapter (binary COPY, DECISIONS #174) | `pgvector==0.5.1` | client package | | ArcadeDB official Python driver (sync, httpx; the served ArcadeDB arms' headline client, CAMPAIGN section 7 row 72) | `arcadedb-driver==0.2.0` | client package | **Held below the newest, and why.** diff --git a/benchmarks/experiments/Dockerfile.pgage b/benchmarks/experiments/Dockerfile.pgage index 93e2f8f8d6a..ff8d652b23d 100644 --- a/benchmarks/experiments/Dockerfile.pgage +++ b/benchmarks/experiments/Dockerfile.pgage @@ -1,4 +1,4 @@ -# PostgreSQL 18 + pgvector 0.8.6 + Apache AGE 1.8.0: the "one engine" rival to +# PostgreSQL 18 + pgvector 0.8.7 + Apache AGE 1.8.0: the "one engine" rival to # the cross-model transaction (pgvector hit, Cypher hop through AGE, row update, # one transaction). Base image pinned by digest; AGE from the PGDG apt repo the # postgres image already carries. Built as dbbench:pg-age by build_images.sh. @@ -24,6 +24,6 @@ # measured on (`dpkg-query` on dbbench:pg-age: 1.8.0~rc0-2.pgdg12+1, which the # rows record as age:1.8.0). If PGDG stops serving it, the build fails here # rather than moving AGE silently. -FROM pgvector/pgvector@sha256:1d50c689b0a6511b9ea0a15615281c81a59fd04a08eb35057ec8646fb3a2118a +FROM pgvector/pgvector@sha256:ad249f9fb9572979643a47694e59b9a8cdcdf6abc6accebf49bb80ad5f20ff9b RUN apt-get update && apt-get install -y --no-install-recommends postgresql-18-age=1.8.0~rc0-2.pgdg12+1 \ && rm -rf /var/lib/apt/lists/* diff --git a/benchmarks/experiments/QUANTIZATION.md b/benchmarks/experiments/QUANTIZATION.md index 57ddb868afc..14c4401484d 100644 --- a/benchmarks/experiments/QUANTIZATION.md +++ b/benchmarks/experiments/QUANTIZATION.md @@ -17,7 +17,7 @@ The quantization survey of DECISIONS #131 item 8 (CAMPAIGN.md section 7 row 42), | LanceDB 0.39.0 | `IVF_HNSW_FLAT` (unquantized HNSW in IVF partitions), `IVF_HNSW_SQ`, `IVF_HNSW_PQ`, `IVF_FLAT`, `IVF_SQ`, `IVF_PQ`, `IVF_RQ` | `lancedb_dense`: `IVF_HNSW_SQ`, INT8 (it kept its name, so its history stays one series); `lancedb_dense_fp32`: `IVF_HNSW_FLAT`, built 2026-10-02. The old premise (SQ as LanceDB's only HNSW offering) is gone from the adapter, runner, and paper-table comments | the SQ arm is the int8 arm; the fp32 arm now exists. Read back: `lancedb_index_type` (`index_stats`) and `lancedb_index_details` (`list_indices`: HNSW 16/100, `compression` sq 8 bits on the SQ arm only). Recall fp32 / int8: 0.9999 / 0.9902 at 5,000, 0.9994 / 0.9871 at 20,000 | no arm: `IVF_HNSW_PQ` trains its codebook per IVF partition, not over the corpus, so it is a different PQ setting; the fp32 and SQ arms stay its comparison | image: each type passed to `create_index` on a 3,000-row table in `dbbench:dense`; `HNSW_FLAT` and `HNSW_SQ` (without IVF) are refused; smoke: micro, tiny | | Chroma 1.5.9 | hnswlib, fp32 only | fp32 | none shipped | none shipped | adapter (only `hnsw:*` collection metadata exists); docs | | DuckDB VSS (duckdb 1.5.4) | HNSW over `FLOAT[]` arrays; no quantization setting | fp32 | none shipped | none shipped | image: `duckdb_settings()` in `dbbench:duckdb` lists only `hnsw_ef_search` and `hnsw_enable_experimental_persistence` | -| pgvector 0.8.6 | `vector` (fp32), `halfvec` (fp16), `bit` (binary, by an expression index over `binary_quantize`), `sparsevec` | fp32 | no int8 type; fp16 and binary exist but are not int8, so no arm under #53 | none shipped (`halfvec` and `bit` are not PQ) | docs | +| pgvector 0.8.7 | `vector` (fp32), `halfvec` (fp16), `bit` (binary, by an expression index over `binary_quantize`), `sparsevec` | fp32 | no int8 type; fp16 and binary exist but are not int8, so no arm under #53 | none shipped (`halfvec` and `bit` are not PQ) | docs | | Neo4j 2026.08.1 | `vector.quantization.type` `NONE`, `SCALAR`, `BINARY`; **unset means `BINARY`**, with `vector.default_search_expansion_factor` 3.0 | until 2026-10-02 **recorded fp32, ran BINARY**: `neo4j_dense` and `neo4j_e2` created their index without the key (BUGS F164). Now `neo4j_dense` and `neo4j_e2` set `NONE`, and both read the applied type back (`neo4j_vector_quantization`) | built: `neo4j_dense_int8` sets `SCALAR` (repin-prep `32c0a011da`). Each type brings its own `vector.default_search_expansion_factor` (NONE 1.0, SCALAR 1.5, BINARY 3.0 on 2026.08.1 and 2026.09.0, the candidate multiple searched before re-scoring); the lane matches search effort at ef 100 (Neo4j asks for 100 candidates and keeps 10), so both arms set 1.0 and read it back (`neo4j_vector_search_expansion`), as the Elasticsearch int8 arm sets the minimum oversample. 2026.09.0 (the re-pin candidate, `neo4j@sha256:eaab5879…`) keeps the BINARY default (`.notes` repros/neo4j-202609-quant) | none shipped (`NONE`, `SCALAR`, `BINARY`) | image: `SHOW VECTOR INDEXES` on an index created with exactly our options reports provider `vector-2026.08` and `vector.quantization.type: "BINARY"`; `INT8` and `FLOAT32` are syntax errors | | MongoDB 8.2.12 + mongot-community 1.70.4 | `vectorSearch` index `quantization`: `none`, `scalar`, `binary` | `mongodb_dense`: `none` | built: `mongodb_dense_int8` sets `scalar`. Both read the field back from mongot's index definition (`mongot_vector_quantization`, `none` / `scalar`). Recall fp32 / int8: 0.9996 / 0.969 at 5,000, 0.9977 / 0.9699 at 20,000 | none shipped (`none`, `scalar`, `binary`) | adapter; smoke: micro, tiny (`scalar` accepted, built, queryable); docs for `binary` | | SurrealDB (core 2.3.10 embedded, 3.2.4 served) | HNSW `TYPE` F64, F32, I64, I32, I16: element types of the stored array, not quantization of float vectors | `TYPE F32` | none shipped | none shipped | adapter; docs | @@ -45,7 +45,7 @@ Smoke, 2026-10-21, laptop, the first 5,000 siftsmall vectors and 80 timed querie | Qdrant v1.19.1 | sparse `datatype` `float32` (default), `float16`, `uint8`, `turbo4` | `qdrant_sparse`: `float32`, now set in the definition | built: `qdrant_sparse_uint8`, `uint8`; both read the type back from the collection (`qdrant_sparse_datatype`). Recall fp32 / uint8: 1.0 / 1.0 on the 5,000-document synthetic corpus, 1.0 / 0.9952 on the 100,000-document Big-ANN SPLADE slice (once the arms' settle waits for every point, 2026-10-02: before that, both searched a collection still applying its last batches and read 0.93 and 0.88) | image: `qdrant_client.models.Datatype` in `dbbench:client`; smoke: synthetic micro, Big-ANN tiny | | Milvus v3.0.1 | `SPARSE_FLOAT_VECTOR` with `SPARSE_INVERTED_INDEX` | float32 weights | none documented | adapter; docs | | Elasticsearch 9.5.4 | `sparse_vector`; weights kept at reduced precision by the engine, no setting | the default | none to choose | docs | -| pgvector 0.8.6 | `sparsevec`, fp32 | fp32 | none shipped | docs | +| pgvector 0.8.7 | `sparsevec`, fp32 | fp32 | none shipped | docs | ## What follows diff --git a/benchmarks/experiments/build_images.sh b/benchmarks/experiments/build_images.sh index ec0958d338f..52b7130b787 100755 --- a/benchmarks/experiments/build_images.sh +++ b/benchmarks/experiments/build_images.sh @@ -66,7 +66,7 @@ declare -A PKGS=( # arm's mutation phase after a deleted vector key is re-inserted, and 0.21.1 # carries #1074 (COPY linear for compressible columns) and #1075 (the pool # sized from the cgroup limit). 0.21.2 (2026-10-01) is the newest stable. - [client]="requests==2.34.2 psycopg[binary]==3.3.6 $HARNESS_PKGS surrealdb==2.0.0 qdrant-client==1.19.1 pymilvus[bulk_writer]==3.0.2 minio==7.2.20 azure-storage-blob==12.31.0 ml-dtypes==0.6.0 elasticsearch==9.5.1 neo4j==6.3.1 ladybug==0.21.2 pymongo==4.18.1 python-arango==8.3.5 falkordb==1.7.1 redis==8.1.0 neo4j-rust-ext==6.3.1.0 pgvector==0.5.0 arcadedb-driver==0.2.0" + [client]="requests==2.34.2 psycopg[binary]==3.3.6 $HARNESS_PKGS surrealdb==2.0.0 qdrant-client==1.19.1 pymilvus[bulk_writer]==3.0.2 minio==7.2.20 azure-storage-blob==12.31.0 ml-dtypes==0.6.0 elasticsearch==9.5.1 neo4j==6.3.1 ladybug==0.21.2 pymongo==4.18.1 python-arango==8.3.5 falkordb==1.7.1 redis==8.1.0 neo4j-rust-ext==6.3.1.0 pgvector==0.5.1 arcadedb-driver==0.2.0" [dense]="chromadb==1.5.9 lancedb==0.39.0 sqlite-vec==0.1.9 duckdb==1.5.4 $HARNESS_PKGS" ) # A GUARD, not a comment. The dev pin above survived because nothing checked diff --git a/benchmarks/experiments/l2_graph.py b/benchmarks/experiments/l2_graph.py index a10c64b71e5..b0276dc2415 100644 --- a/benchmarks/experiments/l2_graph.py +++ b/benchmarks/experiments/l2_graph.py @@ -1736,7 +1736,7 @@ class PgAgeGraph(Base): """PostgreSQL with Apache AGE, served (DECISIONS #128): every graph read and write is the lane's own Cypher text, run through AGE's `cypher()` table function, on the `dbbench:pg-age` image the cross-model lane already - pins (PostgreSQL 18 + pgvector 0.8.6 + AGE 1.8.0~rc0; the row records the + pins (PostgreSQL 18 + pgvector 0.8.7 + AGE 1.8.0~rc0; the row records the versions the server reports). THE TEXTS RUN AS WRITTEN (laptop probe 2026-10-02, diff --git a/benchmarks/experiments/runner.py b/benchmarks/experiments/runner.py index 04d6ec045c9..c14cd120fa2 100644 --- a/benchmarks/experiments/runner.py +++ b/benchmarks/experiments/runner.py @@ -604,7 +604,7 @@ def _milvus_arm(dense): "timescaledb": { "topology": "client_server", "image": "dbbench:client", - "server_image": "timescale/timescaledb@sha256:f7036933154c52dbc500f7b08ff8e28404a8ccabbf8ce3528142cde6ab253eef", # 2.30.1-pg18 + "server_image": "timescale/timescaledb@sha256:26e20a5e5a7ab6bec7ff217a52ffe21c9f4799603816718b2a573e7be839d8b4", # 2.31.0-pg18 "server_env": ["-e", "POSTGRES_PASSWORD=dbbenchpass", "-e", "POSTGRES_DB=bench"], "server_cmd": ["-c", "synchronous_commit=off", "-c", "shared_buffers={sb}", "-c", "effective_cache_size={ecs}", "-c", "maintenance_work_mem={mwm}", "-c", "max_wal_size=4GB"] + PG_FIT_CMD, @@ -953,7 +953,7 @@ def _milvus_arm(dense): "pgage_graph": { "topology": "client_server", "image": "dbbench:client", - "server_image": "dbbench:pg-age", # PostgreSQL 18 + pgvector 0.8.6 + AGE 1.8.0, built from Dockerfile.pgage + "server_image": "dbbench:pg-age", # PostgreSQL 18 + pgvector 0.8.7 + AGE 1.8.0, built from Dockerfile.pgage "server_env": ["-e", "POSTGRES_PASSWORD=dbbenchpass", "-e", "POSTGRES_DB=bench"], "server_cmd": ["-c", "synchronous_commit=off", "-c", "shared_buffers={sb}", "-c", "effective_cache_size={ecs}", "-c", "maintenance_work_mem={mwm}", "-c", "max_wal_size=4GB"] + PG_FIT_CMD, @@ -980,7 +980,7 @@ def _milvus_arm(dense): "pg_age_e2": { "topology": "client_server", "image": "dbbench:client", - "server_image": "dbbench:pg-age", # PostgreSQL 18 + pgvector 0.8.6 + AGE 1.8.0, built from Dockerfile.pgage + "server_image": "dbbench:pg-age", # PostgreSQL 18 + pgvector 0.8.7 + AGE 1.8.0, built from Dockerfile.pgage "server_env": ["-e", "POSTGRES_PASSWORD=dbbenchpass", "-e", "POSTGRES_DB=bench"], "server_cmd": ["-c", "synchronous_commit=off", "-c", "shared_buffers={sb}", "-c", "effective_cache_size={ecs}", "-c", "maintenance_work_mem={mwm}", "-c", "max_wal_size=4GB"] + PG_FIT_CMD, @@ -1429,7 +1429,7 @@ def _milvus_arm(dense): "server_port": 6333, "ready_regex": r"Qdrant (HTTP|gRPC) listening|Actix runtime found", }, - # pgvector 0.8.6 on PostgreSQL 18. maintenance_work_mem at half the cap: + # pgvector 0.8.7 on PostgreSQL 18. maintenance_work_mem at half the cap: # the HNSW build spills to a slow path when the graph outgrows it, and at # the 64 MB default a 10M build does not finish inside the envelope. # shared_buffers a quarter of the cap so the index can be resident, the @@ -1437,7 +1437,7 @@ def _milvus_arm(dense): "pgvector_dense": { "topology": "client_server", "image": "dbbench:client", - "server_image": "pgvector/pgvector@sha256:1d50c689b0a6511b9ea0a15615281c81a59fd04a08eb35057ec8646fb3a2118a", # 0.8.6-pg18 + "server_image": "pgvector/pgvector@sha256:ad249f9fb9572979643a47694e59b9a8cdcdf6abc6accebf49bb80ad5f20ff9b", # 0.8.7-pg18 "server_env": ["-e", "POSTGRES_PASSWORD=dbbenchpass", "-e", "POSTGRES_DB=bench"], "server_cmd": ["-c", "synchronous_commit=off", "-c", "shared_buffers={sb}", "-c", "effective_cache_size={ecs}", "-c", "maintenance_work_mem={mwm}", "-c", "max_wal_size=8GB"] + PG_FIT_CMD, @@ -1448,7 +1448,7 @@ def _milvus_arm(dense): "pgvector_sparse": { "topology": "client_server", "image": "dbbench:client", - "server_image": "pgvector/pgvector@sha256:1d50c689b0a6511b9ea0a15615281c81a59fd04a08eb35057ec8646fb3a2118a", # 0.8.6-pg18 + "server_image": "pgvector/pgvector@sha256:ad249f9fb9572979643a47694e59b9a8cdcdf6abc6accebf49bb80ad5f20ff9b", # 0.8.7-pg18 "server_env": ["-e", "POSTGRES_PASSWORD=dbbenchpass", "-e", "POSTGRES_DB=bench"], "server_cmd": ["-c", "synchronous_commit=off", "-c", "shared_buffers={sb}", "-c", "effective_cache_size={ecs}", "-c", "maintenance_work_mem={mwm}", "-c", "max_wal_size=8GB"] + PG_FIT_CMD, diff --git a/benchmarks/experiments/stage_drafts_2611.sha256 b/benchmarks/experiments/stage_drafts_2611.sha256 index d70d2b55c3a..4e61a1bc010 100644 --- a/benchmarks/experiments/stage_drafts_2611.sha256 +++ b/benchmarks/experiments/stage_drafts_2611.sha256 @@ -14,19 +14,19 @@ ffa1ce8fcf59cdef2db6fdac12a7fb62026f344343f1474c8d63f3118a1ee794 arcadedb-relea 741d0a3518c3ba90666afa9495ac21148644423cb997c96a4fae5481753abcd8 comparators-paper-5reps/qU02.sh a41a74e6d6ff0733b3e59f5c8a1efbb2160133639614a5d71f424410a7762dff comparators-paper-5reps/qU03.sh f28ff18b4a830df753e2087e0103e5b294f83d5ad370ca5c5e2f059bd7177727 comparators-paper-5reps/qU04.sh -b4b3b9514ade64c245a0e2ae199e635a4b4a5cece481cf4906704189a4630f3c comparators-paper-5reps/qU05.sh -0caad6f919a1a19db5ad04b5397608bdf8973e292e0be1adf65026a37d39c72b comparators-paper-5reps/qU06.sh -75d37fe01be94e4bac59eca71310f59bde5753e8d0988234d3d18fbae08a16eb comparators-paper-5reps/qU07.sh +dd799f7477c67a4e5556a5364fbc66997ec1545740a115b36eba72a3393dcf24 comparators-paper-5reps/qU05.sh +265281756435dc8129db19f856724645cb94e4f2909722cd83c2bbd9cee01686 comparators-paper-5reps/qU06.sh +a6ea4d4d852e88248e2f927728d7b403ca4144f98977af2b29570b583d40f9cd comparators-paper-5reps/qU07.sh 724104505b46dba372beba53b626173ea6e3bf607171d97dcd707bbe3b140729 comparators-paper-5reps/qU08.sh 3bbaa1c5b2c2d319e917a4c8688d99b3a9531ee9c3e352eb1421e38b156ec151 comparators/qS01.sh 6d4b10f667c3c2dd5a42c04e47388b7468b842a13c4afd3b7393282eb07b2bbe comparators/qS02.sh 225d969b481a930a79097879df676eef1ae1385d3d3d78e4a8e8d3cca6048ba7 comparators/qS03.sh -2836bd3d69923b3fa52e38d5746244811752eb67c75441b43ae9a21fb964ac59 comparators/qS04.sh +7d20043a65ee244ea2217af31570b3f576ceaf4b378230a16addf4f1e8ee79b9 comparators/qS04.sh 7f5f6878f8e4e121d5c107e9cc5d0976f53d42b517190e471c97de18ed0a71a6 comparators/qS05.sh -3f13190d57232af92e181b62ad4a11d7967625c0af79db1d883e0004d40377dd comparators/qS06.sh +15aad98c816ddefe96767cc1f70d1d2a8ec6c9c0581c940dca5ceb3149a90646 comparators/qS06.sh a73dae1fa5c57e808886e1dc19b44d090477e5eb4803d238137144f400057c19 comparators/qS07.sh d86cb9af49ff51f26713f96d8346cbfc85673976378fa934c7f68e042c7797cb comparators/qS08.sh a444bcb39221f9d38192f87288d0aad0279d48b568b32c6d7ad77cc72db70518 comparators/qS09.sh 902f580f2ebdf2aee403697cd9e6909f0a4135df838cf214ff979079cadc86fd comparators/qS10.sh -d5fada14d1f3336fbf69a83c24932d7d9b487d298dc7f84665494acb3215301e comparators/qS11.sh -f47edb43082f1c74888db94c06c98ed07aaa56ee667f4de54d513e43994e74a6 comparators/qS12.sh +3864eb5fe2f24cc3d8996ca7079b927b33ee2edd30ff328210fc5f89512d9781 comparators/qS11.sh +1e06c05db299ddc369f8ebe27b58cce4d1ea63c182998a726b8696e64d148126 comparators/qS12.sh diff --git a/benchmarks/experiments/test_sparse_mp_versions.py b/benchmarks/experiments/test_sparse_mp_versions.py index 51b0f24db06..b32087490d7 100644 --- a/benchmarks/experiments/test_sparse_mp_versions.py +++ b/benchmarks/experiments/test_sparse_mp_versions.py @@ -24,7 +24,7 @@ @pytest.mark.parametrize("backend,adapter_says", [ ("qdrant_sparse", "qdrant:1.19.1"), ("qdrant_sparse_uint8", "qdrant:1.19.1"), - ("milvus_sparse", "milvus:3.0.1"), ("pgvector_sparse", "pgvector:0.8.6 on 18"), + ("milvus_sparse", "milvus:3.0.1"), ("pgvector_sparse", "pgvector:0.8.7 on 18"), ("elasticsearch_sparse", "9.5.4"), # the one adapter that reports the bare number ]) def test_a_comparator_pass_keeps_a_lib_version_that_starts_with_its_engine(backend, adapter_says):