Skip to content
This repository was archived by the owner on Nov 21, 2026. It is now read-only.

Benchmarks: Redis 8.10.2 joins the dense table (Query Engine HNSW, vendor settings, both durability classes) - #481

Merged
tae898 merged 4 commits into
mainfrom
bench/redis-comparator
Oct 23, 2026
Merged

tae898 merged 4 commits into
mainfrom
bench/redis-comparator

Conversation

@tae898

@tae898 tae898 commented Oct 23, 2026 •

Copy link
Copy Markdown

Adds Redis 8.10.2 (the official image, Redis Query Engine) as a comparator on the dense table, as the first of the two multi-model engines decided on 2026-10-23 (CrateDB follows in its own PR). Candidates and licence reading: .notes/bench/MULTIMODEL-CANDIDATES-20261023.md.

What it does to existing rows

Nothing. redis_dense is a new arm: runner.LANES["l3d"], l3d_dense.BACKENDS, a new redis engine in make_2611_stages.ENGINES. The default comparator stage drafts gain the Redis cells, so stage_drafts_2611.sha256 is rewritten (reviewed: only the l3d stages and the BACKENDS lines move). No published row is edited.

Lanes

Lane In / out Why
dense (l3d) in, fp32 FT.CREATE ... VECTOR HNSW (Query Engine). Same code path as every dense arm: build, id index (the record key is the id), cold pass, ef sweep 50/100/200/400 with EF_RUNTIME per query, matched-recall cells, insert and delete mutation at the 1M tier, per-phase memory probes, both durability classes
dense int8 out HNSW INT8/UINT8 only types vectors the client already quantized; SVS-VAMANA LVQ/LeanVec is Intel's and the vector reference says it is not in Redis Open Source
documents (l1tpc) out of this PR Expressible (JSON + FT.AGGREGATE; TPC-C new-order/payment would be Lua, atomic but with no rollback, so the OLTP cells would be declared out). Not built in this PR: the adapter and digest work did not fit the call budget; next PR
time series (l4) out of this PR Expressible (TS.MADD, TS.MRANGE). Not built in this PR; next PR
graph, cross-model out no graph model (RedisGraph ended; FalkorDB, already on the page, runs the graph)
sparse out Query Engine vector fields are dense only
lifecycle, restart out no in-process mode; restart lane roster unchanged

Settings: ours, the existing engines', the vendor's

Setting Our value Existing engines' equivalent Vendor doc
Image redis@sha256:13ad9044b4b2fd2abe4c5aad8c384b0504a71006fa33f023fc68998453a2eeca (8.10.2, amd64, equals redis:latest on 2026-10-23); search module 8.10.1 every comparator pinned by amd64 digest; FalkorDB 6.0.1 runs on the same Redis 8.10.2 https://hub.docker.com/_/redis
Client redis==8.1.0 (already in dbbench:client, newest on PyPI), protocol=2 (list replies) Qdrant: gRPC (the vendor's fast path); FalkorDB: same redis package https://redis.io/docs/latest/develop/clients/redis-py/
maxmemory 90% of the cell cap (7372 MB at 8g) Memgraph --memory-limit = {mem90_mib} redis.conf, MEMORY MANAGEMENT: https://raw.githubusercontent.com/redis/redis/8.10.2/redis.conf
maxmemory-policy noeviction (the documented default, set explicitly) none (no other dense engine evicts) same redis.conf
io-threads cpuset - 1 (7 on the 8-CPU smoke cell; 11 at 0-11); default is 1 FalkorDB THREAD_COUNT {ncpu}, Memgraph Bolt workers {ncpu} (F6) redis.conf, THREADED I/O ("four cores, 3 I/O threads; 8 cores, 7")
search-workers cpuset - 1; default 0 PostgreSQL parallel workers = cpuset - 1 https://redis.io/docs/latest/develop/ai/search-and-query/administration/configuration/#search-workers
protected-mode no (private cell network, no password) the other servers run without auth redis.conf, SECURITY
Index HNSW, TYPE FLOAT32, DISTANCE_METRIC L2, M 16 (hnswlib's; layer 0 holds twice that, degree_family hnswlib_m_doubled_at_base), EF_CONSTRUCTION 100 Qdrant m / ef_construct, FalkorDB M / efConstruction: the lane's COMPARATOR_M, EF_CONSTRUCTION https://redis.io/docs/latest/develop/ai/search-and-query/vectors/
Beam EF_RUNTIME in the query, the lane's ef sweep Qdrant hnsw_ef per query same page, runtime parameters
Load HSET pipeline of BATCH (10,000), no MULTI, index created before the load Qdrant upsert(Batch) of 10,000
Wait for the index INFO search: no active write thread and no index indexing (7 and 1 right after the load, 0 once drained), inside the build timer Qdrant status green; FalkorDB OPERATIONAL https://redis.io/docs/latest/commands/info/
Durability relaxed image default: appendonly no, save 3600 1 300 100 60 10000; read back with CONFIG GET FalkorDB (same Redis): identical string https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/
Durability strict --appendonly yes --appendfsync always; read back FalkorDB strict (same switches) same page
Disk reading relaxed cell runs redis-cli SAVE before the final reading (pre_disk_cmd) FalkorDB, same

Every override above is on the row (redis_io_threads, redis_search_workers, redis_maxmemory, redis_maxmemory_policy, redis_protocol, redis_search_module, durability, server_cmd) and in PROTOCOL.md section 7 and COMPARATORS.md. Equal answers (F11) on the dense lane are recall against the exact ground truth, as for every dense arm; no digest exists on this lane.

Smoke (laptop, Docker through quiet_build.sh, cores 8-15, micro and the 20k laptop corpus, then the 1M corpus)

All rows rc 0, into a scratch file (not committed). The laptop dense corpus holds 20,000 vectors, so micro, tiny and small all ran on 5,000 to 20,000 vectors; the real 1M corpus run was queued behind other agents' runner lock (one runner per host) and did not start before the call cap, so no 1M or 10M number exists yet.

Cell durability read back build_s recall@10 p50 ms (ef 100 / ef 400) notes
micro, relaxed appendonly=no, RDB save '3600 1 300 100 60 10000' (image default) 0.44 0.9996 0.179 / 0.313 degree_param 16, hnswlib_m_doubled_at_base, matched-recall cells reached
micro, strict appendonly=yes, appendfsync=always 1.17 0.9996 0.172 / 0.299 server_cmd carries the AOF flags; class on the row equals the class asked
20k corpus at tiny relaxed 0.78 0.9991 0.275 / 0.653
20k corpus at small (mutation tier) relaxed 0.75 0.9993 0.283 / 0.688 delete: 0 deleted hits, recall 0.9976; re-insert: recall 0.9972

Per-phase memory probes (load, index_build, warmup) are on the rows. fairness_check.py itself gates published rows of a stage and refuses a smoke file (it needs BENCH_ENGINE_COMMIT and the stage state), so its checks are held by the unit tests below, not by a run on the smoke rows.

Tests

uv run --no-project --with pytest --with numpy --with pandas --with pyarrow --with requests python -m pytest -q -p no:cacheprovider . in benchmarks/experiments: 1016 passed, 24 skipped (includes fairness_check and test_memory_probe.py, test_overrides.py, test_stage_generator_2611.py).

Hours at N=3 (an estimate, scaled from smoke timings)

From the smoke: build 26.7k vectors/s at 20k vectors, query p50 0.28 ms at ef 100. HNSW inserts slow with corpus size, so I assume 3 to 5x slower at 1M (150 to 250 s build) and 10 to 15x at 10M (1 to 1.5 h build, includes the 10M npy load and the 3.8 GB in the cap). With the multipass driver (one build, five timed passes) a rep is roughly 0.1 h at small and 1 to 1.5 h at deep10m. At N=3: about 3 to 5 h of mini for the dense lane (small about 0.3 h, deep10m about 3 to 4.5 h). This is an estimate, not a measurement: the 1M run must replace it. Documents and time series are not in this PR and add nothing yet.

🤖 Generated with Claude Code

tae898 and others added 4 commits October 23, 2026 16:09
…ndor settings, both durability classes)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898
tae898 merged commit 1801b6f into main Oct 23, 2026
4 checks passed
@tae898
tae898 deleted the bench/redis-comparator branch October 23, 2026 08:30
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant