This repository was archived by the owner on Nov 21, 2026. It is now read-only.
Repository navigation
Benchmarks: Redis 8.10.2 joins the dense table (Query Engine HNSW, vendor settings, both durability classes) - #481
Merged
Merged
Conversation
…ndor settings, both durability classes) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Oct 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Redis 8.10.2 (the official image, Redis Query Engine) as a comparator on the dense table, as the first of the two multi-model engines decided on 2026-10-23 (CrateDB follows in its own PR). Candidates and licence reading:
.notes/bench/MULTIMODEL-CANDIDATES-20261023.md.What it does to existing rows
Nothing.
redis_denseis a new arm:runner.LANES["l3d"],l3d_dense.BACKENDS, a newredisengine inmake_2611_stages.ENGINES. The default comparator stage drafts gain the Redis cells, sostage_drafts_2611.sha256is rewritten (reviewed: only the l3d stages and the BACKENDS lines move). No published row is edited.Lanes
FT.CREATE ... VECTOR HNSW(Query Engine). Same code path as every dense arm: build, id index (the record key is the id), cold pass, ef sweep 50/100/200/400 withEF_RUNTIMEper query, matched-recall cells, insert and delete mutation at the 1M tier, per-phase memory probes, both durability classesINT8/UINT8only types vectors the client already quantized;SVS-VAMANALVQ/LeanVec is Intel's and the vector reference says it is not in Redis Open SourceFT.AGGREGATE; TPC-C new-order/payment would be Lua, atomic but with no rollback, so the OLTP cells would be declared out). Not built in this PR: the adapter and digest work did not fit the call budget; next PRTS.MADD,TS.MRANGE). Not built in this PR; next PRSettings: ours, the existing engines', the vendor's
redis@sha256:13ad9044b4b2fd2abe4c5aad8c384b0504a71006fa33f023fc68998453a2eeca(8.10.2, amd64, equalsredis:lateston 2026-10-23); search module 8.10.1redis==8.1.0(already indbbench:client, newest on PyPI),protocol=2(list replies)redispackage--memory-limit={mem90_mib}noeviction(the documented default, set explicitly)THREAD_COUNT {ncpu}, Memgraph Bolt workers{ncpu}(F6)no(private cell network, no password)TYPE FLOAT32,DISTANCE_METRIC L2,M16 (hnswlib's; layer 0 holds twice that,degree_familyhnswlib_m_doubled_at_base),EF_CONSTRUCTION100m/ef_construct, FalkorDBM/efConstruction: the lane'sCOMPARATOR_M,EF_CONSTRUCTIONEF_RUNTIMEin the query, the lane's ef sweephnsw_efper queryBATCH(10,000), no MULTI, index created before the loadupsert(Batch)of 10,000INFO search: no active write thread and no index indexing (7 and 1 right after the load, 0 once drained), inside the build timerappendonly no,save 3600 1 300 100 60 10000; read back withCONFIG GET--appendonly yes --appendfsync always; read backredis-cli SAVEbefore the final reading (pre_disk_cmd)Every override above is on the row (
redis_io_threads,redis_search_workers,redis_maxmemory,redis_maxmemory_policy,redis_protocol,redis_search_module,durability,server_cmd) and in PROTOCOL.md section 7 and COMPARATORS.md. Equal answers (F11) on the dense lane are recall against the exact ground truth, as for every dense arm; no digest exists on this lane.Smoke (laptop, Docker through
quiet_build.sh, cores 8-15, micro and the 20k laptop corpus, then the 1M corpus)All rows
rc 0, into a scratch file (not committed). The laptop dense corpus holds 20,000 vectors, somicro,tinyandsmallall ran on 5,000 to 20,000 vectors; the real 1M corpus run was queued behind other agents' runner lock (one runner per host) and did not start before the call cap, so no 1M or 10M number exists yet.appendonly=no, RDB save '3600 1 300 100 60 10000' (image default)degree_param 16,hnswlib_m_doubled_at_base, matched-recall cells reachedappendonly=yes, appendfsync=alwaysserver_cmdcarries the AOF flags; class on the row equals the class askedtinysmall(mutation tier)Per-phase memory probes (load, index_build, warmup) are on the rows.
fairness_check.pyitself gates published rows of a stage and refuses a smoke file (it needsBENCH_ENGINE_COMMITand the stage state), so its checks are held by the unit tests below, not by a run on the smoke rows.Tests
uv run --no-project --with pytest --with numpy --with pandas --with pyarrow --with requests python -m pytest -q -p no:cacheprovider .inbenchmarks/experiments: 1016 passed, 24 skipped (includesfairness_checkandtest_memory_probe.py,test_overrides.py,test_stage_generator_2611.py).Hours at N=3 (an estimate, scaled from smoke timings)
From the smoke: build 26.7k vectors/s at 20k vectors, query p50 0.28 ms at ef 100. HNSW inserts slow with corpus size, so I assume 3 to 5x slower at 1M (150 to 250 s build) and 10 to 15x at 10M (1 to 1.5 h build, includes the 10M npy load and the 3.8 GB in the cap). With the multipass driver (one build, five timed passes) a rep is roughly 0.1 h at small and 1 to 1.5 h at deep10m. At N=3: about 3 to 5 h of mini for the dense lane (small about 0.3 h, deep10m about 3 to 4.5 h). This is an estimate, not a measurement: the 1M run must replace it. Documents and time series are not in this PR and add nothing yet.
🤖 Generated with Claude Code