Skip to content
This repository was archived by the owner on Nov 21, 2026. It is now read-only.

Bindings: PRODUCT quantization after ArcadeData/arcadedb#10545 (SQL path, find_nearest, new index settings, docs) - #446

Merged
tae898 merged 4 commits into
mainfrom
pq-bindings-10545
Oct 22, 2026
Merged

tae898 merged 4 commits into
mainfrom
pq-bindings-10545

Conversation

@tae898

@tae898 tae898 commented Oct 22, 2026 •

Copy link
Copy Markdown

Closes #443

Brings the bindings in line with ArcadeData#10545 (engine issue ArcadeData#10514), already synced into main. DSL-first: the statements are documented and tested, and the Python surface changes only where it has to.

find_nearest decision. VectorIndex.find_nearest called findNeighborsFromVector, the exact walk, so on a PRODUCT index it disagreed with SQL vectorNeighbors. It now calls findNeighbors, the entry point SQL, Cypher and the select share, on a PRODUCT index (walk on the PQ codes, re-rank the best k * rerankOversample candidates on the stored vectors, exact distances returned); every other quantization keeps the exact walk. I did not route it through SQL: the wrapper takes an RID set and the engine call is the same one SQL makes, so one index answers the same way through both doors. It gets no rerank argument: a per-query value is the SQL rerank option. find_nearest_approximate stays and its docs now say it is the PQ scores alone, with no beam and no re-rank.

create_vector_index and get_metadata. Two new keyword arguments, rerank_oversample and fused_pq, ride the metadata JSON the helper already sends (rerankOversample, fusedPQ) and require quantization="PRODUCT" like the other PQ arguments; get_metadata() returns rerank_oversample and fused_pq. Negative rerankOversample is refused by the engine.

Docs. guide/vectors.md gains "Product quantization (PQ) in SQL": the CREATE INDEX ... METADATA statement with every PQ key mapped to its helper argument, the rerank option (also in the "Search Options in SQL" table), and the measured numbers. The "PQ is not recommended: no speedup, files larger" text from #395 is gone from the guide and from api/database.md. The numbers are SIFT-1M, 32 subspaces, on ArcadeData#10545's head: PRODUCT index files 170.0 MiB against 144.9 MiB for NONE (1.17x, 7.9x before the fix); recall@10 at efSearch 10/50/100/200/400 is NONE .764/.965/.990/.997/.999, PRODUCT with the default rerankOversample 10 .986/.986/.986/.996/.999, rerankOversample 0 .617/.713/.720/.722/.722. No speed claim is made: no ab.py timing of PQ search exists yet, and the guide says so. api/vector.md and development/testing.md are updated. No example uses PQ beyond passing --quantization PRODUCT through to the SQL statement, and known-issues does not list PQ.

Tests. New tests/test_vector_product_quantization.py (13 tests): the METADATA settings reach the index and default to 10 / off, a negative value is refused, a default search returns the exact distances with recall >= 0.9, rerank: 0 gives PQ estimates and lower recall, a negative rerank is refused, a fusedPQ index returns k results, find_nearest equals SQL on a PRODUCT index, and the helper arguments and metadata fields. Push and pull_request runs of the bindings workflow build their wheel from the arcadedata/arcadedb:26.11.1-SNAPSHOT image, last published before ArcadeData#10545 (upstream's deploy is broken, ArcadeData#10530), with the same version string as an engine that has it. So a fixture has_pq_rerank probes whether the engine accepts the rerankOversample setting, and on an engine without ArcadeData#10545 the tests assert the old answer (settings and rerank refused, find_nearest and SQL both the exact walk), with no skips, as test_vector_sql.py does for where and #445 does for ArcadeData#10448. The old-answer branches go once 26.11.1 ships. For the same reason find_nearest keeps the exact walk when the index has no findNeighbors, and get_metadata() leaves out rerank_oversample and fused_pq on such an engine.

Local results, tests on cores 12-15: on the synced 26.11.1.dev0 engine with this branch's src/ the new file and the existing vector tests pass (121); on the 26.10.2 engine with this branch's src/ they pass (106, old-answer branch); with main's src/ on the new engine 5 fail (the find_nearest test and the four helper-argument tests). Strict mkdocs build and pre-commit pass. Run of the bindings workflow dispatched on this branch with engine-from-source=true, which shows the new-engine path: https://github.com/humemai/arcadedb-embedded-python/actions/runs/39704687271

🤖 Generated with Claude Code

tae898 and others added 2 commits October 22, 2026 09:01
find_nearest takes the engine entry point SQL vectorNeighbors takes on a PRODUCT index (walk on the PQ codes, re-rank on the stored vectors); create_vector_index passes rerankOversample and fusedPQ and get_metadata reports them; the vector guide documents the CREATE INDEX METADATA statement and the per-query rerank option with the measured size and recall, and drops the claim that PQ gives no speedup and writes larger files.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…#10545 (#443)

CI builds its wheel from the 26.11.1-SNAPSHOT image, which was last published before ArcadeData#10545 and carries the same version string. A fixture probes for the rerankOversample setting; without it the tests assert what the old engine does, find_nearest keeps the exact walk, and get_metadata leaves out the two new keys.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
tae898 and others added 2 commits October 22, 2026 09:39
test_jvm_payload flags .tolist() before to_java_float_array(): the helper takes the array and crosses it in one copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tae898
tae898 merged commit 23fb23c into main Oct 22, 2026
80 of 86 checks passed
@tae898
tae898 deleted the pq-bindings-10545 branch October 22, 2026 03:55
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bindings: PRODUCT quantization after ArcadeData/arcadedb#10545 (SQL path, find_nearest, new index settings, docs)

1 participant