This repository was archived by the owner on Nov 21, 2026. It is now read-only.
Repository navigation
Bindings: PRODUCT quantization after ArcadeData/arcadedb#10545 (SQL path, find_nearest, new index settings, docs) - #446
Merged
Merged
Conversation
find_nearest takes the engine entry point SQL vectorNeighbors takes on a PRODUCT index (walk on the PQ codes, re-rank on the stored vectors); create_vector_index passes rerankOversample and fusedPQ and get_metadata reports them; the vector guide documents the CREATE INDEX METADATA statement and the per-query rerank option with the measured size and recall, and drops the claim that PQ gives no speedup and writes larger files. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…#10545 (#443) CI builds its wheel from the 26.11.1-SNAPSHOT image, which was last published before ArcadeData#10545 and carries the same version string. A fixture probes for the rerankOversample setting; without it the tests assert what the old engine does, find_nearest keeps the exact walk, and get_metadata leaves out the two new keys. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This was referenced Oct 22, 2026
test_jvm_payload flags .tolist() before to_java_float_array(): the helper takes the array and crosses it in one copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #443
Brings the bindings in line with ArcadeData#10545 (engine issue ArcadeData#10514), already synced into main. DSL-first: the statements are documented and tested, and the Python surface changes only where it has to.
find_nearest decision.
VectorIndex.find_nearestcalledfindNeighborsFromVector, the exact walk, so on a PRODUCT index it disagreed with SQLvectorNeighbors. It now callsfindNeighbors, the entry point SQL, Cypher and the select share, on a PRODUCT index (walk on the PQ codes, re-rank the bestk * rerankOversamplecandidates on the stored vectors, exact distances returned); every other quantization keeps the exact walk. I did not route it through SQL: the wrapper takes an RID set and the engine call is the same one SQL makes, so one index answers the same way through both doors. It gets norerankargument: a per-query value is the SQLrerankoption.find_nearest_approximatestays and its docs now say it is the PQ scores alone, with no beam and no re-rank.create_vector_index and get_metadata. Two new keyword arguments,
rerank_oversampleandfused_pq, ride the metadata JSON the helper already sends (rerankOversample,fusedPQ) and requirequantization="PRODUCT"like the other PQ arguments;get_metadata()returnsrerank_oversampleandfused_pq. NegativererankOversampleis refused by the engine.Docs.
guide/vectors.mdgains "Product quantization (PQ) in SQL": theCREATE INDEX ... METADATAstatement with every PQ key mapped to its helper argument, thererankoption (also in the "Search Options in SQL" table), and the measured numbers. The "PQ is not recommended: no speedup, files larger" text from #395 is gone from the guide and fromapi/database.md. The numbers are SIFT-1M, 32 subspaces, on ArcadeData#10545's head: PRODUCT index files 170.0 MiB against 144.9 MiB for NONE (1.17x, 7.9x before the fix); recall@10 at efSearch 10/50/100/200/400 is NONE .764/.965/.990/.997/.999, PRODUCT with the default rerankOversample 10 .986/.986/.986/.996/.999, rerankOversample 0 .617/.713/.720/.722/.722. No speed claim is made: no ab.py timing of PQ search exists yet, and the guide says so.api/vector.mdanddevelopment/testing.mdare updated. No example uses PQ beyond passing--quantization PRODUCTthrough to the SQL statement, and known-issues does not list PQ.Tests. New
tests/test_vector_product_quantization.py(13 tests): the METADATA settings reach the index and default to 10 / off, a negative value is refused, a default search returns the exact distances with recall >= 0.9,rerank: 0gives PQ estimates and lower recall, a negativererankis refused, afusedPQindex returns k results,find_nearestequals SQL on a PRODUCT index, and the helper arguments and metadata fields. Push and pull_request runs of the bindings workflow build their wheel from thearcadedata/arcadedb:26.11.1-SNAPSHOTimage, last published before ArcadeData#10545 (upstream's deploy is broken, ArcadeData#10530), with the same version string as an engine that has it. So a fixturehas_pq_rerankprobes whether the engine accepts thererankOversamplesetting, and on an engine without ArcadeData#10545 the tests assert the old answer (settings andrerankrefused,find_nearestand SQL both the exact walk), with no skips, astest_vector_sql.pydoes forwhereand #445 does for ArcadeData#10448. The old-answer branches go once 26.11.1 ships. For the same reasonfind_nearestkeeps the exact walk when the index has nofindNeighbors, andget_metadata()leaves outrerank_oversampleandfused_pqon such an engine.Local results, tests on cores 12-15: on the synced 26.11.1.dev0 engine with this branch's
src/the new file and the existing vector tests pass (121); on the 26.10.2 engine with this branch'ssrc/they pass (106, old-answer branch); with main'ssrc/on the new engine 5 fail (thefind_nearesttest and the four helper-argument tests). Strict mkdocs build and pre-commit pass. Run of the bindings workflow dispatched on this branch withengine-from-source=true, which shows the new-engine path: https://github.com/humemai/arcadedb-embedded-python/actions/runs/39704687271🤖 Generated with Claude Code