[kamronis]

Summary

MinHash candidate selection scanned every active bucket head for each emitted document. For sparse searches with many buckets, this repeated work dominates.

• Select an adaptive heap merge after a bounded overlap sample, and use linear merging when a scan has fewer than 32 active buckets or window overlap makes the heap ineffective.
• Preserve heap heads across bounded window refills and reset selection state on seek, maintaining ordered document IDs and exact shared-band counts.
• Add sparse, dense, skewed, partially overlapping, and sparse/dense transition coverage. A real 300-document, 256-band, one-row-page index now verifies heap admission, production window refill, and ordered candidates/shared counts against an independent bands-file oracle.
• Add a Criterion benchmark for cold and prewarmed searches.

Performance

Same 20,000-document harness, current main, and the repository's release-with-debug profile. Two alternating runs per implementation; values are the range of Criterion time estimates.

Prewarmed case	Main	This change	Change
sparse_256 query	7.59–7.61 ms	3.32 ms	2.28–2.29x faster
default_16 query	6.65–6.79 us	7.11–7.76 us	+4.6% to +16.7%
dense_64 query	3.05–3.10 ms	3.09–3.20 ms	-0.5% to +4.7%

The stable gain is limited to the prewarmed sparse 256-band workload. The low-band default_16 query control repeated the same direction in both runs: 6.7927 -> 7.1054 us (+4.6%, +0.31 us) and 6.6546 -> 7.7645 us (+16.7%, +1.11 us). This is an accepted bounded cost in this fixture, not evidence of a universal default-config throughput regression. dense_64 varied with run order. No general or cold-query speedup is claimed. Measurements use local files and prewarmed caches, not physically cold disk or remote storage.

The selector was measured at 0b00653e6568414c836d322ea4afc43f028c1242; the later 134ba47cb1e1dd24f85f558c300f0a6fa35103b7 commit is test-only and does not enter the benchmark binary.

Validation

• cargo fmt --all -- --check
• cargo clippy --profile ci --locked --features <all workspace features> --all-targets -- -D warnings
• cargo test --profile ci -p lance-index --lib minhash_lsh — 63 passed

The new real-index regression admits the heap after the bounded sample, consumes a window supplied by the production refill path, and matches every ordered candidate and shared-band count with an independent bands-file oracle. A rejecting-mask cursor walk was also measured as a diagnostic, but is not presented here as whole-dataset throughput.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[zhangyue19921010]

Thanks for the PR. Two concerns:

1. Prefix sampling is unstable

• The strategy is chosen from the first HEAP_SAMPLE_CANDIDATES (64) candidates in doc-id order, and falling back to SelectionStrategy::Linear is one-way: the scan never goes back to the heap.
• A few dense candidates near the start therefore disable the heap for the whole scan. With 256 bands, 2 exact duplicates of the query already exceed the 2 × active_cursors threshold. In heap mode, ~8 full-match candidates in any 64-candidate window do the same.
• Near-duplicate clusters are the normal workload for this index.

2. Regressions in some cases

Prewarmed criterion runs, same text generator as benches/minhash_lsh.rs; each value is this PR's time ÷ main's time (<1 = faster):

case	PR / main
256/256, 20k docs, sparse	0.53x
same, first 8 rows = copies of the query	0.97x
128/64, 20k docs, sparse	1.01x (heap never engaged)
128/32, 20k docs	1.04x
default 128/16, 20k docs	1.10x (42.8 → 47.1 µs)

The default config can never use the heap (HEAP_MIN_BUCKETS = 32), yet it is consistently ~10% slower.

[kamronis]

@zhangyue19921010
Thanks for checking this! I changed the selection so a dense prefix no longer disables the heap for the rest of the scan. It can switch back when the data becomes sparse. Small scans also skip sampling and take a direct linear path.

With these changes, the 256-band case with eight duplicates at the start is about 2.1× faster. The default configuration and 64-band query are about the same. The 64-band full walk is still around 1.7% slower, down from roughly 4%, so there’s a small tradeoff remaining.

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The production selector is unchanged. The new sparse-index regression checks heap admission and consumed page refills against an independent bands-file oracle. Refreshed verification supports ordered candidates and exact shared-band counts through strategy changes, exhaustion, and seek.

Earlier paired prewarmed benchmarks showed roughly 2× improvement for the 256-band fixture with eight leading duplicates, with default query fixtures near parity. The acknowledged small 64-band full-walk overhead remains a non-blocking trade-off. Performance was not remeasured on this head; gains remain workload-dependent, and cold-query speedup is unestablished.

[Xuanwo]

Reviewed the latest MinHash selector, heap/window/seek state transitions, and the new real-index refill and posting-oracle regression. The retained can_remap behavior, remap tests, and benchmark registrations match current main. Local MinHash tests and workspace Clippy passed, and the applicable hosted checks are complete. The measured sparse-workload benefit and bounded default-control cost are documented.