build: bump lance to v11.0.0 - #77
Merged
Merged
Conversation
Summary: Intent: - Move the lance git pins from e934cc2c to ab6b5bbe (lance v11.0.0 release tag) so lance-c tracks a released upstream version instead of an arbitrary commit. - Pick up the v11 blob APIs (read_blob_ranges, Option-based take_blobs results) needed to answer #76 without a second pin bump. Changes: - Point all lance, lance-core, lance-file, lance-index, lance-io, lance-linalg, lance-table, lance-datafusion, and lance-datagen dependencies at ab6b5bbe. - Re-resolve Cargo.lock; blake3, jiff, and reqwest 0.13 were unlocked explicitly because v11 raised their minimum versions, the rest follows from lance v11 (opendal 0.58, lance-namespace-reqwest-client 0.11, etc.). - Adapt to upstream signature changes: build_global_bm25_scorer takes an optional metrics collector, MatchQueryExec/PhraseQueryExec::new_with_segments are now fallible, DataFile::new takes a ConcreteFileVersion, and pb::IndexMetadata gained covering_fields. - Keep the DOT PQ strict-subset guard; upstream make_global_pq is unchanged at v11, so only the referenced revision in the comment and error text moved. Test Plan: - cargo fmt, cargo check --all-targets, cargo clippy --all-targets -D warnings. - cargo test: 367 passed, 0 failed, 2 ignored. - cargo test --test compile_and_run_test -- --ignored: 2 passed (C and C++ compile-and-run against the rebuilt library). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This was referenced Sep 17, 2026
jja725
pushed a commit
that referenced
this pull request
Sep 17, 2026
Second PR in the #55 sequence (PR 1 / E1+E2 was #57): the C-side commit half of the distributed build loop. ## Summary Adds `lance_dataset_commit_index_segments`: commits previously built uncommitted segments (protobuf-encoded `pb::IndexMetadata`, as produced by `lance_index_segment_builder_execute_uncommitted`) as one logical index in a single dataset version — the same loop the Java SDK exposes as `Dataset.commitExistingIndexSegments`. Thin FFI over `DatasetIndexExt::commit_existing_index_segments` (lance v11.0.0 `ab6b5bbe`, `rust/lance/src/index.rs:1995`); the prost decode + range validation is shared with `lance_index_segment_metadata_parse` via a common `decode_segment_metadata()` helper. Segment-set validation (empty set, duplicate UUIDs, overlapping fragment coverage, keyed-field mismatch) is left to the core; the FFI boundary validates NULL/empty inputs with descriptive errors. ## Deviation from the issue §6 sketch: no `replace` flag The sketch predates the v11 bump (#77). At v11 there is no `replace` parameter — replacement is automatic and coverage-driven, and the header documents the exact semantics (verified against `index.rs:2077-2130` and locked by tests): - Same-type, same name: existing segments whose fragment coverage is fully covered by the incoming set are replaced; disjoint-coverage segments are retained as deltas; partial overlap (orphaning fragments) is rejected. - Type change: a full-coverage commit whose index type differs from the existing same-name index replaces that index entirely; a partial-coverage type change is rejected. - Every segment must declare `column` as its keyed field (i.e., have been built for `column`); mismatched or unknown columns fail. ## Bindings - C: declaration + docs in `include/lance/lance.h`. - C++: `Dataset::commit_index_segments(index_name, column, segment_metadata)` in `include/lance/lance.hpp`. - Rust FFI: `src/index_segment.rs`, alongside the PR-1 segment APIs. ## Tests 9 new Rust tests in `tests/c_api_test.rs`: multi-segment happy path (2 IvfFlat segments over disjoint fragments → one version bump → k-NN through the committed multi-segment index), duplicate UUIDs, overlapping coverage, malformed/truncated metadata, 9 NULL/empty boundary cases, unknown column, wrong column (keyed-field mismatch), full-coverage replacement, and type-change (full-coverage replace + partial-coverage rejection). Plus C and C++ compile-and-run tests in `tests/cpp/`. - `cargo test`: 319 passed, 0 failed (all suites) - `cargo clippy --all-targets -- -D warnings` / `cargo fmt --check`: clean - `cargo test --test compile_and_run_test -- --ignored`: 3 passed Does not close #55 — PR 3 (E3 progress) and PR 4 (E6 listing) remain tracked there. ## Open question carried from #55 Physical segment merge (`Dataset::merge_existing_index_segments`) is still not exposed; per the tracker its exposure scope is to be decided alongside this PR. ## Update: vector segment compatibility (review follow-up) The pinned Lance validator compared only `index_details.type_url`, so mixed-metric vector segments could be committed as one logical index and silently ranked under the wrong metric. Lance core now enforces the query path's `validate_vector_query_compatibility` (metric, dimension, sub-index type, quantizer kind) across the coexisting segment set — incoming plus retained existing segments, after replacement selection — rejecting incompatible commits with `InvalidInput` before any manifest change. Companion Lance PR: lance-format/lance#9351 (targets `release/v11.0`); the pin moved from `ab6b5bbe` (v11.0.0) to `356acb0d`, the head of that PR, and will move to the upstream merge commit once it lands. New tests: mixed-metric rejection in a single commit and against a retained segment (version/manifest untouched, retained segment still answers indexed k-NN identically), same-metric delta commit, and full-coverage metric replacement. Docs in `lance.h`/`lance.hpp` describe the compatibility rule.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Intent:
e934cc2ctoab6b5bbe, which is the lance v11.0.0 release tag, so lance-c tracks a released upstream version instead of an arbitrary commit.read_blob_ranges,Option-basedtake_blobsresults) needed to answer Blob v2: how to read payload bytes from C? #76 without a second pin bump.Changes:
ab6b5bbe.Cargo.lock.blake3,jiff, andreqwest0.13 were unlocked explicitly because v11 raised their minimum versions; the rest follows from lance v11 (opendal 0.58, lance-namespace-reqwest-client 0.11, etc.).build_global_bm25_scorertakes an optional metrics collector,MatchQueryExec/PhraseQueryExec::new_with_segmentsare now fallible,DataFile::newtakes aConcreteFileVersion, andpb::IndexMetadatagainedcovering_fields.make_global_pqis unchanged at v11, so only the referenced revision in the comment and error text moved.Test Plan
cargo fmt,cargo check --all-targets,cargo clippy --all-targets -- -D warningsare clean.cargo test: 367 passed, 0 failed, 2 ignored.cargo test --test compile_and_run_test -- --ignored: 2 passed (C and C++ compile-and-run against the rebuilt library).🤖 Generated with Claude Code