feat(index): compose FRI mapping readers with planner safety boundaries

[LuQQiu]

What this adds

This PR adds the dataset-level reader for FRI index version 1. Datasets without FRI and datasets using version 0 continue through the existing legacy read path.

FRI index metadata
        │
        ▼
decode validated ledger
        │
        ▼
bind one MappingReader per supported transition
        │
        ▼
compose mappings across fragment lineage
        ├── derive query coverage
        └── translate physical row IDs

Ordered-compaction transitions use the existing bitmap/rank mapping. Stable-partition transitions bind their immutable Lance row-map file and load label blocks only when translation reaches that transition.

Coverage planning

Stored index bitmaps describe the fragments from which each index segment was built. The reader groups segments by logical index, combines their usable source coverage, and walks lineage to determine which current destination fragments are completely covered.

The planner receives an in-memory copy of each segment’s derived coverage; persisted index metadata is not changed. Empty coverage is removed so the query scans those fragments. Complete coverage for one destination is retained even when another destination cannot be covered.

When a newer segment directly covers an intermediate or destination fragment, older segments stop contributing along that path. The same exclusion is applied during address translation to avoid duplicate results when lineage branches reconverge.

Address translation

For each physical row ID, the reader follows the relevant mappings forward until the address reaches a current live fragment. Deleted rows return None. Paths containing an unsupported mapping cannot claim derived coverage and fall back to scanning.

Cache and compatibility boundaries

Decoded immutable histories are cached by FRI UUID. Mapping readers can be reused across history updates through their content fingerprint and storage binding, while live fragments remain specific to the dataset snapshot.

The first version-1 FRI commit sets and preserves the paired reader/writer feature flag. Manifest publication verifies that version-1 metadata and the flags agree. Writers and maintenance operations that cannot preserve version-1 history fail with an upgrade error. Restore preserves the selected snapshot’s index metadata and external references.

This PR does not modify scalar or vector index loaders. Until consumer integration in #9107, segments requiring version-1 address translation are excluded and their fragments are scanned.

Validation

Tests cover legacy-path isolation, inline and external history, multi-step and branching lineage, partial destination coverage, direct destination coverage, duplicate suppression, deleted rows, unsupported mappings and versions, cache reuse and snapshot isolation, feature-flag publication, restore and clone boundaries, and rejection of unsupported maintenance operations.

Validated with workspace Clippy and cargo fmt --all.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

Keep table-level system metadata outside fragment-coverage filtering, and reserve coverage remapping for queryable user-index segments. Also move the existing unknown-flag fixtures above the new supported bit. These two changes preserve MemWAL availability on tagged snapshots and keep the existing unsupported-feature guards meaningful.

[lance-gatekeeper]

This branch also sends the __lance_mem_wal system entry through the fragment-coverage groups. MemWAL metadata intentionally has fragment_bitmap: None, so may_need_translation(None) is true and the entry is silently omitted at lines 60–63. Every MemWAL API built on load_index_by_name then sees an initialized dataset as uninitialized and refuses writers. Preserve the MemWAL system entry unchanged, or otherwise restrict coverage rewriting to queryable user-index segments.

Reproducer

I added this regression beside the existing FRI reader tests, using their fixture, prepare, and install helpers:

let mut dataset = fixture().await;
let mem_wal = lance_table::system_index::mem_wal::new_mem_wal_index_meta(
    dataset.manifest.version,
    Default::default(),
).unwrap();
dataset.apply_commit(
    Transaction::new(
        dataset.manifest.version,
        Operation::CreateIndex {
            new_indices: vec![mem_wal],
            removed_indices: vec![],
        },
        None,
    ),
    &Default::default(),
    &Default::default(),
).await.unwrap();
assert!(dataset.load_index_by_name(
    lance_table::system_index::mem_wal::MEM_WAL_INDEX_NAME,
).await.unwrap().is_some());

let (transition, destinations) = prepare(&dataset).await;
let content = InlineContent {
    legacy_versions: vec![],
    transitions: vec![transition],
}.encode_to_vec();
install(&mut dataset, content, destinations, false).await;
assert!(dataset.load_index_by_name(
    lance_table::system_index::mem_wal::MEM_WAL_INDEX_NAME,
).await.unwrap().is_some(), "tagged FRI must not hide the MemWAL system index");

Command: cargo test -p lance index::frag_reuse_reader::tests::tagged_history_keeps_mem_wal_system_index_visible -- --exact

The first lookup succeeded; the final assertion failed with tagged FRI must not hide the MemWAL system index.

[lance-gatekeeper]

FLAG_UNKNOWN << 1 is no longer an unsupported sentinel: it is exactly this newly supported bit (512). Three existing unsupported-writer fixtures still use that expression, so they now construct a one-sided FRI flag pair instead of an unknown feature. In both clone regressions, manifest setup returns CorruptFile before the tests can exercise the intended NotSupported clone boundary. Move all of those fixtures to a genuinely unsupported bit above the highest supported flag, such as FLAG_FRAGMENT_REUSE_INDEX << 1.

Reproducer

Command: cargo test -p lance clone_rejects_unsupported_writer_before -- --nocapture

Expected: both clone guard tests pass. Observed: both failed while writing the fixture manifest with FRI requires both reader and writer feature flags. The same stale sentinel also appears in rust/lance-namespace-impls/src/dir/manifest.rs.

[LuQQiu]

Adjudicated both points.

System metadata in coverage filtering: confirmed, fixed in 433d8c4 on #9139 (this branch is frozen while the format vote runs, so the fix rides the stack tip). frag_reuse_reader::load_indices exempted only the FRI by name; MemWAL's metadata carries no fragment bitmap and no remap plugin, so tagged snapshots silently dropped it, taking open_mem_wal_index, catch-up, and statistics with it. The exemption is now is_system_index, so table-level system metadata bypasses coverage filtering and remapping entirely. Regression test: a real MemWAL entry survives tagged filtering and opens, while a user index without a bitmap is still dropped.

Unknown-flag fixtures: not applicable as stated. FLAG_UNKNOWN remains 1 << 8 and the supported set is carved explicitly as (FLAG_UNKNOWN - 1) | FLAG_FRAGMENT_REUSE_INDEX, so the existing unknown-flag guards still exercise rejected bits (256, 512) and none is vacuous. The bit move did leave four stale & 512 == 0 assertions in frag_reuse_reader tests (trivially true, wrong flag); those now assert against the FLAG_FRAGMENT_REUSE_INDEX constant in the same commit.