feat(format): define a unified tagged fragment reuse history

[LuQQiu]

FRI keeps existing indices usable after fragment rewrites by translating old physical row addresses to new ones. Today it supports order-preserving compaction. This proposal extends the same system index to support stable partitioning, with both mapping types sharing one fragment-lineage history.

Following the unified FRI proposal, source/destination lineage stays in FRI details, while large mapping payloads remain in separate immutable files.

One history, multiple mappings

Continue storing FRI information in a single __lance_frag_reuse system-index entry. Keep the existing InlineContent / ExternalFile envelope and the original field number for legacy versions. Add tagged transitions alongside them:

FragmentReuseIndexDetails
└── InlineContent, stored inline or in external details.binpb
    ├── legacy_versions[]
    └── transitions[]
        ├── ordered sources[]
        ├── ordered destinations[]
        └── mapping
            ├── OrderedCompaction: surviving-row bitmap
            └── StablePartition: immutable row-map reference

Sources and destinations define the common rewrite graph. Each mapping defines how to translate row offsets. Legacy groups can be read as ordered-compaction transitions; mixed histories follow fragment lineage, not the order of records or dataset version numbers.

Lightweight metadata, external row maps

Ordered compaction retains its compact bitmap representation. Stable partition assigns each physical source row a nullable uint16 destination label, preserving source order within each destination. A null label means the row was deleted. A counts matrix lets readers reconstruct destination offsets without reading all preceding labels.

Stable-partition metadata records map_id, map_size_bytes, and optional base_id. The labels and counts are stored in _fri/<map_id>/stable_partition.lance. Mapping identity is independent of the FRI index UUID: updating the history rewrites its metadata, but does not rewrite existing row-map files. The history can be opened without loading labels; address translation reads the required blocks.

Publication and compatibility

AppendFragmentReuseTransitions expresses a transition delta. Combined atomically with a fragment rewrite, it lets the commit apply the delta to the current history and publish destination fragments and their mappings together. The persisted FRI details remain a snapshot of that history.

• Index version 0: existing compaction format and read/write behavior remain unchanged.
• Index version 1: supports legacy groups and tagged transitions in one history.
• The first commit publishing index version 1 sets reader and writer flag 512. The reader flag prevents old clients from partially interpreting the history; the writer flag prevents them from dropping mappings during metadata maintenance. Subsequent manifests retain both bits.

Scope and validation

This PR contains protobuf definitions, the corresponding format documentation, the proposed flag constant, and minimal compile adapters. It does not enable tagged-history reads or writes. Mapping implementations and reader integration follow in #9106 → #9064 → #9067 → #9068 → #9107.

Replaces #9065 as the standalone spec at the bottom of native stack #9137, based on main 31d78d170.

cargo fmt --all and whitespace checks pass. Clippy and tests are blocked by dependency resolution: main requires object_store_opendal 0.60.1, while the crates.io index currently offers only up to 0.60.0.

[github-actions]

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer) and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Status: ❌ Blocked — 1 of 3 required approvals

Approvals (this commit)	@Xuanwo (1/3)
Vetoes	none
Voting period	ends Tue 2026-09-15 18:18 UTC (11:18 PDT)

_{Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).}

[LuQQiu]

Re the gate concern about durable transaction semantics for the rewrite mapping:

This is how the FRI has operated since it launched, and this PR does not change it. A deferred rewrite has never carried its mapping in the serialized transaction message: the durable record is the FRI entry in the manifest's index section, replaced in the same commit (manifest_build retain-by-name + push). The transaction file records the rewrite groups; the mapping's source of truth is the committed manifest.

Independent writers do not need to reconstruct another writer's delta from its transaction file, because that delta is already materialized in its committed manifest entry. The correctness requirement is on the writer side: assemble the entry against the current manifest at each commit attempt, never splice an entry pre-assembled from a stale snapshot. With that rule, a retrying writer reloads the latest entry (which contains the other writer's record) and appends its own in-memory delta, so both records survive. Double consumption is detectable from the rewrite groups' fragment sets, which are serialized.

The reproducer is real, and it demonstrates an implementation defect in the current legacy writer (stale-snapshot splicing), not a format gap: the tagged-history writer in the implementation stack already reassembles at every commit attempt inside conflict resolution, and a matching fix for the legacy path can follow separately. The spec text has been adjusted to state only the durable fact (the entry replacement in the manifest) and leaves the transport to the implementation, where it has always lived.

[Xuanwo]

The standalone AppendFragmentReuseTransitions operation seems premature here, since Rewrite can already publish the FRI entry atomically. Could we defer it until there is a concrete caller for separate mapping publication or composite transactions? That would let us review the durable operation’s semantics alongside its actual use, while keeping this PR focused on the unified history format.

[LuQQiu]

This is pre-added for composite transaction V2 at the assumption that transaction V2 will be landed soon. In that case, having a separate AppendFragmentReuseTransitions can help make the transaction more clear. I can remove it now and add it back later when transaction V2 is ready

[LuQQiu]

Agreed, removed in 66232d4: the operation, its oneof entry, and the related doc sections (the Appending FRI Transitions section and the separate-commit wording). Rewrite remains the only publication path, atomic with its own commit. We can bring a durable append back together with a concrete caller or composite transactions.

[lance-gatekeeper]

✅ Gate recommendation: approve.

Deferring the standalone append operation keeps this format change focused on the concrete atomic path: a rewrite publishes its FRI mapping in the same commit as fragment replacement. The unified tagged history, compatibility fence, and fail-closed adapter behavior remain coherent, and the removed field had no released compatibility obligation.

Please mark this PR with the breaking-change label.

[Xuanwo]

+1 binding from me