Conversation
github-actions
Bot
added
A-format
This was referenced Sep 8, 2026
LuQQiu
removed this pull request from stack #9066
September 9, 2026 19:15
LuQQiu
added this pull request to stack #9108
September 9, 2026 19:16
Document FLAG_FRAGMENT_REUSE_INDEX with its paired sticky semantics, the index-version negotiation and transition lineage invariants, the stable-partition row map file schema with its counts matrix layout and reader navigation, and the append_fri_transitions operation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016RynpAxtwGB9Q9CL4JCvR4
The flag table row and the required-columns already state the contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016RynpAxtwGB9Q9CL4JCvR4
Restructure around the two use cases, specify version negotiation and shared transition metadata, define both mapping encodings with full translation algorithms and validation rules, and state the reader and writer compatibility contract.
LuQQiu
removed this pull request from stack #9108
September 10, 2026 03:15
LuQQiu
added this pull request to stack #9117
September 10, 2026 03:15
LuQQiu
removed this pull request from stack #9117
September 10, 2026 18:17
This was referenced Sep 10, 2026
LuQQiu
added a commit
that referenced
this pull request
Sep 15, 2026
FRI keeps existing indices usable after fragment rewrites by translating old physical row addresses to new ones. Today it supports order-preserving compaction. This proposal extends the same system index to support stable partitioning, with both mapping types sharing one fragment-lineage history. Following [the unified FRI proposal](#8972 (comment)), source/destination lineage stays in FRI details, while large mapping payloads remain in separate immutable files. ### One history, multiple mappings Continue storing FRI information in a single `__lance_frag_reuse` system-index entry. Keep the existing `InlineContent` / `ExternalFile` envelope and the original field number for legacy versions. Add tagged transitions alongside them: ```text FragmentReuseIndexDetails └── InlineContent, stored inline or in external details.binpb ├── legacy_versions[] └── transitions[] ├── ordered sources[] ├── ordered destinations[] └── mapping ├── OrderedCompaction: surviving-row bitmap └── StablePartition: immutable row-map reference ``` Sources and destinations define the common rewrite graph. Each mapping defines how to translate row offsets. Legacy groups can be read as ordered-compaction transitions; mixed histories follow fragment lineage, not the order of records or dataset version numbers. ### Lightweight metadata, external row maps Ordered compaction retains its compact bitmap representation. Stable partition assigns each physical source row a nullable `uint16` destination label, preserving source order within each destination. A null label means the row was deleted. A counts matrix lets readers reconstruct destination offsets without reading all preceding labels. Stable-partition metadata records `map_id`, `map_size_bytes`, and optional `base_id`. The labels and counts are stored in `_fri/<map_id>/stable_partition.lance`. Mapping identity is independent of the FRI index UUID: updating the history rewrites its metadata, but does not rewrite existing row-map files. The history can be opened without loading labels; address translation reads the required blocks. ### Publication and compatibility `AppendFragmentReuseTransitions` expresses a transition delta. Combined atomically with a fragment rewrite, it lets the commit apply the delta to the current history and publish destination fragments and their mappings together. The persisted FRI details remain a snapshot of that history. - **Index version 0:** existing compaction format and read/write behavior remain unchanged. - **Index version 1:** supports legacy groups and tagged transitions in one history. - The first commit publishing index version 1 sets reader and writer flag **512**. The reader flag prevents old clients from partially interpreting the history; the writer flag prevents them from dropping mappings during metadata maintenance. Subsequent manifests retain both bits. ### Scope and validation This PR contains protobuf definitions, the corresponding format documentation, the proposed flag constant, and minimal compile adapters. It does not enable tagged-history reads or writes. Mapping implementations and reader integration follow in #9106 → #9064 → #9067 → #9068 → #9107. Replaces #9065 as the standalone spec at the bottom of native stack #9137, based on main `31d78d170`. `cargo fmt --all` and whitespace checks pass. Clippy and tests are blocked by dependency resolution: main requires `object_store_opendal 0.60.1`, while the crates.io index currently offers only up to 0.60.0. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Superseded by #9136, which puts this format proposal directly on main as the first PR in native stack #9137. Existing discussion is retained here.
Add tagged transitions alongside legacy versions in the existing FRI details envelope. A transition contains ordered source/destination digests and an explicit mapping: ordered compaction or an immutable stable-partition row-map reference. Index version 0 retains its existing format; version 1 introduces tagged history.
Scope: proto definitions, minimal compile adapters, and the proposed
FLAG_FRAGMENT_REUSE_INDEX = 1 << 9constant with its reader/writer rationale. Mapping implementations are in #9106/#9064, decoding in #9067, and runtime flag support, stamping, inheritance, validation, and reader behavior in #9068. The three format documentation pages are unchanged, as agreed; format review documentation comes last.Validation: the dependent reader stack compiles against these definitions; workspace Clippy and targeted FRI tests pass.
Latest main synchronization (2026-09-10): merged
d7b031f9cthrough the stack without rewriting history.cargo fmt --alland whitespace checks pass. Workspace Clippy could not run: main now requiresobject_store_opendal 0.60.1, but the crates.io index currently resolves only up to 0.60.0. Tests were not rerun after this synchronization because dependency resolution is blocked. Earlier validation above predates this merge.