Conversation
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The label-plus-cumulative-count encoding fits the stable-partition contract: it preserves per-destination input order without an explicit destination/offset entry per row. The immutable writer/reader and lazy MappingReader implementation keep validation and block traversal within this layer. This stacks cleanly after #9106 supplies the reader contract; downstream integration remains isolated to #9067, #9068, and #9107.
FRI keeps existing indices usable after fragment rewrites by translating old physical row addresses to new ones. Today it supports order-preserving compaction. This proposal extends the same system index to support stable partitioning, with both mapping types sharing one fragment-lineage history. Following [the unified FRI proposal](#8972 (comment)), source/destination lineage stays in FRI details, while large mapping payloads remain in separate immutable files. ### One history, multiple mappings Continue storing FRI information in a single `__lance_frag_reuse` system-index entry. Keep the existing `InlineContent` / `ExternalFile` envelope and the original field number for legacy versions. Add tagged transitions alongside them: ```text FragmentReuseIndexDetails └── InlineContent, stored inline or in external details.binpb ├── legacy_versions[] └── transitions[] ├── ordered sources[] ├── ordered destinations[] └── mapping ├── OrderedCompaction: surviving-row bitmap └── StablePartition: immutable row-map reference ``` Sources and destinations define the common rewrite graph. Each mapping defines how to translate row offsets. Legacy groups can be read as ordered-compaction transitions; mixed histories follow fragment lineage, not the order of records or dataset version numbers. ### Lightweight metadata, external row maps Ordered compaction retains its compact bitmap representation. Stable partition assigns each physical source row a nullable `uint16` destination label, preserving source order within each destination. A null label means the row was deleted. A counts matrix lets readers reconstruct destination offsets without reading all preceding labels. Stable-partition metadata records `map_id`, `map_size_bytes`, and optional `base_id`. The labels and counts are stored in `_fri/<map_id>/stable_partition.lance`. Mapping identity is independent of the FRI index UUID: updating the history rewrites its metadata, but does not rewrite existing row-map files. The history can be opened without loading labels; address translation reads the required blocks. ### Publication and compatibility `AppendFragmentReuseTransitions` expresses a transition delta. Combined atomically with a fragment rewrite, it lets the commit apply the delta to the current history and publish destination fragments and their mappings together. The persisted FRI details remain a snapshot of that history. - **Index version 0:** existing compaction format and read/write behavior remain unchanged. - **Index version 1:** supports legacy groups and tagged transitions in one history. - The first commit publishing index version 1 sets reader and writer flag **512**. The reader flag prevents old clients from partially interpreting the history; the writer flag prevents them from dropping mappings during metadata maintenance. Subsequent manifests retain both bits. ### Scope and validation This PR contains protobuf definitions, the corresponding format documentation, the proposed flag constant, and minimal compile adapters. It does not enable tagged-history reads or writes. Mapping implementations and reader integration follow in #9106 → #9064 → #9067 → #9068 → #9107. Replaces #9065 as the standalone spec at the bottom of native stack #9137, based on main `31d78d170`. `cargo fmt --all` and whitespace checks pass. Clippy and tests are blocked by dependency resolution: main requires `object_store_opendal 0.60.1`, while the crates.io index currently offers only up to 0.60.0. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A reordered rewrite (reclustering) distributes live rows of n source fragments across m destination fragments in scan order, so unlike compaction the destination of a row cannot be derived from row order. This adds the standalone format capability that records and replays that mapping, with no transaction or read-path integration yet: - lance-core/utils/stable_partition: pure translation arithmetic. CountsMatrix stores cumulative per-destination row counts at every 64K-row block boundary; a point lookup is counts base + label rank in one block, a sweep is counter[label]++ per row seeded from any block boundary. Encode/decode for the on-disk form plus content validation. - lance-index/frag_reuse/row_map: the row map file. One Lance file with a single nullable u16 label column (one row per physical source row, NULL = deleted at source) and the encoded counts in a global buffer, so open costs one tail read. RowMapWriter interleaves NULLs from the source deletion vectors while the caller streams live-row labels; RowMapReader offers point, coalesced-batch and sweep translation. - benches/stable_partition_row_map: encoded size and translation costs. 2M rows / 1000 destinations on V2_1: 11.15 bits/row uniform-random labels (worst case, nominal 10) and 6.27 bits/row with 16-destination block locality; sweep ~170M rows/s, full-block label rank 3.8us. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review follow-ups on the stable-partition row map: - The counts header now carries a representation tag. Only the dense grid is written (exact, data-independent size: ~600KB for a 50M-row rewrite across 500 destinations, ~61MB at 1B rows across 1000 — trivial beside the rewrite either way); unknown tags are rejected with a clear error, so sparser encodings can be added later without breaking readers. - RowMapReader::open() now fails loudly on a bad file instead of translating rows to wrong addresses or panicking: it checks the label column's schema, decodes with exact structural checks (magic, version, supported representation, shape, precise payload length), runs the full counts consistency validation, and reconciles label row count against the counts. Batch column casts return errors instead of panicking. - Documented the stable-partition ordering contract the arithmetic rests on (labels in source physical-row order, destinations filled in that same order and never re-sorted, destination list fixed), mirroring the Ordering section of row_addr_remap.rs. - translate_many now subtracts block starts in u64 like translate. - Replaced three copies of a hand-rolled LCG with seeded StdRng; rand is already a workspace dependency. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Deduplicate the ordering-contract paragraph in the module docs and drop a redundant explicit rustdoc link. - open() enforces the full schema contract it claims: exactly one column, named label, u16, nullable. - Document the remaining public writer/reader methods. - Deterministic NULL edge-case test: fully-deleted source, zero-row source, deleted tail drained by finish(), empty translate_many and sweep inputs. - State that sweep's one-block-at-a-time IO is intentional (bounded memory); prefetch belongs to read integration. - Fix the 50M x 500 counts size in docs (1.5 MB, not 600 KB) and allow the size-probe printlns in the bench. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review follow-up: make explicit that the row map represents stable partitions only. A rewrite that sorts rows within a destination cannot be expressed by destination labels (equal labels would rank in source order, not output order) and would need a per-row final-offset encoding as a separate format. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
d0e6779 to
9f735c1
Compare
There was a problem hiding this comment.
✅ Gate recommendation: approve.
Labels plus cumulative counts remain the right fit for the stable-partition contract: they preserve per-destination input order without an explicit destination/offset entry for every row. #9136 now establishes the unified history format; this PR is safely sequenced after #9106, while decoding, planner, and consumer integration stays isolated to #9067, #9068, and #9107.
What this adds
A stable-partition rewrite reads source fragments in scan order and distributes their live rows across an ordered list of destination fragments. Row order is preserved within each destination, but the destination cannot be inferred from source row order as it can for ordered compaction.
This PR implements the mapping as one immutable Lance file and exposes it through the
MappingReaderinterface introduced in #9106. It does not integrate dataset lineage traversal or change the legacy compaction reader/writer.Encoding
The row-map file contains one entry for every physical source row, ordered by source fragment and then row offset:
For example, a label of
2selectsdestinations[2]; it is not a fragment ID or destination row offset.The writer receives labels for live rows in source scan order and inserts nulls from the source deletion vectors. This preserves one file row per physical source row, including rows already deleted when the rewrite ran.
The counts matrix divides the label column into 65,536-row blocks. Its on-disk layout is:
The Lance schema metadata key
lance:stable_partition:counts_buffer_indexidentifies this global buffer.Address translation
For source row
(fragment_id, row_offset), the reader first converts it to its position in the concatenated source layout. A null label returns a deleted row. Otherwise:Because each destination is written in source scan order, this reconstructs its physical destination offset without storing a complete source-to-destination address table. Reordering rows within a destination is outside this encoding and requires a different mapping type.
Opening the mapping reads and validates the counts metadata without reading labels. Point and batch translations lazily read only the touched label blocks; batch requests sweep each touched block once.
Validation
The reader validates the label schema, counts header and dimensions, monotonic cumulative counts, source and destination row totals, label bounds, and per-block label/count agreement.
Validated with the stable-partition row-map and mapping-reader tests, workspace Clippy, and
cargo fmt --all.