Skip to content

feat(index): implement stable partition mapping readers - #9064

Open
LuQQiu wants to merge 7 commits into
lu/fri-mapping-readerfrom
lu/fri-row-map
Open

LuQQiu wants to merge 7 commits into
lu/fri-mapping-readerfrom
lu/fri-row-map

Conversation

@LuQQiu

@LuQQiu LuQQiu commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

What this adds

A stable-partition rewrite reads source fragments in scan order and distributes their live rows across an ordered list of destination fragments. Row order is preserved within each destination, but the destination cannot be inferred from source row order as it can for ordered compaction.

This PR implements the mapping as one immutable Lance file and exposes it through the MappingReader interface introduced in #9106. It does not integrate dataset lineage traversal or change the legacy compaction reader/writer.

Encoding

The row-map file contains one entry for every physical source row, ordered by source fragment and then row offset:

stable_partition.lance
├── label: nullable UInt16
│   ├── 0..N-1 → position in the ordered destination list
│   └── null   → source row was deleted
└── global buffer: cumulative counts matrix

For example, a label of 2 selects destinations[2]; it is not a fragment ID or destination row offset.

The writer receives labels for live rows in source scan order and inserts nulls from the source deletion vectors. This preserves one file row per physical source row, including rows already deleted when the rewrite ran.

The counts matrix divides the label column into 65,536-row blocks. Its on-disk layout is:

28-byte header
├── magic:                "LSPC"
├── version:              u32
├── representation:       u32 (0 = dense)
├── num_destinations:     u32
├── rows_per_block:       u32
└── total_source_rows:    u64

dense grid
└── cumulative u32 count for each (block, destination), block-major

The Lance schema metadata key lance:stable_partition:counts_buffer_index identifies this global buffer.

Address translation

For source row (fragment_id, row_offset), the reader first converts it to its position in the concatenated source layout. A null label returns a deleted row. Otherwise:

destination fragment = destinations[label]
destination offset   = count(label before this block)
                     + rank(label earlier in this block)

Because each destination is written in source scan order, this reconstructs its physical destination offset without storing a complete source-to-destination address table. Reordering rows within a destination is outside this encoding and requires a different mapping type.

Opening the mapping reads and validates the counts metadata without reading labels. Point and batch translations lazily read only the touched label blocks; batch requests sweep each touched block once.

Validation

The reader validates the label schema, counts header and dimensions, monotonic cumulative counts, source and destination row totals, label bounds, and per-block label/count agreement.

Validated with the stable-partition row-map and mapping-reader tests, workspace Clippy, and cargo fmt --all.

@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer enhancement New feature or request labels Sep 8, 2026
@LuQQiu
LuQQiu removed this pull request from stack #9066 September 9, 2026 19:15
@LuQQiu LuQQiu changed the title feat(core,index): fragment reuse row map for stable partitions feat(index): implement stable partition mapping readers Sep 9, 2026
@LuQQiu
LuQQiu changed the base branch from main to lu/fri-mapping-reader September 9, 2026 19:16
@LuQQiu
LuQQiu added this pull request to stack #9108 September 9, 2026 19:16
@LuQQiu
LuQQiu removed this pull request from stack #9108 September 10, 2026 03:15
@LuQQiu
LuQQiu added this pull request to stack #9117 September 10, 2026 03:15
@LuQQiu
LuQQiu removed this pull request from stack #9117 September 10, 2026 18:17
@LuQQiu
LuQQiu added this pull request to stack #9137 September 10, 2026 18:17
@LuQQiu
LuQQiu marked this pull request as ready for review September 10, 2026 18:21
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The label-plus-cumulative-count encoding fits the stable-partition contract: it preserves per-destination input order without an explicit destination/offset entry per row. The immutable writer/reader and lazy MappingReader implementation keep validation and block traversal within this layer. This stacks cleanly after #9106 supplies the reader contract; downstream integration remains isolated to #9067, #9068, and #9107.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 10, 2026
LuQQiu added a commit that referenced this pull request Sep 15, 2026
FRI keeps existing indices usable after fragment rewrites by translating
old physical row addresses to new ones. Today it supports
order-preserving compaction. This proposal extends the same system index
to support stable partitioning, with both mapping types sharing one
fragment-lineage history.

Following [the unified FRI
proposal](#8972 (comment)),
source/destination lineage stays in FRI details, while large mapping
payloads remain in separate immutable files.

### One history, multiple mappings

Continue storing FRI information in a single `__lance_frag_reuse`
system-index entry. Keep the existing `InlineContent` / `ExternalFile`
envelope and the original field number for legacy versions. Add tagged
transitions alongside them:

```text
FragmentReuseIndexDetails
└── InlineContent, stored inline or in external details.binpb
    ├── legacy_versions[]
    └── transitions[]
        ├── ordered sources[]
        ├── ordered destinations[]
        └── mapping
            ├── OrderedCompaction: surviving-row bitmap
            └── StablePartition: immutable row-map reference
```

Sources and destinations define the common rewrite graph. Each mapping
defines how to translate row offsets. Legacy groups can be read as
ordered-compaction transitions; mixed histories follow fragment lineage,
not the order of records or dataset version numbers.

### Lightweight metadata, external row maps

Ordered compaction retains its compact bitmap representation. Stable
partition assigns each physical source row a nullable `uint16`
destination label, preserving source order within each destination. A
null label means the row was deleted. A counts matrix lets readers
reconstruct destination offsets without reading all preceding labels.

Stable-partition metadata records `map_id`, `map_size_bytes`, and
optional `base_id`. The labels and counts are stored in
`_fri/<map_id>/stable_partition.lance`. Mapping identity is independent
of the FRI index UUID: updating the history rewrites its metadata, but
does not rewrite existing row-map files. The history can be opened
without loading labels; address translation reads the required blocks.

### Publication and compatibility

`AppendFragmentReuseTransitions` expresses a transition delta. Combined
atomically with a fragment rewrite, it lets the commit apply the delta
to the current history and publish destination fragments and their
mappings together. The persisted FRI details remain a snapshot of that
history.

- **Index version 0:** existing compaction format and read/write
behavior remain unchanged.
- **Index version 1:** supports legacy groups and tagged transitions in
one history.
- The first commit publishing index version 1 sets reader and writer
flag **512**. The reader flag prevents old clients from partially
interpreting the history; the writer flag prevents them from dropping
mappings during metadata maintenance. Subsequent manifests retain both
bits.

### Scope and validation

This PR contains protobuf definitions, the corresponding format
documentation, the proposed flag constant, and minimal compile adapters.
It does not enable tagged-history reads or writes. Mapping
implementations and reader integration follow in #9106#9064#9067#9068#9107.

Replaces #9065 as the standalone spec at the bottom of native stack
#9137, based on main `31d78d170`.

`cargo fmt --all` and whitespace checks pass. Clippy and tests are
blocked by dependency resolution: main requires `object_store_opendal
0.60.1`, while the crates.io index currently offers only up to 0.60.0.

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
LuQQiu and others added 7 commits September 15, 2026 11:46
A reordered rewrite (reclustering) distributes live rows of n source
fragments across m destination fragments in scan order, so unlike
compaction the destination of a row cannot be derived from row order.
This adds the standalone format capability that records and replays that
mapping, with no transaction or read-path integration yet:

- lance-core/utils/stable_partition: pure translation arithmetic.
  CountsMatrix stores cumulative per-destination row counts at every
  64K-row block boundary; a point lookup is counts base + label rank in
  one block, a sweep is counter[label]++ per row seeded from any block
  boundary. Encode/decode for the on-disk form plus content validation.
- lance-index/frag_reuse/row_map: the row map file. One Lance file with
  a single nullable u16 label column (one row per physical source row,
  NULL = deleted at source) and the encoded counts in a global buffer,
  so open costs one tail read. RowMapWriter interleaves NULLs from the
  source deletion vectors while the caller streams live-row labels;
  RowMapReader offers point, coalesced-batch and sweep translation.
- benches/stable_partition_row_map: encoded size and translation costs.
  2M rows / 1000 destinations on V2_1: 11.15 bits/row uniform-random
  labels (worst case, nominal 10) and 6.27 bits/row with 16-destination
  block locality; sweep ~170M rows/s, full-block label rank 3.8us.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review follow-ups on the stable-partition row map:

- The counts header now carries a representation tag. Only the dense grid
  is written (exact, data-independent size: ~600KB for a 50M-row rewrite
  across 500 destinations, ~61MB at 1B rows across 1000 — trivial beside
  the rewrite either way); unknown tags are rejected with a clear error,
  so sparser encodings can be added later without breaking readers.
- RowMapReader::open() now fails loudly on a bad file instead of
  translating rows to wrong addresses or panicking: it checks the label
  column's schema, decodes with exact structural checks (magic, version,
  supported representation, shape, precise payload length), runs the full
  counts consistency validation, and reconciles label row count against
  the counts. Batch column casts return errors instead of panicking.
- Documented the stable-partition ordering contract the arithmetic rests
  on (labels in source physical-row order, destinations filled in that
  same order and never re-sorted, destination list fixed), mirroring the
  Ordering section of row_addr_remap.rs.
- translate_many now subtracts block starts in u64 like translate.
- Replaced three copies of a hand-rolled LCG with seeded StdRng; rand is
  already a workspace dependency.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Deduplicate the ordering-contract paragraph in the module docs and drop
  a redundant explicit rustdoc link.
- open() enforces the full schema contract it claims: exactly one column,
  named label, u16, nullable.
- Document the remaining public writer/reader methods.
- Deterministic NULL edge-case test: fully-deleted source, zero-row
  source, deleted tail drained by finish(), empty translate_many and
  sweep inputs.
- State that sweep's one-block-at-a-time IO is intentional (bounded
  memory); prefetch belongs to read integration.
- Fix the 50M x 500 counts size in docs (1.5 MB, not 600 KB) and allow
  the size-probe printlns in the bench.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review follow-up: make explicit that the row map represents stable
partitions only. A rewrite that sorts rows within a destination cannot
be expressed by destination labels (equal labels would rank in source
order, not output order) and would need a per-row final-offset encoding
as a separate format.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 15, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

Labels plus cumulative counts remain the right fit for the stable-partition contract: they preserve per-destination input order without an explicit destination/offset entry for every row. #9136 now establishes the unified history format; this PR is safely sequenced after #9106, while decoding, planner, and consumer integration stays isolated to #9067, #9068, and #9107.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer enhancement New feature or request K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant