feat(format): feature flag for a fragment reuse index on stable row id tables - #9119
Conversation
github-actions
Bot
added
A-format
0fe8685 to
f1655f5
Compare
…d tables Reserve reader/writer feature flag 1 << 9 for a table version that uses stable row ids and carries a fragment reuse index. The index records how physical row addresses moved; entries of an index that stores stable row ids must not be remapped through it, so builds without domain-aware index loading must refuse such a table. This build defines the bit but does not support it yet; support lands with the implementation. Part of lance-format#8977
f1655f5 to
059faf8
Compare
|
Important Format specification voteThis PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer) and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process. Status: ❌ Blocked — 0 of 3 required approvals
Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the |
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The format-only reservation establishes the necessary compatibility fence: bit 9 is documented for stable-row-ID fragment reuse, while this revision still rejects bits 8 and 9 until their implementations land. PR #9120 is correctly separated as the dependent implementation.
lance-gatekeeper
Bot
added
the
K-approved
Bit 9 is taken by the stable-row-id FRI compatibility flag (#9119), so the tagged-history capability moves to bit 1024. A rewrite and its transition may land in separate commits: transitions reference only committed fragments, and unmapped destinations are served by scanning, so correctness never depends on the mapping being present; atomic composition is deferred to a future composite transaction mechanism. Tables using stable row IDs do not support tagged histories. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016RynpAxtwGB9Q9CL4JCvR4
Bit 9 is taken by the stable-row-id FRI compatibility flag (#9119). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016RynpAxtwGB9Q9CL4JCvR4
|
updated my flag number to avoid contradiction with #9119 |
2 participants
Proposes a reader/writer feature flag, FLAG_FRAG_REUSE_WITH_STABLE_ROW_IDS, for a table version that uses stable row IDs and carries a Fragment Reuse Index (FRI).
The FRI records how physical row addresses moved during compaction. On a stable row ID table most indices store stable row IDs, not addresses, and those entries must never be remapped through the FRI. An implementation that applies the FRI to such an index returns wrong rows when reading and persists a corrupt index when merging or optimizing. Setting this flag on affected table versions makes older implementations fail to open the table with an unsupported error, rather than applying the FRI to every index.
Today compaction does not write an FRI for stable row ID tables. We plan to change that and need the flag in place first so older readers and writers fail closed. This PR defines and documents the flag, and in a follow-up PR I will enable support alongside the index loading changes, so no intermediate build can open such a table and misread it.
Part of #8977