fix(encoding): prevent bits_per_value=0 in FullZipLayout for FSL with AllNull child

[westonpace]

If we had a fixed-size-list array where each list was non-null but consisted only of null elements (e.g. FSL<2> = [[NULL, NULL], [NULL, NULL], [NULL, NULL]]) then we would store it with bits_per_values=0 and the resulting file would be unreadable.

This PR fixes the issue. Arguably we could do it more efficiently if we turn this into a special case (we currently still store the validity bytes which is redundant information) but this is probably a unique enough case to not worry about too much yet.

Details of fix with four coordinated changes in value.rs:

1. fsl_to_encoding: instead of returning constant(None) early for AllNull children, fall through to the inner-encoding match, set has_validity=true, and emit fsl(dim, has_validity=true, constant(None)). This tells the decoder that validity bytes are present per row.

2. nullable_per_value_fsl AllNull arm: add an all-zero validity buffer to validity_iters and increment bytes_per_row by cum_dim.div_ceil(8). bytes_per_row is now >= 1, so bits_per_value > 0 in the layout.

3. ValueDecompressor::from_fsl: handle Compression::Constant as a terminal case. Only the validity bytes count toward bits_per_value; bits_per_item=0 signals the all-null path to unzip_decompress.

4. unzip_decompress: when bits_per_item==0 produce DataBlock::AllNull for the inner block instead of FixedWidth{bits_per_value:0}. AllNull::into_arrow creates a properly-typed all-null array for any element type.

Backward compatibility: existing files whose AllNull-child FSL used constant(None) directly as the inner encoding still decode via ConstantDecompressor unchanged; only new files take the new path.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The current-reader round trip is fixed, but stable writers must keep emitting value-encoding shapes understood by the latest released stable reader. Preserve the v2.1 forward-compatibility contract—for example, retain a FixedSizeList → Flat terminal with masked leaf-width payload bytes, or restrict the new encoding composition to an unstable file version.

[lance-gatekeeper]

This makes stable v2.1 writers emit FixedSizeList(has_validity=true) → Constant(None). Lance v11.0.0 is the latest stable release and resolves stable to v2.1, but its ValueDecompressor::from_fsl accepts only nested FixedSizeList and a terminal Flat; this terminal reaches _ => unreachable!(). Files written through this new path therefore panic in that released reader. The same-version round-trip test cannot detect this forward-compatibility break. Please keep stable writes on an encoding shape v11 understands (for example, a Flat terminal with masked payload bytes at the actual leaf width), or version-gate this composition to an unstable format.

Reproducer against v11.0.0

#[test]
fn accepts_nested_constant_fsl_value_encoding() {
    let encoding = super::ProtobufUtils21::fsl(
        4,
        true,
        super::ProtobufUtils21::constant(None),
    );
    let Compression::FixedSizeList(fsl) = encoding.compression.unwrap() else {
        panic!("expected fixed-size-list encoding");
    };

    let _ = ValueDecompressor::from_fsl(fsl.as_ref());
}

cargo test -p lance-encoding accepts_nested_constant_fsl_value_encoding -- --nocapture

Expected: the released stable reader accepts the stable v2.1 encoding. Observed: internal error: entered unreachable code in ValueDecompressor::from_fsl.

[westonpace]

The old writer would have crashed without this change. We are not making things worse in terms of compatibility.

[lance-gatekeeper]

The old writer failure is the bug being fixed, but it does not make the new stable encoding forward-compatible: this patch makes a v2.1 file readable by the current reader while the released stable v11 reader still panics on its nested Constant terminal. The repository contract requires every new stable-format write to remain readable by the released stable reader, even when the old writer could not successfully produce that edge case, so this finding remains.

[westonpace]

I accept this as a known risk. Masked payload bytes would be inefficient and this is a fairly niche corner case already.

[westonpace]

Actually, I guess I'll try and fix it.

[lance-gatekeeper]

The follow-up commit only permits zero-width pages in the current FullZip scheduler; it does not change fsl_to_encoding, which still emits stable v2.1 FixedSizeList → Constant. The released v11 reader therefore still reaches its unsupported terminal, so the compatibility finding remains.

[westonpace]

Correct. I am accepting (and have documented) this compatibility gap. The old writer created files that failed anyways. The new writer creates files which panic instead of error, which is unfortunate. However, the compatibility path could lead to 1000x+ increase in storage cost for these types and that is unacceptable. A break in compatibility is the best path forward for this niche case.

[lance-gatekeeper]

The storage-cost concern rules out the padded stable encoding, but not the other cited path: put this composition behind an unstable format version. Stable-format forward compatibility is an explicit repository contract, so documenting a v2.1 exception cannot waive it; this finding remains.

[lance-gatekeeper]

The current head contains only a formatting change and does not alter this assessment. Version-gating the new composition avoids the padded storage cost while preserving the explicit stable-format compatibility contract, so the finding remains.

[github-actions]

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer) and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Status: ❌ Blocked — 0 of 3 required approvals

Approvals (this commit)	none (0/3)
Vetoes	none
Voting period	ends Wed 2026-09-16 13:56 UTC (06:56 PDT)

_{Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).}

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The author has declined a padded v11-readable stable encoding because of its storage cost. The remaining compatible path is to emit this FixedSizeList → Constant composition only under an unstable format version. Keeping it in v2.1 still violates the stable forward-compatibility contract; author acceptance and documentation cannot waive that storage boundary.

[westonpace]

I have applied format-waived since the change is documenting current behavior and not a format change.

[Xuanwo]

Thank you!