[Variant] Track and remove native Variant compatibility workarounds after upstream fixes

[peterxcli]

Track removal of compatibility code introduced by #5868. Parent roadmap: #5438. Keep completed prerequisites checked and their remaining Comet integration steps visible.

As of 2026-10-02, Comet main resolves Arrow/Parquet 59.3.0 and DataFusion 55.1.0. Dependency upgrades remain deferred. An upstream merge completes its prerequisite; the workaround stays open until Comet adopts the fix and passes the existing regressions.

Completed milestones

• Land native whole-value Variant scans and compatibility handling — feat: support direct Variant projection in native Parquet scans #5868.
• Resolve Unicode column-name matching — Support Spark-compatible Unicode case-insensitive Parquet field matching #5495, merged feat: support unicode case sensitive field names for reading parquet #5602.

Upstream-dependent cleanup — deferred

• Remove local decoding of encoded metadata.
  • Upstream accepts Dictionary/RunEndEncoded metadata — merged fix(variant): accept Dictionary and RunEndEncoded metadata in VariantArray arrow-rs#10810.
  • Adopt a containing Arrow release and remove the corresponding metadata normalization.
• Remove Dictionary value / typed_value normalization.
  • Add upstream support — [Variant] Support dictionary-encoded value and typed_value inputs arrow-rs#11068, open PR Support dictionary-encoded Variant value and typed_value inputs arrow-rs#11071.
  • Adopt that support and remove normalization, or demonstrate that every supported reader entry point excludes these inputs.
• Remove empty-key canonicalization and retry.
  • Fix upstream empty-key acceptance — merged Add proptest fuzzing to parquet-variant and implement fixes for findings arrow-rs#10352.
  • Adopt the fix, remove canonicalize_spark_empty_key_metadata and the retry, and retain fallible full validation and empty-key regressions.
• Remove wide physical decimal narrowing.
  • Handle wide DECIMAL input upstream — [Variant] Wide physical DECIMAL with precision <= 38 is inferred as Decimal256 and rejected arrow-rs#11067, open PR Fix Variant reads of wide physical decimals arrow-rs#11072.
  • Adopt the replacement for Decimal256(p,s) -> Decimal128(p,s) when p <= 38; verify positive/negative values in 17- and 32-byte physical fields.
• Remove Parquet footer reconstruction for physical/ENUM inference.
  • Support a known per-file Arrow schema — merged Allow specifying an arrow schema for PartitionedFile datafusion#22360.
  • Provide the schema hook after lazy footer loading — merged feat: Support lazy per-file Parquet Arrow schema derivation datafusion#25343.
  • Adopt ParquetFileSchemaProvider, supply the full per-file schema and remove footer reconstruction.

For the schema-provider step, derive types from the physical Parquet schema, ignore advisory ARROW:schema, map ENUM to Utf8 and retain raw BINARY. Preserve metadata/index/decryptor state and unread-column pruning. Reassess the encrypted-scan fallback after encryption tests pass.

Spark compatibility and performance

• [Variant] Remove the extra Spark byte-reconstruction pass #5978 — emit Spark-compatible bytes during unshredding and remove the second reconstruction pass.
  • Reduce allocations through buffer reuse — merged perf: reuse Variant reconstruction buffers #6342.
  • Expose the upstream row decoder — [Variant] Expose a row decoder for custom unshredding output arrow-rs#11260.
  • Integrate the writer and meet the compatibility and whole-value performance targets in [Variant] Remove the extra Spark byte-reconstruction pass #5978.
• [Variant] Consolidate Spark-compatible missing-value validation and errors #5977 — check missing states during reconstruction and remove duplicated prevalidation; keep Spark error translation in Comet.
• [Variant] Replace missing-key metadata repair with native permissive reconstruction #5979 — reconstruct missing-key inputs and remove extend_shredded_metadata.
  • Replace the upstream panic with an error — [Variant] unshred_variant panics when a shredded field name is absent from metadata arrow-rs#11069, open PR fix: return errors for missing Variant metadata field names arrow-rs#11106.
  • Build output metadata during native permissive reconstruction. Returning an error alone does not replace Spark-compatible handling.
• [Variant] Preserve typed-value precedence without residual pre-rewriting #5980 — apply scalar/list typed precedence before residual decoding and remove residual masking.
• [Variant] Remove Variant UTF-16 output rewriting #5474 — remove UTF-16 output ordering when all supported Spark profiles accept canonical ordering; preserve legacy input reads.

These child issues retain the concrete algorithms and acceptance tests.

Compatibility to retain

Unsigned typed_value is excluded by the canonical specification (apache/arrow#50810); apache/arrow-rs#10417 closed without merging. Keep Spark's unsigned widening, including UInt64 -> Decimal(20,0). Millisecond timestamps, unannotated fixed-length binary and FixedSizeList conversion also remain Spark compatibility work.

Field metadata/FFI export, ordinary Binary children in [value, metadata] order, parent nulls, fallback gates and embedded-NUL handling at the C ABI remain integration requirements.

Complete this tracker when each temporary workaround is removed or has a documented retention reason. Each removal must pass its existing native and Spark regressions.