Skip to content

[Variant] Normalize marked Variant arrays at the native Parquet boundary #5549

Closed

Description

@peterxcli

Problem

Once Variant identity exists in the native schema, ordinary Parquet can still return metadata, residual value, and optional typed_value in a physical Struct that is not Spark's required whole-value layout.

Parent: #5546
Depends on: #5548 (merged in #5631)
Implementation: #5715 (in review)
Related: #3983

Solution

The JVM scan gate remains closed in this task. It must not add subfield pruning, predicate pushdown, shredded writing, or JVM-side unshredding.

Tests

  • Canonical unshredded, fully shredded, and partially shredded values agree.
  • Inputs whose children are not in Spark output order are reordered.
  • Objects, arrays, scalars, JSON null, SQL NULL, nested residuals, and nullable parents round-trip.
  • typed_value never reaches the output.
  • The output Field marker and parent null bitmap are preserved.
  • [Variant] Remove Variant UTF-16 output rewriting #5474 is linked to removable legacy ordering compatibility code.

Activity

  1. peterxcli commented on Aug 29, 2026

    @peterxcli
    MemberAuthor

    take

  2. changed the title [-][FEATURE] Normalize marked Variant arrays at the native Parquet boundary[/-] [+][Variant] Normalize marked Variant arrays at the native Parquet boundary[/+] on Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:scanParquet scan / data readingenhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions