Support top-level Variant inputs and outputs in Comet MapInArrow and MapInPandas. The scan/FFI foundation landed in #5868; the Python boundary remains unsupported.
Implementation
Translate at the Python boundary between Comet's parent arrow.parquet.variant marker and Spark/PyArrow's Variant child marker. Follow Spark's Arrow schema and PySpark conversion.
Preserve [value, metadata] Binary children, field names, SQL-null parent validity and Variant-null payloads in both directions, including zero-row streams. Validate layouts before admitting them; nested Variant and other Python operator families retain fallback.
Completion
Parity and plan tests must cover both identity functions, Python-produced Variant output, nulls and adjacent projected fields. Assert Spark receives VariantType and retains value/metadata bytes.
Parent: #5438. #5425 defines native expression-result Fields. #5386 and #5123 cover other Python operator families.
Support top-level Variant inputs and outputs in Comet MapInArrow and MapInPandas. The scan/FFI foundation landed in #5868; the Python boundary remains unsupported.
Implementation
Translate at the Python boundary between Comet's parent
arrow.parquet.variantmarker and Spark/PyArrow's Variant child marker. Follow Spark's Arrow schema and PySpark conversion.Preserve
[value, metadata]Binary children, field names, SQL-null parent validity and Variant-null payloads in both directions, including zero-row streams. Validate layouts before admitting them; nested Variant and other Python operator families retain fallback.Completion
Parity and plan tests must cover both identity functions, Python-produced Variant output, nulls and adjacent projected fields. Assert Spark receives
VariantTypeand retains value/metadata bytes.Parent: #5438. #5425 defines native expression-result Fields. #5386 and #5123 cover other Python operator families.