Support direct, top-level Variant results from native expressions, starting with literal-path two-argument variant_get and try_variant_get. Reuse extraction from #5424 and the scan/FFI contract merged in #5868.
SELECT variant_get(v, '$.customer') FROM t;
Implementation
A physical Struct<value, metadata> datatype alone loses Variant identity. Return a Field carrying ARROW:extension:name=arrow.parquet.variant, with ordinary Binary children in [value, metadata] order and the correct parent nullability.
Reuse Comet's to_arrow_field and DataFusion's existing PhysicalExpr::return_field / projection metadata propagation. Supply that marked Field when constructing ScalarFunctionExpr; its current generic Field::new(...) is unannotated. These APIs already exist in the pinned dependency.
Rebuild the extracted value with only its used dictionary entries, matching Spark's Variant-target conversion. Admit only expressions that guarantee this output contract.
Completion
Assert native evaluation, the extension marker through aliases/FFI, exact child layout and Spark VariantType output. Compare objects, arrays, scalars, Variant null and SQL NULL with Spark, including adjacent projected columns. Unsupported producers and downstream consumers retain fallback.
Dynamic paths and nested targets belong to #5426; other producers and transport boundaries remain separate items in #5438.
Support direct, top-level Variant results from native expressions, starting with literal-path two-argument
variant_getandtry_variant_get. Reuse extraction from #5424 and the scan/FFI contract merged in #5868.Implementation
A physical
Struct<value, metadata>datatype alone loses Variant identity. Return a Field carryingARROW:extension:name=arrow.parquet.variant, with ordinary Binary children in[value, metadata]order and the correct parent nullability.Reuse Comet's
to_arrow_fieldand DataFusion's existingPhysicalExpr::return_field/ projection metadata propagation. Supply that marked Field when constructingScalarFunctionExpr; its current genericField::new(...)is unannotated. These APIs already exist in the pinned dependency.Rebuild the extracted value with only its used dictionary entries, matching Spark's Variant-target conversion. Admit only expressions that guarantee this output contract.
Completion
Assert native evaluation, the extension marker through aliases/FFI, exact child layout and Spark
VariantTypeoutput. Compare objects, arrays, scalars, Variant null and SQL NULL with Spark, including adjacent projected columns. Unsupported producers and downstream consumers retain fallback.Dynamic paths and nested targets belong to #5426; other producers and transport boundaries remain separate items in #5438.