[EPIC] first class support for struct field / Variant access in Parquet

[friendlymatthew]

This issue tracks the hard problems that block efficient querying of struct field columns in Parquet

To investigate, I used an MRE based on a variant column represented as a nested struct. Though, the findings generalize to any schema where queries access leaf fields inside struct columns

• All struct-aware optimizations are hardcoded to GetFieldFunc #21306
• ProjectionExec produces unknown statistics for all ScalarFunctionExpr outputs #21307
• Teach row group pruning about struct field predicates #21003

[adriangb]

I've opened #24119 as an EPIC for the Parquet-side struct access work — projection pruning, filter pushdown, row-group pruning and the statistics that feed them — since a batch of concrete follow-ups just came out of #24090 (nested schema pruning for narrowed declared schemas) and they needed a home.

That leaves some overlap with this issue, so a suggestion: would it make sense to reframe this EPIC to focus specifically on Variant? The two linked issues here point in slightly different directions:

• All struct-aware optimizations are hardcoded to GetFieldFunc #21306 (struct-aware optimizations are hardcoded to GetFieldFunc, so variant_get gets none of them) is squarely the Variant blocker, and it's the one that needs a general capability API rather than more struct-specific machinery. It's relevant to both EPICs; I've listed it under [EPIC] Improvements to struct access in Parquet #24119 as shared rather than claiming it.
• ProjectionExec produces unknown statistics for all ScalarFunctionExpr outputs #21307 (ProjectionExec produces unknown statistics for all ScalarFunctionExpr outputs) is really a physical-expr / statistics issue that happens to bite struct and Variant queries hardest — arguably it belongs to neither EPIC exclusively.

Splitting along "Variant as a first-class type" vs "make Parquet read less for struct columns" seems like the cleaner line to me, but you filed this one and have the fuller picture of where Variant is heading — happy to fold #24119 back in here instead if you'd rather keep it as one umbrella.

cc @alamb

[alamb]

Splitting along "Variant as a first-class type" vs "make Parquet read less for struct columns" seems like the cleaner line to me, but you filed this one and have the fuller picture of where Variant is heading —

i agree this split makes sene to me