All struct-aware optimizations are hardcoded to `GetFieldFunc`

[friendlymatthew]

Is your feature request related to a problem or challenge?

Every optimization for struct field querying (row level pushdown, leaf-level projection pruning, PushdownChecker, resolve_struct_field_leaves) is gated on the GetFieldFunc udf.

A custom UDF like variant_get gets none of these optimizations. There's no trait or capability API for a UDF to declare "I'm a struct field access."

This blocks Variant support entirely unless variant_get decomposes to get_field chains or the Parquet source learns a general mechanism for struct-access UDFs

[adriangb]

There's no trait or capability API for a UDF to declare "I'm a struct field access."

There is:

datafusion/datafusion/expr/src/udf.rs

Lines 972 to 984 in 1416ed4

	/// Returns placement information for this function.
	///
	/// This is used by optimizers to make decisions about expression placement,
	/// such as whether to push expressions down through projections.
	///
	/// The default implementation returns [`ExpressionPlacement::KeepInPlace`],
	/// meaning the expression should be kept where it is in the plan.
	///
	/// Override this method to indicate that the function can be pushed down
	/// closer to the data source.
	fn placement(&self, _args: &[ExpressionPlacement]) -> ExpressionPlacement {
	ExpressionPlacement::KeepInPlace
	}

[liningpan]

I have a query calling a UDF that returns a struct (e.g., fields a and b in the struct). If I manually unnest the struct (e.g., func(x)['a'] and func(x)['b']), datafusion will call my UDF twice and can not optimize it into one call (if I'm reading the execution plan and metrics correctly). It will do the same even if I take the UDF call to a CTE or separate view, so it seems more like an issue with optimization.

I'm not sure if this is the root cause of this behavior.

[adriangb]

@liningpan I don't think that's related to this issue, but it is a very valid point. Could you maybe open an issue to track that? Off the top of my head it feels like something the CSE (Common Sub-expression Elimination) optimizer might be able to handle. A good first step would be an MRE / benchmark showing the perf issue (you could write a UDF that has a sleep in it or something).