chore(deps): Bump DataFusion to 54.0.0#2648
Conversation
xanderbailey
force-pushed
the
xb/datafusion_54
branch
3 times, most recently
from
38323e6 to
a59888d
Compare
CTTY
left a comment
CTTY
left a comment
There was a problem hiding this comment.
Overall LGTM. we can wait until the 0.10 release is done
| } | ||
|
|
||
| fn as_any(&self) -> &dyn Any { | ||
| self |
| } | ||
| Expr::Cast(c) => { | ||
| if c.data_type == DataType::Date32 || c.data_type == DataType::Date64 { | ||
| if *c.field.data_type() == DataType::Date32 || *c.field.data_type() == DataType::Date64 |
There was a problem hiding this comment.
we may need to fix the data type comparison here as well: https://github.com/apache/iceberg-rust/blob/main/crates/integrations/datafusion/src/physical_plan/expr_to_predicate.rs#L265
There was a problem hiding this comment.
Thanks @xanderbailey! Took a pass with the 54.0 upgrade guide open. The as_any removal looks complete and correct across all the trait impls and the downcast call sites, and I confirmed the other guide items don't apply here: partition_statistics isn't implemented (so no Arc<Statistics> change), the to_iceberg_predicate match is non-exhaustive so the new Expr variants (Lambda / HigherOrderFunction / LambdaVariable) are safely ignored, and there's no PruningStatistics / datafusion-proto / comparison_coercion usage in the integration.
One outstanding item, which is the same thing @CTTY flagged: DF54 also renamed logical Expr::Cast's data_type field to field: FieldRef. This PR adapts expr_to_predicate.rs:143 but there's a second site at expr_to_predicate.rs:265 in resolve_nan_preserving_reference (added in #2592, after this branch) that still uses cast.data_type. On current main this won't compile until that one also becomes cast.field.data_type(). So I think this just needs a rebase on main + that one-line fix.
…Fusion 54 DataFusion 54 removed as_any() from several traits including CatalogProviderList. The datafusion integration crate was updated but the playground crate was missed.
xanderbailey
force-pushed
the
xb/datafusion_54
branch
from
5e2205e to
e9f49ef
Compare
4 participants
Which issue does this PR close?
DataFusion to 54.0.0 was recently released, we should try and track latests releases.
What changes are included in this PR?
ExecutionPlan TableProvider, SchemaProvider, CatalogProvider, CatalogProviderList and PhysicalExpr have all had their
as_anyremoved from the trait. That's really the only change needed here. Upgrade guide documents thisAre these changes tested?