Repository navigation
Bump to Arrow 60 - #3257
Open
gabotechs wants to merge 1 commit into
Open
Bump to Arrow 60#3257gabotechs wants to merge 1 commit into
gabotechs wants to merge 1 commit into
Conversation
Collaborator
|
Nice PR, thanks @gabotechs looks like a smooth transition, are you planning to merge the update closer to the release? |
gabotechs
marked this pull request as ready for review
September 24, 2026 06:18
Contributor
Author
Whenever maintainers find more convenient, on my side no blockers to merge this as soon as possible. |
|
@comphead @kevinjqliu would it make sense to merge this now? Arrow/Parquet 60 has some pretty useful reader improvements around adaptive row-group pruning. I'm exploring live runtime pruning in the Iceberg reader, and the newer APIs make the implementation substantially cleaner and more effective. If the bump is otherwise ready, it would be really helpful to have this on main. |
This was referenced Oct 4, 2026
unikdahal
pushed a commit
to unikdahal/datafusion-comet
that referenced
this pull request
Oct 5, 2026
…store 0.14 Point the DataFusion dependencies at a git revision of apache/datafusion main (4a5d58006e620bf6a175bb578be6652c3b330da9), which has moved to arrow/parquet 60.0.0 and object_store 0.14.2. Bump arrow, parquet, object_store and object_store_opendal (0.59.0, the first release on object_store 0.14) to match. iceberg-rust main is still on arrow 59, so pin iceberg and iceberg-storage-opendal to the head of apache/iceberg-rust#3257 (Arrow 60), which is not merged yet. Source changes for the new APIs: - Replace AlignedArrowStreamReader with arrow's ArrowArrayStreamReader. Arrow 60 makes the FFI_ArrowArrayStream callbacks private, and since arrow 59 from_ffi_and_data_type realigns under-aligned buffers itself, which is the reason the custom reader existed. The Decimal128 realignment test moves to scan.rs. - Parquet 60 moves the page index behind ParquetMetaData::page_index(). - object_store 0.14 adds GetResult::extensions. - OffsetBuffer::first/last return values, schema metadata is arrow's Metadata type, fb_to_schema is deprecated for try_fb_to_schema, and FFI_ArrowSchema::with_metadata is unsafe. - HashJoinExec::with_dynamic_filter_expr is deprecated with no replacement; allow it in DynamicFilterJoinExec. - Arrow 60 accepts empty entries in unsorted Variant dictionaries (apache/arrow-rs#10352), so two metadata encodings the empty-key workaround used to reject now pass through unchanged, as they do in Spark.
thorrester
added a commit
to bohmian-ai/iceberg-rust
that referenced
this pull request
Oct 7, 2026
Bumps arrow and parquet to 60.0 and pins DataFusion to bohmian-ai/datafusion 202c5e101c8e (v55.1.0 rebased on apache main with arrow 60). Ports the parquet 60 page-index API from apache#3257, takes TableProvider::scan projections as slices, and keeps the pre-60 Arrow map child names explicit in the name-mapping fixture. Claude-Session: https://claude.ai/code/session_01KGc5N2RkiRMFtCmHBpXqRV
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
What changes are included in this PR?
Bumps the
arrowdependency to 60, addressing all breaking API changes.Are these changes tested?
Yes, by existing tests
AI Disclosure
The code is fully generated by Codex and reviewed by me