Repository navigation
Conversation
Execution engines learn restrictions while a query runs, such as a completed join build's key range or an improving TopK or MIN/MAX bound. Add RuntimePredicateSnapshot and RuntimePredicateProvider so engines can publish them as Iceberg predicates, and ArrowReaderBuilder::with_runtime_predicate_provider to apply them. When a data-file task starts, the reader checks the provider's generation. Only when it changed does the reader take a snapshot, push NOT down to the leaves and bind the predicate. Refreshes are serialized, so a generation is snapshotted and bound once, and the cache never moves to an older generation. The bound predicate is shared, without copying, by later tasks of the scan with the same generation and schema. The runtime predicate is planned separately from the planned and equality-delete predicates, so the shared bound tree is never copied. They are then combined into one Arrow row filter, which decodes shared columns once, and their row-group, bloom-filter and page-index selections are intersected, row groups in linear time. A bloom filter is read at most once per row group and column, even when both predicates use it. Runtime predicates are advisory. Provider, binding and planning failures, and page-index failures, drop only the runtime part for that file; the planned and delete filtering are unchanged. A predicate on a column missing from the file (its initial default or null is invisible to physical filters) or stored with a type that needs promotion (a literal cast before promotion could round or overflow) is ignored for that file. Positional deletes keep their original file positions. The provider contract documents that data skipped under a published predicate must never be required later, that a later None only stops runtime pruning for tasks that start afterwards, that generations increase monotonically and are never reused, and that provider, snapshot, binding, planning and page-pruning failures are advisory. The reader accepts the provider; passing one through TableScan is a follow-up.
21 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
What changes are included in this PR?
Lets execution engines hand restrictions learned during a query (a join build's key range, a TopK or MIN/MAX bound) to
ArrowReaderas Iceberg predicates.RuntimePredicateProvider(a cheapgeneration()plussnapshot()),RuntimePredicateSnapshot, andArrowReaderBuilder::with_runtime_predicate_provider. Without a provider, behaviour is unchanged.Nonemeans, generation ordering) is documented on the trait, with an example.This is the reader layer only.
TableScan::to_arrowcannot pass a provider yet; that wiring is a follow-up.Are these changes tested?
Yes. Unit tests cover the cache under concurrent tasks and generation changes, row-group, page and byte savings, fail-open paths, schema evolution and type promotion, delete handling, bloom filters, and
Some->None-> tighterSometransitions. CI passes on currentmain.AI Disclosure
Assisted by Claude Code, reviewed by me.