[EPIC] Execution-time adaptive pruning for Iceberg readers #3343

What's the feature are you trying to implement?

Summary

Add generic execution-time predicate support to iceberg-rust so readers can progressively avoid file, row-group, page, and payload reads as predicates become available during query execution.

The reader remains producer-agnostic: callers publish Iceberg predicates and generations, while iceberg-rust handles safe application through the existing reader pipeline.

This epic tracks the upstream reader infrastructure required by execution engines such as DataFusion Comet.

Related work

• Runtime predicate discussion: feat(datafusion): runtime filter / dynamic predicate pushdown into IcebergTableScan #2376
• Live predicate discussion: Explore live runtime predicate refresh during Arrow reads #3336
• Arrow 60 upgrade: Bump to Arrow 60 #3257

Work items

Dependencies

• Bump to Arrow 60 #3257

Reader foundations

• Resolve unknown delete-file sizes lazily — Support unknown delete-file sizes in externally constructed scan tasks #3344, PR feat(reader): resolve unknown delete-file sizes lazily #3345
• Reuse the inclusive statistics evaluator with borrowed metrics — Evaluate inclusive metrics from borrowed file statistics #3346, PR refactor(expr): evaluate inclusive metrics from borrowed statistics #3347
• Add the generic runtime predicate publication API — Support execution-time predicates in ArrowReader #3350, PR feat(reader): support execution-time predicates in ArrowReader #3351
• Apply runtime predicates at file-task startup — Support execution-time predicates in ArrowReader #3350, PR feat(reader): support execution-time predicates in ArrowReader #3351
• Add opt-in file statistics retention to scan tasks

Adaptive pruning

• Reject file tasks before footer and delete-file I/O
• Refresh predicates and prune unread row groups
• Represent static page selections per row group
• Preserve position and equality deletes during live pruning
• Refresh page pruning and row filters after predicate updates
• Preserve late materialization with runtime predicates

Correctness and robustness

• Bound refresh overhead and cover reader edge cases
• Add adaptive pruning observability and reader-level metrics
• Complete delete, split-task, schema-evolution, and fallback coverage

Correctness principles

• Runtime predicates are conservative restrictions.
• Previously skipped data is never required by a later predicate publication.
• Missing or unusable statistics retain candidates.
• Runtime predicate failures must not disable planned filtering or delete handling.
• Position deletes and metadata row positions preserve original file ordinals.
• Equality deletes remain active across predicate refreshes.
• Existing reader and schema adaptation paths should be reused rather than duplicated.

Definition of done

• No private Arrow/Parquet patch is required.
• Generic task-start and live runtime predicate APIs are upstream.
• Whole-file, row-group, page, and row-filter pruning use the existing reader pipeline.
• Position deletes, equality deletes, and combined deletes are supported.
• Stable predicates add minimal reader overhead.
• Each work item above is linked to its corresponding issue or PR.

Willingness to contribute

I can contribute to this feature independently