Merge upstream/main: bloom filters, row lineage, new REST auth architecture

[gbrgr]

Summary

Merges apache/iceberg-rust@main into the fork, bringing in ~8 months of upstream changes while preserving all fork-only features (incremental scan, vended credentials, custom REST authenticator, storage-credential refresh, OverwriteAction, binary type promotion).

Notable upstream features adopted:

• Row lineage (_row_id metadata column), including full support in the incremental-scan reader path
• Bloom-filter row-group pruning (ArrowReaderBuilder::with_bloom_filter_enabled), also wired into the incremental-scan reader path (IncrementalTableScanBuilder::with_bloom_filter_enabled)
• Parquet Modular Encryption plumbing through the reader pipeline
• _partition / _last_updated_sequence_number metadata columns
• New REST-catalog AuthSession/AuthManager architecture (fork's CustomAuthenticator/vended-credentials support ported onto it)
• KMS / custom-Runtime wiring, DeleteFileIndexState::Failed fallible delete-index population

Fork-only features carried forward unchanged in behavior, updated only where required to compile against upstream's new APIs:

• Incremental table scan (crates/iceberg/src/scan/incremental, crates/iceberg/src/arrow/incremental.rs)
• REST-catalog vended credentials / storage-credential refresh
• OverwriteAction (this PR also cherry-picks fix(transaction): fix correctness bugs in OverwriteAction #81's outstanding correctness fixes for it — see below)
• Binary type promotion

Notable fixes made during conflict resolution

• record_batch_transformer.rs: adopted upstream's _pos metadata-column fix (the original motivation for this merge — a same-type column reorder was being mislabeled instead of reordered).
• arrow/delete_filter.rs: fixed a latent bug (file_scan_task.deletes → .deletes()) left over from FileScanTask's fields becoming private with accessor methods upstream.
• delete_file_index.rs: two fork-only methods (get_equality_deletes_for_data_file, all_deletes) were not handling the new DeleteFileIndexState::Failed variant (non-exhaustive match); made both fallible and propagated the error at their two call sites in scan/incremental/mod.rs.
• arrow/reader/pipeline.rs: preserved the fork's helper-function architecture (shared between the full scan and incremental scan) while folding in bloom-filter support and _row_id synthesis/coalescing, mirroring the existing _last_updated_sequence_number pattern.
• arrow/incremental.rs: threaded bloom-filter and RowNumber-virtual-column (_row_id positional-fallback) support into the incremental append-task reader, matching the full-scan path.
• catalog/rest/src/catalog.rs: fixed with_token_authenticator wrapping CustomAuthenticatorManager in an extra Arc that with_auth_manager<M: AuthManager> didn't expect.

Follow-up commits after initial review

• fix(incremental-scan): expose bloom-filter row-group pruning on IncrementalTableScanBuilder — the incremental reader path threaded bloom_filter_enabled all the way through, but IncrementalTableScan::to_arrow()/to_unzipped_arrow() never called .with_bloom_filter_enabled(), leaving it permanently disabled and unreachable. Added a builder method mirroring TableScanBuilder's, defaulting to disabled.
• fix(ci): drop crates.io version pin from iceberg-storage-refreshable workspace dep — crates/storage/refreshable is publish = false (fork-only, never published), but the workspace dependency table gave it a version like every published crate, which broke cargo publish --dry-run for iceberg-catalog-rest (its only, dev-only consumer). Dropped the version, matching the existing pattern for iceberg_test_utils.
• fix(scan): tolerate a dropped partition source column in FileScanTask::validate() — a real regression caught by a post-merge audit: upstream's new FileScanTask::validate() (added in refactor(scan): validate FileScanTask construction apache/iceberg-rust#3131, landed here via an unconflicted auto-merge of scan/task.rs) unconditionally calls partition_spec.partition_type(&schema), which errors for a historical partition spec whose source column has since been dropped — a legitimate v2+ state the fork already tolerates one layer up in PartitionFilterCache::get. During conflict resolution of scan/mod.rs, the corresponding regression test was updated to assert the new error instead of catching the incompatibility. Fixed validate() to mirror PartitionFilterCache::get's tolerance, and restored the test to its original success-path assertions.
• Cherry-picked fix(transaction): fix correctness bugs in OverwriteAction #81 (fix(transaction): fix correctness bugs in OverwriteAction) into this branch: a resurrection bug where an already-deleted manifest entry could be silently resurrected on a later, unrelated manifest rewrite; wrong schema/partition-spec-id stamped on rewritten manifests (used the table's current schema instead of the manifest's own); manifest naming/location bypassing metadata_location()/commit-uuid convention; and SnapshotProducer::summary() wrongly applying full-table-truncate semantics to OverwriteAction's partial overwrites.

Test plan

• cargo check --workspace --all-targets --all-features clean
• cargo test -p iceberg --lib: 1791 passed, 0 failed
• cargo test --workspace --all-features (excluding python bindings): all unit/doc tests pass; only live-infra integration tests fail locally (S3/minio, HMS thrift, REST catalog — no server running in this sandbox), consistent with existing CI-only test setup
• cargo clippy --workspace --all-targets --all-features -- -D warnings clean
• cargo fmt --all -- --check clean
• cargo machete clean
• make check-public-api clean (one new public method added and reflected: IncrementalTableScanBuilder::with_bloom_filter_enabled)
• Two independent post-merge audits done: (1) incremental-scan feature parity vs. the full-scan path, (2) fork-only-feature preservation + a systematic before/after dropped-test check across every file touched by the merge — both findings addressed above
• CI green (see checks below)

🤖 Generated with Claude Code