[test](iceberg) Add partition evolution schema coverage - #65870
Merged
Gabriel39 merged 1 commit intoJul 28, 2026
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Automated review conclusion: approval opinion; no blocking correctness issue found.
- Goal and scope: This behavior-neutral refactor preserves the existing all-column Iceberg schema initialization and adds a focused FE unit test showing that a non-projected column remains in
history_schema_info. The patch is small, clear, and aligned with the stated partition-evolution invariant. - Data correctness and parallel paths: For an older non-identity-spec file, FE emits no identity partition constant for the evolved source column. Classic Parquet/ORC readers and format-v2 therefore use the complete schema metadata to resolve and read the physical field; newer identity-spec files can use their split constant. Eager/lazy split assignment and both scanner generations retain the same shared params. No missed parallel path was found.
- Concurrency, lifecycle, and errors: No new threads, locks, shared mutable lifecycle, status handling, or exception boundary are introduced.
UserExceptionpropagation and scan-parameter lifetime are unchanged. - Compatibility, persistence, configuration, and writes: No Thrift field, persisted format, configuration, transaction, data-write path, or rolling-upgrade contract changes. The existing Iceberg semantics-version boundary remains intact.
- Tests and results: The new deterministic unit test directly protects the FE all-column serialization seam even when the evolved column is absent from query slots. Existing partition-evolution regression coverage exercises old physical data after adding an identity partition. An exact bucket-to-identity ORC case across both scanner modes would expand the matrix but is not required to establish this behavior-neutral change. Per the review-runner contract, I performed static review only and did not run builds or tests locally.
- Performance and observability: Runtime work and allocations are unchanged; no new hot-path cost, logging, or metric need was introduced.
- User focus: No additional user-provided focus points were supplied.
All applicable Doris critical checkpoints were reviewed, and no actionable finding remains.
Contributor
TPC-H: Total hot run time: 29339 ms |
Contributor
TPC-DS: Total hot run time: 177180 ms |
Contributor
ClickBench: Total hot run time: 25.12 s |
Gabriel39
changed the title
924060929
approved these changes
Jul 28, 2026
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
github-actions Bot
pushed a commit
that referenced
this pull request
Jul 28, 2026
### What problem does this PR solve? Related PR: #65502 Problem Summary: #65502 already fixed the Iceberg scan correctness issue by sending the complete current table schema to readers. Although that change was introduced for equality-delete dependencies, the same all-column schema invariant is also required for partition evolution: a column classified as an identity partition column by a newer partition spec can still be stored physically in files written by an older spec. This follow-up does not change scan behavior. It documents the partition-evolution invariant, extracts the existing schema initialization into a testable helper, and adds a focused FE unit test verifying that a non-projected evolved partition column remains in the reader schema. ### Release note None ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [x] No. - [ ] Yes. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
morningman
added a commit
that referenced
this pull request
Jul 28, 2026
…ector-iceberg Upstream #65870 is behaviour-inert: it extracts IcebergScanNode's schema-info initialization into a @VisibleForTesting helper, documents that an identity partition column added by a newer spec can still be stored physically in files written by an older spec, and pins the all-column reader schema with one FE unit test. This branch deleted the fe-core iceberg subsystem, so the fe-core edit is moot; what matters is whether the connector upholds the same invariant. It does, by a different route, so no capability had to be migrated: identity partition columns are unioned over ALL partition specs into path_partition_keys, FileQueryScanNode.classifyColumn turns those into TColumnCategory.PARTITION_KEY (non-file slots), and BE resolves through the field-id dictionary only the slots it decodes from the file. The upstream test therefore cannot be transplanted verbatim — it asserts an all-column dictionary the connector deliberately prunes. Instead pin the property the connector actually relies on, which had no multi-spec coverage: getIdentityPartitionColumns must union every spec, not just the current one. Mutating table.specs() to table.spec() turns the new test red and leaves the other 62 green. Also record the reasoning at the dictionary construction site. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hubgeter
added a commit
to hubgeter/doris
that referenced
this pull request
Jul 28, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#65851, apache#65870 Problem Summary: Rebasing the Iceberg V3 default-value implementation onto current master conflicted with partition-evolution schema coverage. The upstream helper serialized Doris catalog columns, while this PR must keep defaults, binary markers, required-field metadata, equality-delete history, and partition-evolution fields bound to the statement-pinned Iceberg schema. Extract the final carrier initialization behind a testable schema-aware helper, update the partition-evolution unit test to exercise that path, and remove a generated trailing blank line found by the final diff check. ### Release note None ### Check List (For Author) - Test: Unit Test / Regression test / Build - Focused FE tests: 132/132 passed on apache/master 7488bc7 - Regression suites: test_iceberg_write_evolution_refs, test_iceberg_initial_defaults, test_iceberg_schema_ref_actions_matrix, and iceberg_branch_tag_schema_change_extended passed - Full ASAN FE+BE build: ./build.sh --be --fe -j16 passed before the final unrelated CCR-only master commit - Style: Maven checkstyle, build-support/check-format.sh, and git diff --check passed - Behavior changed: No. This preserves the existing Iceberg schema-carrier contract while integrating current master. - Does this need documentation: No
yiguolei
pushed a commit
that referenced
this pull request
Jul 29, 2026
…6176) ## Summary - Backport #65955 as the prerequisite for Paimon table option passthrough. - Backport #66065 for Paimon read/write P0 coverage. - Backport #65984 for query-level Paimon dynamic options. - Backport #65870 for Iceberg partition-evolution schema coverage. - Preserve the master merge order and adapt the implementation only for branch-4.1 API compatibility. Regression cases and expected results are unchanged from the original PRs. ## Validation - Full Release FE/BE build: passed. - Targeted FE unit tests: 188 tests passed, 0 failures/errors/skips. - Targeted BE ASAN unit tests: 22 tests passed across `PaimonJniReaderTest` and `PaimonReaderTest`. - External Paimon regression: 8 suites passed, 0 failed/fatal/skipped scripts. --------- Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
morningman
added a commit
that referenced
this pull request
Jul 29, 2026
…ector-iceberg Upstream #65870 is behaviour-inert: it extracts IcebergScanNode's schema-info initialization into a @VisibleForTesting helper, documents that an identity partition column added by a newer spec can still be stored physically in files written by an older spec, and pins the all-column reader schema with one FE unit test. This branch deleted the fe-core iceberg subsystem, so the fe-core edit is moot; what matters is whether the connector upholds the same invariant. It does, by a different route, so no capability had to be migrated: identity partition columns are unioned over ALL partition specs into path_partition_keys, FileQueryScanNode.classifyColumn turns those into TColumnCategory.PARTITION_KEY (non-file slots), and BE resolves through the field-id dictionary only the slots it decodes from the file. The upstream test therefore cannot be transplanted verbatim — it asserts an all-column dictionary the connector deliberately prunes. Instead pin the property the connector actually relies on, which had no multi-spec coverage: getIdentityPartitionColumns must union every spec, not just the current one. Mutating table.specs() to table.spec() turns the new test red and leaves the other 62 green. Also record the reasoning at the dictionary construction site. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
morningman
added a commit
that referenced
this pull request
Jul 29, 2026
…ector-iceberg Upstream #65870 is behaviour-inert: it extracts IcebergScanNode's schema-info initialization into a @VisibleForTesting helper, documents that an identity partition column added by a newer spec can still be stored physically in files written by an older spec, and pins the all-column reader schema with one FE unit test. This branch deleted the fe-core iceberg subsystem, so the fe-core edit is moot; what matters is whether the connector upholds the same invariant. It does, by a different route, so no capability had to be migrated: identity partition columns are unioned over ALL partition specs into path_partition_keys, FileQueryScanNode.classifyColumn turns those into TColumnCategory.PARTITION_KEY (non-file slots), and BE resolves through the field-id dictionary only the slots it decodes from the file. The upstream test therefore cannot be transplanted verbatim — it asserts an all-column dictionary the connector deliberately prunes. Instead pin the property the connector actually relies on, which had no multi-spec coverage: getIdentityPartitionColumns must union every spec, not just the current one. Mutating table.specs() to table.spec() turns the new test red and leaves the other 62 green. Also record the reasoning at the dictionary construction site. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
28 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related PR: #65502
Problem Summary:
#65502 already fixed the Iceberg scan correctness issue by sending the complete current table schema to readers. Although that change was introduced for equality-delete dependencies, the same all-column schema invariant is also required for partition evolution: a column classified as an identity partition column by a newer partition spec can still be stored physically in files written by an older spec.
This follow-up does not change scan behavior. It documents the partition-evolution invariant, extracts the existing schema initialization into a testable helper, and adds a focused FE unit test verifying that a non-projected evolved partition column remains in the reader schema.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)