Skip to content

branch-4.1: [fix](multi-catalog) Preserve external partition metadata - #66012

Open
suxiaogang223 wants to merge 2 commits into
apache:branch-4.1from
suxiaogang223:codex/fix-4.1-per-range-partition-values
Open

branch-4.1: [fix](multi-catalog) Preserve external partition metadata#66012
suxiaogang223 wants to merge 2 commits into
apache:branch-4.1from
suxiaogang223:codex/fix-4.1-per-range-partition-values

Conversation

@suxiaogang223

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: None

Related PR: #62821, #65581, #66010

Problem Summary: branch-4.1 already backported the Paimon and BE partition metadata changes through #65581, but Hive-style scans, Hudi, Iceberg, and file load paths still had incomplete FE metadata propagation. Hive default partitions were encoded as \N, which lost the distinction between NULL and a literal \N. Iceberg attached partition metadata only when runtime partition pruning was enabled, used unstable per-map ordering, and did not safely handle files written with different partition specs.

This change:

  • propagates aligned partition values and explicit NULL flags through Hive-style scans, Hudi, and both legacy and Nereids file load paths;
  • preserves a literal \N as data while representing Hive default partitions as NULL;
  • sends stable Iceberg identity-partition metadata for every split, keyed by both spec ID and partition data;
  • restricts Iceberg path partition keys to identity columns shared by all partition specs, which is safe for both FileScannerV2 and the legacy scanner's scan-level slot mapping.

Release note

Fix partition-column materialization, NULL handling, and Iceberg partition evolution compatibility for external table scans and file loads on branch-4.1.

Check List (For Author)

  • Test: Unit Test
    • ./run-fe-ut.sh --run org.apache.doris.common.util.BrokerUtilTest,org.apache.doris.datasource.hive.source.HiveScanNodeTest,org.apache.doris.datasource.iceberg.IcebergUtilsTest,org.apache.doris.datasource.iceberg.source.IcebergScanNodeTest,org.apache.doris.datasource.paimon.source.PaimonScanNodeTest
  • Behavior changed: Yes. External partition metadata and NULL values are materialized consistently per split.
  • Does this need documentation: No

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 52.48% (53/101) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 71.29% (72/101) 🎉
Increment coverage report
Complete coverage report

@suxiaogang223
suxiaogang223 marked this pull request as ready for review July 27, 2026 02:01
@suxiaogang223
suxiaogang223 requested a review from yiguolei as a code owner July 27, 2026 02:01
@suxiaogang223
suxiaogang223 force-pushed the codex/fix-4.1-per-range-partition-values branch from 8d7da1a to 19633ca Compare July 30, 2026 06:27
Backport the remaining external partition fixes to branch-4.1 while adapting them to its FileScannerV2 infrastructure. Preserve explicit NULL metadata for Hive-style scans, Hudi, and load paths, and make Iceberg identity partition metadata stable and safe across partition spec evolution.
@suxiaogang223
suxiaogang223 force-pushed the codex/fix-4.1-per-range-partition-values branch from 19633ca to eb977fc Compare July 30, 2026 06:32
@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 70.71% (70/99) 🎉
Increment coverage report
Complete coverage report

@suxiaogang223

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 53.70% (58/108) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 21.30% (23/108) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants