[fix](rewrite rule) Reject mismatched compare plan in PullUpJoinFromUnionAll - #65472
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
TPC-H: Total hot run time: 29097 ms |
TPC-DS: Total hot run time: 180174 ms |
ClickBench: Total hot run time: 24.99 s |
FE UT Coverage ReportIncrement line coverage |
|
/review |
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
There was a problem hiding this comment.
I found one test-coverage issue in the new unit test. The implementation change itself is narrowly scoped and appears correct, but the added tests do not fail against the old project-only guard, so they do not prove the behavior that this PR actually changes.
Critical checkpoints:
- Goal/test proof: the code rejects mismatched output arity before comparator slot mapping, but the current test proof is incomplete because it uses
LogicalProjectcommon sides that old code already rejected. - Scope/focus: the code change is small and focused; no unrelated behavior change found.
- Nereids rewrite semantics: no rewrite-loop, mark-join, non-inner join, or join-condition remapping issue found in the changed path.
- Concurrency/lifecycle/config/persistence/FE-BE protocol: not applicable to this FE rewrite/test-only change.
- Parallel paths: checked scan/filter/project comparator paths; the missing coverage is specifically the supported non-project path affected by this PR.
- Performance/observability: no new concern found.
- User focus: no additional user-provided focus was present.
Validation: static review only, per the review-runner instruction not to build or modify code. Round 2 normal and risk-focused convergence subagents both returned NO_NEW_VALUABLE_FINDINGS for the current ledger/comment set.
| void comparatorRejectsDifferentProjectOutputSizes() { | ||
| LogicalOlapScan smallScan = newScan(1, "common_small"); | ||
| LogicalOlapScan largeScan = newScan(1, "common_large"); | ||
| LogicalProject<LogicalOlapScan> smallProject = project(selectSlots(smallScan.getOutput(), 0), smallScan); |
There was a problem hiding this comment.
These tests still wrap the mismatched common side in LogicalProject, so they do not distinguish the new guard from the old code. Before this patch, comparePlan already had a LogicalProject-local output-size check, so both this comparator test and the rewrite-skip test would still pass when commonSmall has one project output and commonLarge has two. The moved guard matters for supported non-project nodes: for example, two same-table LogicalOlapScan common sides whose outputs have been pruned/cached to different arity, optionally under the same LogicalFilter. Please make at least one test use that non-project shape so it fails without the new top-level output-size check.
There was a problem hiding this comment.
Updated the tests to use a non-project shape. The new cases cover LogicalFilter over same-table scans with different cached output arity, so they validate the top-level output-size guard instead of relying on the old LogicalProject-local check.
|
run buildall |
TPC-H: Total hot run time: 29361 ms |
TPC-DS: Total hot run time: 181841 ms |
ClickBench: Total hot run time: 25.27 s |
FE UT Coverage ReportIncrement line coverage |
|
/review |
|
Codex automated review failed and did not complete. Error: Review step was skipped (possibly timeout or cancelled) Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
FE Regression Coverage ReportIncrement line coverage |
|
/review |
|
PR approved by at least one committer and no changes requested. |
There was a problem hiding this comment.
Automated review completed for PR #65472 at head b9e5064.
I found one correctness issue in the new catalog-scan semantic comparison: two equal OLAP TABLESAMPLE clauses without REPEATABLE are independent random scan occurrences, but the comparator permits the rewrite to collapse them into one shared sample.
Critical checkpoint conclusions:
- Goal and proof: the patch substantially strengthens common-scan comparison, but equality of the stored sample parameters does not prove semantic equivalence for an unseeded OLAP sample. The focused test varies seeded sample parameters and does not cover this case.
- Scope and completeness: all 12 authoritative changed paths and every catalog-relation subclass used by the comparator were reviewed. The accepted issue is on a changed line and is not covered by an existing thread.
- Nereids plan semantics: I traced the reduced
UnionAll(Project(Join(...)), Project(Join(...)))tree through binding,PullUpJoinFromUnionAll, physical OLAP translation, and tablet sampling. Before the rewrite there are two random selections; afterwardsconstructNewJoin()retains one sampled scan. - Parallel paths and lifecycle: file sampling is deterministic from its stored seed, Hudi comparison is conservative, and stream comparison includes the captured read state. No additional concurrency, cleanup, retry, or ownership regression was found.
- Interfaces, compatibility, persistence, protocols, and data writes: no public API, persisted-format, replay, mixed-version, protocol, or write-path change is introduced by this patch.
- Performance and observability: comparison remains bounded by plan/output metadata; no additional material hot-path, logging, or metrics issue was found.
- Validation limits: this was a static review only, as required by the review runner; no build or test command was run. The focused FE test source and claimed proof paths were inspected directly.
Convergence: after candidate validation and duplicate suppression, two full-review agents and one dedicated risk challenger independently returned NO_NEW_VALUABLE_FINDINGS against this exact head, ledger, and final comment set.
…tor-migrated Hudi scan path Upstream #65472 added a hasSameScanSemantics/hasSameScanState guard to the union pull-up rewrite, with a PullUpJoinFromUnionAllTest case (comparatorRejectsDifferentHudiIncrementalRelations) that builds a TestLogicalHudiScan extends LogicalHudiScan and compares two Hudi scans by their datasource.hudi.source.IncrementalRelation. On this branch Hudi has been migrated to fe-connector-hudi: LogicalHudiScan is deleted and every external file table (Hudi included) binds to a plain LogicalFileScan (BindRelation), with the incremental-read window carried by scanParams (the incr(...) params) rather than a resolved IncrementalRelation on a dedicated logical node. So the upstream test no longer compiles here. Migrate the case to comparatorRejectsDifferentFileScanParams: a TestLogicalFileScan extends LogicalFileScan built via the protected SelectedPartitions ctor, comparing two scans that differ only in scanParams (INCREMENTAL_READ end_ts 10 vs 20). This preserves the test's intent ("a scan with a different incremental window must not be treated as the same scan") and gives real coverage of LogicalFileScan.hasSameScanState -- the exact path that now handles Hudi -- instead of dropping the case. Assertions map one-to-one: identical scanParams -> equal, different scanParams -> rejected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T2pc9qUhHnjcvLtzzuHaRE
…tor-migrated Hudi scan path Upstream #65472 added a hasSameScanSemantics/hasSameScanState guard to the union pull-up rewrite, with a PullUpJoinFromUnionAllTest case (comparatorRejectsDifferentHudiIncrementalRelations) that builds a TestLogicalHudiScan extends LogicalHudiScan and compares two Hudi scans by their datasource.hudi.source.IncrementalRelation. On this branch Hudi has been migrated to fe-connector-hudi: LogicalHudiScan is deleted and every external file table (Hudi included) binds to a plain LogicalFileScan (BindRelation), with the incremental-read window carried by scanParams (the incr(...) params) rather than a resolved IncrementalRelation on a dedicated logical node. So the upstream test no longer compiles here. Migrate the case to comparatorRejectsDifferentFileScanParams: a TestLogicalFileScan extends LogicalFileScan built via the protected SelectedPartitions ctor, comparing two scans that differ only in scanParams (INCREMENTAL_READ end_ts 10 vs 20). This preserves the test's intent ("a scan with a different incremental window must not be treated as the same scan") and gives real coverage of LogicalFileScan.hasSameScanState -- the exact path that now handles Hudi -- instead of dropping the case. Assertions map one-to-one: identical scanParams -> equal, different scanParams -> rejected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T2pc9qUhHnjcvLtzzuHaRE
…tor-migrated Hudi scan path Upstream #65472 added a hasSameScanSemantics/hasSameScanState guard to the union pull-up rewrite, with a PullUpJoinFromUnionAllTest case (comparatorRejectsDifferentHudiIncrementalRelations) that builds a TestLogicalHudiScan extends LogicalHudiScan and compares two Hudi scans by their datasource.hudi.source.IncrementalRelation. On this branch Hudi has been migrated to fe-connector-hudi: LogicalHudiScan is deleted and every external file table (Hudi included) binds to a plain LogicalFileScan (BindRelation), with the incremental-read window carried by scanParams (the incr(...) params) rather than a resolved IncrementalRelation on a dedicated logical node. So the upstream test no longer compiles here. Migrate the case to comparatorRejectsDifferentFileScanParams: a TestLogicalFileScan extends LogicalFileScan built via the protected SelectedPartitions ctor, comparing two scans that differ only in scanParams (INCREMENTAL_READ end_ts 10 vs 20). This preserves the test's intent ("a scan with a different incremental window must not be treated as the same scan") and gives real coverage of LogicalFileScan.hasSameScanState -- the exact path that now handles Hudi -- instead of dropping the case. Assertions map one-to-one: identical scanParams -> equal, different scanParams -> rejected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T2pc9qUhHnjcvLtzzuHaRE
…tor-migrated Hudi scan path Upstream #65472 added a hasSameScanSemantics/hasSameScanState guard to the union pull-up rewrite, with a PullUpJoinFromUnionAllTest case (comparatorRejectsDifferentHudiIncrementalRelations) that builds a TestLogicalHudiScan extends LogicalHudiScan and compares two Hudi scans by their datasource.hudi.source.IncrementalRelation. On this branch Hudi has been migrated to fe-connector-hudi: LogicalHudiScan is deleted and every external file table (Hudi included) binds to a plain LogicalFileScan (BindRelation), with the incremental-read window carried by scanParams (the incr(...) params) rather than a resolved IncrementalRelation on a dedicated logical node. So the upstream test no longer compiles here. Migrate the case to comparatorRejectsDifferentFileScanParams: a TestLogicalFileScan extends LogicalFileScan built via the protected SelectedPartitions ctor, comparing two scans that differ only in scanParams (INCREMENTAL_READ end_ts 10 vs 20). This preserves the test's intent ("a scan with a different incremental window must not be treated as the same scan") and gives real coverage of LogicalFileScan.hasSameScanState -- the exact path that now handles Hudi -- instead of dropping the case. Assertions map one-to-one: identical scanParams -> equal, different scanParams -> rejected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T2pc9qUhHnjcvLtzzuHaRE
…tor-migrated Hudi scan path Upstream #65472 added a hasSameScanSemantics/hasSameScanState guard to the union pull-up rewrite, with a PullUpJoinFromUnionAllTest case (comparatorRejectsDifferentHudiIncrementalRelations) that builds a TestLogicalHudiScan extends LogicalHudiScan and compares two Hudi scans by their datasource.hudi.source.IncrementalRelation. On this branch Hudi has been migrated to fe-connector-hudi: LogicalHudiScan is deleted and every external file table (Hudi included) binds to a plain LogicalFileScan (BindRelation), with the incremental-read window carried by scanParams (the incr(...) params) rather than a resolved IncrementalRelation on a dedicated logical node. So the upstream test no longer compiles here. Migrate the case to comparatorRejectsDifferentFileScanParams: a TestLogicalFileScan extends LogicalFileScan built via the protected SelectedPartitions ctor, comparing two scans that differ only in scanParams (INCREMENTAL_READ end_ts 10 vs 20). This preserves the test's intent ("a scan with a different incremental window must not be treated as the same scan") and gives real coverage of LogicalFileScan.hasSameScanState -- the exact path that now handles Hudi -- instead of dropping the case. Assertions map one-to-one: identical scanParams -> equal, different scanParams -> rejected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T2pc9qUhHnjcvLtzzuHaRE
…tor-migrated Hudi scan path Upstream #65472 added a hasSameScanSemantics/hasSameScanState guard to the union pull-up rewrite, with a PullUpJoinFromUnionAllTest case (comparatorRejectsDifferentHudiIncrementalRelations) that builds a TestLogicalHudiScan extends LogicalHudiScan and compares two Hudi scans by their datasource.hudi.source.IncrementalRelation. On this branch Hudi has been migrated to fe-connector-hudi: LogicalHudiScan is deleted and every external file table (Hudi included) binds to a plain LogicalFileScan (BindRelation), with the incremental-read window carried by scanParams (the incr(...) params) rather than a resolved IncrementalRelation on a dedicated logical node. So the upstream test no longer compiles here. Migrate the case to comparatorRejectsDifferentFileScanParams: a TestLogicalFileScan extends LogicalFileScan built via the protected SelectedPartitions ctor, comparing two scans that differ only in scanParams (INCREMENTAL_READ end_ts 10 vs 20). This preserves the test's intent ("a scan with a different incremental window must not be treated as the same scan") and gives real coverage of LogicalFileScan.hasSameScanState -- the exact path that now handles Hudi -- instead of dropping the case. Assertions map one-to-one: identical scanParams -> equal, different scanParams -> rejected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T2pc9qUhHnjcvLtzzuHaRE
…nionAll (apache#65472) Avoid comparing and remapping join inputs as equivalent when the common-side projects expose different output arity. This keeps the union pull-up rewrite aligned with the actual plan shape and adds focused coverage for the comparator and rewrite guard. Key changes: - Add an output-size guard before comparing plan nodes in PullUpJoinFromUnionAll. - Add a FE unit test that covers both the comparator and the rewrite path. Unit Test: - PullUpJoinFromUnionAllTest relate PR: apache#28682
…tor-migrated Hudi scan path Upstream #65472 added a hasSameScanSemantics/hasSameScanState guard to the union pull-up rewrite, with a PullUpJoinFromUnionAllTest case (comparatorRejectsDifferentHudiIncrementalRelations) that builds a TestLogicalHudiScan extends LogicalHudiScan and compares two Hudi scans by their datasource.hudi.source.IncrementalRelation. On this branch Hudi has been migrated to fe-connector-hudi: LogicalHudiScan is deleted and every external file table (Hudi included) binds to a plain LogicalFileScan (BindRelation), with the incremental-read window carried by scanParams (the incr(...) params) rather than a resolved IncrementalRelation on a dedicated logical node. So the upstream test no longer compiles here. Migrate the case to comparatorRejectsDifferentFileScanParams: a TestLogicalFileScan extends LogicalFileScan built via the protected SelectedPartitions ctor, comparing two scans that differ only in scanParams (INCREMENTAL_READ end_ts 10 vs 20). This preserves the test's intent ("a scan with a different incremental window must not be treated as the same scan") and gives real coverage of LogicalFileScan.hasSameScanState -- the exact path that now handles Hudi -- instead of dropping the case. Assertions map one-to-one: identical scanParams -> equal, different scanParams -> rejected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T2pc9qUhHnjcvLtzzuHaRE
…tor-migrated Hudi scan path Upstream #65472 added a hasSameScanSemantics/hasSameScanState guard to the union pull-up rewrite, with a PullUpJoinFromUnionAllTest case (comparatorRejectsDifferentHudiIncrementalRelations) that builds a TestLogicalHudiScan extends LogicalHudiScan and compares two Hudi scans by their datasource.hudi.source.IncrementalRelation. On this branch Hudi has been migrated to fe-connector-hudi: LogicalHudiScan is deleted and every external file table (Hudi included) binds to a plain LogicalFileScan (BindRelation), with the incremental-read window carried by scanParams (the incr(...) params) rather than a resolved IncrementalRelation on a dedicated logical node. So the upstream test no longer compiles here. Migrate the case to comparatorRejectsDifferentFileScanParams: a TestLogicalFileScan extends LogicalFileScan built via the protected SelectedPartitions ctor, comparing two scans that differ only in scanParams (INCREMENTAL_READ end_ts 10 vs 20). This preserves the test's intent ("a scan with a different incremental window must not be treated as the same scan") and gives real coverage of LogicalFileScan.hasSameScanState -- the exact path that now handles Hudi -- instead of dropping the case. Assertions map one-to-one: identical scanParams -> equal, different scanParams -> rejected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T2pc9qUhHnjcvLtzzuHaRE
…tor-migrated Hudi scan path Upstream #65472 added a hasSameScanSemantics/hasSameScanState guard to the union pull-up rewrite, with a PullUpJoinFromUnionAllTest case (comparatorRejectsDifferentHudiIncrementalRelations) that builds a TestLogicalHudiScan extends LogicalHudiScan and compares two Hudi scans by their datasource.hudi.source.IncrementalRelation. On this branch Hudi has been migrated to fe-connector-hudi: LogicalHudiScan is deleted and every external file table (Hudi included) binds to a plain LogicalFileScan (BindRelation), with the incremental-read window carried by scanParams (the incr(...) params) rather than a resolved IncrementalRelation on a dedicated logical node. So the upstream test no longer compiles here. Migrate the case to comparatorRejectsDifferentFileScanParams: a TestLogicalFileScan extends LogicalFileScan built via the protected SelectedPartitions ctor, comparing two scans that differ only in scanParams (INCREMENTAL_READ end_ts 10 vs 20). This preserves the test's intent ("a scan with a different incremental window must not be treated as the same scan") and gives real coverage of LogicalFileScan.hasSameScanState -- the exact path that now handles Hudi -- instead of dropping the case. Assertions map one-to-one: identical scanParams -> equal, different scanParams -> rejected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T2pc9qUhHnjcvLtzzuHaRE
…tor-migrated Hudi scan path Upstream #65472 added a hasSameScanSemantics/hasSameScanState guard to the union pull-up rewrite, with a PullUpJoinFromUnionAllTest case (comparatorRejectsDifferentHudiIncrementalRelations) that builds a TestLogicalHudiScan extends LogicalHudiScan and compares two Hudi scans by their datasource.hudi.source.IncrementalRelation. On this branch Hudi has been migrated to fe-connector-hudi: LogicalHudiScan is deleted and every external file table (Hudi included) binds to a plain LogicalFileScan (BindRelation), with the incremental-read window carried by scanParams (the incr(...) params) rather than a resolved IncrementalRelation on a dedicated logical node. So the upstream test no longer compiles here. Migrate the case to comparatorRejectsDifferentFileScanParams: a TestLogicalFileScan extends LogicalFileScan built via the protected SelectedPartitions ctor, comparing two scans that differ only in scanParams (INCREMENTAL_READ end_ts 10 vs 20). This preserves the test's intent ("a scan with a different incremental window must not be treated as the same scan") and gives real coverage of LogicalFileScan.hasSameScanState -- the exact path that now handles Hudi -- instead of dropping the case. Assertions map one-to-one: identical scanParams -> equal, different scanParams -> rejected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T2pc9qUhHnjcvLtzzuHaRE
…tor-migrated Hudi scan path Upstream #65472 added a hasSameScanSemantics/hasSameScanState guard to the union pull-up rewrite, with a PullUpJoinFromUnionAllTest case (comparatorRejectsDifferentHudiIncrementalRelations) that builds a TestLogicalHudiScan extends LogicalHudiScan and compares two Hudi scans by their datasource.hudi.source.IncrementalRelation. On this branch Hudi has been migrated to fe-connector-hudi: LogicalHudiScan is deleted and every external file table (Hudi included) binds to a plain LogicalFileScan (BindRelation), with the incremental-read window carried by scanParams (the incr(...) params) rather than a resolved IncrementalRelation on a dedicated logical node. So the upstream test no longer compiles here. Migrate the case to comparatorRejectsDifferentFileScanParams: a TestLogicalFileScan extends LogicalFileScan built via the protected SelectedPartitions ctor, comparing two scans that differ only in scanParams (INCREMENTAL_READ end_ts 10 vs 20). This preserves the test's intent ("a scan with a different incremental window must not be treated as the same scan") and gives real coverage of LogicalFileScan.hasSameScanState -- the exact path that now handles Hudi -- instead of dropping the case. Assertions map one-to-one: identical scanParams -> equal, different scanParams -> rejected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T2pc9qUhHnjcvLtzzuHaRE
…tor-migrated Hudi scan path Upstream #65472 added a hasSameScanSemantics/hasSameScanState guard to the union pull-up rewrite, with a PullUpJoinFromUnionAllTest case (comparatorRejectsDifferentHudiIncrementalRelations) that builds a TestLogicalHudiScan extends LogicalHudiScan and compares two Hudi scans by their datasource.hudi.source.IncrementalRelation. On this branch Hudi has been migrated to fe-connector-hudi: LogicalHudiScan is deleted and every external file table (Hudi included) binds to a plain LogicalFileScan (BindRelation), with the incremental-read window carried by scanParams (the incr(...) params) rather than a resolved IncrementalRelation on a dedicated logical node. So the upstream test no longer compiles here. Migrate the case to comparatorRejectsDifferentFileScanParams: a TestLogicalFileScan extends LogicalFileScan built via the protected SelectedPartitions ctor, comparing two scans that differ only in scanParams (INCREMENTAL_READ end_ts 10 vs 20). This preserves the test's intent ("a scan with a different incremental window must not be treated as the same scan") and gives real coverage of LogicalFileScan.hasSameScanState -- the exact path that now handles Hudi -- instead of dropping the case. Assertions map one-to-one: identical scanParams -> equal, different scanParams -> rejected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T2pc9qUhHnjcvLtzzuHaRE
Avoid comparing and remapping join inputs as equivalent when the common-side projects expose different output arity. This keeps the union pull-up rewrite aligned with the actual plan shape and adds focused coverage for the comparator and rewrite guard.
Key changes:
Unit Test:
relate PR: #28682