branch-incremental-computation: pick the merged incremental-computation PRs from master in merge order (#68138) - #68303
Merged
Conversation
…pache#68138) ### What problem does this PR solve? Issue Number: None Related PR: apache#67181 Problem Summary: A cloud incremental read may wait for its target transaction to become visible before query planning continues. During that wait, the FE `CloudPartition` cache can still contain `PARTITION_INIT_VERSION` for a partition even though the transaction has already become visible in MetaService. `PruneEmptyPartition` previously relied only on that cached version. It could therefore classify the partition as empty and remove it before scan-node planning. The later version refresh in `ScanNode.setVisibleVersionForOlapScanNodes` could observe the new version, but it was already too late because the partition had been pruned from the scan. This PR changes empty-partition pruning for direct cloud incremental reads as follows: - A partition that is already known to be non-empty from the FE cache is retained directly. Partition versions are monotonic, so a cached version greater than `PARTITION_INIT_VERSION` remains sufficient for this pruning decision. - Only cached-empty or unknown candidate partitions are refreshed from MetaService. This catches transactions that became visible after the FE cache entry was populated while avoiding unnecessary RPC work for known non-empty partitions. - The refreshed and cached results are combined in the original candidate-partition order. - Normal reads and incremental reads backed by an `OlapTableWrapper` with fixed visible versions continue to use their existing snapshot-aware path. The refresh in `ScanNode.setVisibleVersionForOlapScanNodes` is intentionally retained. `PruneEmptyPartition` only needs an empty-or-non-empty decision, while scan execution still needs the exact snapshot versions of all final selected partitions. ### Release note Fix cloud incremental reads that could miss newly visible data because a stale FE partition-version cache caused premature empty-partition pruning. ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - `PruneEmptyPartitionTest` - `OlapScanNodeTest` - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [x] Yes. Direct cloud incremental reads refresh cached-empty or unknown partition versions from MetaService before empty-partition pruning. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label (cherry picked from commit 695c88b)
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
morningman
merged commit Sep 21, 2026
71679ac
into
apache:branch-incremental-computation
24 of 26 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from #68138
Batch pick of every merged PR carrying the
incremental-computationlabel thatbranch-incremental-computationdoes not have yet (noincremental-computation-pickedlabel), in the order they landed on master (git log --first-parent). One commit per PR, each ending with(cherry picked from commit <master sha>). Follows the same convention as #67830, #68017, #68073, #68151 and #68236. This round has a single PR.Not included on purpose:
incremental-computation-picked(every other closed PR with the label).Prerequisite check
e5a4e725fac) is before the fork pointefedf10c7e3. Everything the pick uses already exists on this branch:OlapTableWrapper.selectNonEmptyPartitionIds(Collection<Long>, Optional<StreamReadMode>)/hasFixedVisibleVersions(), the two-argumentCloudPartition.getSnapshotVisibleVersionFromMs(List, boolean),StreamReadMode, the threeRowBinlogTableWrapperconstructors, and the test helpers (BinlogTestUtils.newTestRowBinlogConfig,PlanConstructor.newOlapTable,MemoTestUtils.createConnectContext,PlanChecker.from(ConnectContext, Plan)).cherry-pick -xapplied cleanly (one auto-merge inCloudPartition.java, no conflict).Drift check against master
RowBinlogTableWrapper.java,OlapTableWrapperTest.javaand the newPruneEmptyPartitionTest.javaare byte-identical to master at695c88b5772.CloudPartition.javadiffers from master only by the unlabelled [improvement](fe) Reduce cloud version sync config #66296 ("Reduce cloud version sync config": themaxAttemptsoverload ofgetSnapshotVisibleVersionFromMs/getSnapshotVisibleVersionand theVariableMgr.getDefaultSessionVariable()fallback forcloudPartitionVersionCacheTtlMs). Applying [improvement](fe) Reduce cloud version sync config #66296's hunks for this file on top of the branch in a temporary index gives a zero-line diff against master, so nothing of the pick is missing; the pick itself only calls the two-argumentgetSnapshotVisibleVersionFromMs(partitions, false), which is the same on both sides. [improvement](fe) Reduce cloud version sync config #66296 stays out, as in the previous rounds.Verification
run-fe-ut.sh --runon this branch (regenerates thrift/protobuf, compiles fe-core main + test) with the two touched test classes plus every test class that exercisesselectNonEmptyPartitionIds/getSnapshotVisibleVersionFromMs/RowBinlogTableWrapper: 8 classes, 108 tests, 0 failures, 0 errors, BUILD SUCCESS —CloudGlobalTransactionMgrTest37,ExplainTableStreamPlanTest24,PhysicalPlanTranslatorTest17,OlapScanNodeTest12,PartitionCompensatorTest12,OlapTableWrapperTest4 (the two new tests included),PruneEmptyPartitionTest1 (new),TableBinlogFunctionAuthTest1. The@Testcounts of the two touched classes equal master's.