[fix](cloud) Wait for running transactions before incremental reads - #67181
Merged
luwei16 merged 2 commits intoSep 2, 2026
Merged
Conversation
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: In cloud mode, time-based incremental reads treated an empty committed-transaction list as proof that a read window was complete. A transaction could already have a commit timestamp in the window while its delete bitmap and partition version were still being published, allowing the query to return an empty result and downstream consumers to close the window. Capture a MetaService transaction ID watermark at query start, wait for earlier target-table transactions to finish through the existing conflict check, and fetch fresh visible versions for incremental scans after the wait.
### Release note
Cloud time-based incremental reads now wait for transactions registered before query start to finish and read the latest visible partition versions before scanning.
### Check List (For Author)
- Test: Unit Test
- ./run-fe-ut.sh --run org.apache.doris.qe.TimeBasedChangeVisibleWaiterTest,org.apache.doris.planner.OlapScanNodeTest
- ./build.sh --fe
- Behavior changed: Yes. Cloud time-based incremental reads wait for query-start transactions and fail on wait/check timeout or error instead of succeeding against an incomplete snapshot.
- Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The latest master added runtime-filter bucket tests in the same section of OlapScanNodeTest as the incremental-read visible-version test, causing a pull request conflict. Merge the latest master and preserve both the upstream tests and the incremental-read regression test without changing the product fix.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- ./run-fe-ut.sh --run org.apache.doris.qe.TimeBasedChangeVisibleWaiterTest,org.apache.doris.planner.OlapScanNodeTest (15 tests passed)
- Behavior changed: No
- Does this need documentation: No
gavinchou
reviewed
Aug 27, 2026
| versions = CloudPartition.getSnapshotVisibleVersion(partitions); | ||
| // A time-based change read may have just waited for an old transaction to finish. | ||
| // Bypass the FE cache so the scan uses the version made visible by that transaction. | ||
| versions = hasIncrementalRead |
Contributor
Author
|
run buildall |
Contributor
Author
|
run performance |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run performance |
69 tasks
Contributor
Author
|
run performance |
Contributor
TPC-H: Total hot run time: 16578 ms |
Contributor
TPC-DS: Total hot run time: 82299 ms |
Contributor
ClickBench: Total hot run time: 14.62 s |
morrySnow
approved these changes
Sep 2, 2026
luwei16
added a commit
to luwei16/Doris
that referenced
this pull request
Sep 10, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: The committed-TSO window check made getFlightInfoStatement pass through every FlightRuntimeException. Other Flight failures therefore lost the original INTERNAL wrapper, message prefix and cause chain, and unrelated status codes could reach clients unchanged. Only pass through an exception carrying the ERR_INCR_WINDOW_NOT_READY business code; retain the original wrapping for all other exceptions. ### Release note Preserve the existing Arrow Flight SQL error wrapping for failures other than ERR_INCR_WINDOW_NOT_READY. Window-not-ready errors still expose the retryable status and committed TSO details. ### Check List (For Author) - Test: Unit Test (all 7 DorisFlightSqlProducerTest tests passed via run-fe-ut.sh; the new cases reproduce the previous wrapping failures); FE Checkstyle passed with 0 violations - Behavior changed: Yes (restore the original INTERNAL wrapper for other Flight errors) - Does this need documentation: No (restore existing error handling)
luwei16
added a commit
to luwei16/Doris
that referenced
this pull request
Sep 10, 2026
…l reads ### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: A slow commit on one table holds the global committed TSO and rejects otherwise complete incremental windows on unrelated tables. Keep the durable-prefix fast path, reject ends after the current TSO immediately, and let intermediate windows wait for a fixed snapshot of registered transactions involving their tables. Capture the snapshot under the allocator lock and release that lock during the wait; real terminal notifications and reconciliation wake readers without another journal flush. Preserve the recovery guard and distinguish future/recovering windows from visibility wait timeouts through follower RPC, MySQL and Arrow Flight SQL. ### Release note Bounded strongly consistent cloud incremental reads can proceed above the durable committed TSO when their relevant transactions are finished. Visibility wait timeouts return error 5101 (ERR_INCR_VISIBLE_WAIT_TIMEOUT); future or recovering windows retain error 5100 (ERR_INCR_WINDOW_NOT_READY). Both include the current and committed TSO and retry details. ### Check List (For Author) - Test: 65 distinct focused FE unit tests; full FE build and Checkstyle; test_committed_tso SQL regression generated and verified; live MySQL and Flight statement/prepared checks on one master and two followers, including transaction visibility wakeup. - Behavior changed: Yes (table-scoped waiting above the durable prefix and distinct visibility-timeout error). - Does this need documentation: Yes (docs/committed-tso.md updated).
Merged
9 tasks
luwei16
added a commit
to luwei16/Doris
that referenced
this pull request
Sep 11, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: A new FE master previously waited for all pre-failover transactions, including ordinary PREPARED transactions that could remain open until timeout. Publish a monotonic per-instance TSO fence after persisting the new allocation window, reject stale binlog commits in Meta Service, retry them with the new master's TSO, and recover only pre-fence COMMITTED TSO transactions. ### Release note Fence stale binlog transaction commit TSOs across FE master failover and remove the fixed recovery delay and PREPARED-transaction wait. ### Check List (For Author) - Test: Unit Test - FE TSO service/tracker and cloud transaction manager tests - Cloud Meta Service TSO fence/recovery, key, compatibility, and lazy commit tests - Behavior changed: Yes. Binlog commits with a TSO from an older FE master are retried with a fresh TSO; recovery no longer waits for PREPARED or non-TSO transactions. - Does this need documentation: No
luwei16
added a commit
to luwei16/Doris
that referenced
this pull request
Sep 14, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: Incremental reads track TSO-bearing commit attempts in FE memory. A commit RPC can finish without a conclusive response, leaving the attempt in the tracker indefinitely, while scanning Meta Service transactions during startup and periodic reconciliation adds complexity and can block unrelated reads. Fence an uncertain attempt in Meta Service before removing its exact TSO from the tracker, release definite responses immediately, and let incremental scans wait for pending lazy commits on their selected partitions. Disable transport retries for TSO-bearing commit RPCs so every uncertain delivery is fenced. Startup calibration advances the same per-instance fence, so stale commits from an old master are rejected without a recovery scan. ### Release note Bounded incremental reads now fence uncertain commit TSO attempts and no longer require startup or periodic transaction recovery scans. ### Check List (For Author) - Test: Unit Test - FE unit tests for TSO tracking, commit response handling, Meta Service retry behavior, and incremental scan version waits - Cloud unit tests for TSO fence monotonicity, fence-key encoding, and response-code compatibility - ./build.sh -j32 - ./build.sh --cloud -j32 - Cloud clang-tidy and build hygiene checks - Behavior changed: Yes. Uncertain TSO commit attempts are fenced before release, definite failures are released directly, and incremental scans wait for selected-partition pending lazy commits. - Does this need documentation: Yes. Connector-facing error-code documentation is included in the PR description.
luwei16
added a commit
to luwei16/Doris
that referenced
this pull request
Sep 15, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: The commit TSO fence check lacked request-level and Meta Service configuration gates, its error name was unclear, the fence key was unavailable through Meta Service HTTP KV tooling, and the persisted window time did not state its unit. Add both fence-check gates to every Cloud commit path, expose the fence key to HTTP encode/get/set, rename the stale TSO error, and clarify the FE tracker and durable-state names. ### Release note Cloud commit TSO fence checks can be controlled by the request and the mutable Meta Service configuration. Rejected stale commit TSOs now use TXN_COMMIT_TSO_EXPIRED. ### Check List (For Author) - Test: Unit Test - Targeted FE unit tests: 108 passed - Cloud MetaService TSO fence test: passed - Cloud HTTP encode/get/set tests: 11 passed - Full ./build.sh -j32: passed - Cloud clang-tidy on changed lines: passed - Behavior changed: Yes. Commit TSO fencing requires both request and Meta Service gates, and the stale TSO error was renamed. - Does this need documentation: No
luwei16
added a commit
to luwei16/Doris
that referenced
this pull request
Sep 16, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: A master merge added an earlier FlightRuntimeException catch in getFlightInfoStatement, which bypassed the existing error-code filter and exposed every Flight error directly. Remove that catch so only incremental-window errors retain their retryable status while unrelated errors keep the established INTERNAL wrapper. ### Release note None ### Check List (For Author) - Test: Unit Test - DorisFlightSqlProducerTest: 7 passed - Full ./build.sh -j32: passed - Behavior changed: No. This restores the intended Arrow Flight error handling. - Does this need documentation: No
luwei16
added a commit
to luwei16/Doris
that referenced
this pull request
Sep 16, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: Recent master changes extended Cloud transaction tests and changed the test environment to a delegating CloudEnv mock, which conflicted with the committed TSO branch. Merge current master, preserve both sets of test dependencies, and stub the TSO service through the mock so the fence retry test uses the intended service. ### Release note None ### Check List (For Author) - Test: Unit Test - CloudGlobalTransactionMgrTest: 37 passed - DorisFlightSqlProducerTest: 7 passed - Full ./build.sh -j32: passed - Behavior changed: No. This resolves the master merge while preserving the existing TSO fence behavior. - Does this need documentation: No
luwei16
added a commit
to luwei16/Doris
that referenced
this pull request
Sep 17, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: Recent master changes to Arrow Flight session handling and statement executor tests conflicted with committed TSO error propagation. Merge current master, preserve the new session and deferred-query behavior while continuing to expose only incremental-window errors, and extend TSOTransactionTracker tests for repeated registration, fence replacement, waits, exact-attempt abandon semantics, metrics, and preconditions. ### Release note None ### Check List (For Author) - Test: Unit Test - TSOTransactionTrackerTest: 9 passed - DorisFlightSqlProducerTest: 12 passed - StmtExecutorTest: 26 passed - Full `./build.sh -j32`: passed - Behavior changed: No. This resolves the master merge and adds coverage. - Does this need documentation: No
luwei16
added a commit
to luwei16/Doris
that referenced
this pull request
Sep 17, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: A Meta Service KV_TXN_MAYBE_COMMITTED response means the underlying commit attempt is no longer in flight: it either committed or will never commit. Advancing the TSO fence before releasing this TSO adds an unnecessary Meta Service write. Release the tracked TSO directly for this response while retaining fence advancement for FE-to-Meta-Service RPC failures whose requests may still complete later. ### Release note None ### Check List (For Author) - Test: Unit Test - CloudCommittedTsoTest: 3 passed - Full `./build.sh -j32`: passed - Behavior changed: Yes. KV_TXN_MAYBE_COMMITTED releases its tracked TSO without advancing the fence. - Does this need documentation: No
luwei16
added a commit
to luwei16/Doris
that referenced
this pull request
Sep 17, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#67181 Problem Summary: Incremental empty-partition pruning only needs MetaService to recheck partitions whose cached version does not prove they contain data. Refreshing partitions already known to be non-empty adds unnecessary planning RPC work. Keep cached non-empty partitions, directly refresh only cached-empty or unknown candidates, and preserve the original candidate order when combining the results. ### Release note Reduce MetaService version lookups during cloud incremental-read planning. ### Check List (For Author) - Test: Unit Test - PruneEmptyPartitionTest - Behavior changed: No. The same non-empty partitions are retained with a smaller MetaService refresh batch. - Does this need documentation: No
luwei16
added a commit
to luwei16/Doris
that referenced
this pull request
Sep 17, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#67181 Problem Summary: During PruneEmptyPartition, an incremental read only needs to determine whether each candidate partition is empty. A cached visible version greater than PARTITION_INIT_VERSION already proves that the same partition ID is non-empty. Partition versions are monotonic, so this fact cannot become stale even when the cached version is not the latest one. Refreshing those known non-empty partitions from MetaService therefore cannot change the pruning decision and only increases the planning RPC payload. Keep them directly, and call getSnapshotVisibleVersionFromMs only for cached-empty or unknown candidates, because a transaction may have become visible after those cache entries were populated. Merge the cached and refreshed results in the original candidate order. This does not remove the version refresh in ScanNode.setVisibleVersionForOlapScanNodes. That later refresh still queries the final selected partitions because scan execution needs their exact snapshot versions, not just a non-empty decision. ### Release note Reduce MetaService version lookups during cloud incremental-read planning. ### Check List (For Author) - Test: Unit Test - PruneEmptyPartitionTest - Behavior changed: No. The same non-empty partitions are retained with a smaller MetaService refresh batch. - Does this need documentation: No
16 tasks
luwei16
added a commit
to luwei16/Doris
that referenced
this pull request
Sep 17, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#67181 Problem Summary: Centralize the conditions that select direct MetaService partition-version refreshes during empty-partition pruning. The helper makes the cloud incremental-read and fixed-snapshot exclusions explicit without changing behavior. ### Release note None ### Check List (For Author) - Test: Unit Test - PruneEmptyPartitionTest - Behavior changed: No. - Does this need documentation: No
luwei16
added a commit
that referenced
this pull request
Sep 18, 2026
…nded incremental reads (#67820) ### What problem does this PR solve? Issue Number: None Related PR: #67181, #67594 Problem Summary: Bounded cloud `@incr` reads need a safe end timestamp. Waiting by transaction ID can include transactions outside the requested time window, while one slow transaction can hold a global committed prefix and block unrelated tables. FE failover also needs to prevent an old master or an uncertain commit RPC from later committing a TSO that FE has already treated as finished. This PR introduces a durable committed TSO and keeps the active commit-attempt set in the FE master: - Allocate and register a commit TSO immediately before the first Meta Service commit RPC, under the TSO allocator lock. Retries after a fenced response replace the registration with a fresh TSO above the returned fence. - On an explicit success, already-visible result or already-aborted result, remove the transaction registration. On another explicit failure, remove only the exact attempted TSO. - On `KV_TXN_MAYBE_COMMITTED`, an RPC timeout, connection loss or another uncertain result, synchronously advance the per-instance Meta Service fence to the attempted TSO before removing it. If fencing fails, deactivate the TSO service so reads and allocations cannot use an unsafe prefix. - Disable transport-level retries for commits carrying a positive TSO. `MS_TOO_BUSY` remains retryable because Meta Service explicitly rejects it before processing. Non-TSO transactions keep the existing retry behavior. - Derive the committed TSO from the current allocated TSO and the oldest registered attempt, and persist it together with the allocation-window end in one BDB journal record. The default persistence window is reduced from five seconds to one second. - Remove the startup transaction scan, recovery RPC, recovery state, and periodic `checkTransactions()`. A new master persists a fresh window and advances the same global fence before enabling TSO service, so commits using an older TSO are rejected without waiting for PREPARED transactions. - A successful lazy commit can be released from FE memory. Incremental scans ask Meta Service to wait for pending transactions on the selected partitions before fixing their visible versions, preserving readability without a global recovery list. For a strongly consistent cloud read with an explicit end on every incremental relation, FE applies these rules: | Requested end | Behavior | | --- | --- | | After the current TSO physical time | Return `5100 / ERR_INCR_WINDOW_NOT_READY` immediately. | | At or before the durable committed TSO physical time | Admit the window directly. | | Between committed and current TSO physical times | Wait only for the captured FE registrations whose TSO is within the window and whose tables are queried. Return `5101 / ERR_INCR_VISIBLE_WAIT_TIMEOUT` when the wait expires. | After admission, version resolution waits only for pending transactions on the selected partitions. Unrelated tables do not participate in either wait. `information_schema.tso_status` exposes `COMMITTED_TSO` and `COMMITTED_TSO_PHYSICAL_TIME`; the latter is epoch milliseconds and is the maximum globally admitted end timestamp. The existing current TSO and allocation-window columns remain available. The Meta Service fence is one monotonic, non-versioned current-state key in the instance transaction namespace. Commit reads it in the same FDB transaction as the metadata commit and rejects `commit_tso <= fence` with `TXN_COMMIT_TSO_EXPIRED`, returning the effective fence. The check runs only when both the FE request flag and the mutable Meta Service configuration are enabled. Cloud snapshot, clone and rollback do not restore this leadership/commit fence; a restored or cloned instance publishes its current fence when its FE master initializes. Upgrade Meta Service before FE. A missing or unavailable fence RPC keeps a new FE TSO service uninitialized. The new Meta Service error code uses `actual_code` while old clients receive the compatible fallback code. #### Connector error handling | Code | Name | Meaning | Connector action | | --- | --- | --- | --- | | 5100 | `ERR_INCR_WINDOW_NOT_READY` | The requested end is after current TSO, or the FE master/TSO generation changed while establishing the window. | Retry the same end timestamp after `retryAfterMs`; do not advance the incremental offset. Route a retry through the current master when the reason is `TSO_MASTER_CHANGED`. | | 5101 | `ERR_INCR_VISIBLE_WAIT_TIMEOUT` | Relevant commit attempts did not finish within `change_visible_timeout_ms`. | Retry the same end timestamp; do not advance the incremental offset. | MySQL returns SQLSTATE `HY000`; the message contains `reason`, requested end, current TSO, committed TSO, their physical times, timeout and retry delay. Arrow Flight SQL returns `UNAVAILABLE` only for these two errors and includes `doris-error-code` and `doris-error-name` metadata. Other Flight failures retain the original `INTERNAL` wrapping. ### Release note Improve bounded, strongly consistent cloud incremental reads with a durable committed TSO, table-scoped waits and per-instance commit TSO fencing. Expose the committed TSO through `information_schema.tso_status`, distinguish an unavailable window (5100) from a visibility wait timeout (5101), and remove FE startup/periodic transaction recovery scans. Upgrade Meta Service before FE. ### Check List (For Author) - Test - [x] FE unit tests for TSO service/tracker, commit-result handling, Meta Service retry behavior, incremental version waits, MySQL error propagation and Arrow Flight SQL error mapping. - [x] Cloud ASAN unit tests for fence monotonicity and stale-commit rejection, fence-key encoding, and response-code compatibility. - [x] Regression tests `test_committed_tso` and `test_binlog_changes_syntax` passed earlier in this PR with outputs generated by the standard runner. - [x] `./build.sh -j32` and `./build.sh --cloud -j32`. - [x] clang-format 16, check-format, Cloud clang-tidy and build-hygiene checks. - Behavior changed: Yes. Bounded cloud reads use committed TSO/table-scoped waits; uncertain TSO commit attempts are fenced before release; FE no longer scans transactions during startup or periodically. - Does this need documentation: Yes. Connector-facing error handling is documented above.
morningman
added a commit
that referenced
this pull request
Sep 20, 2026
…on PRs from master in merge order (#67820 #68088) (#68236) Cherry-picked from #67820, #68088 Batch pick of every merged PR carrying the `incremental-computation` label that `branch-incremental-computation` does not have yet (no `incremental-computation-picked` label), in the order they landed on master (`git log --first-parent`). One commit per PR, each ending with `(cherry picked from commit <master sha>)`. Follows the same convention as #67830, #68017, #68073 and #68151. | # | Master commit | PR | Title | |---|---|---|---| | 1 | 44e3ae2 | #67820 | [fix](binlog) Track committed TSO and fence uncertain commits for bounded incremental reads | | 2 | 3de3a75 | #68088 | [fix](binlog) Require SELECT privilege for binlog TVF | Not included on purpose: - The 25 labelled PRs that already carry `incremental-computation-picked`. - #68012 carries the label but is a PR against this branch itself (merged as `6f7c87fa892`); nothing to pick. ### How #67820 was picked This branch already carried #67820 through #67861, which backported the PR's first seven commits (up to `17272039558`) before the PR was merged. Between that cut and the merge the PR gained six more commits and five master merges, and the final design differs from what #67861 brought: the Meta Service recovery RPC (`get_tso_recovery_transactions`) and the FE startup/periodic recovery scan are gone, replaced by a durable per-instance commit-TSO fence (`txn_tso_fence_key`, `advance_tso_fence`, commit-time `TXN_COMMIT_TSO_EXPIRED` check, `enable_check_commit_tso_fence`), plus the review-feedback and "release maybe-committed TSO" fixes. A plain `cherry-pick -x 44e3ae2` conflicts in 23 files because the branch holds the intermediate design, so commit 1 was built by replaying, on top of the branch, exactly what the PR gained after the cut: - the PR's later commits `d16e67197b2`, `e53291d196b`, `864546d5531`, `8c55eb086e9`, `8d264ce294a`, `fc4f9969202`; - the content that landed inside its master merges, identified by diffing each merge against its `git merge-tree` automerge result: `c491a293426` (`setEnvTSOService` replaced by `Mockito.doReturn(tsoService).when(masterEnv).getTSOService()` — `masterEnv` is the same Mockito delegating mock here since #67813) and `29f133584b5` (the extra `TSOTransactionTrackerTest` coverage and two comment removals in `DorisFlightSqlProducer`). The other three merges only resolved import blocks against master-only code. The replay then got squashed into one commit with the master PR's message, the original author, and the `(cherry picked from commit 44e3ae2)` trailer. Adaptations, all recorded in the commit message: - `DorisFlightSqlProducer` / `DorisFlightSqlProducerTest` stay under `service/arrowflight` (the branch lacks the package move of #67866). - `8d264ce294a` is a no-op here: it removes a `catch (FlightRuntimeException)` block that #67883 added on master and this branch never had. - The branch-only `get_tso_recovery_transactions` RPC, its recovery scan, the five `TsoRecovery*` Meta Service tests and the NOLINT suppressions #67861 had added for them are removed, as on master. - The round-3 `mockVersionHelper()` adaptation in `CloudGlobalTransactionMgrTest` (no `VersionHelper.getVersionFromMeta(req, maxAttempts)` overload here, #66296) is kept. ### Prerequisite check - **#67820** declares #67181 and #67594 as related; #67181 (`e5a4e725fac`) is before the fork point and #67594 came with #67830. The rest of what the commit touches on master is import-block and neighbouring-code drift from unlabelled commits (#67866 / #67883 / #67966 Arrow Flight and session refactors, #67761 `get_prepare_txn_by_coordinator`, #66598 pre-rowset delete bitmaps, repair-tablet-index changes); none of it is used by the fence/committed-TSO logic. - **#68088** declares no related PR. The hook it implements (`TableValuedFunctionIf.checkAuth`) and the caller chain (`CheckPrivileges.visitLogicalTVFRelation` → `TableValuedFunction.checkAuth` → catalog function) are byte-identical between this branch and master, and the five-argument `AccessControllerManager.checkTblPriv` overload exists. Applied cleanly. ### Drift check against master - **#67820**: every one of the 54 files the master commit touches now contains the pick's content — the master commit reverse-applies cleanly per file onto this branch (50 files), and the four files where only the surrounding context differs (`config.h`, `meta_service_txn.cpp`, `StmtExecutor.java`, `StmtExecutorTest.java`) contain every added line and none of the removed ones. 33 of the 54 files are byte-identical to master at `44e3ae2b951`, including all of `fe/.../tso/`, `CloudGlobalTransactionMgr.java`, `MetaServiceProxy.java`, `keys.{h,cpp}`, `meta_service.h` and the regression suite/output. The remaining differences are unrelated master-only or branch-only code from unlabelled commits (#67761, #66598, #67866/#67883/#67966, repair-tablet-index, meta-cache columns, recycler configs) plus the documented `mockVersionHelper()` adaptation; no line in the committed-TSO/fence domain is left over from #67861. - **#68088**: all three files are byte-identical to master at `3de3a756f74`. ### Verification - FE: `run-fe-ut.sh --run` on this branch (regenerates thrift/protobuf, compiles fe-core main + test) with every test class the picks touch: 12 classes, 175 tests, 0 failures, 0 errors, BUILD SUCCESS — `TSOServiceTest` 38, `CloudGlobalTransactionMgrTest` 37, `StmtExecutorTest` 27, `MetaServiceProxyTest` 22, `OlapScanNodeTest` 12, `TimeBasedChangeVisibleWaiterTest` 12, `TSOTransactionTrackerTest` 9 (the four tests added inside the PR's last master merge included), `DorisFlightSqlProducerTest` 7, `TsoStatusMetadataGeneratorTest` 5, `CloudCommittedTsoTest` 3, `SchemaTableTest` 2, `TableBinlogFunctionAuthTest` 1 (#68088). The `@Test` counts of the touched classes equal master's. - FE checkstyle on fe-core: 0 violations. - Meta Service: `-fsyntax-only` with the flags of `cloud/CMakeLists.txt` (`-Wall -Werror`, regenerated `gen_cpp/cloud.pb.h` with `AdvanceTsoFence*` / `TxnTsoFencePB` and without `GetTsoRecoveryTransactions*`) on `meta_service_txn.cpp`, `keys.cpp`, `http_encode_key.cpp`, `bvars.cpp` and, with `-DUNIT_TEST -DBE_TEST -fno-access-control`, on `meta_service_test.cpp`, `keys_test.cpp`, `http_encode_key_test.cpp`, `meta_service_helper_test.cpp`, `txn_lazy_commit_test.cpp`: no diagnostic in any line the pick touches (the only errors are the pre-existing macOS-only `pthread_setname_np` / `int64_t`-vs-`long` ones in untouched 2024/2025 code). clang-format 16 is clean on all 14 touched cloud files. - No BE file changes in this round (the BE side of #67820 was already byte-identical to master via #67861). - The new `auth_p0/test_binlog_tvf_auth.groovy` and the two #67820 suites parse cleanly (groovy parser check). --------- Co-authored-by: Luwei <814383175@qq.com> Co-authored-by: morrySnow <zhangwenxin@selectdb.com>
luwei16
added a commit
to luwei16/Doris
that referenced
this pull request
Sep 20, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#67181 Problem Summary: A cloud row-binlog scan can begin immediately after waiting for its target transaction to become visible. Empty-partition pruning could still use an older cached initial version and remove a partition that already contains newly visible binlog data. The previous implementation placed cloud incremental-read details directly in PruneEmptyPartition. Keep PruneEmptyPartition generic and limit the behavior change to RowBinlogTableWrapper. In cloud mode, a row-binlog wrapper without fixed visible versions refreshes only cached-empty or unknown partitions directly from MetaService before pruning. Cached-known-non-empty partitions still avoid the RPC. Wrappers with fixed visible versions and the OlapTableWrapper base behavior continue to delegate to the origin table unchanged. The existing CloudPartition selection path also retains its previous behavior and ordering. ### Release note Fix cloud incremental reads that could miss newly visible data because stale partition-version cache entries caused premature empty-partition pruning. ### Check List (For Author) - Test: Unit Test - OlapTableWrapperTest - PruneEmptyPartitionTest - Behavior changed: Yes. Only cloud row-binlog scans without fixed visible versions refresh cached-empty or unknown partitions before pruning. - Does this need documentation: No
luwei16
added a commit
that referenced
this pull request
Sep 21, 2026
…68138) ### What problem does this PR solve? Issue Number: None Related PR: #67181 Problem Summary: A cloud incremental read may wait for its target transaction to become visible before query planning continues. During that wait, the FE `CloudPartition` cache can still contain `PARTITION_INIT_VERSION` for a partition even though the transaction has already become visible in MetaService. `PruneEmptyPartition` previously relied only on that cached version. It could therefore classify the partition as empty and remove it before scan-node planning. The later version refresh in `ScanNode.setVisibleVersionForOlapScanNodes` could observe the new version, but it was already too late because the partition had been pruned from the scan. This PR changes empty-partition pruning for direct cloud incremental reads as follows: - A partition that is already known to be non-empty from the FE cache is retained directly. Partition versions are monotonic, so a cached version greater than `PARTITION_INIT_VERSION` remains sufficient for this pruning decision. - Only cached-empty or unknown candidate partitions are refreshed from MetaService. This catches transactions that became visible after the FE cache entry was populated while avoiding unnecessary RPC work for known non-empty partitions. - The refreshed and cached results are combined in the original candidate-partition order. - Normal reads and incremental reads backed by an `OlapTableWrapper` with fixed visible versions continue to use their existing snapshot-aware path. The refresh in `ScanNode.setVisibleVersionForOlapScanNodes` is intentionally retained. `PruneEmptyPartition` only needs an empty-or-non-empty decision, while scan execution still needs the exact snapshot versions of all final selected partitions. ### Release note Fix cloud incremental reads that could miss newly visible data because a stale FE partition-version cache caused premature empty-partition pruning. ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - `PruneEmptyPartitionTest` - `OlapScanNodeTest` - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason - Behavior changed: - [ ] No. - [x] Yes. Direct cloud incremental reads refresh cached-empty or unknown partition versions from MetaService before empty-partition pruning. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
morningman
added a commit
that referenced
this pull request
Sep 21, 2026
…on PRs from master in merge order (#68138) (#68303) Cherry-picked from #68138 Batch pick of every merged PR carrying the `incremental-computation` label that `branch-incremental-computation` does not have yet (no `incremental-computation-picked` label), in the order they landed on master (`git log --first-parent`). One commit per PR, each ending with `(cherry picked from commit <master sha>)`. Follows the same convention as #67830, #68017, #68073, #68151 and #68236. This round has a single PR. | # | Master commit | PR | Title | |---|---|---|---| | 1 | 695c88b | #68138 | [fix](binlog) Refresh incremental partition versions before pruning | Not included on purpose: - The 28 labelled PRs that already carry `incremental-computation-picked` (every other closed PR with the label). ### Prerequisite check - **#68138** declares #67181 as related; #67181 (`e5a4e725fac`) is before the fork point `efedf10c7e3`. Everything the pick uses already exists on this branch: `OlapTableWrapper.selectNonEmptyPartitionIds(Collection<Long>, Optional<StreamReadMode>)` / `hasFixedVisibleVersions()`, the two-argument `CloudPartition.getSnapshotVisibleVersionFromMs(List, boolean)`, `StreamReadMode`, the three `RowBinlogTableWrapper` constructors, and the test helpers (`BinlogTestUtils.newTestRowBinlogConfig`, `PlanConstructor.newOlapTable`, `MemoTestUtils.createConnectContext`, `PlanChecker.from(ConnectContext, Plan)`). `cherry-pick -x` applied cleanly (one auto-merge in `CloudPartition.java`, no conflict). ### Drift check against master - `RowBinlogTableWrapper.java`, `OlapTableWrapperTest.java` and the new `PruneEmptyPartitionTest.java` are byte-identical to master at `695c88b5772`. - `CloudPartition.java` differs from master only by the unlabelled #66296 ("Reduce cloud version sync config": the `maxAttempts` overload of `getSnapshotVisibleVersionFromMs` / `getSnapshotVisibleVersion` and the `VariableMgr.getDefaultSessionVariable()` fallback for `cloudPartitionVersionCacheTtlMs`). Applying #66296's hunks for this file on top of the branch in a temporary index gives a zero-line diff against master, so nothing of the pick is missing; the pick itself only calls the two-argument `getSnapshotVisibleVersionFromMs(partitions, false)`, which is the same on both sides. #66296 stays out, as in the previous rounds. ### Verification - FE: `run-fe-ut.sh --run` on this branch (regenerates thrift/protobuf, compiles fe-core main + test) with the two touched test classes plus every test class that exercises `selectNonEmptyPartitionIds` / `getSnapshotVisibleVersionFromMs` / `RowBinlogTableWrapper`: 8 classes, 108 tests, 0 failures, 0 errors, BUILD SUCCESS — `CloudGlobalTransactionMgrTest` 37, `ExplainTableStreamPlanTest` 24, `PhysicalPlanTranslatorTest` 17, `OlapScanNodeTest` 12, `PartitionCompensatorTest` 12, `OlapTableWrapperTest` 4 (the two new tests included), `PruneEmptyPartitionTest` 1 (new), `TableBinlogFunctionAuthTest` 1. The `@Test` counts of the two touched classes equal master's. - FE checkstyle on fe-core: 0 violations. - No BE, cloud or regression-suite changes in this PR. Co-authored-by: Luwei <814383175@qq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: In cloud mode, time-based incremental reads treated an empty committed-transaction list as proof that a read window was complete. A transaction could already have a commit timestamp in the window while its delete bitmap and partition version were still being published, allowing the query to return an empty result and downstream consumers to close the window. Capture a MetaService transaction ID watermark at query start, wait for earlier target-table transactions to finish through the existing conflict check, and fetch fresh visible versions for incremental scans after the wait.
Release note
Cloud time-based incremental reads now wait for transactions registered before query start to finish and read the latest visible partition versions before scanning.
Check List (For Author)