[fix](binlog) Track committed TSO and fence uncertain commits for bounded incremental reads - #67820
Merged
luwei16 merged 18 commits intoSep 18, 2026
Merged
Conversation
### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: Strongly consistent cloud incremental reads drain earlier transactions even when those transactions fall outside the requested historical window. A delayed write can therefore block an otherwise readable window. Track commit TSO allocations and unfinished transactions under the allocator lock, and publish a readable prefix together with the reserved TSO window only after their journal write succeeds. Accepted bounded windows skip transaction watermark/conflict polling and retain the MetaService visible-version refresh. Allocate commit TSO after bitmap preparation, callbacks and metadata validation. Preserve the earliest registration across submission retries and release it only after real VISIBLE/ABORTED, including lazy-commit reconciliation. On master recovery, retain the durable prefix until a fixed, instance-wide transaction bound passes a strict MetaService check after the configured recovery delay. The fixed wait retains the agreed old-master fencing limitation. ### Release note In cloud mode, strongly consistent incremental queries with an explicit end on every incremental relation use committed TSO. Unready windows return MySQL error 5100 (ERR_INCR_WINDOW_NOT_READY), or Flight UNAVAILABLE with business metadata; clients should retry the same window. information_schema.tso_status exposes COMMITTED_TSO and COMMITTED_TSO_PHYSICAL_TIME. The TSO persistence window defaults to one second. Upgrade MetaService before enabling prefix recovery on new FEs. ### Check List (For Author) - Test: Unit Test / Regression test / Manual test - 79 distinct focused FE tests, 9 BE scanner tests and 3 MetaService tests passed. - test_committed_tso and test_binlog_changes_syntax passed; new output generated by the standard regression runner. - Real master/follower MySQL and Flight statement/prepared error contracts passed. - Three-FE failover with an allocated, PREPARED transaction preserved the old prefix and historical reads; confirmed abort released recovery and advanced the prefix. - FE build and Checkstyle, ASAN BE/Cloud builds, clang-format 16 and build hygiene passed. clang-tidy reported no changed-line diagnostics with a matching-toolchain wrapper and an analysis-only overlay for an existing unmatched suppression comment; five unchanged scanner diagnostics remain. - No throughput or latency benchmark was run. - Behavior changed: Yes (bounded cloud read admission, visible-prefix system columns, later commit TSO allocation and one-second persistence) - Does this need documentation: Yes (included docs/committed-tso.md)
### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: The committed-TSO window check made getFlightInfoStatement pass through every FlightRuntimeException. Other Flight failures therefore lost the original INTERNAL wrapper, message prefix and cause chain, and unrelated status codes could reach clients unchanged. Only pass through an exception carrying the ERR_INCR_WINDOW_NOT_READY business code; retain the original wrapping for all other exceptions. ### Release note Preserve the existing Arrow Flight SQL error wrapping for failures other than ERR_INCR_WINDOW_NOT_READY. Window-not-ready errors still expose the retryable status and committed TSO details. ### Check List (For Author) - Test: Unit Test (all 7 DorisFlightSqlProducerTest tests passed via run-fe-ut.sh; the new cases reproduce the previous wrapping failures); FE Checkstyle passed with 0 violations - Behavior changed: Yes (restore the original INTERNAL wrapper for other Flight errors) - Does this need documentation: No (restore existing error handling)
…l reads ### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: A slow commit on one table holds the global committed TSO and rejects otherwise complete incremental windows on unrelated tables. Keep the durable-prefix fast path, reject ends after the current TSO immediately, and let intermediate windows wait for a fixed snapshot of registered transactions involving their tables. Capture the snapshot under the allocator lock and release that lock during the wait; real terminal notifications and reconciliation wake readers without another journal flush. Preserve the recovery guard and distinguish future/recovering windows from visibility wait timeouts through follower RPC, MySQL and Arrow Flight SQL. ### Release note Bounded strongly consistent cloud incremental reads can proceed above the durable committed TSO when their relevant transactions are finished. Visibility wait timeouts return error 5101 (ERR_INCR_VISIBLE_WAIT_TIMEOUT); future or recovering windows retain error 5100 (ERR_INCR_WINDOW_NOT_READY). Both include the current and committed TSO and retry details. ### Check List (For Author) - Test: 65 distinct focused FE unit tests; full FE build and Checkstyle; test_committed_tso SQL regression generated and verified; live MySQL and Flight statement/prepared checks on one master and two followers, including transaction visibility wakeup. - Behavior changed: Yes (table-scoped waiting above the durable prefix and distinct visibility-timeout error). - Does this need documentation: Yes (docs/committed-tso.md updated).
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Remove docs/committed-tso.md from the change as requested. The implementation and tests are unchanged. ### Release note None ### Check List (For Author) - Test: No need to test (documentation deletion only); git diff --check passed. - Behavior changed: No - Does this need documentation: No
luwei16
requested review from
924060929,
gavinchou,
liaoxin01,
morrySnow and
mymeiyi
as code owners
September 10, 2026 14:22
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
luwei16
changed the title
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
### What problem does this PR solve? Issue Number: None Related PR: apache#67820 Problem Summary: The committed TSO change adds two columns to information_schema.tso_status, but SchemaTableTest still expects four columns and fails FE unit CI. Expect all six columns and verify the names and positions of the two new columns while retaining the original column checks. ### Release note None ### Check List (For Author) - Test: Unit Test; reproduced the original SchemaTableTest failure, then passed all 7 SchemaTableTest and TsoStatusMetadataGeneratorTest cases with run-fe-ut.sh. FE Checkstyle and git diff --check passed. - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run cloudut |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
luwei16
changed the title
### What problem does this PR solve? Issue Number: None Related PR: apache#67820 Problem Summary: After FE failover, one unfinished old transaction blocks every bounded incremental read above the durable committed TSO, including reads of unrelated tables. Reuse check_txn_conflict to fetch old running transactions in bounded batches after capturing the fixed recovery transaction-ID bound. Once the complete list is loaded, wait only for related tables and known TSO boundaries. Retain unknown-TSO transactions for their tables and keep the global committed TSO frozen until all recovered transactions are VISIBLE or ABORTED. Preserve concurrent registrations and resume failed scans from the last successful batch without opening an incomplete recovery. ### Release note After cloud FE failover, unrelated tables can read bounded incremental windows once the old transaction list is loaded, while relevant transactions still wait with the existing visibility timeout. Upgrade MetaService before FE to support fetching recovery transactions in batches. ### Check List (For Author) - Test: 86 focused FE unit tests; 6 ASAN MetaService/recovery/lazy-commit tests; test_committed_tso regression; three-FE failover and original-window retry; FE/MS product builds, FE Checkstyle, clang-format 16 and Cloud clang-tidy. - Behavior changed: Yes, enable table-scoped waits while old recovery transactions remain pending. - Does this need documentation: No
Contributor
Author
|
run buildall |
### What problem does this PR solve? Issue Number: None Related PR: apache#67820 Problem Summary: TSO recovery had added strict-check and batch modes to the existing transaction-conflict RPC, mixing recovery scans with table-scoped conflict checks. Add get_tso_recovery_transactions with its own request, response, client wrappers, handler and metrics. Restore check_txn_conflict and its messages to their original definitions. Preserve fixed-bound batch scanning, table-scoped waits and committed-TSO advancement. An unavailable new RPC or incomplete batch keeps recovery closed without falling back to the old conflict check. ### Release note Upgrade MetaService before FE to provide the dedicated TSO recovery RPC. Incremental-query waiting behavior and error codes remain unchanged. ### Check List (For Author) - Test: 108 FE unit tests; 9 ASAN MetaService/recovery/legacy-conflict/lazy-commit tests; test_committed_tso regression; live old/new RPC and three-FE failover verification; FE/MS product builds, FE Checkstyle, clang-format 16 and Cloud clang-tidy. - Behavior changed: Yes, the internal recovery RPC changes; SQL behavior is unchanged. - Does this need documentation: No
Contributor
Author
|
run buildall |
morningman
pushed a commit
that referenced
this pull request
Sep 11, 2026
…ounded incremental reads (#67861) ### What problem does this PR solve? Issue Number: None Related PR: #67820 Problem Summary: Backport the complete seven-commit change from PR #67820 through 1727203 onto branch-incremental-computation. A slow commit can otherwise block bounded incremental reads of unrelated tables. Preserve the committed-TSO fast path, table-scoped waits, separate future-window and visibility-timeout errors, and recovery through the dedicated get_tso_recovery_transactions RPC. Keep the target branch's existing TSO range validation and exclude the deleted design document. The change applies without conflicts. Add a documented, targeted function-size suppression to keep the bounded recovery scan and its KV snapshot/error lifetime together without behavioral refactoring. Also suppress macro-expanded cognitive complexity for the cohesive two-batch recovery test. ### Release note Bounded strongly consistent cloud incremental reads use the durable committed TSO or wait only for related transactions. information_schema.tso_status exposes the committed TSO. Errors 5100 and 5101 distinguish unavailable windows from visibility wait timeouts. Upgrade MetaService before FE for the recovery RPC. ### Check List (For Author) - Test: 163 focused FE unit tests, 9 ASAN MetaService recovery/legacy-conflict/ lazy-commit tests, and 9 ASAN BE scanner tests passed. FE/BE/MS product builds, FE Checkstyle, C++ formatting and build hygiene passed. C++ changed-line static analysis passed with a toolchain wrapper and an analysis-only overlay for an existing unmatched suppression. Raw diagnostics were independently checked to avoid the script's large-output filtering issue. No live regression or new remote CI run was performed for this backport. - Behavior changed: Yes (backport committed-TSO admission and table-scoped waits). - Does this need documentation: No (backport existing PR; design document excluded).
Contributor
Author
schema: doris-repo-review/v1
status: PASS
pr: apache/doris#67820
commit: 29f133584b5372d97632de85c0ba19a0c5abac18
base: c4dee4bd5e82c6f0f380e0243cbaf265999e6a6a
reviewed_at: 2026-09-17T12:58:42+08:00
reviewer: luwei16
model: gpt-6-astra
effort: xhigh
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 2
converged: true |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 27932 ms |
Contributor
TPC-DS: Total hot run time: 154155 ms |
Contributor
ClickBench: Total hot run time: 23.91 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run p0 |
Contributor
Author
|
run cloud_p0 |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: A Meta Service KV_TXN_MAYBE_COMMITTED response means the underlying commit attempt is no longer in flight: it either committed or will never commit. Advancing the TSO fence before releasing this TSO adds an unnecessary Meta Service write. Release the tracked TSO directly for this response while retaining fence advancement for FE-to-Meta-Service RPC failures whose requests may still complete later. ### Release note None ### Check List (For Author) - Test: Unit Test - CloudCommittedTsoTest: 3 passed - Full `./build.sh -j32`: passed - Behavior changed: Yes. KV_TXN_MAYBE_COMMITTED releases its tracked TSO without advancing the fence. - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
Review conclusion: PASS for No Blocker or Major findings remain after two review rounds. The earlier concerns about One non-blocking Minor remains: the final Meta Service lazy-commit wait follows the version-RPC timeout rather than strictly sharing schema: doris-repo-review/v1
status: PASS
pr: apache/doris#67820
commit: fc4f996920282b095a09313ba2eac18d32ab00a4
base: c4dee4bd5e82c6f0f380e0243cbaf265999e6a6a
reviewed_at: 2026-09-17T19:12:19+08:00
reviewer: luwei16
model: gpt-5.6-sol
effort: xhigh
findings: {blocker: 0, major: 0, minor: 1, nit: 0}
rounds: 2
converged: true |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 27862 ms |
Contributor
TPC-DS: Total hot run time: 153490 ms |
Contributor
ClickBench: Total hot run time: 23.99 s |
Contributor
Author
|
run feut |
Contributor
FE UT Coverage ReportIncrement line coverage |
morningman
added a commit
that referenced
this pull request
Sep 18, 2026
…on PRs from master in merge order (#67783 #68034 #68033 #68057 #68094 #68087) (#68151) Cherry-picked from #67783, #68034, #68033, #68057, #68094, #68087 Batch pick of every merged PR carrying the `incremental-computation` label that `branch-incremental-computation` does not have yet (no `incremental-computation-picked` label), in the order they landed on master (`git log --first-parent`). One commit per PR, each created with `git cherry-pick -x` so the message ends with `(cherry picked from commit <master sha>)`. Follows the same convention as #67830, #68017 and #68073. | # | Master commit | PR | Title | |---|---|---|---| | 1 | e545b13 | #67783 | [fix](policy) Enforce row policies on MOW time travel | | 2 | f7a0842 | #68034 | [fix](binlog) Fix missing DELETE events in row binlog | | 3 | e85575e | #68033 | [fix](binlog) Preserve row binlog compaction policy | | 4 | 21160d7 | #68057 | [fix](binlog) Decouple row binlog compaction from CCR binlog config | | 5 | 04aa5a1 | #68094 | [fix](binlog) Persist row binlog config updates in cloud mode | | 6 | da8feed | #68087 | [fix](binlog) Report streams with missing base tables as stale | Not included on purpose: - The 20 labelled PRs that already carry `incremental-computation-picked` (#62606 in the fork point, #67508 via #67712, the nine of #67830, the six of #68017, the two of #68073, and #68050 whose content this branch got directly through #68012). - #68012 carries the label but is a PR against this branch itself (merged as `6f7c87fa892`); nothing to pick. - #67820 is still open on master; this branch already carries its content via #67861. ### Prerequisite check For every pick I listed the master commits between the fork point (`efedf10c7e3`) and the pick that touch the same files and are not on this branch, and checked whether the pick's behavior depends on them. - **#67783** declares no related PR. It builds on the MOW time-travel rewrite (`BindRelation.buildMowTimeTravelUnion`, #67480) which is already here. Two unlabelled master commits overlap: #66770 (authorization plugin SPI: reworks the data-mask / row-filter API of `LogicalCheckPolicy` and its test) and #67811 (removes the `isPlayNereidsDump()` check in `CheckPolicy`). Neither is a functional prerequisite: the fix consists of `getPolicyTable()` unwrapping `OlapTableWrapper`, `CheckPolicy` collecting the whole filter chain below the policy mask, and `BindRelation` putting a `LogicalCheckPolicy` above each union branch — none of that uses the #66770 API. Not picked; adapted instead (see below). - **#68034**, **#68033**, **#68087** (#68087 relates to #67173, which is before the fork point): no unpicked master commit touches any of their files; the picks applied cleanly and are byte-identical to master. - **#68057**: only `be/test/cloud/cloud_compaction_test.cpp` overlaps with the unrelated #67972 (refresh tablet meta of continuously ingested tablets); auto-merged, the pick only adds new `TEST_F` blocks. Main-code hunks are byte-identical to master. - **#68094**: overlaps with #68090 / #67972 / #66598 (`cloud_tablet.cpp`), #67295 / #67618 / #68090 (`base_tablet.{h,cpp}`), #66598 / #67637 (`meta_service.cpp`), #66598 (`cloud.proto`, `cloud_tablet_test.cpp`) and #67761 / #66598 / #67637 (`meta_service_test.cpp`). All auto-merged. The fix — `BaseTablet::binlog_config()` under the meta lock, `BinlogConfig::operator==`, `CloudTablet::sync_meta()` refreshing the binlog config, `update_tablet` in the meta service accepting `binlog_config`, and `CloudSchemaChangeHandler` routing ROW-binlog property updates through it — does not use anything those commits add. Its hunks are byte-identical to master except one trailing context line in `sync_meta()` (`last_sync_tablet_meta_time_s` comes from #67972). Only #67783 needed adaptation, recorded in its commit message: - `LogicalCheckPolicy.java`: the conflicting context was master's `parsePolicyExpression()` helper (#66770), which does not exist here; the new `getPolicyTable()` is inserted in the same place without it. - `CheckRowPolicyTest.java`: the data-mask mock is written against this branch's per-column `AccessControllerManager.evalDataMaskPolicy(..., column) -> Optional<DataMaskPolicy>` instead of master's `evalDataMaskPolicies(..., Set<String>) -> Map<String, DataMaskSpec>`. The masks are the same (concat for the random-distribution table; the non-movable `k2` mask plus identity masks for the hidden reconstruction columns of the MOW table). The master-only `Or` import (#66770) is not carried; `Collections` / `Locale` imports were added because they arrive with #66770 on master. ### Drift check against master Each pick's `+`/`-` lines are identical to the master commit's, except for the #67783 adaptation above. After the six picks, the touched files still differ from master at `da8feed859d` in: `cloud_tablet.cpp`, `base_tablet.{h,cpp}`, `cloud_compaction_test.cpp`, `cloud_tablet_test.cpp`, `meta_service.cpp`, `meta_service_test.cpp`, `cloud.proto`, `CheckPolicy.java`, `LogicalCheckPolicy.java`, `CheckRowPolicyTest.java`. Replaying the nine unpicked master commits listed above (#67637, #67618, #66770, #66598, #67972, #67811, #67761, #67295, #68090) in a temporary index and removing this branch's own #67861 (`GetTsoRecoveryTransactions` in `cloud.proto` / `meta_service_test.cpp`) brings every file to zero diff against master, except the two `LogicalCheckPolicy.java` / `CheckRowPolicyTest.java` hunks of #66770 that overlap the adaptation. Nothing else is left over. ### Verification - FE: `run-fe-ut.sh --run` on this branch (regenerates thrift/protobuf, compiles fe-core main + test) with the test classes touched by the picks plus `DeleteFromCommandTest` from #68034's checklist: 5 classes, 58 tests, 0 failures, 0 errors, BUILD SUCCESS — `CheckRowPolicyTest` 9 (the three tests #67783 adds included), `ExplainTableStreamPlanTest` 24, `CloudSchemaChangeHandlerTest` 18 (the tests of #68033 and #68094 included), `DropTableStreamTest` 5, `DeleteFromCommandTest` 2. - FE checkstyle on fe-core: 0 violations. - BE: `-fsyntax-only` with the flags of the Release build (`compile_commands.json`, regenerated `gen_cpp` headers incl. the new `TabletMetaInfoPB.binlog_config`) passes for `cloud/cloud_tablet.cpp`, `cloud/cloud_storage_engine.cpp`, `storage/olap_server.cpp`, `storage/tablet/tablet.cpp`, `storage/tablet/base_tablet.cpp`, and with `-DBE_TEST -fno-access-control` for `test/cloud/cloud_compaction_test.cpp`, `test/cloud/cloud_tablet_test.cpp`, `test/storage/compaction/compaction_task_test.cpp`, `test/storage/tablet/tablet_test.cpp`. - Meta service: the two-line `update_tablet` hunk of #68094 only uses the generated `TabletMetaInfoPB::has_binlog_config()/binlog_config()` and `TabletMetaCloudPB::mutable_binlog_config()` accessors, all present in the regenerated headers. - The two new groovy suites (`row_binlog_p0/test_row_binlog_mow_light_delete`, `time_travel_p0/test_mow_time_travel_row_policy`) parse cleanly (groovy parser check). --------- Co-authored-by: morrySnow <zhangwenxin@selectdb.com> Co-authored-by: Luwei <814383175@qq.com>
gavinchou
approved these changes
Sep 18, 2026
morrySnow
approved these changes
Sep 18, 2026
Merged
6 tasks
morningman
added a commit
that referenced
this pull request
Sep 20, 2026
… status through again (#68221) ### What problem does this PR solve? Issue Number: None Related PR: #67883, #67820, #67966, #68101 Problem Summary: `DorisFlightSqlProducer.getFlightInfoStatement` used to rethrow a `FlightRuntimeException` as is (#67883), so the status the session layer chose reaches the client: `UNAVAILABLE` from the session's command lock and `UNAUTHENTICATED` from a closed session (#67900, #67966), and whatever a refused session is answered with. #67820 rewrote that catch block into a catch-all that lets only its two incremental-window errors through (by `doris-error-code` metadata) and wraps every other `FlightRuntimeException` as `INTERNAL: get flight info statement failed, <message>` -- the wrapping #67883 had removed. The producer's other entry points (`setSessionOptions`, `streamMetadata`) still let the status through and say they do it "as in getFlightInfoStatement". This PR restores the passthrough ahead of the catch-all. The window errors are `FlightRuntimeException`s built by `queryFailure`, so #67820's metadata special case is subsumed and removed; a non-Flight failure is still wrapped as `INTERNAL` with the same message. Found through #68101: its `test_connection_quota` asserts the `RESOURCE_EXHAUSTED` the connection pool answers a refused Flight session with, and since the pipelines compile a PR merged into master, every run after #67820 landed got the wrapped `INTERNAL` instead.
morningman
added a commit
that referenced
this pull request
Sep 20, 2026
…on PRs from master in merge order (#67820 #68088) (#68236) Cherry-picked from #67820, #68088 Batch pick of every merged PR carrying the `incremental-computation` label that `branch-incremental-computation` does not have yet (no `incremental-computation-picked` label), in the order they landed on master (`git log --first-parent`). One commit per PR, each ending with `(cherry picked from commit <master sha>)`. Follows the same convention as #67830, #68017, #68073 and #68151. | # | Master commit | PR | Title | |---|---|---|---| | 1 | 44e3ae2 | #67820 | [fix](binlog) Track committed TSO and fence uncertain commits for bounded incremental reads | | 2 | 3de3a75 | #68088 | [fix](binlog) Require SELECT privilege for binlog TVF | Not included on purpose: - The 25 labelled PRs that already carry `incremental-computation-picked`. - #68012 carries the label but is a PR against this branch itself (merged as `6f7c87fa892`); nothing to pick. ### How #67820 was picked This branch already carried #67820 through #67861, which backported the PR's first seven commits (up to `17272039558`) before the PR was merged. Between that cut and the merge the PR gained six more commits and five master merges, and the final design differs from what #67861 brought: the Meta Service recovery RPC (`get_tso_recovery_transactions`) and the FE startup/periodic recovery scan are gone, replaced by a durable per-instance commit-TSO fence (`txn_tso_fence_key`, `advance_tso_fence`, commit-time `TXN_COMMIT_TSO_EXPIRED` check, `enable_check_commit_tso_fence`), plus the review-feedback and "release maybe-committed TSO" fixes. A plain `cherry-pick -x 44e3ae2` conflicts in 23 files because the branch holds the intermediate design, so commit 1 was built by replaying, on top of the branch, exactly what the PR gained after the cut: - the PR's later commits `d16e67197b2`, `e53291d196b`, `864546d5531`, `8c55eb086e9`, `8d264ce294a`, `fc4f9969202`; - the content that landed inside its master merges, identified by diffing each merge against its `git merge-tree` automerge result: `c491a293426` (`setEnvTSOService` replaced by `Mockito.doReturn(tsoService).when(masterEnv).getTSOService()` — `masterEnv` is the same Mockito delegating mock here since #67813) and `29f133584b5` (the extra `TSOTransactionTrackerTest` coverage and two comment removals in `DorisFlightSqlProducer`). The other three merges only resolved import blocks against master-only code. The replay then got squashed into one commit with the master PR's message, the original author, and the `(cherry picked from commit 44e3ae2)` trailer. Adaptations, all recorded in the commit message: - `DorisFlightSqlProducer` / `DorisFlightSqlProducerTest` stay under `service/arrowflight` (the branch lacks the package move of #67866). - `8d264ce294a` is a no-op here: it removes a `catch (FlightRuntimeException)` block that #67883 added on master and this branch never had. - The branch-only `get_tso_recovery_transactions` RPC, its recovery scan, the five `TsoRecovery*` Meta Service tests and the NOLINT suppressions #67861 had added for them are removed, as on master. - The round-3 `mockVersionHelper()` adaptation in `CloudGlobalTransactionMgrTest` (no `VersionHelper.getVersionFromMeta(req, maxAttempts)` overload here, #66296) is kept. ### Prerequisite check - **#67820** declares #67181 and #67594 as related; #67181 (`e5a4e725fac`) is before the fork point and #67594 came with #67830. The rest of what the commit touches on master is import-block and neighbouring-code drift from unlabelled commits (#67866 / #67883 / #67966 Arrow Flight and session refactors, #67761 `get_prepare_txn_by_coordinator`, #66598 pre-rowset delete bitmaps, repair-tablet-index changes); none of it is used by the fence/committed-TSO logic. - **#68088** declares no related PR. The hook it implements (`TableValuedFunctionIf.checkAuth`) and the caller chain (`CheckPrivileges.visitLogicalTVFRelation` → `TableValuedFunction.checkAuth` → catalog function) are byte-identical between this branch and master, and the five-argument `AccessControllerManager.checkTblPriv` overload exists. Applied cleanly. ### Drift check against master - **#67820**: every one of the 54 files the master commit touches now contains the pick's content — the master commit reverse-applies cleanly per file onto this branch (50 files), and the four files where only the surrounding context differs (`config.h`, `meta_service_txn.cpp`, `StmtExecutor.java`, `StmtExecutorTest.java`) contain every added line and none of the removed ones. 33 of the 54 files are byte-identical to master at `44e3ae2b951`, including all of `fe/.../tso/`, `CloudGlobalTransactionMgr.java`, `MetaServiceProxy.java`, `keys.{h,cpp}`, `meta_service.h` and the regression suite/output. The remaining differences are unrelated master-only or branch-only code from unlabelled commits (#67761, #66598, #67866/#67883/#67966, repair-tablet-index, meta-cache columns, recycler configs) plus the documented `mockVersionHelper()` adaptation; no line in the committed-TSO/fence domain is left over from #67861. - **#68088**: all three files are byte-identical to master at `3de3a756f74`. ### Verification - FE: `run-fe-ut.sh --run` on this branch (regenerates thrift/protobuf, compiles fe-core main + test) with every test class the picks touch: 12 classes, 175 tests, 0 failures, 0 errors, BUILD SUCCESS — `TSOServiceTest` 38, `CloudGlobalTransactionMgrTest` 37, `StmtExecutorTest` 27, `MetaServiceProxyTest` 22, `OlapScanNodeTest` 12, `TimeBasedChangeVisibleWaiterTest` 12, `TSOTransactionTrackerTest` 9 (the four tests added inside the PR's last master merge included), `DorisFlightSqlProducerTest` 7, `TsoStatusMetadataGeneratorTest` 5, `CloudCommittedTsoTest` 3, `SchemaTableTest` 2, `TableBinlogFunctionAuthTest` 1 (#68088). The `@Test` counts of the touched classes equal master's. - FE checkstyle on fe-core: 0 violations. - Meta Service: `-fsyntax-only` with the flags of `cloud/CMakeLists.txt` (`-Wall -Werror`, regenerated `gen_cpp/cloud.pb.h` with `AdvanceTsoFence*` / `TxnTsoFencePB` and without `GetTsoRecoveryTransactions*`) on `meta_service_txn.cpp`, `keys.cpp`, `http_encode_key.cpp`, `bvars.cpp` and, with `-DUNIT_TEST -DBE_TEST -fno-access-control`, on `meta_service_test.cpp`, `keys_test.cpp`, `http_encode_key_test.cpp`, `meta_service_helper_test.cpp`, `txn_lazy_commit_test.cpp`: no diagnostic in any line the pick touches (the only errors are the pre-existing macOS-only `pthread_setname_np` / `int64_t`-vs-`long` ones in untouched 2024/2025 code). clang-format 16 is clean on all 14 touched cloud files. - No BE file changes in this round (the BE side of #67820 was already byte-identical to master via #67861). - The new `auth_p0/test_binlog_tvf_auth.groovy` and the two #67820 suites parse cleanly (groovy parser check). --------- Co-authored-by: Luwei <814383175@qq.com> Co-authored-by: morrySnow <zhangwenxin@selectdb.com>
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #67181, #67594
Problem Summary:
Bounded cloud
@incrreads need a safe end timestamp. Waiting by transaction ID can include transactions outside the requested time window, while one slow transaction can hold a global committed prefix and block unrelated tables. FE failover also needs to prevent an old master or an uncertain commit RPC from later committing a TSO that FE has already treated as finished.This PR introduces a durable committed TSO and keeps the active commit-attempt set in the FE master:
KV_TXN_MAYBE_COMMITTED, an RPC timeout, connection loss or another uncertain result, synchronously advance the per-instance Meta Service fence to the attempted TSO before removing it. If fencing fails, deactivate the TSO service so reads and allocations cannot use an unsafe prefix.MS_TOO_BUSYremains retryable because Meta Service explicitly rejects it before processing. Non-TSO transactions keep the existing retry behavior.checkTransactions(). A new master persists a fresh window and advances the same global fence before enabling TSO service, so commits using an older TSO are rejected without waiting for PREPARED transactions.For a strongly consistent cloud read with an explicit end on every incremental relation, FE applies these rules:
5100 / ERR_INCR_WINDOW_NOT_READYimmediately.5101 / ERR_INCR_VISIBLE_WAIT_TIMEOUTwhen the wait expires.After admission, version resolution waits only for pending transactions on the selected partitions. Unrelated tables do not participate in either wait.
information_schema.tso_statusexposesCOMMITTED_TSOandCOMMITTED_TSO_PHYSICAL_TIME; the latter is epoch milliseconds and is the maximum globally admitted end timestamp. The existing current TSO and allocation-window columns remain available.The Meta Service fence is one monotonic, non-versioned current-state key in the instance transaction namespace. Commit reads it in the same FDB transaction as the metadata commit and rejects
commit_tso <= fencewithTXN_COMMIT_TSO_EXPIRED, returning the effective fence. The check runs only when both the FE request flag and the mutable Meta Service configuration are enabled. Cloud snapshot, clone and rollback do not restore this leadership/commit fence; a restored or cloned instance publishes its current fence when its FE master initializes.Upgrade Meta Service before FE. A missing or unavailable fence RPC keeps a new FE TSO service uninitialized. The new Meta Service error code uses
actual_codewhile old clients receive the compatible fallback code.Connector error handling
ERR_INCR_WINDOW_NOT_READYretryAfterMs; do not advance the incremental offset. Route a retry through the current master when the reason isTSO_MASTER_CHANGED.ERR_INCR_VISIBLE_WAIT_TIMEOUTchange_visible_timeout_ms.MySQL returns SQLSTATE
HY000; the message containsreason, requested end, current TSO, committed TSO, their physical times, timeout and retry delay. Arrow Flight SQL returnsUNAVAILABLEonly for these two errors and includesdoris-error-codeanddoris-error-namemetadata. Other Flight failures retain the originalINTERNALwrapping.Release note
Improve bounded, strongly consistent cloud incremental reads with a durable committed TSO, table-scoped waits and per-instance commit TSO fencing. Expose the committed TSO through
information_schema.tso_status, distinguish an unavailable window (5100) from a visibility wait timeout (5101), and remove FE startup/periodic transaction recovery scans. Upgrade Meta Service before FE.Check List (For Author)
test_committed_tsoandtest_binlog_changes_syntaxpassed earlier in this PR with outputs generated by the standard runner../build.sh -j32and./build.sh --cloud -j32.Check List (For Reviewer who merge this PR)