[fix](row binlog) make time-based incremental reads use a safe fence by luwei16 · Pull Request #67594 · apache/doris

luwei16 · 2026-09-07T08:21:46Z

What problem does this PR solve?

Issue Number: None

Related PR: #65850

Problem Summary:

Time-based @incr reads previously allowed an endTimestamp later than the FE's current TSO. Such a future boundary cannot be closed by the transaction waiting logic: a transaction may start after the request begins but still fall before that future boundary, so the scan cannot guarantee a complete result.

In addition, waiting only for currently running transactions is insufficient unless the TSO boundary, transaction watermark, transaction visibility, and table visible versions are coordinated under one authoritative fence. Otherwise, especially on a follower FE, a transaction can already be VISIBLE while the local catalog still exposes an older visible version, causing the incremental scan to miss rows.

This PR establishes a safe read fence for both classic and cloud modes:

Acquire the current TSO from the master FE.
Capture the transaction ID watermark after the TSO snapshot.
Wait for transactions at or below the watermark that involve the scanned OLAP tables to reach a final visible or aborted state.
In classic mode, synchronize table publishers and wait for follower journal replay so catalog visible versions cross the same fence.
In cloud mode, refresh the latest visible versions from MetaService after the transaction wait.
Reject an endTimestamp later than the captured CURRENT_TSO_PHYSICAL_TIME.
Convert a physical timestamp P to TSO boundary (P, 0), preserving half-open range semantics: [startTimestamp, endTimestamp).

Future endTimestamp error example

Request:

SELECT id, value, __DORIS_BINLOG_OP__
FROM example_table@incr(
    "startTimestamp" = "2026-09-01 00:00:00",
    "endTimestamp" = "2999-01-01 00:00:00",
    "incrementType" = "DETAIL"
);

Response:

ERROR 1105 (HY000): errCode = 2, detailMessage =
endTimestamp exceeds the maximum supported time for an INCR read:
requestedEndTimestampMs=<requested epoch milliseconds>,
CURRENT_TSO_PHYSICAL_TIME=<maximum supported epoch milliseconds>

The maximum currently supported boundary can be obtained from:

SELECT CURRENT_TSO_PHYSICAL_TIME
FROM information_schema.tso_status;

Release note

Time-based @incr reads now reject endTimestamp values after CURRENT_TSO_PHYSICAL_TIME. Timestamp ranges use half-open semantics [startTimestamp, endTimestamp), with each physical timestamp mapped to logical counter zero.

Check List (For Author)

Test: FE unit tests, FE and BE builds, and row_binlog_p0/test_binlog_changes_syntax regression test
Behavior changed: Yes
- Future endTimestamp values are rejected with the maximum supported TSO physical time in the error message.
- Physical timestamp boundaries are interpreted as (physicalTime, 0).
Does this need documentation: No

@incr

### What problem does this PR solve? Issue Number: None Related PR: apache#65850 Problem Summary: Future end timestamps and incomplete transaction visibility fencing could make @incr reads miss rows. Capture an authoritative master TSO before a transaction watermark, wait for target-table transactions in classic and cloud modes, synchronize classic publisher metadata and journal replay, reject unsupported future ends, and map physical timestamp boundaries to logical zero. ### Release note Time-based @incr reads reject endTimestamp values after CURRENT_TSO_PHYSICAL_TIME and interpret timestamp ranges as [startTimestamp, endTimestamp). ### Check List (For Author) - Test: FE unit tests, FE and BE builds, row_binlog_p0/test_binlog_changes_syntax regression test

hello-stephen · 2026-09-07T08:21:53Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

luwei16 · 2026-09-07T08:22:07Z

run buildall

### What problem does this PR solve? Issue Number: None Related PR: apache#67594 Problem Summary: The new TransactionIdGeneratorTest used JUnit 4 imports, which are rejected by the FE checkstyle banJUnit4 rule. Migrate the test to JUnit 5 without changing test behavior. ### Release note None ### Check List (For Author) - Test: FE checkstyle and TransactionIdGeneratorTest - Behavior changed: No - Does this need documentation: No

hello-stephen · 2026-09-07T09:25:14Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	77.69% (2055/2645)
Line Coverage	65.71% (37552/57152)
Region Coverage	52.97% (35046/66167)
Branch Coverage	56.40% (11254/19954)

luwei16 · 2026-09-07T09:59:58Z

run buildall

hello-stephen · 2026-09-07T13:15:25Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	63.05% (29598/46944)
Line Coverage	48.11% (310610/645563)
Region Coverage	43.69% (250896/574211)
Branch Coverage	45.28% (116852/258038)

### What problem does this PR solve? Issue Number: None Related PR: apache#67594 Problem Summary: The PR added three JUnit 4 Assert calls in test methods while the latest master migrated the containing test classes to JUnit 5. The synthetic PR merge therefore failed FE test compilation because Assert was no longer imported. Use JUnit 5 Assertions for the new checks so both the branch and merged PR compile successfully. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.tso.TSOTimestampTest,org.apache.doris.cloud.transaction.CloudGlobalTransactionMgrTest - ./build.sh --fe -j 16 - Behavior changed: No - Does this need documentation: No

luwei16 · 2026-09-08T04:21:19Z

run buildall

### What problem does this PR solve? Issue Number: None Related PR: apache#67594 Problem Summary: Importing JUnit 5 Assertions in a test class that master simultaneously migrated from JUnit 4 caused a textual merge conflict. Use the fully qualified JUnit 5 assertion class for the two new checks so the feature branch compiles independently and merges cleanly with the JUnit 5 migration. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-fe-ut.sh --run org.apache.doris.tso.TSOTimestampTest,org.apache.doris.cloud.transaction.CloudGlobalTransactionMgrTest - Behavior changed: No - Does this need documentation: No

luwei16 · 2026-09-08T04:28:25Z

run buildall

hello-stephen · 2026-09-08T06:12:18Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-09-08T07:33:03Z

TPC-H: Total hot run time: 16975 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 934b1e52f50e6a193c77cb24389262f9749ca0ba, data reload: false

------ Round 1 ----------------------------------
============================================
q1	16990	3063	3075	3063
q2	2076	261	222	222
q3	10092	894	533	533
q4	4660	259	203	203
q5	7747	554	377	377
q6	149	120	93	93
q7	539	504	394	394
q8	9313	911	924	911
q9	3486	2401	2379	2379
q10	6502	874	709	709
q11	407	195	179	179
q12	621	260	201	201
q13	18117	1532	1189	1189
q14	164	145	137	137
q15	q16	435	394	361	361
q17	1521	884	894	884
q18	3072	2234	2234	2234
q19	1322	902	798	798
q20	363	288	213	213
q21	5309	1669	1865	1669
q22	328	265	226	226
Total cold run time: 93213 ms
Total hot run time: 16975 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3437	3359	3362	3359
q2	497	389	361	361
q3	2178	2315	2114	2114
q4	1177	1160	885	885
q5	2155	2099	2106	2099
q6	170	122	86	86
q7	1015	925	843	843
q8	1583	1404	1380	1380
q9	3114	3066	3099	3066
q10	1847	1765	1606	1606
q11	366	270	249	249
q12	455	420	342	342
q13	1494	1538	1138	1138
q14	168	176	172	172
q15	q16	396	394	359	359
q17	3558	3264	3230	3230
q18	4811	4383	4678	4383
q19	855	781	899	781
q20	1051	980	835	835
q21	3857	3074	3328	3074
q22	385	353	329	329
Total cold run time: 34569 ms
Total hot run time: 30691 ms

hello-stephen · 2026-09-08T07:39:29Z

TPC-DS: Total hot run time: 81203 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 934b1e52f50e6a193c77cb24389262f9749ca0ba, data reload: false

query5	4269	422	344	344
query6	384	136	120	120
query7	4950	407	231	231
query8	318	122	118	118
query9	8711	2889	2879	2879
query10	389	217	183	183
query11	5388	1021	919	919
query12	118	74	69	69
query13	1190	447	318	318
query14	6192	2235	2082	2082
query14_1	1964	1940	1957	1940
query15	177	126	113	113
query16	928	372	359	359
query17	1047	444	351	351
query18	2397	324	237	237
query19	171	156	106	106
query20	83	70	70	70
query21	200	104	89	89
query22	5406	5291	5426	5291
query23	6759	6260	5907	5907
query23_1	6122	6096	6121	6096
query24	7181	1092	779	779
query24_1	765	766	748	748
query25	426	291	257	257
query26	1229	225	128	128
query27	2802	400	255	255
query28	4667	1497	1471	1471
query29	933	450	381	381
query30	252	158	131	131
query31	822	393	328	328
query32	125	72	78	72
query33	445	218	166	166
query34	1001	846	486	486
query35	400	390	336	336
query36	567	558	552	552
query37	118	75	80	75
query38	1011	845	797	797
query39	469	456	458	456
query39_1	466	443	458	443
query40	198	88	74	74
query41	53	51	53	51
query42	73	70	71	70
query43	242	241	210	210
query44	1013	536	557	536
query45	112	103	97	97
query46	748	855	519	519
query47	762	778	709	709
query48	315	310	237	237
query49	538	242	182	182
query50	786	254	197	197
query51	8050	7983	7875	7875
query52	70	69	59	59
query53	193	205	143	143
query54	207	153	139	139
query55	74	58	53	53
query56	199	184	155	155
query57	670	685	659	659
query58	206	164	159	159
query59	1319	1250	1101	1101
query60	241	175	172	172
query61	105	110	109	109
query62	348	203	172	172
query63	168	139	140	139
query64	2691	682	636	636
query65	1568	1549	1686	1549
query66	1867	269	199	199
query67	10077	9550	9658	9550
query68	3032	1155	700	700
query69	343	225	198	198
query70	657	606	626	606
query71	264	176	167	167
query72	2256	1644	1475	1475
query73	651	589	348	348
query74	2013	1233	1129	1129
query75	1181	1103	942	942
query76	2349	753	538	538
query77	270	264	213	213
query78	3872	3646	3138	3138
query79	1182	781	589	589
query80	1150	328	257	257
query81	499	153	131	131
query82	594	118	95	95
query83	275	208	195	195
query84	241	107	88	88
query85	771	327	281	281
query86	387	173	171	171
query87	1012	952	888	888
query88	2757	2113	2088	2088
query89	280	195	176	176
query90	1949	131	129	129
query91	126	117	96	96
query92	76	62	68	62
query93	1263	1037	733	733
query94	638	258	231	231
query95	509	254	298	254
query96	772	569	271	271
query97	1066	1067	1005	1005
query98	144	140	135	135
query99	409	343	306	306
Total cold run time: 176121 ms
Total hot run time: 81203 ms

hello-stephen · 2026-09-08T07:43:50Z

ClickBench: Total hot run time: 14.45 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 934b1e52f50e6a193c77cb24389262f9749ca0ba, data reload: false

query1	0.01	0.00	0.01
query2	0.07	0.03	0.04
query3	0.26	0.11	0.11
query4	1.61	0.10	0.09
query5	0.18	0.16	0.16
query6	1.26	0.68	0.66
query7	0.03	0.00	0.01
query8	0.04	0.03	0.03
query9	0.29	0.21	0.22
query10	0.36	0.34	0.33
query11	0.17	0.11	0.12
query12	0.15	0.12	0.12
query13	0.30	0.31	0.31
query14	0.47	0.44	0.44
query15	0.37	0.35	0.35
query16	0.20	0.23	0.23
query17	0.65	0.67	0.70
query18	0.18	0.16	0.18
query19	1.14	1.17	1.03
query20	0.02	0.01	0.01
query21	15.44	0.16	0.12
query22	5.06	0.05	0.04
query23	16.16	0.25	0.10
query24	2.97	0.31	0.25
query25	0.09	0.04	0.04
query26	0.77	0.16	0.13
query27	0.03	0.03	0.03
query28	3.63	0.56	0.27
query29	12.50	3.23	2.57
query30	0.25	0.11	0.12
query31	2.75	0.37	0.16
query32	3.53	0.31	0.23
query33	1.42	1.37	1.46
query34	15.37	2.18	1.75
query35	1.75	1.71	1.71
query36	0.47	0.30	0.28
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.11	0.08	0.08
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 90.33 s
Total hot run time: 14.45 s

gavinchou · 2026-09-09T03:58:54Z

As discussed offline, the following two major findings will be addressed in follow-up fixes. Recording the details here for tracking. Reviewed commit: 934b1e52f50e6a193c77cb24389262f9749ca0ba.

1. [major] Classic-mode incremental reads can wait for their own uncommitted transaction

With enable_eventual_consistent_change=false, an explicit transaction that writes to a table and then reads that table through @incr can time out while planning the second statement. For example, with row binlog enabled on t and compatible tables in the same database:

BEGIN;
INSERT INTO t SELECT id FROM seed;
INSERT INTO dst
SELECT id FROM t@incr('endTimestamp'='2026-01-01 00:00:00');
COMMIT;

The first insert registers transaction T and leaves it in PREPARE. When planning the second insert, the new transaction watermark includes T, and isPreviousTransactionsFinished treats T as unfinished. The statement waits for T, but the same connection cannot execute COMMIT until that statement finishes. This ends in a timeout (about 10 seconds with the default setting), even when the requested end timestamp is far in the past. The query only needs already committed history; it does not require reading its own uncommitted writes.

This is a regression in the classic path: the previous waiter only selected COMMITTED transactions with a matching commit TSO and skipped T. A minimal Java probe using the actual base/head methods confirmed that the old predicate skips the same PREPARE transaction while the new waiter times out. This was not a full-cluster SQL test.

Relevant code: new waiter call, unfinished-transaction check.

Suggested follow-up: handle the current session's uncommitted transaction explicitly without weakening the fence for other transactions that already obtained an in-range commit TSO, and add a multi-statement transaction regression test. Simply skipping every PREPARE transaction would need a separate correctness argument for the TSO-allocation/state-transition window.

2. [major] INCR reads inside a CTE still bypass the fence and future-end validation

WITH c AS (
    SELECT id FROM t@incr('endTimestamp'='2999-01-01 00:00:00')
)
SELECT * FROM c;

collectChangeReadInfo uses plan.foreach, which only traverses children(). A LogicalCTE stores its CTE definitions in aliasQueries, exposed through extraPlans():

LogicalCTE
  children:   Project -> UnboundRelation(c)
  extraPlans: SubQueryAlias(c) -> Project -> UnboundRelation(t@incr)

Although CollectRelation collects the underlying table separately, the waiter finds no incremental relation here and returns early. The later timestamp conversion in BindRelation does not repeat the future-end validation. Consequently, this shape skips both the new validation and the read fence. The actual foreach method was checked with a minimal matching tree; the SQL-level conclusion is based on the complete static call chain, not a cluster execution.

The traversal omission predates this PR. This finding is an existing gap left uncovered by the fix, not a newly introduced traversal regression.

Relevant code: change-read collection, foreach traversal, CTE plan storage.

Suggested follow-up: accumulate incremental-read information during complete relation collection, or explicitly traverse CTE definitions, expression subqueries, and expanded view plans. Add a CTE future-end rejection test and a test showing that nested reads actually wait for relevant unfinished transactions.

gavinchou

Approved as discussed offline. The two major findings documented in #67594 (comment) will be addressed in follow-up fixes.

luwei16 · 2026-09-09T07:05:23Z

run beut

luwei16 · 2026-09-09T07:05:36Z

run cloud_ut

luwei16 · 2026-09-09T09:04:42Z

run cloudut

hello-stephen · 2026-09-09T10:19:09Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	63.42% (29907/47157)
Line Coverage	48.47% (314117/648074)
Region Coverage	43.97% (253302/576129)
Branch Coverage	45.60% (118167/259165)

hello-stephen · 2026-09-09T10:23:08Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	76.41% (34882/45651)
Line Coverage	61.45% (393470/640350)
Region Coverage	57.60% (330458/573681)
Branch Coverage	58.44% (150891/258215)

luwei16 · 2026-09-09T10:36:50Z

run buildall

### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: The committed-TSO window check made getFlightInfoStatement pass through every FlightRuntimeException. Other Flight failures therefore lost the original INTERNAL wrapper, message prefix and cause chain, and unrelated status codes could reach clients unchanged. Only pass through an exception carrying the ERR_INCR_WINDOW_NOT_READY business code; retain the original wrapping for all other exceptions. ### Release note Preserve the existing Arrow Flight SQL error wrapping for failures other than ERR_INCR_WINDOW_NOT_READY. Window-not-ready errors still expose the retryable status and committed TSO details. ### Check List (For Author) - Test: Unit Test (all 7 DorisFlightSqlProducerTest tests passed via run-fe-ut.sh; the new cases reproduce the previous wrapping failures); FE Checkstyle passed with 0 violations - Behavior changed: Yes (restore the original INTERNAL wrapper for other Flight errors) - Does this need documentation: No (restore existing error handling)

…l reads ### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: A slow commit on one table holds the global committed TSO and rejects otherwise complete incremental windows on unrelated tables. Keep the durable-prefix fast path, reject ends after the current TSO immediately, and let intermediate windows wait for a fixed snapshot of registered transactions involving their tables. Capture the snapshot under the allocator lock and release that lock during the wait; real terminal notifications and reconciliation wake readers without another journal flush. Preserve the recovery guard and distinguish future/recovering windows from visibility wait timeouts through follower RPC, MySQL and Arrow Flight SQL. ### Release note Bounded strongly consistent cloud incremental reads can proceed above the durable committed TSO when their relevant transactions are finished. Visibility wait timeouts return error 5101 (ERR_INCR_VISIBLE_WAIT_TIMEOUT); future or recovering windows retain error 5100 (ERR_INCR_WINDOW_NOT_READY). Both include the current and committed TSO and retry details. ### Check List (For Author) - Test: 65 distinct focused FE unit tests; full FE build and Checkstyle; test_committed_tso SQL regression generated and verified; live MySQL and Flight statement/prepared checks on one master and two followers, including transaction visibility wakeup. - Behavior changed: Yes (table-scoped waiting above the durable prefix and distinct visibility-timeout error). - Does this need documentation: Yes (docs/committed-tso.md updated).

…on PRs from master in merge order (#67533 #67594 #67646 #67669 #67665 #67710 #67775 #67575 #67480) (#67830) Cherry-picked from #67533, #67594, #67646, #67669, #67665, #67710, #67775, #67575, #67480 Batch pick of every merged PR carrying the `incremental-computation` label that `branch-incremental-computation` does not have yet, in the order they landed on master (`git log --first-parent`). One commit per PR, each created with `git cherry-pick -x` so the message ends with `(cherry picked from commit <master sha>)`. | # | Master commit | PR | Title | |---|---|---|---| | 1 | 300d532 | #67533 | [fix](table stream) Preserve table stream offsets during cleanup | | 2 | bfe46ec | #67594 | [fix](row binlog) make time-based incremental reads use a safe fence | | 3 | c1bff0d | #67646 | [fix](ivm) Fail aggregate IVM refresh when the previous refresh txn is not visible yet | | 4 | 4f3abce | #67669 | [fix](ivm) Resolve IVM identity keys by slot identity and materialize unprojected keys | | 5 | 7129a3e | #67665 | [fix](ivm) Propagate and compensate failures of the IVM excluded-trigger-tables ALTER | | 6 | a565aca | #67710 | [fix](regression) Stop MTMV task waits from latching onto the previous task | | 7 | 8565db2 | #67775 | [fix](ivm) Repair the removed rewrite-context constructor call in IvmNormalizeMTMVJoinTest | | 8 | 0ded66a | #67575 | [feature](ivm) Support incremental refresh for array_agg and collect_list aggregates | | 9 | efc929a | #67480 | [fix](table stream) fix table stream TSO boundary semantics | Not included on purpose: - #62606 (IVM feature) is already in the branch's fork point (`efedf10c7e3`). - #67508 already landed on this branch via #67712. ### Cherry-pick notes - All nine picks applied without conflicts. Each pick's diff is identical to its master commit (compared with `index`/`@@` lines stripped). - Every touched file is byte-identical to master at `efc929aa7af` except `Env.java`, `FrontendServiceImpl.java` and `FrontendService.thrift`, whose remaining differences come only from unrelated master commits that are not part of this label (#67708, #66770, #67572, #67520). - #67594 is the safe-read-fence prerequisite of #67480; picking both in master order is what makes #67480 apply cleanly here (it replaces the earlier stand-alone attempt #67828, which had to hand-adapt around the missing fence). - #67775 is needed because #67646 (removes the `IvmRewriteContext` constructor) and #67669 (test that still used it) are both picked. ### Verification - `be/src/exec/scan/olap_scanner.cpp` (the only BE change): syntax-only compile with the Release flags, clean. The new thrift RPC `acquireTimeBasedChangeReadFence` is FE-only; nothing in `be/` or `cloud/` references it. - FE: `run-fe-ut.sh --run` on this branch (regenerates thrift, compiles fe-core main + test) with the 19 test classes touched by the picks: 19 classes, 450 tests, 0 failures, 0 errors, BUILD SUCCESS — `TableStreamManagerCleanupTest` 9, `CloudGlobalTransactionMgrTest` 22, `AlterMTMVTest` 24, `IvmAggArrayAggProcessorTest` 2, `IvmAggCollectListProcessorTest` 1, `IvmAggDeltaHandlerTest` 33, `IvmDeltaRewriterTest` 20, `IvmJoinDeltaHandlerTest` 23, `IvmLinearDeltaHandlerTest` 39, `IvmNormalizeMTMVJoinTest` 44, `PhysicalPlanTranslatorTest` 17, `IvmIncrRefreshMTMVTest` 13, `IvmNormalizeMTMVTest` 52, `CreateMTMVCommandTest` 94, `ExplainTableStreamPlanTest` 23, `OlapScanNodeTest` 12, `TimeBasedChangeVisibleWaiterTest` 7, `TransactionIdGeneratorTest` 1, `TSOTimestampTest` 14. - Regression suites touched parse cleanly (groovy parser check). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_018m1ARNXtGSWucJTy34uwe1 --------- Co-authored-by: TsukiokaKogane <cby141994@gmail.com> Co-authored-by: Luwei <814383175@qq.com> Co-authored-by: yujun <yujun@selectdb.com>

### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: A new FE master previously waited for all pre-failover transactions, including ordinary PREPARED transactions that could remain open until timeout. Publish a monotonic per-instance TSO fence after persisting the new allocation window, reject stale binlog commits in Meta Service, retry them with the new master's TSO, and recover only pre-fence COMMITTED TSO transactions. ### Release note Fence stale binlog transaction commit TSOs across FE master failover and remove the fixed recovery delay and PREPARED-transaction wait. ### Check List (For Author) - Test: Unit Test - FE TSO service/tracker and cloud transaction manager tests - Cloud Meta Service TSO fence/recovery, key, compatibility, and lazy commit tests - Behavior changed: Yes. Binlog commits with a TSO from an older FE master are retried with a fresh TSO; recovery no longer waits for PREPARED or non-TSO transactions. - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: Incremental reads track TSO-bearing commit attempts in FE memory. A commit RPC can finish without a conclusive response, leaving the attempt in the tracker indefinitely, while scanning Meta Service transactions during startup and periodic reconciliation adds complexity and can block unrelated reads. Fence an uncertain attempt in Meta Service before removing its exact TSO from the tracker, release definite responses immediately, and let incremental scans wait for pending lazy commits on their selected partitions. Disable transport retries for TSO-bearing commit RPCs so every uncertain delivery is fenced. Startup calibration advances the same per-instance fence, so stale commits from an old master are rejected without a recovery scan. ### Release note Bounded incremental reads now fence uncertain commit TSO attempts and no longer require startup or periodic transaction recovery scans. ### Check List (For Author) - Test: Unit Test - FE unit tests for TSO tracking, commit response handling, Meta Service retry behavior, and incremental scan version waits - Cloud unit tests for TSO fence monotonicity, fence-key encoding, and response-code compatibility - ./build.sh -j32 - ./build.sh --cloud -j32 - Cloud clang-tidy and build hygiene checks - Behavior changed: Yes. Uncertain TSO commit attempts are fenced before release, definite failures are released directly, and incremental scans wait for selected-partition pending lazy commits. - Does this need documentation: Yes. Connector-facing error-code documentation is included in the PR description.

### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: The commit TSO fence check lacked request-level and Meta Service configuration gates, its error name was unclear, the fence key was unavailable through Meta Service HTTP KV tooling, and the persisted window time did not state its unit. Add both fence-check gates to every Cloud commit path, expose the fence key to HTTP encode/get/set, rename the stale TSO error, and clarify the FE tracker and durable-state names. ### Release note Cloud commit TSO fence checks can be controlled by the request and the mutable Meta Service configuration. Rejected stale commit TSOs now use TXN_COMMIT_TSO_EXPIRED. ### Check List (For Author) - Test: Unit Test - Targeted FE unit tests: 108 passed - Cloud MetaService TSO fence test: passed - Cloud HTTP encode/get/set tests: 11 passed - Full ./build.sh -j32: passed - Cloud clang-tidy on changed lines: passed - Behavior changed: Yes. Commit TSO fencing requires both request and Meta Service gates, and the stale TSO error was renamed. - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: A master merge added an earlier FlightRuntimeException catch in getFlightInfoStatement, which bypassed the existing error-code filter and exposed every Flight error directly. Remove that catch so only incremental-window errors retain their retryable status while unrelated errors keep the established INTERNAL wrapper. ### Release note None ### Check List (For Author) - Test: Unit Test - DorisFlightSqlProducerTest: 7 passed - Full ./build.sh -j32: passed - Behavior changed: No. This restores the intended Arrow Flight error handling. - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: Recent master changes extended Cloud transaction tests and changed the test environment to a delegating CloudEnv mock, which conflicted with the committed TSO branch. Merge current master, preserve both sets of test dependencies, and stub the TSO service through the mock so the fence retry test uses the intended service. ### Release note None ### Check List (For Author) - Test: Unit Test - CloudGlobalTransactionMgrTest: 37 passed - DorisFlightSqlProducerTest: 7 passed - Full ./build.sh -j32: passed - Behavior changed: No. This resolves the master merge while preserving the existing TSO fence behavior. - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: Recent master changes to Arrow Flight session handling and statement executor tests conflicted with committed TSO error propagation. Merge current master, preserve the new session and deferred-query behavior while continuing to expose only incremental-window errors, and extend TSOTransactionTracker tests for repeated registration, fence replacement, waits, exact-attempt abandon semantics, metrics, and preconditions. ### Release note None ### Check List (For Author) - Test: Unit Test - TSOTransactionTrackerTest: 9 passed - DorisFlightSqlProducerTest: 12 passed - StmtExecutorTest: 26 passed - Full `./build.sh -j32`: passed - Behavior changed: No. This resolves the master merge and adds coverage. - Does this need documentation: No

### What problem does this PR solve? Issue Number: None Related PR: apache#67181, apache#67594 Problem Summary: A Meta Service KV_TXN_MAYBE_COMMITTED response means the underlying commit attempt is no longer in flight: it either committed or will never commit. Advancing the TSO fence before releasing this TSO adds an unnecessary Meta Service write. Release the tracked TSO directly for this response while retaining fence advancement for FE-to-Meta-Service RPC failures whose requests may still complete later. ### Release note None ### Check List (For Author) - Test: Unit Test - CloudCommittedTsoTest: 3 passed - Full `./build.sh -j32`: passed - Behavior changed: Yes. KV_TXN_MAYBE_COMMITTED releases its tracked TSO without advancing the fence. - Does this need documentation: No

…nded incremental reads (#67820) ### What problem does this PR solve? Issue Number: None Related PR: #67181, #67594 Problem Summary: Bounded cloud `@incr` reads need a safe end timestamp. Waiting by transaction ID can include transactions outside the requested time window, while one slow transaction can hold a global committed prefix and block unrelated tables. FE failover also needs to prevent an old master or an uncertain commit RPC from later committing a TSO that FE has already treated as finished. This PR introduces a durable committed TSO and keeps the active commit-attempt set in the FE master: - Allocate and register a commit TSO immediately before the first Meta Service commit RPC, under the TSO allocator lock. Retries after a fenced response replace the registration with a fresh TSO above the returned fence. - On an explicit success, already-visible result or already-aborted result, remove the transaction registration. On another explicit failure, remove only the exact attempted TSO. - On `KV_TXN_MAYBE_COMMITTED`, an RPC timeout, connection loss or another uncertain result, synchronously advance the per-instance Meta Service fence to the attempted TSO before removing it. If fencing fails, deactivate the TSO service so reads and allocations cannot use an unsafe prefix. - Disable transport-level retries for commits carrying a positive TSO. `MS_TOO_BUSY` remains retryable because Meta Service explicitly rejects it before processing. Non-TSO transactions keep the existing retry behavior. - Derive the committed TSO from the current allocated TSO and the oldest registered attempt, and persist it together with the allocation-window end in one BDB journal record. The default persistence window is reduced from five seconds to one second. - Remove the startup transaction scan, recovery RPC, recovery state, and periodic `checkTransactions()`. A new master persists a fresh window and advances the same global fence before enabling TSO service, so commits using an older TSO are rejected without waiting for PREPARED transactions. - A successful lazy commit can be released from FE memory. Incremental scans ask Meta Service to wait for pending transactions on the selected partitions before fixing their visible versions, preserving readability without a global recovery list. For a strongly consistent cloud read with an explicit end on every incremental relation, FE applies these rules: | Requested end | Behavior | | --- | --- | | After the current TSO physical time | Return `5100 / ERR_INCR_WINDOW_NOT_READY` immediately. | | At or before the durable committed TSO physical time | Admit the window directly. | | Between committed and current TSO physical times | Wait only for the captured FE registrations whose TSO is within the window and whose tables are queried. Return `5101 / ERR_INCR_VISIBLE_WAIT_TIMEOUT` when the wait expires. | After admission, version resolution waits only for pending transactions on the selected partitions. Unrelated tables do not participate in either wait. `information_schema.tso_status` exposes `COMMITTED_TSO` and `COMMITTED_TSO_PHYSICAL_TIME`; the latter is epoch milliseconds and is the maximum globally admitted end timestamp. The existing current TSO and allocation-window columns remain available. The Meta Service fence is one monotonic, non-versioned current-state key in the instance transaction namespace. Commit reads it in the same FDB transaction as the metadata commit and rejects `commit_tso <= fence` with `TXN_COMMIT_TSO_EXPIRED`, returning the effective fence. The check runs only when both the FE request flag and the mutable Meta Service configuration are enabled. Cloud snapshot, clone and rollback do not restore this leadership/commit fence; a restored or cloned instance publishes its current fence when its FE master initializes. Upgrade Meta Service before FE. A missing or unavailable fence RPC keeps a new FE TSO service uninitialized. The new Meta Service error code uses `actual_code` while old clients receive the compatible fallback code. #### Connector error handling | Code | Name | Meaning | Connector action | | --- | --- | --- | --- | | 5100 | `ERR_INCR_WINDOW_NOT_READY` | The requested end is after current TSO, or the FE master/TSO generation changed while establishing the window. | Retry the same end timestamp after `retryAfterMs`; do not advance the incremental offset. Route a retry through the current master when the reason is `TSO_MASTER_CHANGED`. | | 5101 | `ERR_INCR_VISIBLE_WAIT_TIMEOUT` | Relevant commit attempts did not finish within `change_visible_timeout_ms`. | Retry the same end timestamp; do not advance the incremental offset. | MySQL returns SQLSTATE `HY000`; the message contains `reason`, requested end, current TSO, committed TSO, their physical times, timeout and retry delay. Arrow Flight SQL returns `UNAVAILABLE` only for these two errors and includes `doris-error-code` and `doris-error-name` metadata. Other Flight failures retain the original `INTERNAL` wrapping. ### Release note Improve bounded, strongly consistent cloud incremental reads with a durable committed TSO, table-scoped waits and per-instance commit TSO fencing. Expose the committed TSO through `information_schema.tso_status`, distinguish an unavailable window (5100) from a visibility wait timeout (5101), and remove FE startup/periodic transaction recovery scans. Upgrade Meta Service before FE. ### Check List (For Author) - Test - [x] FE unit tests for TSO service/tracker, commit-result handling, Meta Service retry behavior, incremental version waits, MySQL error propagation and Arrow Flight SQL error mapping. - [x] Cloud ASAN unit tests for fence monotonicity and stale-commit rejection, fence-key encoding, and response-code compatibility. - [x] Regression tests `test_committed_tso` and `test_binlog_changes_syntax` passed earlier in this PR with outputs generated by the standard runner. - [x] `./build.sh -j32` and `./build.sh --cloud -j32`. - [x] clang-format 16, check-format, Cloud clang-tidy and build-hygiene checks. - Behavior changed: Yes. Bounded cloud reads use committed TSO/table-scoped waits; uncertain TSO commit attempts are fenced before release; FE no longer scans transactions during startup or periodically. - Does this need documentation: Yes. Connector-facing error handling is documented above.

…on PRs from master in merge order (#67820 #68088) (#68236) Cherry-picked from #67820, #68088 Batch pick of every merged PR carrying the `incremental-computation` label that `branch-incremental-computation` does not have yet (no `incremental-computation-picked` label), in the order they landed on master (`git log --first-parent`). One commit per PR, each ending with `(cherry picked from commit <master sha>)`. Follows the same convention as #67830, #68017, #68073 and #68151. | # | Master commit | PR | Title | |---|---|---|---| | 1 | 44e3ae2 | #67820 | [fix](binlog) Track committed TSO and fence uncertain commits for bounded incremental reads | | 2 | 3de3a75 | #68088 | [fix](binlog) Require SELECT privilege for binlog TVF | Not included on purpose: - The 25 labelled PRs that already carry `incremental-computation-picked`. - #68012 carries the label but is a PR against this branch itself (merged as `6f7c87fa892`); nothing to pick. ### How #67820 was picked This branch already carried #67820 through #67861, which backported the PR's first seven commits (up to `17272039558`) before the PR was merged. Between that cut and the merge the PR gained six more commits and five master merges, and the final design differs from what #67861 brought: the Meta Service recovery RPC (`get_tso_recovery_transactions`) and the FE startup/periodic recovery scan are gone, replaced by a durable per-instance commit-TSO fence (`txn_tso_fence_key`, `advance_tso_fence`, commit-time `TXN_COMMIT_TSO_EXPIRED` check, `enable_check_commit_tso_fence`), plus the review-feedback and "release maybe-committed TSO" fixes. A plain `cherry-pick -x 44e3ae2` conflicts in 23 files because the branch holds the intermediate design, so commit 1 was built by replaying, on top of the branch, exactly what the PR gained after the cut: - the PR's later commits `d16e67197b2`, `e53291d196b`, `864546d5531`, `8c55eb086e9`, `8d264ce294a`, `fc4f9969202`; - the content that landed inside its master merges, identified by diffing each merge against its `git merge-tree` automerge result: `c491a293426` (`setEnvTSOService` replaced by `Mockito.doReturn(tsoService).when(masterEnv).getTSOService()` — `masterEnv` is the same Mockito delegating mock here since #67813) and `29f133584b5` (the extra `TSOTransactionTrackerTest` coverage and two comment removals in `DorisFlightSqlProducer`). The other three merges only resolved import blocks against master-only code. The replay then got squashed into one commit with the master PR's message, the original author, and the `(cherry picked from commit 44e3ae2)` trailer. Adaptations, all recorded in the commit message: - `DorisFlightSqlProducer` / `DorisFlightSqlProducerTest` stay under `service/arrowflight` (the branch lacks the package move of #67866). - `8d264ce294a` is a no-op here: it removes a `catch (FlightRuntimeException)` block that #67883 added on master and this branch never had. - The branch-only `get_tso_recovery_transactions` RPC, its recovery scan, the five `TsoRecovery*` Meta Service tests and the NOLINT suppressions #67861 had added for them are removed, as on master. - The round-3 `mockVersionHelper()` adaptation in `CloudGlobalTransactionMgrTest` (no `VersionHelper.getVersionFromMeta(req, maxAttempts)` overload here, #66296) is kept. ### Prerequisite check - **#67820** declares #67181 and #67594 as related; #67181 (`e5a4e725fac`) is before the fork point and #67594 came with #67830. The rest of what the commit touches on master is import-block and neighbouring-code drift from unlabelled commits (#67866 / #67883 / #67966 Arrow Flight and session refactors, #67761 `get_prepare_txn_by_coordinator`, #66598 pre-rowset delete bitmaps, repair-tablet-index changes); none of it is used by the fence/committed-TSO logic. - **#68088** declares no related PR. The hook it implements (`TableValuedFunctionIf.checkAuth`) and the caller chain (`CheckPrivileges.visitLogicalTVFRelation` → `TableValuedFunction.checkAuth` → catalog function) are byte-identical between this branch and master, and the five-argument `AccessControllerManager.checkTblPriv` overload exists. Applied cleanly. ### Drift check against master - **#67820**: every one of the 54 files the master commit touches now contains the pick's content — the master commit reverse-applies cleanly per file onto this branch (50 files), and the four files where only the surrounding context differs (`config.h`, `meta_service_txn.cpp`, `StmtExecutor.java`, `StmtExecutorTest.java`) contain every added line and none of the removed ones. 33 of the 54 files are byte-identical to master at `44e3ae2b951`, including all of `fe/.../tso/`, `CloudGlobalTransactionMgr.java`, `MetaServiceProxy.java`, `keys.{h,cpp}`, `meta_service.h` and the regression suite/output. The remaining differences are unrelated master-only or branch-only code from unlabelled commits (#67761, #66598, #67866/#67883/#67966, repair-tablet-index, meta-cache columns, recycler configs) plus the documented `mockVersionHelper()` adaptation; no line in the committed-TSO/fence domain is left over from #67861. - **#68088**: all three files are byte-identical to master at `3de3a756f74`. ### Verification - FE: `run-fe-ut.sh --run` on this branch (regenerates thrift/protobuf, compiles fe-core main + test) with every test class the picks touch: 12 classes, 175 tests, 0 failures, 0 errors, BUILD SUCCESS — `TSOServiceTest` 38, `CloudGlobalTransactionMgrTest` 37, `StmtExecutorTest` 27, `MetaServiceProxyTest` 22, `OlapScanNodeTest` 12, `TimeBasedChangeVisibleWaiterTest` 12, `TSOTransactionTrackerTest` 9 (the four tests added inside the PR's last master merge included), `DorisFlightSqlProducerTest` 7, `TsoStatusMetadataGeneratorTest` 5, `CloudCommittedTsoTest` 3, `SchemaTableTest` 2, `TableBinlogFunctionAuthTest` 1 (#68088). The `@Test` counts of the touched classes equal master's. - FE checkstyle on fe-core: 0 violations. - Meta Service: `-fsyntax-only` with the flags of `cloud/CMakeLists.txt` (`-Wall -Werror`, regenerated `gen_cpp/cloud.pb.h` with `AdvanceTsoFence*` / `TxnTsoFencePB` and without `GetTsoRecoveryTransactions*`) on `meta_service_txn.cpp`, `keys.cpp`, `http_encode_key.cpp`, `bvars.cpp` and, with `-DUNIT_TEST -DBE_TEST -fno-access-control`, on `meta_service_test.cpp`, `keys_test.cpp`, `http_encode_key_test.cpp`, `meta_service_helper_test.cpp`, `txn_lazy_commit_test.cpp`: no diagnostic in any line the pick touches (the only errors are the pre-existing macOS-only `pthread_setname_np` / `int64_t`-vs-`long` ones in untouched 2024/2025 code). clang-format 16 is clean on all 14 touched cloud files. - No BE file changes in this round (the BE side of #67820 was already byte-identical to master via #67861). - The new `auth_p0/test_binlog_tvf_auth.groovy` and the two #67820 suites parse cleanly (groovy parser check). --------- Co-authored-by: Luwei <814383175@qq.com> Co-authored-by: morrySnow <zhangwenxin@selectdb.com>

luwei16 requested review from 924060929, englefly, gavinchou, morrySnow and starocean999 as code owners September 7, 2026 08:21

morrySnow previously approved these changes Sep 7, 2026

View reviewed changes

luwei16 dismissed morrySnow’s stale review via 51cf858 September 7, 2026 08:48

luwei16 added the area/mysql-binlog label Sep 7, 2026

luwei16 added binlog and removed area/mysql-binlog labels Sep 8, 2026

luwei16 added the incremental-computation label Sep 8, 2026

gavinchou approved these changes Sep 9, 2026

View reviewed changes

morrySnow approved these changes Sep 9, 2026

View reviewed changes

luwei16 merged commit bfe46ec into apache:master Sep 9, 2026
42 of 44 checks passed

luwei16 mentioned this pull request Sep 10, 2026

[fix](binlog) Track committed TSO and fence uncertain commits for bounded incremental reads #67820

Merged

9 tasks

morningman added the incremental-computation-picked label Sep 11, 2026

morningman mentioned this pull request Sep 20, 2026

branch-incremental-computation: pick the merged incremental-computation PRs from master in merge order (#67820 #68088) #68236

Merged

Conversation

luwei16 commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Future endTimestamp error example

Release note

Check List (For Author)

Uh oh!

hello-stephen commented Sep 7, 2026

Uh oh!

luwei16 commented Sep 7, 2026

Uh oh!

hello-stephen commented Sep 7, 2026

Cloud UT Coverage Report

Uh oh!

luwei16 commented Sep 7, 2026

Uh oh!

hello-stephen commented Sep 7, 2026

BE UT Coverage Report

Uh oh!

luwei16 commented Sep 8, 2026

Uh oh!

luwei16 commented Sep 8, 2026

Uh oh!

hello-stephen commented Sep 8, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Sep 8, 2026

Uh oh!

hello-stephen commented Sep 8, 2026

Uh oh!

hello-stephen commented Sep 8, 2026

Uh oh!

gavinchou commented Sep 9, 2026

1. [major] Classic-mode incremental reads can wait for their own uncommitted transaction

2. [major] INCR reads inside a CTE still bypass the fence and future-end validation

Uh oh!

gavinchou left a comment

Choose a reason for hiding this comment

Uh oh!

luwei16 commented Sep 9, 2026

Uh oh!

luwei16 commented Sep 9, 2026

Uh oh!

luwei16 commented Sep 9, 2026

Uh oh!

hello-stephen commented Sep 9, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Sep 9, 2026

BE Regression && UT Coverage Report

Uh oh!

luwei16 commented Sep 9, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

luwei16 commented Sep 7, 2026 •

edited

Loading