[feat](TSO) Improve TSO status access - #65850
Merged
Merged
Conversation
AntiTopQuark
requested review from
924060929,
englefly,
morrySnow and
starocean999
as code owners
July 21, 2026 08:46
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29118 ms |
Contributor
TPC-DS: Total hot run time: 177063 ms |
Contributor
ClickBench: Total hot run time: 24.77 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29120 ms |
Contributor
TPC-DS: Total hot run time: 176933 ms |
Contributor
ClickBench: Total hot run time: 25 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
Member
|
/review |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a new FE/Nereids SHOW TSO STATUS SQL command so SQL clients can query the same TSO status fields that were previously only available via the HTTP /api/tso endpoint. The command forwards to the master FE and returns four BIGINT fields describing the current TSO and its physical/logical components, while surfacing “disabled” and “not calibrated” states as user-visible errors.
Changes:
- Extend the Nereids SQL grammar to parse
SHOW TSO STATUSand introduceTSOas a keyword (kept usable as an identifier vianonReserved). - Implement
ShowTsoStatusCommand(metadata + execution + master-forwarding) and wire it through plan type, visitor, and logical plan builder. - Add FE unit tests and extend the existing TSO regression suite to validate the SQL interface.
Critical checkpoints (Part 1.3) — conclusions:
- Goal & correctness: The new command returns the four required fields without allocating a new TSO (uses
getCurrentTSO()/getWindowEndTSO(), notgetTSO()), and is forwarded to master viaFORWARD_NO_SYNC. Covered by both regression and unit tests. - Scope/focus: Changes are narrowly scoped to parsing, command plumbing, and tests.
- Concurrency: No new shared-state mutation is introduced; reads rely on existing
TSOServiceconcurrency controls. - Config changes: No new configuration items added.
- Compatibility: No FE↔BE protocol changes; this is FE-only command parsing/execution.
- Test coverage: Regression test validates SQL result shape and basic invariants; unit tests cover parse, metadata, redirect, disabled, and not-calibrated cases.
- Observability: No new operational surface added beyond user-visible errors; acceptable for a read-only SHOW command.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| regression-test/suites/tso_p0/test_tso_api.groovy | Adds regression validation for SHOW TSO STATUS output fields and basic invariants. |
| fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 | Adds SHOW TSO STATUS statement and marks TSO as non-reserved identifier keyword. |
| fe/fe-sql-parser/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4 | Adds TSO lexer token. |
| fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowTsoStatusCommandTest.java | Unit tests for parsing, metadata, redirect behavior, disabled/not-calibrated handling, and “no allocation” behavior. |
| fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/CommandVisitor.java | Wires visitor entrypoint for the new command. |
| fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/PlanType.java | Adds SHOW_TSO_STATUS_COMMAND plan type. |
| fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowTsoStatusCommand.java | Implements the new SHOW command and forwards to master FE. |
| fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java | Adds parse-to-plan construction for SHOW TSO STATUS. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Request changes: the command's readiness check can return a successful but impossible TSO status while calibration is still uninitialized.
Critical checkpoints: The parser, non-reserved keyword handling, builder/plan/visitor registration, master forwarding, proxy result propagation, cloud-disabled behavior, four BIGINT fields, and non-allocating read all match the feature goal. The change is otherwise small and focused. The applicable concurrency/lifecycle checkpoint is not satisfied: calibration publishes the current timestamp before the persisted window and isInitialized flag, while the command treats only zero as uncalibrated; MAIN-TSO-1 gives the concrete startup/failover/journal-failure sequence. No new configuration, FE-BE protocol, storage format, transaction, or data-write compatibility surface is introduced. Lock scope and steady-state performance are lightweight; no deadlock or other lifecycle issue was found. Existing logging/metrics are adequate once the false-success path is removed.
Tests: the new unit tests cover parsing, metadata, non-allocation, disabled/zero states, and redirect status, and the regression test covers the dynamic happy path and composition. A real nonzero-current/uninitialized calibration failure or blocked-publication test is missing. Per the review-only runner contract, no local build or test was run; current GitHub results show compile, FE UT, P0, and nonConcurrent regression success, while the FE coverage context is failing at review time.
User focus: no additional focus was provided; the full PR was reviewed.
Overall: one P2 correctness finding; request changes.
Contributor
Author
|
run buildall |
morrySnow
changed the title
Contributor
|
maybe a tso table in information_schema is a better way |
Contributor
TPC-H: Total hot run time: 29731 ms |
Contributor
Author
I believe using the
|
Contributor
TPC-DS: Total hot run time: 177050 ms |
Contributor
ClickBench: Total hot run time: 24.87 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
AntiTopQuark
force-pushed
the
support_show_status
branch
from
July 28, 2026 13:25
22e0053 to
43b2c7b
Compare
Contributor
Author
|
run check_coverage |
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run beut |
Contributor
Author
|
run buildall |
github-actions
Bot
removed
the
approved
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run performance |
1 similar comment
Contributor
Author
|
run performance |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 29119 ms |
Contributor
TPC-DS: Total hot run time: 166378 ms |
Contributor
ClickBench: Total hot run time: 23.81 s |
morrySnow
approved these changes
Aug 6, 2026
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
Author
|
/review |
luwei16
approved these changes
Aug 7, 2026
wyxxxcat
pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 17, 2026
### What problem does this PR solve? Issue Number: close apache#65849 Related PR: #xxx Problem Summary: The /api/tso endpoint only served requests on the master FE, so requests sent to a follower failed instead of reaching the authoritative TSO service. TSO status also needed a queryable system-table interface without adding a dedicated SHOW command. This change forwards /api/tso requests to the master FE by default, adds local=true to inspect the receiving FE without forwarding, and reads all response fields from one TSO status snapshot. It also exposes the master TSO state through information_schema.tso_status using the existing schema-table RPC path.
luwei16
added a commit
that referenced
this pull request
Sep 9, 2026
…67594) ### What problem does this PR solve? Issue Number: None Related PR: #65850 Problem Summary: Time-based `@incr` reads previously allowed an `endTimestamp` later than the FE's current TSO. Such a future boundary cannot be closed by the transaction waiting logic: a transaction may start after the request begins but still fall before that future boundary, so the scan cannot guarantee a complete result. In addition, waiting only for currently running transactions is insufficient unless the TSO boundary, transaction watermark, transaction visibility, and table visible versions are coordinated under one authoritative fence. Otherwise, especially on a follower FE, a transaction can already be `VISIBLE` while the local catalog still exposes an older visible version, causing the incremental scan to miss rows. This PR establishes a safe read fence for both classic and cloud modes: 1. Acquire the current TSO from the master FE. 2. Capture the transaction ID watermark after the TSO snapshot. 3. Wait for transactions at or below the watermark that involve the scanned OLAP tables to reach a final visible or aborted state. 4. In classic mode, synchronize table publishers and wait for follower journal replay so catalog visible versions cross the same fence. 5. In cloud mode, refresh the latest visible versions from MetaService after the transaction wait. 6. Reject an `endTimestamp` later than the captured `CURRENT_TSO_PHYSICAL_TIME`. 7. Convert a physical timestamp `P` to TSO boundary `(P, 0)`, preserving half-open range semantics: `[startTimestamp, endTimestamp)`. #### Future endTimestamp error example Request: ```sql SELECT id, value, __DORIS_BINLOG_OP__ FROM example_table@incr( "startTimestamp" = "2026-09-01 00:00:00", "endTimestamp" = "2999-01-01 00:00:00", "incrementType" = "DETAIL" ); ``` Response: ```text ERROR 1105 (HY000): errCode = 2, detailMessage = endTimestamp exceeds the maximum supported time for an INCR read: requestedEndTimestampMs=<requested epoch milliseconds>, CURRENT_TSO_PHYSICAL_TIME=<maximum supported epoch milliseconds> ``` The maximum currently supported boundary can be obtained from: ```sql SELECT CURRENT_TSO_PHYSICAL_TIME FROM information_schema.tso_status; ``` ### Release note Time-based `@incr` reads now reject `endTimestamp` values after `CURRENT_TSO_PHYSICAL_TIME`. Timestamp ranges use half-open semantics `[startTimestamp, endTimestamp)`, with each physical timestamp mapped to logical counter zero. ### Check List (For Author) - Test: FE unit tests, FE and BE builds, and `row_binlog_p0/test_binlog_changes_syntax` regression test - Behavior changed: Yes - Future `endTimestamp` values are rejected with the maximum supported TSO physical time in the error message. - Physical timestamp boundaries are interpreted as `(physicalTime, 0)`. - Does this need documentation: No
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #65849
Related PR: #xxx
Problem Summary:
The /api/tso endpoint only served requests on the master FE, so requests sent to a follower failed instead of reaching the authoritative TSO service. TSO status also needed a queryable system-table interface without adding a dedicated SHOW command.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)