Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
TPC-H: Total hot run time: 29107 ms |
TPC-DS: Total hot run time: 176900 ms |
ClickBench: Total hot run time: 24.73 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
1590e7e to
b28476c
Compare
|
run buildall |
b28476c to
2020c8d
Compare
|
run buildall |
2020c8d to
8241ada
Compare
|
run buildall |
1 similar comment
|
run buildall |
FE UT Coverage ReportIncrement line coverage |
|
run buildall |
TPC-H: Total hot run time: 17359 ms |
TPC-DS: Total hot run time: 83576 ms |
ClickBench: Total hot run time: 14.52 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
|
run buildall |
8241ada to
93dd6ad
Compare
|
run buildall |
TPC-H: Total hot run time: 16571 ms |
TPC-DS: Total hot run time: 81064 ms |
ClickBench: Total hot run time: 14.5 s |
93dd6ad to
c0f940a
Compare
|
run buildall |
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 16836 ms |
Inject a constant tinyint 1 match flag project on the null side of each outer join (LOJ right, ROJ left, FULL both, inner/cross none) during IVM normalize. The join's null filling turns the flag NULL for unmatched rows, so hash(left_rid, [left_flag], right_rid, [right_flag]) decouples "NULL row id <=> unmatched" and prepares for a future PR that makes child row ids nullable (single-column MOW key used directly as largeint). - addJoinNullSideMatchedColumn appends Alias(TinyIntLiteral(1), flag) to the join child; flags are looked up by name from the join output (post null-fill) and consumed by the compose project above the join - identity keys are captured before flag-wrap and re-registered on the wrapped child so the flag is never part of the keys - IvmDeltaRewriteHelper.findSlotByName widened to public for name lookup - IvmNormalizeMTMVJoinTest covers key order for LOJ/ROJ/FULL/INNER and nested joins, flag consumption by compose projects, and output hygiene Co-Authored-By: Claude <noreply@anthropic.com>
buildRowIdHash returns cast(key AS LARGEINT) directly when the row-id has a single key that widens losslessly to largeint, instead of hashing it: no hash collision, and the row-id is NULL exactly when the key is NULL. Callers must treat row-ids as nullable: - agg apply join matches row-ids with null-safe equality - net-zero filter signals unmatched delta rows via the MV-side group count (COUNT(*) is never NULL) instead of IsNull(row-id), so the NULL-key group (NULL row-id) is not mistaken for an unmatched row Fixes stale test assertions (Cast instead of MurmurHash3128 for single-key row-ids, NullSafeEqual join conjuncts, pooled hidden-column reuse) and regenerates regression expected output (row-id value and nullability) for agg_shared_columns, refresh_dry_run, mtmv_row_binlog. Adds test_ivm_nullable_row_id covering NULL-key MOW rows and NULL group-by keys. Co-Authored-By: Claude <noreply@anthropic.com>
…ntal refresh Move the stored-vs-current IVM plan signature check from IvmIncrRefreshMTMV into IvmNormalizeMTMV: once normalization completes and the signature is computed, an INCREMENTAL-mode statement validates it against the stored layout baseline and fails fast with PLAN_SIGNATURE_MISMATCH instead of attempting a delta rewrite against a stale baseline. Key changes: - IvmNormalizeMTMV.rewriteRoot validates the signature after computing it when the rewrite context mode is INCREMENTAL - IvmIncrRefreshMTMV drops its validatePlanSignature call and method Unit Test: - IvmNormalizeMTMVTest: testIncrementalSignatureMismatchThrowsAtNormalize, testIncrementalSignatureMatchPassesNormalize - IvmIncrRefreshMTMVTest: removed stale testSignatureMismatchThrowsBeforeDeltaRewrite
…d of largeint buildRowIdHash previously passed a single key through only when it could be losslessly widened to largeint (cast to LARGEINT); any other single key fell back to a murmur hash. Since any eligible MOW key type can serve as the row-id without hash collision, the passthrough now applies to every type allowed by ColumnDefinition.isEligibleKeyType, and the row-id keeps the key's own type. A scalar agg (empty key list) now uses TinyIntLiteral(0) instead of LargeIntLiteral(0) for the same reason. Key changes: - IvmUtil.buildRowIdHash: single-key passthrough condition switched from isInjectiveCastTo(LargeIntType) to ColumnDefinition.isEligibleKeyType, returning the key itself without a Cast; empty key list returns TinyIntLiteral(0); javadoc updated - IvmNormalizeMTMV: updated computeScanRowIdAndKeys javadoc to reflect the new passthrough semantics Unit Test: - IvmUtilTest: single eligible INT/VARCHAR/largeint keys pass through, single STRING key still hashes, empty list returns TinyIntLiteral(0) - IvmNormalizeMTMVTest: row-id assertions updated to the key itself for grouped agg and excluded agg-key scans, scalar agg row-id is TinyInt(0) - New regression test_ivm_varchar_key_rowid: non-agg and agg IVM MVs over a single VARCHAR-key MOW table, three incremental refreshes (initial, upsert, delete) plus a final COMPLETE refresh, with DESC (show_hidden_ columns=true) asserting the row-id column keeps the varchar type - Regenerated expected outputs for test_ivm_agg_shared_columns (row-id largeint -> int) and test_ivm_explain_refresh (join shuffle change); full mtmv_p0/ivm regression 77/77 passed
… in IvmNormalizeMTMV Rename ScanRowId.rowIdExpr to mergedRowIdExpr so the field name matches its semantics (hash of all scan key slots), and collect keySlots, remainKeys and baseTableRowId in a single pass over the scan output instead of three separate stream pipelines. Key changes: - Rename ScanRowId.rowIdExpr to mergedRowIdExpr. - Merge keySlots/remainKeys/baseTableRowId collection into one for loop. Unit Test: - mvn compile -pl fe-core -am -DskipTests
…reams Add an IvmStreams column to the mv_infos table function that maps each IVM base table (full catalog.db.table path) to the IVM stream backing it, so users can tell which stream corresponds to which base table. Key changes: - Append IvmStreams(STRING) column to the MvInfosTableValuedFunction schema - Populate it in MetadataGenerator via IvmUtil.streamName and relation base tables - List only streams that actually exist, skipping excluded trigger tables - Return empty string for non-IVM materialized views Unit Test: - test_ivm_basic_mtmv - test_ivm_cross_db_same_name - test_build_mtmv
Fix two issues surfaced by rebasing the ivm branch onto upstream master: the withPreSnapshot comment lost its indentation (checkstyle violation) and the test still referenced ResolveCloudTableStreamReadState, which upstream renamed to CloudTableStreamReadStateHook. Key changes: - LogicalOlapScan: restore 4-space indentation of withPreSnapshot javadoc - InsertIntoTableCommandTableStreamTest: call CloudTableStreamReadStateHook.resolve via Deencapsulation instead of the removed ResolveCloudTableStreamReadState.rewriteRoot Unit Test: - ./build.sh --fe --clean passed
The guard skipping alias injection for views without an explicit column list was restored in 33ceb93, and the second assertion in CreateViewTest.testAlterView was updated at that time, but the first assertion in the same test still expected uppercase AS aliases, so the suite failed on any environment running current behavior. Key changes: - Update the stale expectation to keep the original lowercase as text Unit Test: - run-fe-ut.sh --run CreateViewTest passed 9/9
…fresh EXPLAIN SHAPE PLAN of the IVM incremental refresh chose broadcast on CI but bucketShuffle locally: join distribution is a cost-based decision sensitive to environment such as BE count. Session switches like enable_bucket_shuffle_join cannot pin it here because isBucketShuffleDownGrade short-circuits when the join child is not a GroupPlan, bypassing every downgrade path. Key changes: - Set broadcast_row_count_limit=0 in the suite so the broadcast candidate is structurally excluded; bucketShuffle always beats plain shuffle since it only redistributes one side, independent of BE count Unit Test: - run-regression-test.sh -d mtmv_p0/ivm -s test_ivm_explain_refresh passed twice consecutively with unchanged .out
IVM refresh synthesizes LogicalOlapTableStreamScan after relation binding: COMPLETE rewrites base scans in IvmFullRefreshMTMV and INCREMENTAL creates delta scans via IvmDeltaRewriteState. These scans never pass BindRelation, so CloudTableStreamReadStateHook is never registered for their statements and their cloud read states are never installed. Since apache#66739 tightened StreamConsumptionInfoExtractor, every IVM MV refresh failed in cloud mode with "Cloud Table Stream read state must be installed during relation analysis". Key changes: - Expose the install orchestration as public static CloudTableStreamReadStateHook.installReadStates(Plan): no-op on shared-nothing and on plans without stream scans - Register the hook in IvmFullRefreshMTMV so afterAnalyze batch-installs read states for COMPLETE refresh stream scans - Call installReadStates at the single return of IvmIncrRefreshMTMV.rewriteRoot for INCREMENTAL delta scans - Update the extractor error message accordingly Unit Test: - run-regression-test.sh -d mtmv_p0/ivm -s test_ivm_basic_mtmv passed on a local cloud cluster (1FE+1BE+1MS+1Recycler+FDB), previously failing
These suites assert local-mode-specific behavior and fail on cloud deployments even after the read-state install fix: - test_ivm_refresh_dry_run: __DORIS_SEQUENCE_COL__ derives from txn versioning, which differs between cloud and local (6145 vs 4097) - test_ivm_explain_refresh: the refresh sink plan on cloud has no PhysicalDistribute[DistributionSpecOlapTableSinkHashPartitioned] node - test_ivm_execution_failure_no_fallback: failure-injection debug points target DatabaseTransactionMgr and the shared-nothing insert RPC path, which the cloud transaction manager never hits Key changes: - Skip the three suites via isCloudMode() with the specific difference noted in comments and logs Unit Test: - run-regression-test.sh -d mtmv_p0/ivm with the three suites passed on the local cloud cluster (skipped as intended)
An AUTO refresh built attempts [PARTITIONS, COMPLETE] for every non-IVM MTMV. The PARTITIONS attempt trusts isMTMVSync, whose per-table verdict treats tables not implementing MTMVRelatedTableIf (JDBC plugin tables, views) as always synchronous. A deferred MV over such base tables was judged FRESH on its first AUTO and skipped as NOT_REFRESH while still empty, leaving external regression test_mysql_mtmv failing with an empty MV after a refresh task that reported SUCCESS. Key changes: - buildAttempts: add the PARTITIONS attempt for AUTO only when the MV refresh method is not COMPLETE, so COMPLETE-method MVs always go through the full-refresh attempt and log the routing decision - MTMVTaskTest: pin the attempt chain for COMPLETE-method, non-COMPLETE-method and IVM+INCREMENTAL AUTO requests, and pin that explicit partition lists never upgrade to a full refresh - regression: add mtmv_p0/test_auto_refresh_complete_method using an internal view to reproduce the sync-check blind spot without external environments, asserting task RefreshMode=COMPLETE across repeated AUTO refreshes including after new base data arrives Unit Test: - MTMVTaskTest (47 tests passed) - regression suites mtmv_p0/test_auto_refresh_complete_method and mtmv_p0/test_mysql_mtmv passed on local cluster
- InsertIntoTableCommandTest: duplicated class-closing brace (checkstyle parse error) - CreateTableCommandTest: more than one empty line before testRejectMaxValueInListPartition - DatabaseTransactionMgrTest: wrong import order (IvmInfo must sort before TestLogAppender) Co-Authored-By: Claude <noreply@anthropic.com>
…tream tests 047882e renamed the hook's static method resolve() to the public installReadStates() but the five test call sites in InsertIntoTableCommandTableStreamTest still invoked "resolve", so the four testCloud* tests failed with "No compatible static method found". Pre-existing on the ivm branch, surfaced by the post-rebase FE UT run. Co-Authored-By: Claude <noreply@anthropic.com>
…detail tables DUP_KEYS base tables previously used uuid_numeric() as the IVM row-id, which is non-deterministic and forced delete/update deltas to fall back to full refresh. DUP tables with row binlog always carry the __DORIS_ROW_LSN_COL__ hidden column, so the per-row lsn is now treated like a UNIQUE table's key columns and flows through the same deterministic row-id construction. DUP tables without row binlog keep the legacy uuid row-id and are still rejected by the binlog check. Key changes: - IvmNormalizeMTMV.computeScanRowIdAndKeys: unified keyColNames-based buildDeterministicScanRowIdAndKeys; DUP + row binlog uses the row lsn column as the identity key, DUP without row binlog keeps uuid_numeric() - LogicalOlapTableStreamScan.computeOutput: expose __DORIS_ROW_LSN_COL__ under IVM rewrite for RESET/SNAPSHOT stream scans (incremental scans expose the stream lsn virtual column instead) - IvmLinearDeltaHandler: map the incremental delta scan's STREAM_LSN_COL back to the original scan's ROW_LSN_COL slot - IvmUtil / IvmDeltaRewriteHelper: fail loudly instead of defaulting the row lsn column to 0 when it is missing from a scan output - Propagate the MTMV's excluded trigger tables into the incremental refresh statement contexts (job, dry run, EXPLAIN INCREMENTAL) so excluded tables are not validated for binlog / key-type support Unit Test: - IvmNormalizeMTMVTest (52), IvmNormalizeMTMVJoinTest (43), IvmNormalizeMTMVUnionTest (10) - IvmDeltaRewriterTest (20), IvmLinearDeltaHandlerTest (37) - IvmIncrRefreshManagerTest (11), RefreshMTMVCommandTest (13), ExplainRefreshMTMVCommandTest (23) - New regression test_ivm_dup_keys_lsn_rowid and test_ivm_dup_excluded_no_binlog_fallback; updated test_ivm_complete_refresh_rowid / test_ivm_inner_join_1 for deterministic DUP row-id
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary: Changing excluded_trigger_tables clears the refresh snapshot while leaving a successfully refreshed MTMV in NORMAL state. AUTO refresh could then run the partition freshness check with the new exclusions, incorrectly conclude that no refresh was needed, and leave stale data in the materialized view. Route only non-IVM AUTO refreshes in NORMAL/SUCCESS state without a snapshot directly to COMPLETE so the invalidated baseline is rebuilt, while preserving initial refresh and existing partition refresh behavior.
### Release note
AUTO refresh now rebuilds a successfully established MTMV baseline after it has been invalidated.
### Check List (For Author)
- Test: Unit Test and Regression Test
- ./run-fe-ut.sh --run org.apache.doris.mtmv.MTMVTaskTest
- ./run-regression-test.sh --run -d mtmv_p0 -s test_excluded_trigger_table_mtmv
- Behavior changed: Yes. A non-IVM AUTO refresh rebuilds an invalidated successful baseline with COMPLETE refresh.
- Does this need documentation: No
… refresh Adds an MTMV property that limits IVM incremental refresh of each configured base table to its last N partitions (by partition value). Changes outside the window are ignored on the incremental path, so users with very old partitions they no longer care about can cut refresh computation cost. COMPLETE refresh stays full-table and authoritative; the property is not part of the IVM plan signature, so altering it takes effect immediately and removed windows replay accumulated binlog to catch up. Key changes: - PropertyAnalyzer / MTMVPropertyUtil: parse and validate "tbl:N,...", value-ordered window (range upper bound / list value), excluded_trigger_tables-style name matching - CreateMTMVInfo / AlterMTMVPropertyInfo: base-table membership check at create/alter; non-IVM rejected on ALTER only (AUTO creation may fall back to non-IVM) - IvmDeltaRewriteState / IvmDeltaRewriter: window the stream scan (partitions and tablets), snapshot scans, and excluded trigger tables; hasPendingData only checks window partitions - Regression suite test_ivm_partition_window_limit with window-maintained / window-ignored / shrink / catch-up / excluded+windowed scenarios Unit Test: - AlterMTMVTest: non-IVM create allowed, non-IVM alter rejected, unknown table rejected at create and alter, ivm_use_full_keys alter rejected - MTMVPropertyUtilTest: parse, name matching, duplicate/ambiguous rejection, value-ordered window ids, non-partitioned/unlimited returns - IvmDeltaRewriteStateTest: window restriction, tablet narrowing, excluded-table windowing, selection intersection - Regression test_ivm_partition_window_limit passed on local cluster
Partitioned (FOLLOW_BASE_TABLE) IVM materialized views previously rejected queries containing a OneRowRelation. The restriction was a scope decision from the initial OneRowRelation support, not a known failure mode: the constant side produces no delta, partition routing comes from the real base table, and the initial-refresh COMPLETE mechanism is generic. Removing it lets constant-dimension join fact tables with growing partitions use IVM incremental refresh. Key changes: - CreateMTMVInfo: drop validateIvmOneRowRelationPartition and the now-unused containsOneRowRelation field/state snapshot - CreateMTMVCommandTest: partitioned one-row creation is now allowed; outer join retained-side determinism test updated (row-lsn row-id makes DUP retained sides deterministic) - Regression test_ivm_one_row_relation_partitioned: multi-partition incremental maintenance plus a growing-PCT scenario (base table starts with one partition, ADD PARTITION + insert + refresh, MV follows automatically) Unit Test: - CreateMTMVCommandTest: 93 tests pass (one-row partitioned allowed, outer join with deterministic row-id allowed) - Regression test_ivm_one_row_relation_partitioned passed on local cluster
Upstream removed the schema-based OlapTableStream constructors and resolves the stream base table through the catalog (TableStreamBaseTableInfo), so the IVM delta tests no longer compile and registered test streams can no longer resolve their mock base tables. Key changes: - Use the new OlapTableStream(id, name, baseTable) constructor in IvmDeltaTestBase and IvmDeltaRewriteHelperTest - Register the mock base table into test_db in registerTestStream so the stream resolves it by id, replacing any stale object registered under reused mock ids - Set up the IVM rewrite ConnectContext in testRemapScanOutputForPreSnapshotPreservesExprId so the pre-snapshot stream scan carries __DORIS_ROW_LSN_COL__ Unit Test: - RefreshMTMVCommandTest, RefreshMTMVInfoAnalyzeTest, ExplainRefreshMTMVCommandTest, CreateMTMVCommandTest, AlterMTMVTest, ShowCreateMTMVTest - IvmIncrRefreshManagerTest, MTMVTaskTest, MTMVJobManagerTest, MTMVTaskContextCompatTest, OlapTableStreamUpdateTest - IvmDeltaRewriteHelperTest, IvmAggDeltaHandlerTest, IvmDeltaRewriterTest (324 tests, all passed)
c0f940a to
4bc05c1
Compare
|
run buildall |
FE UT Coverage ReportIncrement line coverage |
… resolution After the dynamic stream schema and the lsn row-id change, IvmJoinDeltaHandlerTest failed with 20 errors and UpdateMvByPartitionCommandTest with 1 failure. Mock streams could not resolve their base tables by id when mocks reused the same name with a different id (Database.registerTable refuses same-name registration), the test-driven delta rewrite ran without the production IVM rewrite ConnectContext so snapshot stream scans dropped __DORIS_ROW_LSN_COL__, and the planner assertion still expected the legacy LARGEINT uuid row-id for a DUP binlog base table. Key changes: - IvmDeltaTestBase.registerTestStream: drop stale mock tables registered under the same name as well as the same id before registering - IvmJoinDeltaHandlerTest: run delta rewrites under a thread-local ConnectContext carrying the IVM rewrite context (runWithIvmRewriteContext helper), matching production incremental refresh - UpdateMvByPartitionCommandTest: expect the BIGINT lsn row-id for the DUP binlog base and add a MOW UNIQUE base MV pinning the LARGEINT multi-key hash row-id Unit Test: - IvmJoinDeltaHandlerTest (23 tests), UpdateMvByPartitionCommandTest (10 tests) - Full Ivm*Test suite (372 tests), checkstyle clean
…es baseline rebuild e234ad2 routes the first AUTO refresh after changing excluded_trigger_tables to a COMPLETE baseline rebuild, because the alter clears the refresh snapshot and the freshness check can no longer prove the baseline. The rebuilt MV also reflects the excluded table's current data. The suite still expected the pre-rebuild semantics (NOT_REFRESH and no row from the excluded table), so it failed. Key changes: - Expect COMPLETE for trigger_table_not_need_refresh (first AUTO refresh after the alter rebuilds the baseline) - Expect the rebuilt MV to contain the excluded table's current row in after_trigger_table Unit Test: - mtmv_p0/test_create_mtmv_with_view regression run with forceGenOut to regenerate the expected output against the current behavior
Restore the apache#66889 behavior: SNAPSHOT reads on DUP_KEYS streams should expose the base __DORIS_ROW_LSN_COL__ column. It was filtered out when hidden column handling moved into computeOutput in 5afd66e, making snapshot queries on the column fail with Unknown column. Key changes: - LogicalOlapTableStreamScan.computeOutput: expose ROW_LSN_COL for SNAPSHOT reads on DUP_KEYS tables, hoist the keys-type check out of the column loop, and correct the forceNullable comment for the SNAPSHOT expansion - test_olap_table_stream_snapshot: add partitioned DUP + show_initial_rows=false case asserting the snapshot image and base row LSN Unit Test: - regression-test/suites/table_stream_p0/test_olap_table_stream_snapshot.groovy (pass)
… enlarged or removed Enlarging or removing ivm_partition_window_limit brings previously lossy partitions back into the refresh range, but their stream backlog was skipped by the windowed refreshes. A strict INCREMENTAL refresh right after the ALTER was wrongly skipped as "all partitions are synced" and returned SUCCESS with stale data (DORIS-28376). The ALTER now invalidates the refresh snapshot and marks the IVM baseline for a complete rebuild, so the next refresh covers the new range; a strict INCREMENTAL is rejected until then. Key changes: - MTMV.alterMvProperties: detect window enlargement/removal per base table and require a complete baseline rebuild, merged with the excluded_trigger_tables invalidation so schemaChangeVersion/snapshot are bumped once - PropertyAnalyzer: document the enlarge/remove contract on the property - test_ivm_partition_window_remove: new regression covering window removal and enlargement (strict INCREMENTAL rejected, AUTO refresh performs COMPLETE and replays the backlog) - test_ivm_partition_window_limit: switch the remove-window refresh to AUTO Unit Test: - regression-test/suites/mtmv_p0/ivm/test_ivm_partition_window_remove.groovy (pass) - regression-test/suites/mtmv_p0/ivm/test_ivm_partition_window_limit.groovy (pass) - regression-test/suites/mtmv_p0/ivm/test_ivm_excluded_trigger_table.groovy (pass)
|
run buildall |
1 similar comment
|
run buildall |
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)