Skip to content

[draft](ivm) for run the test pipeline - #61496

Closed
yujun777 wants to merge 86 commits into
apache:masterfrom
yujun777:ivm-yj-1
Closed

yujun777 wants to merge 86 commits into
apache:masterfrom
yujun777:ivm-yj-1

Conversation

@yujun777

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas

Thearas commented Mar 18, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yujun777

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

Copy link
Copy Markdown
TPC-H: Total hot run time: 26843 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 289882c2417314ec86137188792dc2435d5b89f0, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17620	4500	4281	4281
q2	q3	10649	767	518	518
q4	4675	345	255	255
q5	7552	1197	1019	1019
q6	171	174	146	146
q7	785	852	674	674
q8	9310	1499	1375	1375
q9	4726	4674	4627	4627
q10	6245	1892	1627	1627
q11	446	264	235	235
q12	698	582	467	467
q13	18040	2929	2207	2207
q14	230	240	208	208
q15	q16	718	734	673	673
q17	725	822	486	486
q18	5898	5381	5327	5327
q19	1235	990	614	614
q20	542	494	378	378
q21	4716	1844	1391	1391
q22	345	335	436	335
Total cold run time: 95326 ms
Total hot run time: 26843 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4809	4636	4602	4602
q2	q3	3900	4318	3806	3806
q4	899	1237	791	791
q5	4103	4379	4361	4361
q6	183	175	170	170
q7	1804	1693	1527	1527
q8	2491	2694	2580	2580
q9	7612	7393	7511	7393
q10	3853	4016	3609	3609
q11	518	440	422	422
q12	500	613	459	459
q13	2802	3231	2443	2443
q14	293	306	281	281
q15	q16	706	765	719	719
q17	1141	1333	1282	1282
q18	7178	6894	6708	6708
q19	895	881	902	881
q20	2077	2156	2000	2000
q21	3955	3687	3329	3329
q22	467	417	375	375
Total cold run time: 50186 ms
Total hot run time: 47738 ms

@doris-robot

Copy link
Copy Markdown
TPC-DS: Total hot run time: 168011 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 289882c2417314ec86137188792dc2435d5b89f0, data reload: false

query5	4327	663	514	514
query6	334	228	207	207
query7	4226	462	262	262
query8	334	244	223	223
query9	8735	2694	2683	2683
query10	495	398	361	361
query11	6963	5089	4902	4902
query12	194	134	125	125
query13	1274	468	356	356
query14	5808	3721	3495	3495
query14_1	2863	2862	2908	2862
query15	205	192	181	181
query16	986	470	463	463
query17	898	757	627	627
query18	2452	450	353	353
query19	233	212	190	190
query20	140	130	133	130
query21	216	137	113	113
query22	13178	13406	13154	13154
query23	15901	15482	15804	15482
query23_1	15876	15712	15669	15669
query24	7441	1661	1266	1266
query24_1	1270	1329	1270	1270
query25	613	486	451	451
query26	1388	278	167	167
query27	3326	514	313	313
query28	4467	1822	1806	1806
query29	840	574	479	479
query30	296	221	187	187
query31	1027	952	894	894
query32	93	74	73	73
query33	512	336	280	280
query34	907	901	527	527
query35	638	663	602	602
query36	1087	1132	956	956
query37	131	95	81	81
query38	2962	2920	2886	2886
query39	842	862	822	822
query39_1	800	792	784	784
query40	234	149	140	140
query41	67	58	58	58
query42	261	258	254	254
query43	245	260	216	216
query44	
query45	196	194	186	186
query46	872	980	619	619
query47	2109	2108	2043	2043
query48	306	338	225	225
query49	636	454	387	387
query50	680	274	207	207
query51	4093	4075	4021	4021
query52	267	267	262	262
query53	292	343	281	281
query54	320	278	266	266
query55	92	93	82	82
query56	326	322	329	322
query57	1920	1884	1623	1623
query58	293	279	282	279
query59	2800	2970	2740	2740
query60	339	338	330	330
query61	165	159	159	159
query62	638	592	534	534
query63	310	281	276	276
query64	5079	1301	1042	1042
query65	
query66	1474	468	364	364
query67	24292	24335	24177	24177
query68	
query69	403	316	287	287
query70	918	946	997	946
query71	343	308	304	304
query72	2829	2751	2691	2691
query73	551	550	315	315
query74	9632	9555	9453	9453
query75	2904	2798	2533	2533
query76	2298	1036	684	684
query77	378	409	326	326
query78	11113	11177	10476	10476
query79	1111	751	561	561
query80	853	647	546	546
query81	531	263	220	220
query82	1353	157	131	131
query83	372	265	245	245
query84	301	118	109	109
query85	988	500	482	482
query86	404	305	293	293
query87	3140	3128	3043	3043
query88	3533	2653	2655	2653
query89	421	369	342	342
query90	1859	180	171	171
query91	175	165	136	136
query92	81	75	74	74
query93	929	838	498	498
query94	549	323	306	306
query95	603	395	322	322
query96	638	517	229	229
query97	2440	2460	2387	2387
query98	235	223	220	220
query99	1011	985	914	914
Total cold run time: 250050 ms
Total hot run time: 168011 ms

@yujun777
yujun777 force-pushed the ivm-yj-1 branch 4 times, most recently from ce07174 to 5b0f7f2 Compare March 19, 2026 09:52
@yujun777

Copy link
Copy Markdown
Contributor Author

run buildall

@yujun777

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

Copy link
Copy Markdown
TPC-H: Total hot run time: 26660 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7f78f77cc9745f904f6fd1baade01a526b80af82, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17619	4420	4266	4266
q2	q3	10643	748	523	523
q4	4670	359	253	253
q5	7575	1181	1031	1031
q6	188	176	145	145
q7	779	831	674	674
q8	9298	1461	1301	1301
q9	4937	4691	4712	4691
q10	6292	1900	1635	1635
q11	478	269	242	242
q12	748	579	468	468
q13	18055	2905	2159	2159
q14	227	230	211	211
q15	q16	732	747	665	665
q17	754	817	470	470
q18	5874	5381	5216	5216
q19	1141	956	608	608
q20	547	491	376	376
q21	4494	1841	1410	1410
q22	343	316	450	316
Total cold run time: 95394 ms
Total hot run time: 26660 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4789	4701	4711	4701
q2	q3	3870	4348	3928	3928
q4	879	1199	766	766
q5	4069	4371	4368	4368
q6	192	171	143	143
q7	1792	1727	1520	1520
q8	2496	2745	2652	2652
q9	7564	7541	7423	7423
q10	3774	3923	3606	3606
q11	520	432	421	421
q12	489	602	447	447
q13	2657	3304	2608	2608
q14	294	306	286	286
q15	q16	723	791	717	717
q17	1168	1382	1371	1371
q18	7225	7016	6722	6722
q19	932	855	913	855
q20	2090	2124	1973	1973
q21	3987	3466	3346	3346
q22	447	438	385	385
Total cold run time: 49957 ms
Total hot run time: 48238 ms

@doris-robot

Copy link
Copy Markdown
TPC-DS: Total hot run time: 168858 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7f78f77cc9745f904f6fd1baade01a526b80af82, data reload: false

query5	4349	657	488	488
query6	322	234	209	209
query7	4218	467	258	258
query8	344	247	245	245
query9	8742	2693	2675	2675
query10	570	393	328	328
query11	6997	5097	4895	4895
query12	186	135	126	126
query13	1275	461	356	356
query14	5763	3725	3467	3467
query14_1	2894	2920	2831	2831
query15	205	195	181	181
query16	988	478	442	442
query17	880	744	650	650
query18	2452	454	357	357
query19	222	215	189	189
query20	133	129	124	124
query21	213	139	121	121
query22	13379	14202	14652	14202
query23	16314	15894	15730	15730
query23_1	15769	15694	16214	15694
query24	7253	1608	1252	1252
query24_1	1205	1243	1237	1237
query25	565	528	408	408
query26	1227	258	146	146
query27	2791	486	293	293
query28	4466	1828	1852	1828
query29	873	557	478	478
query30	305	234	193	193
query31	1043	938	875	875
query32	79	72	72	72
query33	513	341	281	281
query34	911	866	524	524
query35	647	672	595	595
query36	1091	1134	999	999
query37	129	90	83	83
query38	2982	2898	2842	2842
query39	872	833	812	812
query39_1	779	812	803	803
query40	232	155	132	132
query41	63	60	58	58
query42	262	259	260	259
query43	245	255	265	255
query44	
query45	199	193	180	180
query46	878	988	608	608
query47	2110	2135	2042	2042
query48	309	312	228	228
query49	644	463	384	384
query50	705	272	209	209
query51	4106	3988	4027	3988
query52	261	267	253	253
query53	286	340	288	288
query54	322	277	285	277
query55	94	88	82	82
query56	323	344	316	316
query57	1932	1768	1566	1566
query58	291	272	279	272
query59	2798	2968	2754	2754
query60	342	356	332	332
query61	159	155	166	155
query62	631	591	543	543
query63	310	281	273	273
query64	5157	1297	1012	1012
query65	
query66	1466	451	364	364
query67	24297	24366	24283	24283
query68	
query69	398	319	286	286
query70	924	956	960	956
query71	358	329	314	314
query72	2861	2674	2463	2463
query73	553	564	317	317
query74	9641	9566	9391	9391
query75	2871	2833	2484	2484
query76	2308	1054	675	675
query77	364	385	307	307
query78	10907	11180	10474	10474
query79	2789	776	593	593
query80	1787	605	565	565
query81	550	259	231	231
query82	998	151	121	121
query83	370	268	246	246
query84	300	129	94	94
query85	915	505	470	470
query86	416	307	313	307
query87	3153	3151	3003	3003
query88	3564	2640	2639	2639
query89	434	371	347	347
query90	2075	177	173	173
query91	167	162	136	136
query92	79	81	69	69
query93	1126	848	500	500
query94	637	319	295	295
query95	583	344	384	344
query96	634	514	221	221
query97	2467	2464	2403	2403
query98	238	224	220	220
query99	1018	1009	908	908
Total cold run time: 252037 ms
Total hot run time: 168858 ms

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 71.92% (315/438) 🎉
Increment coverage report
Complete coverage report

@yujun777

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

Copy link
Copy Markdown
TPC-H: Total hot run time: 26893 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 575af34520e087a570753620630e8f18f20dab58, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17614	4445	4302	4302
q2	q3	10640	786	514	514
q4	4679	346	249	249
q5	7567	1231	1015	1015
q6	177	173	150	150
q7	783	839	681	681
q8	9296	1481	1339	1339
q9	4822	4706	4660	4660
q10	6256	1888	1652	1652
q11	507	273	245	245
q12	701	575	464	464
q13	18015	2916	2215	2215
q14	240	235	210	210
q15	q16	746	744	660	660
q17	740	819	480	480
q18	6073	5335	5271	5271
q19	1109	993	606	606
q20	537	490	384	384
q21	4427	1800	1408	1408
q22	362	388	404	388
Total cold run time: 95291 ms
Total hot run time: 26893 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4790	4575	4614	4575
q2	q3	3881	4358	3786	3786
q4	906	1197	792	792
q5	4078	4394	4338	4338
q6	192	203	145	145
q7	1742	1650	1535	1535
q8	2487	2746	2600	2600
q9	7656	7337	7657	7337
q10	3802	4089	3651	3651
q11	511	440	424	424
q12	510	603	438	438
q13	2662	3201	2361	2361
q14	301	303	273	273
q15	q16	707	763	753	753
q17	1192	1383	1341	1341
q18	7175	6737	6612	6612
q19	912	871	951	871
q20	2169	2232	1990	1990
q21	4019	3650	3295	3295
q22	443	418	380	380
Total cold run time: 50135 ms
Total hot run time: 47497 ms

@doris-robot

Copy link
Copy Markdown
TPC-DS: Total hot run time: 167693 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 575af34520e087a570753620630e8f18f20dab58, data reload: false

query5	4329	642	525	525
query6	344	234	216	216
query7	4225	469	270	270
query8	351	250	229	229
query9	8735	2692	2736	2692
query10	548	407	350	350
query11	6858	5052	4900	4900
query12	182	131	126	126
query13	1279	476	354	354
query14	5660	3717	3484	3484
query14_1	2844	2802	2839	2802
query15	206	195	174	174
query16	979	474	454	454
query17	1055	733	636	636
query18	2450	463	350	350
query19	223	213	188	188
query20	139	126	128	126
query21	221	131	110	110
query22	13312	13887	14554	13887
query23	16311	15944	15642	15642
query23_1	15711	15442	15197	15197
query24	7130	1600	1197	1197
query24_1	1213	1231	1207	1207
query25	544	485	402	402
query26	1234	268	141	141
query27	2773	476	298	298
query28	4419	1841	1845	1841
query29	856	629	474	474
query30	300	214	192	192
query31	1004	948	876	876
query32	85	73	69	69
query33	514	348	279	279
query34	886	893	521	521
query35	629	678	600	600
query36	1082	1114	980	980
query37	137	97	82	82
query38	2905	2904	2862	2862
query39	878	833	819	819
query39_1	788	794	791	791
query40	243	147	134	134
query41	61	62	60	60
query42	266	272	252	252
query43	239	249	230	230
query44	
query45	198	196	183	183
query46	883	985	596	596
query47	2090	2114	2028	2028
query48	323	319	224	224
query49	631	471	389	389
query50	677	285	217	217
query51	4134	3997	3974	3974
query52	262	265	255	255
query53	286	333	281	281
query54	295	272	287	272
query55	102	86	82	82
query56	316	328	307	307
query57	1945	1917	1551	1551
query58	291	279	291	279
query59	2810	2954	2724	2724
query60	350	339	323	323
query61	196	152	157	152
query62	631	572	531	531
query63	316	274	278	274
query64	5104	1267	997	997
query65	
query66	1443	454	359	359
query67	24175	24324	24151	24151
query68	
query69	450	319	311	311
query70	977	985	949	949
query71	361	326	307	307
query72	2832	2726	2447	2447
query73	560	543	330	330
query74	9606	9577	9387	9387
query75	2880	2770	2520	2520
query76	2284	1042	685	685
query77	383	402	327	327
query78	10899	11113	10451	10451
query79	1079	821	597	597
query80	764	678	583	583
query81	484	261	229	229
query82	1379	153	121	121
query83	385	269	261	261
query84	269	118	100	100
query85	935	492	456	456
query86	362	315	325	315
query87	3138	3118	2987	2987
query88	3594	2649	2624	2624
query89	417	372	351	351
query90	1989	189	187	187
query91	174	168	138	138
query92	77	73	72	72
query93	924	872	495	495
query94	465	336	292	292
query95	588	353	330	330
query96	648	519	226	226
query97	2478	2476	2405	2405
query98	261	224	215	215
query99	1016	990	929	929
Total cold run time: 248556 ms
Total hot run time: 167693 ms

@yujun777

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

Copy link
Copy Markdown
TPC-H: Total hot run time: 26963 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b27eefe8d8615caa0af4db45603f2ba23a290bb0, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17621	4472	4299	4299
q2	q3	10641	778	522	522
q4	4684	365	248	248
q5	7578	1223	1037	1037
q6	183	173	146	146
q7	780	866	678	678
q8	9853	1456	1366	1366
q9	5385	4768	4709	4709
q10	6327	1941	1676	1676
q11	445	269	254	254
q12	758	588	481	481
q13	18059	2733	1950	1950
q14	232	231	205	205
q15	q16	745	731	673	673
q17	739	838	448	448
q18	6025	5461	5317	5317
q19	1582	990	634	634
q20	583	492	377	377
q21	4500	1893	1656	1656
q22	472	363	287	287
Total cold run time: 97192 ms
Total hot run time: 26963 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4779	4636	4561	4561
q2	q3	3937	4354	3838	3838
q4	897	1181	794	794
q5	4066	4384	4331	4331
q6	176	176	144	144
q7	1741	1672	1532	1532
q8	2490	2758	2548	2548
q9	7735	7360	7335	7335
q10	3753	3984	3582	3582
q11	500	431	422	422
q12	497	617	440	440
q13	2398	2868	2061	2061
q14	304	310	284	284
q15	q16	744	836	731	731
q17	1149	1341	1410	1341
q18	7124	6796	6668	6668
q19	935	917	920	917
q20	2038	2207	2070	2070
q21	4185	3415	3335	3335
q22	460	455	376	376
Total cold run time: 49908 ms
Total hot run time: 47310 ms

@doris-robot

Copy link
Copy Markdown
TPC-DS: Total hot run time: 169448 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b27eefe8d8615caa0af4db45603f2ba23a290bb0, data reload: false

query5	4330	631	524	524
query6	342	240	219	219
query7	4221	465	270	270
query8	353	257	249	249
query9	8719	2731	2764	2731
query10	517	392	346	346
query11	6968	5165	4895	4895
query12	185	134	131	131
query13	1279	491	350	350
query14	5719	3740	3503	3503
query14_1	2884	2874	2869	2869
query15	206	188	173	173
query16	991	464	452	452
query17	1091	725	608	608
query18	2446	437	351	351
query19	222	200	176	176
query20	139	130	125	125
query21	208	133	114	114
query22	13211	14165	14681	14165
query23	16971	16103	16039	16039
query23_1	15932	15619	15648	15619
query24	7308	1609	1219	1219
query24_1	1220	1235	1233	1233
query25	556	468	405	405
query26	1251	256	153	153
query27	2771	481	297	297
query28	4516	1857	1848	1848
query29	827	566	478	478
query30	301	229	196	196
query31	999	936	874	874
query32	88	73	71	71
query33	519	347	293	293
query34	901	870	516	516
query35	694	682	604	604
query36	1051	1147	995	995
query37	140	94	79	79
query38	2953	2925	2932	2925
query39	850	843	810	810
query39_1	790	813	793	793
query40	233	158	146	146
query41	74	73	70	70
query42	254	260	258	258
query43	244	283	219	219
query44	
query45	205	189	187	187
query46	885	994	605	605
query47	2080	2124	2032	2032
query48	309	323	230	230
query49	630	461	373	373
query50	687	301	215	215
query51	4114	4048	3947	3947
query52	256	268	258	258
query53	289	344	289	289
query54	299	271	268	268
query55	100	87	80	80
query56	307	325	318	318
query57	1956	1896	1764	1764
query58	289	292	272	272
query59	2807	2946	2762	2762
query60	350	335	327	327
query61	159	157	158	157
query62	631	605	538	538
query63	316	288	277	277
query64	5032	1302	1098	1098
query65	
query66	1467	528	347	347
query67	24374	24260	24130	24130
query68	
query69	419	314	277	277
query70	910	920	889	889
query71	340	312	296	296
query72	2908	2628	2438	2438
query73	540	558	313	313
query74	9628	9576	9439	9439
query75	2852	2765	2478	2478
query76	2302	1040	672	672
query77	364	387	327	327
query78	10926	11028	10484	10484
query79	1137	776	564	564
query80	746	665	599	599
query81	478	263	223	223
query82	1354	156	122	122
query83	377	273	253	253
query84	262	131	109	109
query85	942	574	538	538
query86	363	303	300	300
query87	3152	3108	3053	3053
query88	3597	2716	2705	2705
query89	431	378	355	355
query90	1974	188	180	180
query91	186	194	155	155
query92	80	77	77	77
query93	910	845	502	502
query94	464	357	293	293
query95	580	346	319	319
query96	654	512	233	233
query97	2453	2492	2401	2401
query98	230	219	220	219
query99	1021	988	903	903
Total cold run time: 249898 ms
Total hot run time: 169448 ms

@yujun777

Copy link
Copy Markdown
Contributor Author

run buildall

@yujun777

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

Copy link
Copy Markdown
TPC-H: Total hot run time: 26586 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4bab8ed37274e87d0e067991588990b045d5d4a7, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17583	4543	4346	4346
q2	q3	10647	763	521	521
q4	4675	346	248	248
q5	7565	1194	1012	1012
q6	174	174	147	147
q7	758	832	673	673
q8	9300	1475	1306	1306
q9	4771	4696	4728	4696
q10	6261	1903	1638	1638
q11	467	243	263	243
q12	702	595	458	458
q13	18025	2703	1932	1932
q14	233	227	208	208
q15	q16	722	741	685	685
q17	731	834	449	449
q18	5905	5389	5371	5371
q19	1230	972	605	605
q20	524	481	377	377
q21	4737	1828	1373	1373
q22	339	298	370	298
Total cold run time: 95349 ms
Total hot run time: 26586 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4862	4639	4583	4583
q2	q3	3902	4344	3816	3816
q4	899	1199	789	789
q5	4045	4333	4377	4333
q6	186	174	143	143
q7	1801	1670	1558	1558
q8	2466	2703	2546	2546
q9	7565	7387	7384	7384
q10	3775	3963	3601	3601
q11	504	430	407	407
q12	473	598	447	447
q13	2465	2882	2312	2312
q14	297	303	275	275
q15	q16	720	753	709	709
q17	1174	1280	1356	1280
q18	7041	6855	6845	6845
q19	912	870	953	870
q20	2077	2206	2006	2006
q21	3934	3480	3446	3446
q22	467	440	383	383
Total cold run time: 49565 ms
Total hot run time: 47733 ms

@doris-robot

Copy link
Copy Markdown
TPC-DS: Total hot run time: 168564 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4bab8ed37274e87d0e067991588990b045d5d4a7, data reload: false

query5	4343	630	509	509
query6	351	234	202	202
query7	4224	473	263	263
query8	345	254	248	248
query9	8739	2697	2739	2697
query10	521	395	331	331
query11	6953	5091	4925	4925
query12	194	126	134	126
query13	1283	468	346	346
query14	5776	3769	3493	3493
query14_1	2864	2828	2827	2827
query15	204	198	176	176
query16	1016	506	453	453
query17	1133	761	631	631
query18	2684	465	362	362
query19	215	214	192	192
query20	136	123	127	123
query21	217	138	113	113
query22	13232	14103	14776	14103
query23	16584	16151	15798	15798
query23_1	16093	15643	15713	15643
query24	7367	1623	1211	1211
query24_1	1284	1201	1228	1201
query25	554	455	446	446
query26	1240	270	148	148
query27	2756	476	294	294
query28	4427	1853	1840	1840
query29	852	563	477	477
query30	298	219	189	189
query31	1023	994	863	863
query32	86	71	69	69
query33	518	332	280	280
query34	877	864	523	523
query35	650	682	601	601
query36	1075	1132	978	978
query37	136	99	84	84
query38	2941	2891	2881	2881
query39	862	834	802	802
query39_1	802	793	779	779
query40	228	151	137	137
query41	63	59	57	57
query42	268	256	257	256
query43	243	252	215	215
query44	
query45	195	190	182	182
query46	872	991	620	620
query47	2046	2600	2045	2045
query48	306	317	227	227
query49	637	465	377	377
query50	688	290	214	214
query51	4077	4059	3967	3967
query52	263	266	252	252
query53	287	340	294	294
query54	306	272	267	267
query55	97	93	83	83
query56	307	318	309	309
query57	1946	1883	1720	1720
query58	289	307	275	275
query59	2786	2932	2773	2773
query60	336	325	314	314
query61	161	160	154	154
query62	614	587	535	535
query63	306	282	274	274
query64	5062	1288	1009	1009
query65	
query66	1420	465	347	347
query67	24244	24192	24130	24130
query68	
query69	410	304	284	284
query70	979	962	953	953
query71	335	303	296	296
query72	2773	2758	2480	2480
query73	544	542	320	320
query74	9604	9577	9422	9422
query75	2844	2771	2450	2450
query76	2284	1031	674	674
query77	364	386	312	312
query78	10993	11178	10459	10459
query79	1072	826	568	568
query80	1351	631	543	543
query81	539	270	228	228
query82	1339	159	117	117
query83	342	267	269	267
query84	250	120	100	100
query85	945	538	460	460
query86	420	309	331	309
query87	3111	3108	2981	2981
query88	3486	2648	2633	2633
query89	425	363	335	335
query90	1917	176	167	167
query91	171	168	141	141
query92	80	73	69	69
query93	903	826	507	507
query94	545	328	310	310
query95	582	393	320	320
query96	649	521	232	232
query97	2467	2475	2409	2409
query98	239	228	220	220
query99	1023	997	880	880
Total cold run time: 250417 ms
Total hot run time: 168564 ms

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 74.18% (408/550) 🎉
Increment coverage report
Complete coverage report

@yujun777

Copy link
Copy Markdown
Contributor Author

run buildall

yujun777 and others added 13 commits April 16, 2026 08:02
Problem: murmur_hash3_64 implements PropagateNullable, so any NULL group key
argument makes the entire hash return NULL. This causes different groups with
NULL keys to collide (e.g., (NULL,'a') and (NULL,'b') both produce row_id=NULL),
leading to incorrect incremental refresh results.

Key changes:
- Replace direct hash(keys...) with null-safe pattern:
  hash(ifnull(cast(k AS VARCHAR),''), cast(isnull(k) AS VARCHAR), ...) per key
- Add MurmurHash364(List<Expression>) constructor for cleaner list-based construction
- Remove unused IvmUtil.newIvmCountColumnDefinition()
- Update IVM AGENTS.md with null-safe row_id documentation

Unit Test:
- IvmUtilTest: 7 new tests verifying expression tree structure and non-nullability
- All 99 existing IVM FE unit tests pass

Regression Test:
- test_ivm_agg_4: Parts 16-18 covering single/multiple NULL group keys
  and empty-string vs NULL distinction
…ions

Previously IVM rejected compound expressions like SUM(v1+v2) or MIN(v1*2)
inside aggregate functions, requiring bare column Slots only. This relaxes
the constraint to accept arbitrary expressions as aggregate arguments.

Key changes:
- IvmAggMeta.AggTarget: exprSlots (List<Slot>) -> exprArgs (List<Expression>)
- IvmNormalizeMtmv: removed instanceof Slot check in buildHiddenStateForAgg
- IvmAggDeltaStrategy: widened helper method params from Slot to Expression
- Renamed all misleading XXXSlot variables/methods to XXXArg where type is Expression

Unit Test: IvmNormalizeMtmvTest (25), IvmAggDeltaStrategyTest (25), all 103 IVM FE tests pass
Regression Test: all 7 IVM regression suites pass
…code

Replace bare "SUM"/"COUNT"/"MIN"/"MAX" strings used as hidden state slot
keys with a type-safe StateKey enum in IvmAggMeta. This prevents typos and
provides compile-time safety. Also extract addHiddenSumAndCount() and
addHiddenAlias() helper methods in IvmNormalizeMtmv to eliminate SUM/AVG
code duplication.

Key changes:
- Add IvmAggMeta.StateKey enum with SUM, COUNT, MIN, MAX values
- Change AggTarget.hiddenStateSlots from Map<String,Slot> to Map<StateKey,Slot>
- Update all callsites in IvmAggDeltaStrategy and IvmNormalizeMtmv
- Add DELMIN/DELMAX as private static final String constants (transient keys)
- Extract addHiddenSumAndCount() and addHiddenAlias() in IvmNormalizeMtmv
- Update IvmNormalizeMtmvTest to use StateKey

Unit Test: 103 IVM FE unit tests pass
Regression Test: all 7 IVM suites pass
When an IVM INCREMENTAL refresh fails because a deleted row equals
the current MIN or MAX aggregate value, the assert_true guard fires
at runtime. Previously this was caught as a generic
INCREMENTAL_EXECUTION_FAILED reason, making it hard to distinguish
from real execution errors in logs and task error messages.

Key changes:
- IvmRefreshManager.doRefreshInternal() inspects the caught exception
  message for the boundary guard marker ("IVM: deleted row may be
  current") and sets IvmFallbackReason.MIN_MAX_BOUNDARY_HIT, which
  was defined but never used before
- Boundary hits are logged at INFO level (expected path) while other
  execution failures remain at WARN level
- Update regression test error-message assertions (Parts 4 and 9) to
  also check for MIN_MAX_BOUNDARY in the reason string

Unit Test: 103 FE unit tests pass; 7/7 IVM regression suites pass
### What problem does this PR solve?

Issue Number: close #xxx

Problem Summary: Previously, IVM (Incremental View Maintenance) rejected
materialized views with GROUP BY but no aggregate functions, throwing
"GROUP BY without aggregate functions is not supported for IVM".

This is unnecessarily restrictive because the unconditionally-injected
hidden column `__DORIS_IVM_AGG_COUNT_COL__` alone is sufficient to track
group membership for incremental maintenance. A bare GROUP BY is
semantically equivalent to SELECT DISTINCT, and the delta/apply paths
already handle empty aggTargets lists correctly.

### Release note

Support bare GROUP BY (SELECT DISTINCT) queries in IVM materialized views.

### Check List (For Author)

- Test: Unit Test (IvmNormalizeMtmvTest) + Regression test (test_ivm_agg_5)
- Behavior changed: Yes — previously rejected bare GROUP BY with AnalysisException, now accepted
- Does this need documentation: No

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ssion tests

### What problem does this PR solve?

Issue Number: N/A

Problem Summary: IVM agg regression suites (test_ivm_agg_1~3, 5) only
verified MV output after COMPLETE refresh, leaving INCREMENTAL refresh
code paths untested. Also, test_ivm_agg_5 had a logic flaw in Part 1
where consecutive INCREMENTAL refreshes inflated group counts, preventing
group deletion from triggering correctly.

### Release note

None

### Check List (For Author)

- Test: Regression test — all 8 suites under mtmv_p0/ivm pass
- Behavior changed: No
- Does this need documentation: No

Key changes:
- test_ivm_agg_1.groovy: Add 6 order_qt_* assertions after INCREMENTAL refreshes
- test_ivm_agg_2.groovy: Add 8 order_qt_* assertions after INCREMENTAL refreshes
- test_ivm_agg_3.groovy: Add 4 order_qt_* assertions after INCREMENTAL refreshes
- test_ivm_agg_5.groovy: Redesign Part 1 into 3 isolated Scenarios (A/B/C), each
  starting from a fresh COMPLETE to keep counts accurate; redesign Part 2 to combine
  delete+insert in one batch before a single INCREMENTAL; fix Scenario B comment errors
- Regenerate test_ivm_agg_1~3.out and generate new test_ivm_agg_5.out
…S.md

### What problem does this PR solve?

Issue Number: N/A

Problem Summary: The binlog_op mocking guide for IVM regression tests was
only in an untracked AGENTS.md under the regression-test directory. Merge
it into the committed FE IVM AGENTS.md so it is preserved and visible to
all contributors, with an additional note about the COMPLETE-before-delete
requirement for correct group deletion testing.

### Release note

None

### Check List (For Author)

- Test: No need to test (documentation only)
- Behavior changed: No
- Does this need documentation: No
Rewrite the IVM FE unit tests that still depended on JMockit so FE test compilation works with the current test dependencies.

Key changes:
- replace JMockit usage in IvmDeltaExecutorTest with Mockito static mocks
- rewrite RefreshMTMVInfoAnalyzeTest to use Mockito for Env and catalog setup
- migrate the remaining IVM FE tests away from JMockit imports and expectations

Unit Test:
- mvn test -pl fe-core -Dtest="RefreshMTMVInfoAnalyzeTest" -Dmaven.build.cache.enabled=false
- mvn test -pl fe-core -Dtest="IvmDeltaExecutorTest,IvmDeltaRewriterTest,IvmSimpleScanDeltaStrategyTest,IvmRefreshManagerTest" -Dmaven.build.cache.enabled=false
…anagerTest

### What problem does this PR solve?

Problem Summary: IvmRefreshManagerTest still used JMockit annotations
(@mocked, Expectations) which are not available as a dependency,
causing FE compilation failure.

### Release note

None

### Check List (For Author)

- Test: No need to test (build fix only, replacing mock framework usage)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
### What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:
IVM (Incremental View Maintenance) was creating redundant hidden columns
in the MV schema. For several aggregate types, the visible column already
stores the same value as the hidden column, wasting storage and adding
unnecessary complexity:

- COUNT(*): hidden COUNT duplicated the global group count
- COUNT(expr): hidden COUNT duplicated the visible COUNT(expr)
- SUM: hidden SUM duplicated the visible SUM value
- MIN/MAX: hidden MIN/MAX duplicated the visible extremal value

This commit removes these redundant hidden columns. The delta apply logic
now reads old state from the visible column instead. Only genuinely needed
hidden columns are retained:
- SUM/MIN/MAX: hidden COUNT (for assertNonNegative guard and null logic)
- AVG: hidden SUM + COUNT (visible is AVG, not SUM or COUNT)

Additional cleanup:
- Inline addHiddenSumAndCount (only called once for AVG)
- Remove hasIvmHiddenOutputInOutputs/isIvmHiddenOutput private methods
- Simplify group key resolution to direct Slot casting
- Add stateColumnName(StateKey) helper to AggTarget
- Fix toColumn() bug in IvmDeltaTestBase (isVisible/isKey params swapped)
- Update class Javadoc with accurate plan shape

### Release note

None

### Check List (For Author)

- Test: Unit Test (90 IVM FE UTs + 24 CreateMTMVCommandTest all pass) and Regression test (8 IVM regression tests all pass)
- Behavior changed: No (internal schema optimization, no user-visible change)
- Does this need documentation: No

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ey with AggType

### What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:
The IVM (Incremental View Maintenance) code had two redundant abstractions:
1. AggType enum had separate COUNT_STAR and COUNT_EXPR values, but the only
   difference is whether exprArgs is empty. Merging them into a single COUNT
   type with an isCountStar() helper simplifies all switch statements.
2. StateKey enum {SUM, COUNT, MIN, MAX} was a strict subset of AggType
   {COUNT, SUM, AVG, MIN, MAX} (AVG is never used as a hidden-state key).
   Eliminating StateKey removes an unnecessary indirection layer.
3. caseWhenExprNotNull was renamed to ifExprNotNull since it generates an
   IF expression, not a CASE WHEN.

### Release note

None

### Check List (For Author)

- Test: Unit Test (90 IVM FE UTs + 24 CreateMTMVCommandTest) and Regression test (8 IVM suites)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fix compilation and checkstyle errors caused by upstream's TableNameInfo
class relocation (org.apache.doris.info → org.apache.doris.catalog.info)
and duplicate TStorageType import from conflict resolution.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@abcxyz333

Copy link
Copy Markdown

run buildall

…ustness

### What problem does this PR solve?

Problem Summary:
1. Remove empty if-block in buildHiddenStateForAgg COUNT branch — neither
   COUNT(*) nor COUNT(expr) adds hidden columns, so the if was dead code.
2. Use Count.isCountStar() instead of Count.isStar() when determining
   exprArgs. isCountStar() also covers COUNT() and COUNT(literal) forms,
   making the code robust against optimizer rewrites like COUNT(*)->COUNT(1).

### Release note

None

### Check List (For Author)

- Test: Unit Test (IvmNormalizeMtmvTest, IvmAggDeltaStrategyTest, CreateMTMVCommandTest all pass)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 78.56% (960/1222) 🎉
Increment coverage report
Complete coverage report

yujun777 and others added 3 commits April 16, 2026 16:39
…to zero

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
When all non-null rows contributing to a MIN/MAX aggregate are deleted
(hidden non-null count drops to 0), the boundary guard assertion
(assert_true) would incorrectly fire because the deleted extremal value
equals the current extreme. This caused unnecessary COMPLETE fallback.

This change:
- Adds newCount==0 as the first disjunct in the guard OR condition,
  bypassing the boundary check when count is zero (no boundary to protect)
- Replaces nested IF merge logic with CASE WHEN for clarity:
  CASE WHEN newCount=0 THEN NULL
       WHEN old IS NULL THEN deltaInsert
       WHEN deltaInsert IS NULL THEN old
       ELSE LEAST/GREATEST END
- Uses flat Or(ImmutableList.of(...)) instead of nested binary Or
- Updates method Javadoc to document the four-way guard condition
- Adds regression tests (test_ivm_agg_6) for two scenarios:
  A) Delete all rows: cnt=0, min/max=NULL
  B) Delete last non-null row with NULL rows remaining: min/max=NULL

### Release note

None

### Check List (For Author)

- Test: Regression test (test_ivm_agg_6), FE unit test pass
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* [improvement](fe) Add IVM create and alter validation for MTMV (#5)

Issue Number: close #xxx

Problem Summary:
1. Remove redundant pre-analysis in IVM analyzeQuery - single analyzeQueryInternal call
2. Unify IVM base table validation error messages (AGG_KEYS vs UNIQUE without MOW)
3. Add excluded_trigger_tables support in IvmNormalizeMtmv (transient row-id for excluded tables)
4. Block ALTER MTMV refresh method to/from INCREMENTAL
5. Validate base table models when ALTER MTMV excluded_trigger_tables
6. Extract MTMVPropertyUtil.parseTableNameInfos utility
7. Add comprehensive unit tests for all validation paths

* [fix](fe) Fix IVM ExprId collision by reusing parser StatementContext

### What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:
Commit 71a3086 introduced a new StatementContext in analyzeQueryInternal()
and restored the original (parser-created) StatementContext in the finally
block. This caused ExprId collisions during IVM INCREMENTAL refresh because
the parser StatementContext has a much smaller ExprId counter than the
analysis StatementContext, and IvmRefreshManager.doRefreshInternal() reads
exprIdStart from the ConnectContext's StatementContext after analysis.

The fix reverts to the pre-71a3086f pattern: reuse ctx.getStatementContext()
(the parser-created StatementContext) directly for analysis instead of
creating a new one. This way all ExprId allocations accumulate in the same
StatementContext that doRefreshInternal() later reads.

### Release note

None

### Check List (For Author)

- Test: Regression test (test_ivm_agg_2)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* [fix](fe) Fix buildRowId to compute proper row-id for excluded trigger tables

### What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:
buildRowId() short-circuited to UuidNumeric() for excluded trigger tables,
losing the deterministic row-id (hash of unique keys) for MOW and non-MOW
UNIQUE_KEYS tables. The fix removes the early return and only uses the
isExcludedTriggerTable flag to suppress AnalysisException for unsupported
table types (AGG_KEYS etc.), while UNIQUE_KEYS tables always compute
buildRowIdHash(keySlots) regardless of exclusion status.

### Release note

None

### Check List (For Author)

- Test: Unit Test (IvmNormalizeMtmvTest)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* [test](fe) Update IvmNormalizeMtmvTest for excluded MOW table row-id fix

### What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:
Update testExcludedMowTableUsesTransientRowId to expect deterministic
hash-based row-id (Cast expression) instead of UuidNumeric for excluded
MOW tables, matching the fix in buildRowId().

### Release note

None

### Check List (For Author)

- Test: Unit Test (IvmNormalizeMtmvTest - 27 tests pass)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* [fix](fe) Refine ALTER MTMV refresh method compatibility rules

### What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:
The ALTER MTMV refresh method validation was too restrictive (blocking all
changes to/from INCREMENTAL) or too permissive in some cases. The new rules:
- COMPLETE <-> INCREMENTAL: forbidden (must recreate MV)
- COMPLETE/INCREMENTAL -> AUTO: allowed
- AUTO -> COMPLETE/INCREMENTAL: forbidden (must recreate MV)
- Same method (no-op): allowed

### Release note

None

### Check List (For Author)

- Test: Regression test
- Behavior changed: Yes (COMPLETE/INCREMENTAL to AUTO now allowed)
- Does this need documentation: No

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* opt code

* [test](fe) Isolate CreateMTMVCommandTest statement context

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: CreateMTMVCommandTest reused the same StatementContext across multiple statements, so table and excluded_trigger_tables state leaked between test cases and caused false incremental MV validation failures. This change resets the statement context for each statement in the test and adds the missing LinkedHashSet import required by StatementContext.

### Release note

None

### Check List (For Author)

- Test: FE unit test

    - ./run-fe-ut.sh --run org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest,org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest,org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest,org.apache.doris.mtmv.ivm.IvmRefreshManagerTest,org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest,org.apache.doris.mtmv.ivm.IvmUtilTest,org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest,org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest,org.apache.doris.catalog.ShowCreateMTMVTest

- Behavior changed: No

- Does this need documentation: No

* [fix](fe) Fix TableNameInfo imports after rebasing IVM branch

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Rebase onto the latest yujun777/ivm left several FE IVM files importing
org.apache.doris.info.TableNameInfo while the current base branch provides the type in
org.apache.doris.catalog.info.TableNameInfo. This fixes the stale imports so the rebased
branch compiles and the targeted IVM FE tests run successfully again.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest
    - org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest
    - org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest
    - org.apache.doris.mtmv.ivm.IvmRefreshManagerTest
    - org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest
    - org.apache.doris.mtmv.ivm.IvmUtilTest
    - org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest
    - org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest
    - org.apache.doris.catalog.ShowCreateMTMVTest
- Behavior changed: No
- Does this need documentation: No

* [fix](fe) Use deterministic row id for excluded AGG_KEYS tables

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: IVM normalization treated excluded AGG_KEYS base tables as transient-row-id scans
and generated uuid_numeric() row ids. This changed excluded AGG_KEYS row identity across refreshes
and did not follow the base table aggregate keys. Fix buildRowId to hash base-schema key columns for
excluded AGG_KEYS tables, and add focused tests to verify the row id is deterministic and excludes
non-key value columns.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - org.apache.doris.mtmv.ivm.IvmAggDeltaStrategyTest
    - org.apache.doris.mtmv.ivm.IvmDeltaExecutorTest
    - org.apache.doris.mtmv.ivm.IvmDeltaRewriterTest
    - org.apache.doris.mtmv.ivm.IvmRefreshManagerTest
    - org.apache.doris.mtmv.ivm.IvmSimpleScanDeltaStrategyTest
    - org.apache.doris.mtmv.ivm.IvmUtilTest
    - org.apache.doris.nereids.rules.rewrite.IvmNormalizeMtmvTest
    - org.apache.doris.nereids.trees.plans.CreateMTMVCommandTest
    - org.apache.doris.catalog.ShowCreateMTMVTest
- Behavior changed: Yes (excluded AGG_KEYS row-id generation is now deterministic on agg keys)
- Does this need documentation: No

* fix comment

* fix comment

* opt code

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ages

### What problem does this PR solve?

Problem Summary:
1. Remove dead code in IvmNormalizeMtmv.buildRowId() — the isExcludedTriggerTable
   branch at lines 503-505 was unreachable because all KeysType cases (UNIQUE_KEYS,
   DUP_KEYS, AGG_KEYS) are already handled above it.
2. Improve the error message in AlterMTMVRefreshInfo.validateRefreshMethodCompat()
   when attempting to alter the refresh method of an INCREMENTAL materialized view,
   making it clearer that the operation is not allowed.

### Release note

None

### Check List (For Author)

- Test: Regression test / Unit Test
    - FE UT: 94/94 IVM tests passed
    - Regression: 10/10 IVM suites passed (mtmv_p0/ivm)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@abcxyz333

Copy link
Copy Markdown

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 77.48% (1008/1301) 🎉
Increment coverage report
Complete coverage report

### What problem does this PR solve?

Issue Number: close #xxx

Problem Summary: Implements the core multi-bundle delta plan generation for
IVM incremental refresh. This enables producing one delta command bundle per
base table that has pending changes, with correct TSO snapshot binding across
all scans in the normalized plan.

Changes:
- IvmStreamRef: Replaced streamType/consumerId/properties with consumedTso
  (persisted) and latestTso (transient). Added isUpToDate(). Deleted StreamType enum.
- OlapTable: Added getVisibleTso() mock (delegates to getVisibleVersion).
- LogicalOlapScan: Added tso (default -1) and isDelta (default false) fields,
  with withTso()/withIsDelta() methods. Both participate in equals().
- IvmDeltaRewriter: Complete rewrite with generateDeltaPlans() multi-bundle
  logic, rewriteOlapScans() helper, replaceWithDelta() mock. Uses
  rewriteDownShortCircuit + AtomicInteger for deterministic scan traversal.
  TSO binding: j<i → latestTso (v2), j>i → consumedTso (v1).
  Includes latestTso >= consumedTso invariant check.
- IvmSimpleScanDeltaStrategy: isDelta check — non-delta scans skip dml_factor.
- IvmAggDeltaStrategy: ctx made final, set via constructor (single-use).
- IvmRefreshManager: Empty bundles = success (no-op, all tables up to date).
- IvmDeltaRewriteContext: Added baseTableStreams field.

### Release note

None

### Check List (For Author)

- Test: Unit Test (109 IVM tests pass: 21 rewriter, 25 agg strategy, 14 simple strategy, 28 normalize, 7 util, 10 refresh manager, 4 executor)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@abcxyz333

Copy link
Copy Markdown

run buildall

… latestTso reading

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
Implements Steps 6-7 of the multi-bundle IVM plan:

Step 6 - runningIvmRefresh crash recovery flag:
- Add runningIvmRefresh boolean to IvmInfo with @SerializedName("rr")
- Add ALTER_IVM_INFO editlog op type with full AlterMTMV persistence path
- In IvmRefreshManager: set flag=true before bundle execution, clear after
  success with consumedTso advance in one atomic editlog write
- On failure: leave flag set so next task detects and falls back to COMPLETE
- In MTMVTask: detect flag on COMPLETE refresh entry, capture pre-refresh
  TSOs, reset state after successful full refresh
- MTMV.alterIvmInfo() and getIvmInfo() use writeMvLock for thread safety

Step 7 - latestTso reading and baseTableStreams passing:
- populateLatestTso() reads OlapTable.getVisibleTso() for each base table
- ensureBaseTableStreamsInitialized() lazily populates from MTMV relation
  metadata on first incremental refresh (handles empty map from MTMV creation)
- Pass baseTableStreams to IvmDeltaRewriteContext for TSO binding
- Guard advanceConsumedTso: only advance if latestTso >= consumedTso to
  prevent regression when table resolution fails

### Release note

None

### Check List (For Author)

- Test: Unit Test (71 tests pass: IvmRefreshManager 17, IvmDeltaRewriter 21, IvmAggDeltaStrategy 25, AlterMTMV 8) + Regression test (10/10 IVM tests pass)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@abcxyz333

Copy link
Copy Markdown

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 78.25% (1155/1476) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants