Skip to content

branch-3.1: [fix](params-refactor)Enhance Object Storage Parameter Validation and Exception Handling #55014 - #55086

Merged
morrySnow merged 5 commits into
apache:branch-3.1from
CalvinKirs:branch-3.1-55014
Aug 21, 2025
Merged

branch-3.1: [fix](params-refactor)Enhance Object Storage Parameter Validation and Exception Handling #55014#55086
morrySnow merged 5 commits into
apache:branch-3.1from
CalvinKirs:branch-3.1-55014

Conversation

@CalvinKirs

@CalvinKirs CalvinKirs commented Aug 20, 2025

Copy link
Copy Markdown
Member

picked from #55014

@CalvinKirs
CalvinKirs requested a review from morrySnow as a code owner August 20, 2025 15:55
@Thearas

Thearas commented Aug 20, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@CalvinKirs

Copy link
Copy Markdown
Member Author

run buildall

@doris-robot

Copy link
Copy Markdown
TPC-H: Total hot run time: 33029 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d5b42fde3bf66cdc82e7a3918a771fd18b50009f, data reload: false

------ Round 1 ----------------------------------
q1	17644	5566	5768	5566
q2	2033	415	281	281
q3	12418	1264	758	758
q4	10561	875	465	465
q5	9516	2390	2171	2171
q6	189	167	139	139
q7	896	779	622	622
q8	9352	1461	1198	1198
q9	5338	5039	4934	4934
q10	6778	2298	1836	1836
q11	466	288	265	265
q12	336	359	215	215
q13	17798	3640	3037	3037
q14	237	238	203	203
q15	522	472	465	465
q16	425	430	380	380
q17	624	874	373	373
q18	7057	6389	6353	6353
q19	1213	967	584	584
q20	343	348	218	218
q21	2807	2215	1973	1973
q22	1030	1045	993	993
Total cold run time: 107583 ms
Total hot run time: 33029 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5614	5553	5555	5553
q2	239	333	236	236
q3	2221	2705	2354	2354
q4	1368	1815	1405	1405
q5	4436	5149	5033	5033
q6	181	170	133	133
q7	2086	1996	1886	1886
q8	2688	2835	2726	2726
q9	7319	7305	7209	7209
q10	3064	3242	2783	2783
q11	591	528	503	503
q12	639	762	632	632
q13	3467	3783	3169	3169
q14	304	300	267	267
q15	510	494	470	470
q16	476	485	429	429
q17	1264	1781	1275	1275
q18	7605	7556	7352	7352
q19	845	1208	1103	1103
q20	2042	2057	1870	1870
q21	5361	5074	4604	4604
q22	1046	1086	1051	1051
Total cold run time: 53366 ms
Total hot run time: 52043 ms

@doris-robot

Copy link
Copy Markdown
TPC-DS: Total hot run time: 192253 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d5b42fde3bf66cdc82e7a3918a771fd18b50009f, data reload: false

query1	969	398	405	398
query2	6228	1983	1880	1880
query3	8681	200	199	199
query4	33452	23655	23417	23417
query5	3701	591	443	443
query6	287	201	196	196
query7	4193	490	313	313
query8	301	242	235	235
query9	9495	2614	2607	2607
query10	511	329	261	261
query11	18032	15921	15201	15201
query12	171	109	108	108
query13	1558	535	416	416
query14	9825	7114	6831	6831
query15	262	188	183	183
query16	7924	690	509	509
query17	1577	789	611	611
query18	2081	414	319	319
query19	209	200	173	173
query20	123	120	121	120
query21	210	125	107	107
query22	4552	4649	4379	4379
query23	35088	34121	34345	34121
query24	7589	2744	2744	2744
query25	558	494	442	442
query26	1200	302	180	180
query27	2288	472	368	368
query28	5589	2227	2226	2226
query29	775	614	469	469
query30	247	194	169	169
query31	1015	943	811	811
query32	101	87	59	59
query33	502	376	331	331
query34	767	855	534	534
query35	789	824	745	745
query36	1025	1066	969	969
query37	109	92	75	75
query38	4033	4103	4048	4048
query39	1517	1505	1484	1484
query40	210	114	101	101
query41	49	50	51	50
query42	130	110	111	110
query43	508	541	492	492
query44	1311	810	827	810
query45	183	177	171	171
query46	887	1064	679	679
query47	1978	1966	1943	1943
query48	409	434	344	344
query49	775	492	425	425
query50	704	711	435	435
query51	7396	7388	7400	7388
query52	99	107	98	98
query53	242	272	195	195
query54	568	550	473	473
query55	86	80	86	80
query56	278	286	255	255
query57	1261	1306	1198	1198
query58	245	215	221	215
query59	3125	3288	2985	2985
query60	291	282	273	273
query61	108	109	113	109
query62	798	771	716	716
query63	234	201	197	197
query64	4630	976	620	620
query65	3434	3276	3320	3276
query66	933	409	317	317
query67	16357	15728	15381	15381
query68	7592	825	530	530
query69	480	300	271	271
query70	1173	1122	1159	1122
query71	393	294	263	263
query72	5738	3606	3825	3606
query73	614	748	354	354
query74	10226	9212	8998	8998
query75	3393	3155	2727	2727
query76	3272	1181	759	759
query77	766	367	281	281
query78	10490	10502	9750	9750
query79	2833	906	606	606
query80	620	511	432	432
query81	507	258	224	224
query82	577	121	95	95
query83	163	156	143	143
query84	269	97	83	83
query85	828	367	303	303
query86	346	307	294	294
query87	4279	4294	4254	4254
query88	4955	2509	2404	2404
query89	406	339	291	291
query90	1806	191	193	191
query91	135	140	108	108
query92	69	57	53	53
query93	1586	890	544	544
query94	663	412	312	312
query95	354	289	274	274
query96	492	606	279	279
query97	3181	3293	3178	3178
query98	232	214	204	204
query99	1622	1434	1303	1303
Total cold run time: 294823 ms
Total hot run time: 192253 ms

@doris-robot

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.04 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d5b42fde3bf66cdc82e7a3918a771fd18b50009f, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.02
query3	0.24	0.06	0.06
query4	1.61	0.11	0.11
query5	0.52	0.51	0.51
query6	1.13	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.03	0.04
query9	0.56	0.50	0.49
query10	0.55	0.55	0.54
query11	0.14	0.10	0.10
query12	0.13	0.11	0.10
query13	0.61	0.59	0.60
query14	0.79	0.79	0.80
query15	0.87	0.83	0.82
query16	0.39	0.39	0.40
query17	1.01	1.01	1.04
query18	0.24	0.23	0.23
query19	1.97	1.89	1.91
query20	0.02	0.01	0.02
query21	15.38	0.89	0.57
query22	0.77	0.85	0.67
query23	15.04	1.44	0.59
query24	3.00	1.08	2.00
query25	0.16	0.13	0.07
query26	0.27	0.14	0.14
query27	0.07	0.07	0.04
query28	14.61	0.97	0.43
query29	12.60	3.95	3.26
query30	0.25	0.08	0.07
query31	2.82	0.62	0.38
query32	3.24	0.53	0.46
query33	2.99	3.06	3.00
query34	16.75	5.23	4.52
query35	4.62	4.62	4.53
query36	0.64	0.50	0.48
query37	0.10	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.67 s
Total hot run time: 29.04 s

@CalvinKirs

Copy link
Copy Markdown
Member Author

run buildall

@doris-robot

Copy link
Copy Markdown
TPC-H: Total hot run time: 32774 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0dc5c668bad71a1fb77688a60eecf41edef04bcf, data reload: false

------ Round 1 ----------------------------------
q1	17586	5624	5442	5442
q2	2024	393	280	280
q3	11973	1233	751	751
q4	10316	878	458	458
q5	9402	2360	2153	2153
q6	187	168	131	131
q7	907	739	607	607
q8	9331	1420	1127	1127
q9	5221	5038	4951	4951
q10	6768	2269	1819	1819
q11	503	285	266	266
q12	341	352	210	210
q13	17768	3598	3019	3019
q14	230	233	222	222
q15	523	463	462	462
q16	418	439	387	387
q17	614	865	372	372
q18	6873	6436	6404	6404
q19	1216	962	547	547
q20	326	345	212	212
q21	2785	2168	1993	1993
q22	1031	991	961	961
Total cold run time: 106343 ms
Total hot run time: 32774 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5537	5511	5483	5483
q2	229	320	229	229
q3	2269	2622	2351	2351
q4	1348	1791	1339	1339
q5	4416	5005	4946	4946
q6	170	162	126	126
q7	2081	1978	1871	1871
q8	2619	2793	2709	2709
q9	7318	7309	7243	7243
q10	3071	3296	2760	2760
q11	603	533	506	506
q12	696	784	662	662
q13	3499	3807	3206	3206
q14	294	322	291	291
q15	526	486	484	484
q16	439	496	425	425
q17	1258	1756	1309	1309
q18	7624	7396	7334	7334
q19	785	896	1117	896
q20	1999	2083	1893	1893
q21	5369	5001	4594	4594
q22	1052	1020	983	983
Total cold run time: 53202 ms
Total hot run time: 51640 ms

@doris-robot

Copy link
Copy Markdown
TPC-DS: Total hot run time: 192138 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0dc5c668bad71a1fb77688a60eecf41edef04bcf, data reload: false

query1	943	402	404	402
query2	6236	1847	1897	1847
query3	8676	201	196	196
query4	33446	24061	23739	23739
query5	3684	597	439	439
query6	280	184	172	172
query7	4208	485	303	303
query8	299	242	227	227
query9	9318	2617	2602	2602
query10	476	329	263	263
query11	18161	15768	15187	15187
query12	162	107	107	107
query13	1555	541	429	429
query14	9954	7487	6731	6731
query15	234	194	185	185
query16	7974	716	516	516
query17	1578	782	593	593
query18	2168	430	343	343
query19	223	195	162	162
query20	125	126	122	122
query21	215	131	122	122
query22	4550	4631	4380	4380
query23	35095	34520	33952	33952
query24	7410	2680	2729	2680
query25	548	510	436	436
query26	796	296	179	179
query27	2203	484	369	369
query28	5053	2195	2179	2179
query29	736	596	462	462
query30	240	188	164	164
query31	997	894	852	852
query32	89	60	63	60
query33	495	375	317	317
query34	762	864	509	509
query35	791	832	754	754
query36	1020	1082	1001	1001
query37	107	93	66	66
query38	4047	3947	3945	3945
query39	1536	1480	1447	1447
query40	200	115	101	101
query41	54	50	49	49
query42	126	111	109	109
query43	512	521	487	487
query44	1352	817	830	817
query45	191	186	174	174
query46	909	1071	675	675
query47	1942	1981	1969	1969
query48	404	445	365	365
query49	745	504	407	407
query50	675	691	430	430
query51	7402	7440	7166	7166
query52	109	102	89	89
query53	228	268	203	203
query54	548	543	476	476
query55	82	78	76	76
query56	274	267	262	262
query57	1316	1295	1209	1209
query58	226	234	221	221
query59	3008	3220	3046	3046
query60	292	301	264	264
query61	129	114	130	114
query62	789	761	700	700
query63	244	197	191	191
query64	3749	1005	624	624
query65	3404	3352	3304	3304
query66	918	414	313	313
query67	16429	15816	15587	15587
query68	7339	814	532	532
query69	489	311	267	267
query70	1177	1126	1093	1093
query71	384	299	258	258
query72	5708	3726	3757	3726
query73	634	750	354	354
query74	10265	9119	9228	9119
query75	3180	3131	2715	2715
query76	2952	1171	771	771
query77	462	357	279	279
query78	10372	10301	9603	9603
query79	3421	887	582	582
query80	806	520	432	432
query81	500	268	219	219
query82	970	122	87	87
query83	159	163	146	146
query84	236	103	89	89
query85	768	353	303	303
query86	389	311	282	282
query87	4368	4317	4249	4249
query88	5220	2448	2442	2442
query89	417	338	312	312
query90	1701	198	196	196
query91	179	138	112	112
query92	63	57	50	50
query93	2865	873	544	544
query94	716	420	311	311
query95	350	284	276	276
query96	498	625	293	293
query97	3219	3293	3113	3113
query98	218	208	205	205
query99	1538	1438	1311	1311
Total cold run time: 294210 ms
Total hot run time: 192138 ms

@doris-robot

Copy link
Copy Markdown
ClickBench: Total hot run time: 28.75 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0dc5c668bad71a1fb77688a60eecf41edef04bcf, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.61	0.11	0.10
query5	0.52	0.50	0.51
query6	1.14	0.74	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.51	0.51
query10	0.56	0.54	0.55
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	0.78	0.82	0.78
query15	0.84	0.84	0.82
query16	0.41	0.38	0.39
query17	1.07	1.04	1.07
query18	0.25	0.24	0.23
query19	1.89	1.75	1.77
query20	0.02	0.01	0.02
query21	15.40	0.91	0.58
query22	0.73	0.83	0.80
query23	14.87	1.47	0.58
query24	3.75	0.52	0.65
query25	0.19	0.14	0.10
query26	0.30	0.15	0.13
query27	0.05	0.04	0.05
query28	13.37	1.07	0.43
query29	12.55	4.05	3.41
query30	0.25	0.08	0.06
query31	2.82	0.61	0.37
query32	3.23	0.54	0.46
query33	3.04	3.01	3.10
query34	16.65	5.15	4.58
query35	4.56	4.53	4.58
query36	0.66	0.49	0.48
query37	0.09	0.06	0.05
query38	0.05	0.03	0.04
query39	0.03	0.03	0.02
query40	0.17	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 103.86 s
Total hot run time: 28.75 s

@CalvinKirs

Copy link
Copy Markdown
Member Author

run buildall

@doris-robot

Copy link
Copy Markdown
TPC-H: Total hot run time: 32630 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f2a6625b9e126edb01f756f9cf2a9c8e5ad4a908, data reload: false

------ Round 1 ----------------------------------
q1	17602	5517	5786	5517
q2	2050	378	287	287
q3	11745	1236	732	732
q4	10304	867	456	456
q5	9142	2394	2099	2099
q6	182	161	135	135
q7	885	740	615	615
q8	9338	1466	1164	1164
q9	5332	4954	4882	4882
q10	6748	2264	1801	1801
q11	491	281	268	268
q12	343	350	210	210
q13	17788	3587	3006	3006
q14	223	230	215	215
q15	535	476	463	463
q16	408	429	381	381
q17	604	865	373	373
q18	6899	6379	6288	6288
q19	1205	957	533	533
q20	325	362	204	204
q21	3067	2175	2037	2037
q22	1050	1010	964	964
Total cold run time: 106266 ms
Total hot run time: 32630 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5542	5522	5519	5519
q2	234	323	246	246
q3	2274	2622	2277	2277
q4	1340	1751	1358	1358
q5	4400	4941	4990	4941
q6	165	162	132	132
q7	2115	1974	1797	1797
q8	2640	2880	2728	2728
q9	7309	7237	7284	7237
q10	3124	3296	2751	2751
q11	576	495	486	486
q12	731	796	588	588
q13	3459	3780	3169	3169
q14	271	304	272	272
q15	509	475	476	475
q16	445	505	424	424
q17	1233	1721	1269	1269
q18	7661	7465	7241	7241
q19	801	1063	1115	1063
q20	2030	2054	1888	1888
q21	5373	4880	4638	4638
q22	1070	1047	975	975
Total cold run time: 53302 ms
Total hot run time: 51474 ms

@doris-robot

Copy link
Copy Markdown
TPC-DS: Total hot run time: 193123 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f2a6625b9e126edb01f756f9cf2a9c8e5ad4a908, data reload: false

query1	971	419	397	397
query2	6225	1943	1878	1878
query3	8681	198	194	194
query4	33518	24022	23511	23511
query5	3646	595	447	447
query6	296	211	170	170
query7	4197	491	322	322
query8	300	236	234	234
query9	9301	2611	2605	2605
query10	468	333	252	252
query11	18401	15375	15167	15167
query12	160	105	105	105
query13	1565	534	422	422
query14	9291	7695	7225	7225
query15	279	193	186	186
query16	7651	651	470	470
query17	1581	805	594	594
query18	2040	413	337	337
query19	236	193	170	170
query20	126	125	117	117
query21	209	132	112	112
query22	4504	4641	4417	4417
query23	34670	34223	34195	34195
query24	7513	2691	2695	2691
query25	571	494	458	458
query26	826	290	188	188
query27	1977	488	355	355
query28	5660	2213	2170	2170
query29	692	617	467	467
query30	242	201	166	166
query31	981	938	851	851
query32	98	61	60	60
query33	524	372	357	357
query34	765	850	551	551
query35	791	802	745	745
query36	1004	1067	984	984
query37	112	91	67	67
query38	3998	4055	4007	4007
query39	1558	1505	1458	1458
query40	201	116	106	106
query41	49	49	48	48
query42	121	109	108	108
query43	525	522	479	479
query44	1355	827	826	826
query45	184	181	174	174
query46	892	1079	699	699
query47	1995	1960	1940	1940
query48	404	436	370	370
query49	744	496	404	404
query50	692	715	434	434
query51	7425	7224	7230	7224
query52	104	111	95	95
query53	227	261	194	194
query54	561	527	483	483
query55	81	78	78	78
query56	271	283	251	251
query57	1284	1232	1205	1205
query58	251	232	222	222
query59	3048	3161	3084	3084
query60	288	287	274	274
query61	114	124	108	108
query62	795	741	707	707
query63	244	193	201	193
query64	3859	1010	648	648
query65	3416	3315	3340	3315
query66	784	405	305	305
query67	16637	15898	15625	15625
query68	8025	832	534	534
query69	495	300	279	279
query70	1184	1162	1123	1123
query71	444	299	273	273
query72	5152	3760	3848	3760
query73	639	748	354	354
query74	10416	9409	9122	9122
query75	3552	3173	2716	2716
query76	3466	1199	756	756
query77	758	373	280	280
query78	10327	10356	9601	9601
query79	2712	896	616	616
query80	770	523	428	428
query81	498	256	223	223
query82	419	119	89	89
query83	172	162	142	142
query84	276	106	90	90
query85	735	354	303	303
query86	349	315	296	296
query87	4380	4306	4293	4293
query88	3506	2426	2411	2411
query89	401	328	288	288
query90	1907	187	189	187
query91	137	142	106	106
query92	66	56	51	51
query93	1209	850	543	543
query94	648	425	308	308
query95	351	279	268	268
query96	496	608	285	285
query97	3217	3268	3162	3162
query98	231	205	203	203
query99	1504	1434	1332	1332
Total cold run time: 290621 ms
Total hot run time: 193123 ms

@doris-robot

Copy link
Copy Markdown
ClickBench: Total hot run time: 28.64 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f2a6625b9e126edb01f756f9cf2a9c8e5ad4a908, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.62	0.10	0.10
query5	0.51	0.50	0.50
query6	1.13	0.73	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.50	0.50
query10	0.57	0.56	0.55
query11	0.14	0.13	0.11
query12	0.13	0.11	0.11
query13	0.63	0.60	0.60
query14	0.78	0.78	0.80
query15	0.84	0.85	0.83
query16	0.39	0.40	0.40
query17	0.98	1.09	1.10
query18	0.24	0.23	0.22
query19	1.90	1.79	1.81
query20	0.01	0.01	0.02
query21	15.43	0.95	0.55
query22	0.75	0.72	0.63
query23	15.19	1.46	0.56
query24	3.40	0.72	0.69
query25	0.15	0.05	0.14
query26	0.28	0.15	0.13
query27	0.05	0.04	0.03
query28	13.11	1.00	0.44
query29	12.58	3.94	3.29
query30	0.26	0.09	0.08
query31	2.82	0.58	0.39
query32	3.22	0.54	0.45
query33	3.00	3.00	3.05
query34	16.59	5.30	4.54
query35	4.53	4.52	4.60
query36	0.64	0.50	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.14	0.13
query41	0.07	0.03	0.03
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.3 s
Total hot run time: 28.64 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 64.10% (25/39) 🎉
Increment coverage report
Complete coverage report

@morrySnow morrySnow changed the title branch-3.1: [fix](params-refactor)Enhance Object Storage Parameter Validation and Exception Handling (#55014) branch-3.1: [fix](params-refactor)Enhance Object Storage Parameter Validation and Exception Handling #55014 Aug 21, 2025
@morrySnow
morrySnow merged commit f5519f6 into apache:branch-3.1 Aug 21, 2025
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants