[feature](multi-catalog)support hms catalog create and drop table/db by wsjz · Pull Request #30198 · apache/doris

wsjz · 2024-01-22T02:05:23Z

Proposed changes

rename old create/drop table to add/removeMemoryTable
add new create/drop table/db method
support hms catalog create/drop table/db

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

add exception add log

fix

wsjz · 2024-02-23T08:29:49Z

run buildall

wsjz · 2024-02-25T11:57:45Z

run feut

wsjz · 2024-02-25T12:11:55Z

run buildall

doris-robot · 2024-02-25T12:45:24Z

TPC-H: Total hot run time: 40984 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 392b0a1827f8fc52edf8855de5bee046adc51b2a, data reload: false

------ Round 1 ----------------------------------
q1	17646	4953	4881	4881
q2	2035	148	133	133
q3	10577	987	992	987
q4	4654	959	980	959
q5	7613	3145	3162	3145
q6	191	132	136	132
q7	1239	772	761	761
q8	9263	2033	2026	2026
q9	7412	6541	6562	6541
q10	8307	2663	2645	2645
q11	408	225	214	214
q12	711	332	336	332
q13	17947	3620	3621	3620
q14	292	261	279	261
q15	586	520	500	500
q16	480	420	433	420
q17	908	871	850	850
q18	7506	6808	6563	6563
q19	1541	1486	1484	1484
q20	534	274	273	273
q21	6468	3936	3950	3936
q22	851	338	321	321
Total cold run time: 107169 ms
Total hot run time: 40984 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4853	4854	4886	4854
q2	291	184	197	184
q3	3592	3566	3577	3566
q4	2497	2504	2517	2504
q5	5780	5743	5767	5743
q6	210	127	127	127
q7	2247	1636	1682	1636
q8	2993	3060	3097	3060
q9	8682	8678	8692	8678
q10	6777	4217	4229	4217
q11	511	350	369	350
q12	763	549	536	536
q13	4327	3423	3389	3389
q14	273	241	240	240
q15	600	502	546	502
q16	472	425	447	425
q17	1642	1602	1610	1602
q18	8493	7750	7740	7740
q19	1628	1619	1622	1619
q20	2123	1833	1832	1832
q21	6538	6140	6095	6095
q22	571	508	502	502
Total cold run time: 65863 ms
Total hot run time: 59401 ms

doris-robot · 2024-02-25T12:56:28Z

TPC-DS: Total hot run time: 177298 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 392b0a1827f8fc52edf8855de5bee046adc51b2a, data reload: false

query1	932	354	353	353
query2	6508	1728	1641	1641
query3	6721	206	204	204
query4	23182	21273	21112	21112
query5	4286	390	398	390
query6	286	170	176	170
query7	4613	297	294	294
query8	261	204	211	204
query9	8435	2838	2822	2822
query10	419	240	235	235
query11	15017	14667	14393	14393
query12	154	88	87	87
query13	1731	427	418	418
query14	9263	7921	7825	7825
query15	218	189	194	189
query16	7449	262	255	255
query17	1403	552	546	546
query18	1943	269	266	266
query19	205	144	155	144
query20	88	82	81	81
query21	192	133	129	129
query22	4951	4717	4778	4717
query23	32590	31520	31672	31520
query24	12868	3407	3448	3407
query25	651	381	367	367
query26	1884	158	165	158
query27	3030	321	309	309
query28	6592	1816	1814	1814
query29	1188	663	669	663
query30	286	144	154	144
query31	926	757	779	757
query32	97	63	61	61
query33	734	239	235	235
query34	1080	498	501	498
query35	944	856	840	840
query36	1019	857	898	857
query37	182	62	63	62
query38	3321	3187	3181	3181
query39	1375	1356	1314	1314
query40	285	109	110	109
query41	39	37	35	35
query42	108	102	100	100
query43	474	428	438	428
query44	1073	683	703	683
query45	200	183	178	178
query46	1047	787	749	749
query47	1611	1541	1561	1541
query48	429	343	351	343
query49	1226	316	333	316
query50	776	384	378	378
query51	4449	4356	4289	4289
query52	115	100	93	93
query53	409	315	330	315
query54	291	237	245	237
query55	89	81	78	78
query56	233	195	206	195
query57	1058	964	939	939
query58	222	203	209	203
query59	2223	2179	2239	2179
query60	255	226	227	226
query61	94	89	85	85
query62	601	386	384	384
query63	344	313	318	313
query64	6378	3083	3133	3083
query65	3302	3290	3262	3262
query66	1338	335	331	331
query67	14629	14281	14344	14281
query68	5201	550	561	550
query69	528	363	368	363
query70	1269	1193	1183	1183
query71	454	260	258	258
query72	6558	2808	2599	2599
query73	704	310	311	310
query74	6927	6470	6463	6463
query75	3191	2589	2589	2589
query76	3286	1147	1215	1147
query77	368	242	239	239
query78	9462	8897	8791	8791
query79	975	510	521	510
query80	532	375	368	368
query81	434	206	212	206
query82	163	93	90	90
query83	147	125	125	125
query84	233	80	81	80
query85	1037	363	352	352
query86	288	299	311	299
query87	3519	3396	3318	3318
query88	2712	2318	2303	2303
query89	450	365	379	365
query90	1957	166	172	166
query91	156	127	127	127
query92	56	54	51	51
query93	1027	515	504	504
query94	1117	182	176	176
query95	443	344	346	344
query96	573	258	263	258
query97	4482	4253	4288	4253
query98	238	207	209	207
query99	1095	740	689	689
Total cold run time: 269904 ms
Total hot run time: 177298 ms

doris-robot · 2024-02-25T13:01:56Z

ClickBench: Total hot run time: 31.84 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 392b0a1827f8fc52edf8855de5bee046adc51b2a, data reload: false

query1	0.03	0.02	0.02
query2	0.06	0.02	0.02
query3	0.23	0.07	0.07
query4	1.61	0.08	0.08
query5	0.48	0.48	0.48
query6	1.39	0.62	0.63
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.54	0.43	0.45
query10	0.49	0.49	0.48
query11	0.13	0.10	0.10
query12	0.12	0.10	0.10
query13	0.60	0.60	0.59
query14	0.79	0.78	0.79
query15	0.82	0.81	0.80
query16	0.34	0.33	0.33
query17	0.90	0.92	0.89
query18	0.18	0.17	0.17
query19	1.76	1.69	1.69
query20	0.02	0.01	0.02
query21	15.40	0.65	0.59
query22	2.93	3.69	3.20
query23	17.45	1.04	1.18
query24	2.17	0.23	0.23
query25	0.54	0.07	0.07
query26	0.15	0.14	0.14
query27	0.05	0.06	0.05
query28	12.30	0.82	0.83
query29	12.54	3.40	3.32
query30	0.52	0.47	0.46
query31	2.79	0.37	0.38
query32	3.29	0.48	0.47
query33	3.12	3.13	3.17
query34	15.69	4.55	4.51
query35	4.52	4.52	4.52
query36	1.08	0.95	0.96
query37	0.08	0.05	0.05
query38	0.04	0.03	0.02
query39	0.03	0.02	0.01
query40	0.17	0.14	0.14
query41	0.06	0.01	0.02
query42	0.02	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 105.52 s
Total hot run time: 31.84 s

doris-robot · 2024-02-25T13:05:07Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 392b0a1827f8fc52edf8855de5bee046adc51b2a with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       17.3 seconds inserted 10000000 Rows, about 578K ops/s

doris-robot · 2024-02-26T01:54:35Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1231f6410ae5dd83de00e5249a5c48bfee54ac88 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       15.9 seconds inserted 10000000 Rows, about 628K ops/s

morningman

LGTM

morningman · 2024-02-26T10:04:38Z

        "Sample size for hive row count estimation."})
    public static int hive_stats_partition_sample_size = 3000;

+    @ConfField(mutable = true, masterOnly = true, description = {


Modify thess config in your next PR

wsjz · 2024-02-26T10:40:12Z

run buildall

doris-robot · 2024-02-26T10:59:10Z

TPC-H: Total hot run time: 37990 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dd84d6468919ec913bbaa027a34753f11ccd37c8, data reload: false

------ Round 1 ----------------------------------
q1	18101	4150	4101	4101
q2	2213	141	135	135
q3	12247	966	950	950
q4	6692	964	943	943
q5	8311	2947	3048	2947
q6	177	120	118	118
q7	1233	753	743	743
q8	9246	2006	2033	2006
q9	7224	6307	6333	6307
q10	8183	2490	2489	2489
q11	421	220	195	195
q12	716	312	307	307
q13	17948	3200	3210	3200
q14	270	246	246	246
q15	531	503	493	493
q16	474	404	421	404
q17	927	903	890	890
q18	6558	5950	5899	5899
q19	1580	1507	1512	1507
q20	559	278	267	267
q21	6583	3567	3586	3567
q22	796	285	276	276
Total cold run time: 110990 ms
Total hot run time: 37990 ms

----- Round 2, with runtime_filter_mode=off -----
q1	3978	3968	3979	3968
q2	316	212	214	212
q3	2968	2938	2904	2904
q4	1864	1821	1827	1821
q5	5116	5082	5135	5082
q6	199	114	116	114
q7	2114	1673	1740	1673
q8	3132	3175	3182	3175
q9	8262	8267	8209	8209
q10	6236	3660	3644	3644
q11	510	424	423	423
q12	675	529	508	508
q13	3874	3087	3075	3075
q14	262	257	257	257
q15	540	494	496	494
q16	510	473	458	458
q17	1680	1680	1667	1667
q18	7857	7649	7283	7283
q19	1656	1639	1638	1638
q20	2099	1904	1885	1885
q21	4825	4708	4609	4609
q22	522	443	443	443
Total cold run time: 59195 ms
Total hot run time: 53542 ms

doris-robot · 2024-02-26T11:09:48Z

TPC-DS: Total hot run time: 169325 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dd84d6468919ec913bbaa027a34753f11ccd37c8, data reload: false

query1	949	340	327	327
query2	6528	1869	1793	1793
query3	6932	206	204	204
query4	23647	20526	20441	20441
query5	4246	359	356	356
query6	267	171	169	169
query7	4616	302	308	302
query8	266	183	210	183
query9	8460	2209	2212	2209
query10	425	226	207	207
query11	14495	14263	14229	14229
query12	134	79	83	79
query13	1638	444	431	431
query14	8273	6566	6690	6566
query15	213	175	176	175
query16	7091	257	263	257
query17	1037	525	508	508
query18	1899	266	272	266
query19	189	149	152	149
query20	89	79	81	79
query21	198	128	125	125
query22	4745	4607	4584	4584
query23	30502	29791	29851	29791
query24	10964	3024	3111	3024
query25	629	364	348	348
query26	1701	153	164	153
query27	3025	318	322	318
query28	7004	1803	1780	1780
query29	1130	557	555	555
query30	275	129	140	129
query31	871	687	684	684
query32	90	60	59	59
query33	718	230	223	223
query34	1033	484	489	484
query35	851	746	754	746
query36	950	848	881	848
query37	132	60	61	60
query38	3082	3072	2951	2951
query39	1301	1239	1252	1239
query40	278	100	99	99
query41	37	37	34	34
query42	101	96	100	96
query43	448	415	409	409
query44	1073	695	712	695
query45	189	175	174	174
query46	1038	808	802	802
query47	1512	1478	1439	1439
query48	414	345	339	339
query49	1185	297	284	284
query50	766	379	378	378
query51	4329	4250	4306	4250
query52	103	94	84	84
query53	330	270	264	264
query54	301	227	228	227
query55	88	88	84	84
query56	221	201	200	200
query57	957	848	898	848
query58	209	198	200	198
query59	2188	2130	2099	2099
query60	248	214	225	214
query61	95	88	86	86
query62	588	347	357	347
query63	285	251	256	251
query64	6576	3042	3114	3042
query65	3242	3197	3202	3197
query66	1453	321	319	319
query67	14413	14177	13935	13935
query68	4914	568	582	568
query69	541	381	389	381
query70	1235	1252	1213	1213
query71	384	250	252	250
query72	6258	2797	2643	2643
query73	710	327	340	327
query74	6286	5943	5990	5943
query75	3040	2412	2376	2376
query76	2808	1017	1173	1017
query77	359	234	227	227
query78	9131	8544	8534	8534
query79	990	530	530	530
query80	538	369	356	356
query81	457	193	205	193
query82	164	83	84	83
query83	147	124	120	120
query84	265	78	85	78
query85	1039	357	346	346
query86	300	299	296	296
query87	3250	3088	3112	3088
query88	2769	2388	2402	2388
query89	376	327	314	314
query90	1842	162	161	161
query91	146	123	124	123
query92	56	47	50	47
query93	1015	515	513	513
query94	1150	181	185	181
query95	439	337	342	337
query96	578	271	270	270
query97	4236	4135	4145	4135
query98	219	201	214	201
query99	1001	669	661	661
Total cold run time: 259905 ms
Total hot run time: 169325 ms

doris-robot · 2024-02-26T11:15:18Z

ClickBench: Total hot run time: 31.9 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dd84d6468919ec913bbaa027a34753f11ccd37c8, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.02	0.03
query3	0.24	0.06	0.05
query4	1.67	0.10	0.09
query5	0.52	0.52	0.51
query6	1.23	0.68	0.68
query7	0.01	0.01	0.01
query8	0.03	0.03	0.02
query9	0.57	0.52	0.52
query10	0.57	0.56	0.59
query11	0.12	0.09	0.10
query12	0.12	0.10	0.10
query13	0.62	0.60	0.62
query14	0.78	0.82	0.79
query15	0.86	0.83	0.81
query16	0.37	0.38	0.38
query17	0.98	0.98	0.98
query18	0.26	0.25	0.24
query19	1.86	1.74	1.66
query20	0.01	0.01	0.01
query21	15.45	0.68	0.60
query22	2.92	4.34	2.17
query23	17.64	1.13	0.96
query24	2.00	0.37	0.56
query25	0.47	0.20	0.06
query26	0.18	0.15	0.15
query27	0.06	0.05	0.03
query28	12.03	0.86	0.82
query29	12.70	3.27	3.38
query30	0.56	0.56	0.49
query31	2.78	0.34	0.34
query32	3.39	0.48	0.48
query33	3.21	3.19	3.21
query34	15.34	4.76	4.73
query35	4.76	4.74	4.78
query36	1.15	1.08	1.07
query37	0.07	0.05	0.05
query38	0.05	0.03	0.03
query39	0.02	0.02	0.02
query40	0.16	0.13	0.13
query41	0.06	0.02	0.02
query42	0.02	0.01	0.02
query43	0.02	0.03	0.02
Total cold run time: 105.96 s
Total hot run time: 31.9 s

doris-robot · 2024-02-26T11:18:24Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit dd84d6468919ec913bbaa027a34753f11ccd37c8 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       15.6 seconds inserted 10000000 Rows, about 641K ops/s

wsjz · 2024-02-26T12:06:29Z

run p0

github-actions · 2024-02-26T15:01:27Z

PR approved by anyone and no changes requested.

github-actions · 2024-02-27T01:59:14Z

PR approved by at least one committer and no changes requested.

wuwenchi mentioned this pull request Jan 25, 2024

[feature](iceberg)support iceberg catalog to create database and table #30347

Closed

morningman reviewed Jan 29, 2024

View reviewed changes

wsjz marked this pull request as ready for review February 4, 2024 07:40

github-actions Bot added the meta-change label Feb 4, 2024

morningman reviewed Feb 5, 2024

View reviewed changes

wsjz and others added 11 commits February 21, 2024 15:03

[feature](multi-catalog)support hms catalog create and drop table/db

09b1d6b

support for iceberg catalog(hms/hadoop/rest/glu)

91a4ab6

add refresh hms

310e946

fix rebase comple

3c97cc9

add some config

f862a4f

fix rebase

3d2463d

fix rebase

2280a6f

add engineName:iceberg

e69681b

add exception add log

Merge pull request #1 from wuwenchi/hms_create_meta

f29293c

fix

fix create db

e5b30f3

fix refresh

3099f22

morningman reviewed Feb 22, 2024

View reviewed changes

wsjz and others added 5 commits February 23, 2024 11:41

fix

16c88a8

fix

f42b9f2

fix

f54dc70

fix iceberg refresh

a0e2eef

support orc db/tbl

696f569

fix ut

392b0a1

morningman previously approved these changes Feb 26, 2024

View reviewed changes

Merge branch 'master' into hms_create_meta

dd84d64

wsjz dismissed morningman’s stale review via dd84d64 February 26, 2024 10:39

wuwenchi approved these changes Feb 26, 2024

View reviewed changes

github-actions Bot added the reviewed label Feb 26, 2024

morningman approved these changes Feb 27, 2024

View reviewed changes

github-actions Bot added the approved Indicates a PR has been approved by one committer. label Feb 27, 2024

morningman merged commit b2e869c into apache:master Feb 27, 2024

morningman mentioned this pull request Feb 27, 2024

[Feature] Support DDL/DML on external catalog #31442

Closed

3 tasks

wsjz mentioned this pull request Feb 28, 2024

[feature](multi-catalog)support hms catalog create and drop table/db … #31499

Merged

morningman added the dev/2.1.0-merged label Feb 28, 2024

morningman mentioned this pull request Mar 17, 2024

[feature](multi-catalog)support hms catalog create and drop table/db (#30198) #32343

Closed

wsjz deleted the hms_create_meta branch March 22, 2024 07:42

morningman mentioned this pull request Mar 27, 2024

[fix](iceberg) fix iceberg catalog bug and p2 test cases #32898

Merged

Uh oh!

Conversation

wsjz commented Jan 22, 2024 • edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wsjz commented Feb 23, 2024

Uh oh!

wsjz commented Feb 25, 2024

Uh oh!

wsjz commented Feb 25, 2024

Uh oh!

doris-robot commented Feb 25, 2024

Uh oh!

doris-robot commented Feb 25, 2024

Uh oh!

doris-robot commented Feb 25, 2024

Uh oh!

doris-robot commented Feb 25, 2024

Uh oh!

doris-robot commented Feb 26, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

morningman Feb 26, 2024

Choose a reason for hiding this comment

Uh oh!

wsjz commented Feb 26, 2024

Uh oh!

doris-robot commented Feb 26, 2024

Uh oh!

doris-robot commented Feb 26, 2024

Uh oh!

doris-robot commented Feb 26, 2024

Uh oh!

doris-robot commented Feb 26, 2024

Uh oh!

wsjz commented Feb 26, 2024

Uh oh!

github-actions Bot commented Feb 26, 2024

Uh oh!

github-actions Bot commented Feb 27, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wsjz commented Jan 22, 2024 •

edited by morningman

Loading