Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature](kerberos)support custom hadoop conf #34989

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

wsjz
Copy link
Contributor

@wsjz wsjz commented May 17, 2024

Proposed changes

for catalog and tvf, we should support custom hadoop conf dir
add new config, dfs.config.dir

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wsjz
Copy link
Contributor Author

wsjz commented May 22, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40616 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9b3b103d4c3370a3906e921680cd0756bf6d6180, data reload: false

------ Round 1 ----------------------------------
q1	17688	4401	4260	4260
q2	2017	194	186	186
q3	10739	1278	1188	1188
q4	10796	780	715	715
q5	7914	2693	2688	2688
q6	220	134	134	134
q7	964	612	591	591
q8	9761	2104	2078	2078
q9	9269	6617	6553	6553
q10	8954	3686	3721	3686
q11	466	241	233	233
q12	455	218	211	211
q13	17761	2986	2958	2958
q14	264	220	230	220
q15	510	466	461	461
q16	503	377	380	377
q17	966	750	739	739
q18	8016	7423	7424	7423
q19	6039	1535	1475	1475
q20	666	321	318	318
q21	4909	3916	3848	3848
q22	361	274	282	274
Total cold run time: 119238 ms
Total hot run time: 40616 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4327	4173	4197	4173
q2	364	254	275	254
q3	2952	2729	2697	2697
q4	1846	1580	1614	1580
q5	5215	5251	5256	5251
q6	215	122	124	122
q7	2123	1744	1739	1739
q8	3164	3290	3294	3290
q9	8282	8304	8363	8304
q10	3886	3681	3693	3681
q11	583	483	473	473
q12	751	609	601	601
q13	16341	3003	2991	2991
q14	287	263	254	254
q15	502	465	470	465
q16	466	420	402	402
q17	1772	1450	1451	1450
q18	7617	7793	7398	7398
q19	1638	1532	1548	1532
q20	1982	1790	1770	1770
q21	4916	4656	4718	4656
q22	561	486	493	486
Total cold run time: 69790 ms
Total hot run time: 53569 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171153 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9b3b103d4c3370a3906e921680cd0756bf6d6180, data reload: false

query1	910	374	367	367
query2	6455	2344	2283	2283
query3	6658	207	204	204
query4	19259	17526	17308	17308
query5	4137	439	423	423
query6	250	162	157	157
query7	4585	310	297	297
query8	241	193	178	178
query9	8663	2378	2347	2347
query10	464	284	262	262
query11	10436	10026	9976	9976
query12	146	91	95	91
query13	1646	370	374	370
query14	9508	7679	7104	7104
query15	255	167	176	167
query16	8096	268	265	265
query17	1835	528	504	504
query18	1909	271	262	262
query19	211	157	156	156
query20	92	86	105	86
query21	198	129	128	128
query22	4285	3873	3948	3873
query23	33679	33072	32997	32997
query24	12031	2840	2874	2840
query25	651	395	357	357
query26	1754	153	154	153
query27	3048	320	337	320
query28	7572	2042	2039	2039
query29	1018	603	594	594
query30	326	170	170	170
query31	978	760	742	742
query32	97	52	55	52
query33	763	260	271	260
query34	1032	456	470	456
query35	731	620	577	577
query36	1087	931	915	915
query37	280	72	70	70
query38	2930	2824	2731	2731
query39	846	791	791	791
query40	288	132	124	124
query41	48	45	43	43
query42	103	97	99	97
query43	584	533	559	533
query44	1186	738	744	738
query45	182	167	165	165
query46	1071	740	712	712
query47	1857	1743	1779	1743
query48	379	292	291	291
query49	1175	390	382	382
query50	784	394	396	394
query51	6908	6853	6767	6767
query52	101	93	87	87
query53	346	284	286	284
query54	1020	431	442	431
query55	79	72	72	72
query56	294	244	254	244
query57	1121	1072	1060	1060
query58	240	210	208	208
query59	3493	3052	3183	3052
query60	281	250	250	250
query61	88	90	84	84
query62	660	462	441	441
query63	311	294	299	294
query64	9834	2271	1722	1722
query65	3146	3092	3089	3089
query66	1355	332	324	324
query67	15425	14736	14645	14645
query68	4578	544	544	544
query69	444	267	274	267
query70	1111	1050	1146	1050
query71	424	273	269	269
query72	7673	4885	5327	4885
query73	736	318	315	315
query74	6089	5589	5572	5572
query75	3472	2619	2645	2619
query76	2940	1020	949	949
query77	400	273	268	268
query78	10270	9795	9725	9725
query79	2148	518	522	518
query80	922	453	443	443
query81	521	246	237	237
query82	658	97	96	96
query83	244	168	178	168
query84	239	95	88	88
query85	1845	264	263	263
query86	524	288	309	288
query87	3287	3082	3173	3082
query88	4062	2329	2338	2329
query89	478	384	383	383
query90	2017	191	190	190
query91	125	97	99	97
query92	63	49	53	49
query93	2291	518	499	499
query94	1196	192	191	191
query95	396	304	315	304
query96	609	266	264	264
query97	3155	3004	3001	3001
query98	237	225	209	209
query99	1061	844	843	843
Total cold run time: 275636 ms
Total hot run time: 171153 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.1 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9b3b103d4c3370a3906e921680cd0756bf6d6180, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.07	0.08
query5	0.49	0.49	0.49
query6	1.12	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.49	0.48
query10	0.54	0.54	0.54
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.59	0.59	0.60
query14	0.75	0.78	0.79
query15	0.82	0.80	0.79
query16	0.36	0.36	0.35
query17	0.95	1.04	0.96
query18	0.21	0.24	0.23
query19	1.75	1.65	1.71
query20	0.01	0.00	0.01
query21	15.74	0.65	0.66
query22	4.22	8.13	1.65
query23	18.32	1.35	1.31
query24	1.93	0.24	0.20
query25	0.15	0.07	0.08
query26	0.27	0.16	0.18
query27	0.07	0.08	0.08
query28	13.26	1.02	1.00
query29	13.10	3.27	3.26
query30	0.24	0.06	0.06
query31	2.85	0.38	0.38
query32	3.27	0.47	0.46
query33	2.87	2.82	2.90
query34	16.95	4.43	4.50
query35	4.56	4.55	4.50
query36	0.65	0.45	0.48
query37	0.18	0.15	0.16
query38	0.16	0.14	0.14
query39	0.05	0.03	0.04
query40	0.18	0.15	0.14
query41	0.09	0.05	0.04
query42	0.05	0.04	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.71 s
Total hot run time: 30.1 s

@wsjz
Copy link
Contributor Author

wsjz commented May 29, 2024

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants