Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

branch-3.0: [opt](vault) Check hdfs connectivity when creating hdfs storage vault #48369 #48816

Open
wants to merge 1 commit into
base: branch-3.0
Choose a base branch
from

Conversation

github-actions[bot]
Copy link
Contributor

@github-actions github-actions bot commented Mar 7, 2025

Cherry-picked from #48369

…#48369)

When creating an HDFS storage vault, perform connection and permission
checks to avoid outputting error messages that could cause the storage
vault to throw errors during usage after creation.
@github-actions github-actions bot requested a review from dataroaring as a code owner March 7, 2025 06:51
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Mar 7, 2025
@dataroaring dataroaring reopened this Mar 7, 2025
@hello-stephen
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40079 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e63877b1280bf2c8f059e6b7f5fbf0f063989c81, data reload: false

------ Round 1 ----------------------------------
q1	17691	7330	6711	6711
q2	2061	179	190	179
q3	10586	1112	1243	1112
q4	10551	780	802	780
q5	7943	2896	2825	2825
q6	222	135	130	130
q7	977	614	599	599
q8	9356	1972	2017	1972
q9	6594	6394	6393	6393
q10	7044	2227	2254	2227
q11	463	259	262	259
q12	396	210	210	210
q13	17755	2983	2975	2975
q14	240	215	210	210
q15	526	472	461	461
q16	670	603	608	603
q17	954	530	553	530
q18	7306	6654	6781	6654
q19	1410	1048	952	952
q20	473	199	196	196
q21	3954	3118	3122	3118
q22	1095	983	995	983
Total cold run time: 108267 ms
Total hot run time: 40079 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6617	6576	6522	6522
q2	331	230	224	224
q3	2893	2746	2935	2746
q4	2026	1779	1676	1676
q5	5776	5752	5767	5752
q6	205	126	123	123
q7	2187	1818	1780	1780
q8	3358	3563	3476	3476
q9	8991	8876	8854	8854
q10	3561	3535	3476	3476
q11	596	483	483	483
q12	803	618	583	583
q13	9359	3224	3103	3103
q14	299	279	283	279
q15	507	471	449	449
q16	675	638	656	638
q17	1845	1613	1583	1583
q18	8138	7845	7642	7642
q19	1664	1535	1529	1529
q20	2057	1804	1868	1804
q21	5624	5285	5265	5265
q22	1109	1074	1034	1034
Total cold run time: 68621 ms
Total hot run time: 59021 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196420 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e63877b1280bf2c8f059e6b7f5fbf0f063989c81, data reload: false

query1	1295	930	892	892
query2	6269	2010	1943	1943
query3	10810	4259	4444	4259
query4	60373	28547	23363	23363
query5	5194	448	433	433
query6	406	169	185	169
query7	5490	306	310	306
query8	299	225	231	225
query9	8759	2615	2589	2589
query10	471	268	248	248
query11	17734	15149	15655	15149
query12	159	104	103	103
query13	1478	455	418	418
query14	10843	7535	6661	6661
query15	216	181	171	171
query16	7074	485	487	485
query17	1172	588	586	586
query18	1727	338	311	311
query19	210	172	157	157
query20	117	111	113	111
query21	208	104	105	104
query22	4746	4481	4784	4481
query23	34648	34033	34057	34033
query24	6248	2945	2922	2922
query25	526	401	406	401
query26	667	176	171	171
query27	1915	346	367	346
query28	4298	2503	2437	2437
query29	696	440	415	415
query30	248	172	160	160
query31	1009	842	811	811
query32	71	56	59	56
query33	429	277	282	277
query34	920	506	513	506
query35	883	726	721	721
query36	1072	1002	967	967
query37	115	66	68	66
query38	4120	4120	4058	4058
query39	1538	1451	1501	1451
query40	204	99	98	98
query41	48	47	46	46
query42	112	106	105	105
query43	520	493	491	491
query44	1185	832	829	829
query45	183	169	165	165
query46	1172	753	720	720
query47	2060	1872	1932	1872
query48	487	380	383	380
query49	731	385	390	385
query50	843	424	415	415
query51	7304	7572	7158	7158
query52	101	90	95	90
query53	267	185	188	185
query54	579	460	450	450
query55	80	84	74	74
query56	270	277	256	256
query57	1243	1149	1148	1148
query58	212	204	213	204
query59	3235	2897	2967	2897
query60	275	263	249	249
query61	117	108	105	105
query62	770	672	683	672
query63	218	182	180	180
query64	1380	666	617	617
query65	3284	3161	3163	3161
query66	615	299	299	299
query67	15961	15570	15854	15570
query68	4123	603	567	567
query69	423	260	265	260
query70	1184	1145	1114	1114
query71	331	254	263	254
query72	6360	4059	3997	3997
query73	763	352	378	352
query74	9997	9028	9096	9028
query75	3360	2684	2682	2682
query76	1834	1053	1075	1053
query77	466	292	286	286
query78	10422	9561	9625	9561
query79	1266	621	605	605
query80	898	435	416	416
query81	512	232	237	232
query82	1265	91	85	85
query83	239	162	141	141
query84	283	86	76	76
query85	897	295	286	286
query86	328	315	301	301
query87	4374	4359	4314	4314
query88	3586	2407	2373	2373
query89	410	292	293	292
query90	1947	183	184	183
query91	201	163	148	148
query92	67	51	52	51
query93	1465	548	546	546
query94	707	291	296	291
query95	354	256	251	251
query96	620	286	275	275
query97	3314	3142	3180	3142
query98	212	203	203	203
query99	1645	1272	1284	1272
Total cold run time: 311762 ms
Total hot run time: 196420 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.82 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e63877b1280bf2c8f059e6b7f5fbf0f063989c81, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.23	0.06	0.06
query4	1.62	0.10	0.10
query5	0.52	0.51	0.50
query6	1.13	0.73	0.73
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.55	0.52	0.49
query10	0.53	0.55	0.54
query11	0.15	0.10	0.10
query12	0.14	0.12	0.11
query13	0.61	0.60	0.59
query14	2.82	2.72	2.75
query15	0.91	0.82	0.83
query16	0.40	0.40	0.38
query17	1.04	1.06	1.04
query18	0.24	0.22	0.23
query19	1.92	1.78	2.02
query20	0.01	0.02	0.01
query21	15.42	0.62	0.59
query22	2.65	1.74	2.44
query23	17.19	0.98	0.88
query24	2.61	0.76	1.18
query25	0.16	0.14	0.11
query26	0.43	0.14	0.14
query27	0.04	0.04	0.04
query28	10.87	0.53	0.44
query29	12.58	3.21	3.23
query30	0.25	0.06	0.06
query31	2.86	0.39	0.37
query32	3.24	0.46	0.47
query33	3.03	3.03	2.99
query34	17.01	4.45	4.48
query35	4.48	4.52	4.50
query36	0.66	0.48	0.49
query37	0.09	0.06	0.07
query38	0.04	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.03
query42	0.04	0.03	0.03
query43	0.03	0.04	0.03
Total cold run time: 106.94 s
Total hot run time: 31.82 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants