Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement] FE/BE print the node start time at startup #34499

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[Enhancement] FE/BE print the node start time at startup #34499

wants to merge 2 commits into from
+2 −2

Conversation

rohitrs1983
Copy link
Contributor

@rohitrs1983 rohitrs1983 commented May 7, 2024 •
edited

Proposed changes

When FE/BE restarts we cannot know if FE/BE restarted due to node restart or process restart

Issue Number: close #35003

Add node uptime to FE/BE out file during startup

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 7, 2024

sh-checker report

To get the full details, please check in the job output.

shellcheck errors 
'shellcheck ' found no issues.
shfmt errors 

'shfmt ' returned error 1 finding the following formatting issues:

----------
--- bin/start_be.sh.orig
+++ bin/start_be.sh
@@ -237,7 +237,7 @@
 fi
 
 chmod 550 "${DORIS_HOME}/lib/doris_be"
-echo "Start Time: $(date), Node uptime: $(uptime)" >> "${LOG_DIR}/be.out"
+echo "Start Time: $(date), Node uptime: $(uptime)" >>"${LOG_DIR}/be.out"
 
 if [[ ! -f '/bin/limit3' ]]; then
     LIMIT=''
--- bin/start_fe.sh.orig
+++ bin/start_fe.sh
@@ -239,7 +239,7 @@
     coverage_opt="${JACOCO_COVERAGE_OPT}"
 fi
 
-echo "Start Time: $(date), Node uptime: $(uptime)" >> "${LOG_DIR}/fe.out"
+echo "Start Time: $(date), Node uptime: $(uptime)" >>"${LOG_DIR}/fe.out"
 
 if [[ "${HELPER}" != "" ]]; then
     # change it to '-helper' to be compatible with code in Frontend
----------

You can reformat the above files to meet shfmt's requirements by typing:

  shfmt  -w filename

@rohitrs1983
Copy link
Contributor Author

rohitrs1983 commented May 14, 2024 •
edited

@LemonLiTree can u please review the PR

@Nitin-Kashyap
Copy link
Contributor

lgtm

dataroaring
dataroaring approved these changes May 26, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
Copy link
Contributor

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 26, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 39789 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2e9379da3866d68134da3962f0b56641cdaaa36f, data reload: false

------ Round 1 ----------------------------------
q1	17903	4503	4317	4317
q2	2585	201	200	200
q3	12230	1160	1166	1160
q4	10582	821	756	756
q5	7749	2722	2694	2694
q6	226	136	138	136
q7	948	585	583	583
q8	9272	2087	2089	2087
q9	8849	6484	6487	6484
q10	8907	3684	3745	3684
q11	481	240	230	230
q12	464	221	220	220
q13	17764	2981	2994	2981
q14	259	217	211	211
q15	511	487	480	480
q16	509	381	379	379
q17	962	664	668	664
q18	8098	7437	7434	7434
q19	6302	1535	1443	1443
q20	651	298	300	298
q21	4854	3061	3845	3061
q22	343	287	290	287
Total cold run time: 120449 ms
Total hot run time: 39789 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4374	4189	4171	4171
q2	377	277	276	276
q3	2999	2801	2729	2729
q4	1870	1630	1555	1555
q5	5264	5285	5259	5259
q6	205	124	124	124
q7	2082	1780	1747	1747
q8	3211	3333	3302	3302
q9	8339	8406	8316	8316
q10	3870	3678	3727	3678
q11	585	480	486	480
q12	786	584	594	584
q13	17277	2983	2993	2983
q14	315	268	278	268
q15	509	479	473	473
q16	481	431	417	417
q17	1780	1493	1489	1489
q18	7703	7618	7724	7618
q19	1665	1554	1569	1554
q20	1982	1796	1814	1796
q21	4951	4765	4767	4765
q22	586	476	503	476
Total cold run time: 71211 ms
Total hot run time: 54060 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.64% (9015/25293)
Line Coverage: 27.32% (74561/272952)
Region Coverage: 26.53% (38575/145398)
Branch Coverage: 23.39% (19667/84070)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2e9379da3866d68134da3962f0b56641cdaaa36f_2e9379da3866d68134da3962f0b56641cdaaa36f/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 172527 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2e9379da3866d68134da3962f0b56641cdaaa36f, data reload: false

query1	917	398	368	368
query2	6454	2546	2365	2365
query3	6654	214	211	211
query4	19423	17478	17217	17217
query5	4154	423	421	421
query6	245	160	148	148
query7	4600	297	289	289
query8	246	185	191	185
query9	8493	2378	2358	2358
query10	458	278	262	262
query11	10496	10138	10197	10138
query12	135	89	92	89
query13	1650	367	374	367
query14	9287	6982	7002	6982
query15	214	166	179	166
query16	7821	271	269	269
query17	1827	551	533	533
query18	1935	287	279	279
query19	199	158	157	157
query20	96	90	89	89
query21	205	130	126	126
query22	4267	4051	4023	4023
query23	33749	33114	33021	33021
query24	12086	2869	2829	2829
query25	660	358	361	358
query26	1823	154	154	154
query27	3008	328	319	319
query28	7412	2023	2015	2015
query29	1128	631	592	592
query30	279	173	180	173
query31	962	778	763	763
query32	92	55	60	55
query33	767	271	261	261
query34	1055	478	476	476
query35	750	599	615	599
query36	1096	898	911	898
query37	288	67	92	67
query38	2903	2789	2757	2757
query39	846	776	785	776
query40	272	127	120	120
query41	48	43	44	43
query42	104	94	97	94
query43	600	560	581	560
query44	1144	713	735	713
query45	178	164	162	162
query46	1079	737	732	732
query47	1843	1773	1773	1773
query48	365	297	296	296
query49	1207	373	388	373
query50	772	385	389	385
query51	6836	6736	6750	6736
query52	101	86	85	85
query53	355	282	282	282
query54	999	433	420	420
query55	73	75	73	73
query56	256	266	248	248
query57	1132	1015	1064	1015
query58	264	221	210	210
query59	3521	3575	3240	3240
query60	274	260	254	254
query61	91	90	89	89
query62	667	437	470	437
query63	319	279	281	279
query64	9847	2246	1802	1802
query65	3172	3120	3113	3113
query66	1362	346	319	319
query67	15143	15159	14795	14795
query68	5057	534	539	534
query69	482	268	268	268
query70	1160	1131	1120	1120
query71	460	285	270	270
query72	7637	5809	5462	5462
query73	765	327	324	324
query74	6101	5659	5608	5608
query75	3796	2635	2621	2621
query76	3643	945	905	905
query77	609	271	263	263
query78	10317	9858	9713	9713
query79	2429	515	512	512
query80	1348	433	425	425
query81	530	248	240	240
query82	1344	93	95	93
query83	269	171	179	171
query84	229	86	83	83
query85	1556	280	262	262
query86	485	305	299	299
query87	3259	3140	3155	3140
query88	3971	2347	2350	2347
query89	483	382	381	381
query90	1985	189	190	189
query91	123	96	167	96
query92	57	47	49	47
query93	2211	497	499	497
query94	1230	185	186	185
query95	387	305	304	304
query96	580	267	263	263
query97	3232	3050	3060	3050
query98	239	217	216	216
query99	1226	836	856	836
Total cold run time: 277904 ms
Total hot run time: 172527 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.67 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2e9379da3866d68134da3962f0b56641cdaaa36f, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.04	0.05
query3	0.23	0.05	0.05
query4	1.67	0.08	0.08
query5	0.52	0.49	0.50
query6	1.13	0.72	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.49	0.48
query10	0.54	0.54	0.55
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.60	0.58	0.59
query14	0.78	0.77	0.77
query15	0.82	0.82	0.81
query16	0.37	0.38	0.37
query17	1.01	1.02	0.98
query18	0.24	0.23	0.25
query19	1.86	1.69	1.72
query20	0.01	0.02	0.01
query21	15.73	0.67	0.66
query22	4.25	6.64	1.98
query23	18.26	1.40	1.29
query24	1.89	0.24	0.21
query25	0.15	0.08	0.09
query26	0.27	0.16	0.16
query27	0.08	0.08	0.08
query28	13.30	1.00	1.00
query29	13.22	3.32	3.29
query30	0.25	0.06	0.06
query31	2.87	0.39	0.39
query32	3.26	0.47	0.46
query33	2.85	2.90	2.97
query34	17.05	4.39	4.44
query35	4.54	4.51	4.53
query36	0.65	0.46	0.46
query37	0.18	0.15	0.15
query38	0.16	0.15	0.15
query39	0.05	0.03	0.04
query40	0.18	0.14	0.15
query41	0.09	0.04	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.2 s
Total hot run time: 30.67 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Enhancement] FE/BE print the node start time at startup
4 participants
@rohitrs1983 @doris-robot @Nitin-Kashyap @dataroaring