doris icon indicating copy to clipboard operation
doris copied to clipboard

[fix](spill) fix wrong disk usage of spill

Open jacktengg opened this issue 1 year ago • 10 comments

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

jacktengg avatar May 27 '24 04:05 jacktengg

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.

doris-robot avatar May 27 '24 04:05 doris-robot

run buildall

jacktengg avatar May 27 '24 04:05 jacktengg

clang-tidy review says "All clean, LGTM! :+1:"

github-actions[bot] avatar May 27 '24 04:05 github-actions[bot]

clang-tidy review says "All clean, LGTM! :+1:"

github-actions[bot] avatar May 27 '24 04:05 github-actions[bot]

TeamCity be ut coverage result: Function Coverage: 35.73% (9007/25206) Line Coverage: 27.36% (74558/272478) Region Coverage: 26.58% (38570/145085) Branch Coverage: 23.45% (19670/83894) Coverage Report: http://coverage.selectdb-in.cc/coverage/144712af7bebae2f89aa6dc9bdae3ba20f792d69_144712af7bebae2f89aa6dc9bdae3ba20f792d69/report/index.html

doris-robot avatar May 27 '24 05:05 doris-robot

run buildall

jacktengg avatar May 27 '24 06:05 jacktengg

clang-tidy review says "All clean, LGTM! :+1:"

github-actions[bot] avatar May 27 '24 06:05 github-actions[bot]

clang-tidy review says "All clean, LGTM! :+1:"

github-actions[bot] avatar May 27 '24 06:05 github-actions[bot]

TeamCity be ut coverage result: Function Coverage: 35.74% (9009/25206) Line Coverage: 27.36% (74560/272474) Region Coverage: 26.59% (38573/145085) Branch Coverage: 23.44% (19667/83894) Coverage Report: http://coverage.selectdb-in.cc/coverage/9bfdaf470f6bd2ffa2ad2d2224e7b5ff18f71a1d_9bfdaf470f6bd2ffa2ad2d2224e7b5ff18f71a1d/report/index.html

doris-robot avatar May 27 '24 07:05 doris-robot

run performance

jacktengg avatar May 27 '24 15:05 jacktengg

run buildall

jacktengg avatar May 28 '24 02:05 jacktengg

clang-tidy review says "All clean, LGTM! :+1:"

github-actions[bot] avatar May 28 '24 02:05 github-actions[bot]

clang-tidy review says "All clean, LGTM! :+1:"

github-actions[bot] avatar May 28 '24 02:05 github-actions[bot]

TeamCity be ut coverage result: Function Coverage: 35.75% (9009/25199) Line Coverage: 27.36% (74567/272575) Region Coverage: 26.57% (38562/145124) Branch Coverage: 23.44% (19665/83902) Coverage Report: http://coverage.selectdb-in.cc/coverage/cea170917d3916bfbc317b7f4352592372747de3_cea170917d3916bfbc317b7f4352592372747de3/report/index.html

doris-robot avatar May 28 '24 03:05 doris-robot

run performance

jacktengg avatar May 28 '24 07:05 jacktengg

TPC-H: Total hot run time: 41659 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cea170917d3916bfbc317b7f4352592372747de3, data reload: false

------ Round 1 ----------------------------------
q1	17614	4326	4323	4323
q2	2033	199	192	192
q3	10481	1262	1256	1256
q4	10200	878	836	836
q5	7468	2874	2769	2769
q6	235	132	135	132
q7	973	645	607	607
q8	9215	2160	2155	2155
q9	9515	6709	6699	6699
q10	8992	3872	3860	3860
q11	471	257	255	255
q12	440	226	228	226
q13	17205	3269	3203	3203
q14	271	232	221	221
q15	531	466	471	466
q16	487	397	404	397
q17	1008	740	773	740
q18	8508	7965	7824	7824
q19	8356	1578	1555	1555
q20	642	319	320	319
q21	5175	3348	4046	3348
q22	354	276	291	276
Total cold run time: 120174 ms
Total hot run time: 41659 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4633	4357	4408	4357
q2	371	273	291	273
q3	3145	2982	2929	2929
q4	1952	1669	1746	1669
q5	5315	5457	5533	5457
q6	214	125	126	125
q7	2224	1855	1802	1802
q8	3298	3472	3453	3453
q9	8713	8669	8710	8669
q10	4097	3824	3715	3715
q11	593	490	504	490
q12	780	627	665	627
q13	16972	3170	3110	3110
q14	309	261	271	261
q15	534	473	477	473
q16	494	425	442	425
q17	1842	1530	1508	1508
q18	7940	7608	7580	7580
q19	1682	1569	1559	1559
q20	1970	1829	1823	1823
q21	4974	4824	4738	4738
q22	578	497	508	497
Total cold run time: 72630 ms
Total hot run time: 55540 ms

doris-robot avatar May 28 '24 08:05 doris-robot

TPC-DS: Total hot run time: 171880 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cea170917d3916bfbc317b7f4352592372747de3, data reload: false

query1	922	380	370	370
query2	6570	2350	2262	2262
query3	6652	204	205	204
query4	19907	17467	17354	17354
query5	4164	426	436	426
query6	250	163	153	153
query7	4585	304	295	295
query8	247	180	181	180
query9	8460	2446	2431	2431
query10	458	285	268	268
query11	10746	10178	10160	10160
query12	138	97	87	87
query13	1654	372	371	371
query14	9317	6892	6767	6767
query15	205	166	164	164
query16	7126	262	265	262
query17	1126	539	511	511
query18	1948	269	270	269
query19	205	160	151	151
query20	88	88	85	85
query21	203	127	139	127
query22	4345	4017	4017	4017
query23	33850	33282	33090	33090
query24	5045	2788	2894	2788
query25	463	354	365	354
query26	698	155	157	155
query27	1840	319	323	319
query28	3816	2098	2124	2098
query29	862	600	594	594
query30	208	147	152	147
query31	912	770	747	747
query32	59	52	53	52
query33	395	270	263	263
query34	851	486	479	479
query35	705	597	595	595
query36	1070	908	944	908
query37	104	66	69	66
query38	2881	2815	2791	2791
query39	834	802	807	802
query40	198	128	124	124
query41	47	44	43	43
query42	102	95	93	93
query43	587	521	568	521
query44	1097	740	755	740
query45	172	162	161	161
query46	1051	734	719	719
query47	1867	1778	1795	1778
query48	376	308	298	298
query49	759	380	385	380
query50	767	391	397	391
query51	6874	6851	6670	6670
query52	101	93	93	93
query53	350	281	285	281
query54	543	438	429	429
query55	75	71	72	71
query56	251	237	245	237
query57	1146	1043	1056	1043
query58	217	225	221	221
query59	3266	3192	3061	3061
query60	278	248	270	248
query61	90	88	88	88
query62	541	440	443	440
query63	304	287	293	287
query64	2532	1770	1726	1726
query65	3218	3071	3117	3071
query66	668	326	329	326
query67	15106	14891	14951	14891
query68	4520	560	555	555
query69	445	271	271	271
query70	1133	1139	1163	1139
query71	388	276	280	276
query72	7599	5737	4989	4989
query73	710	329	331	329
query74	6015	5593	5641	5593
query75	3336	2612	2599	2599
query76	2186	965	978	965
query77	383	265	267	265
query78	10243	9770	9592	9592
query79	2553	511	520	511
query80	1087	446	440	440
query81	523	222	218	218
query82	916	93	93	93
query83	257	173	173	173
query84	271	85	88	85
query85	930	267	273	267
query86	458	320	301	301
query87	3284	3109	3150	3109
query88	4099	2443	2457	2443
query89	497	380	384	380
query90	2028	188	189	188
query91	123	99	96	96
query92	67	49	51	49
query93	1980	532	515	515
query94	1217	189	195	189
query95	413	315	309	309
query96	593	270	270	270
query97	3196	3015	3006	3006
query98	236	220	211	211
query99	1207	876	861	861
Total cold run time: 249385 ms
Total hot run time: 171880 ms

doris-robot avatar May 28 '24 08:05 doris-robot

ClickBench: Total hot run time: 30.81 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cea170917d3916bfbc317b7f4352592372747de3, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.07	0.07
query4	1.66	0.09	0.09
query5	0.51	0.49	0.49
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.54	0.53	0.53
query11	0.16	0.12	0.11
query12	0.14	0.11	0.11
query13	0.59	0.59	0.60
query14	0.80	0.79	0.79
query15	0.83	0.80	0.82
query16	0.38	0.36	0.35
query17	0.96	0.95	0.96
query18	0.20	0.25	0.24
query19	1.76	1.71	1.70
query20	0.01	0.01	0.01
query21	15.60	0.68	0.65
query22	4.58	6.87	2.00
query23	18.32	1.40	1.30
query24	1.55	0.26	0.27
query25	0.15	0.09	0.08
query26	0.27	0.16	0.16
query27	0.09	0.08	0.07
query28	13.40	1.01	1.00
query29	12.88	3.45	3.41
query30	0.24	0.05	0.05
query31	2.88	0.37	0.37
query32	3.31	0.48	0.47
query33	2.84	2.97	2.90
query34	17.17	4.43	4.45
query35	4.50	4.45	4.57
query36	0.68	0.51	0.48
query37	0.17	0.14	0.16
query38	0.15	0.14	0.15
query39	0.04	0.04	0.03
query40	0.16	0.13	0.14
query41	0.10	0.05	0.04
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.8 s
Total hot run time: 30.81 s

doris-robot avatar May 28 '24 08:05 doris-robot

PR approved by at least one committer and no changes requested.

github-actions[bot] avatar May 28 '24 09:05 github-actions[bot]

PR approved by anyone and no changes requested.

github-actions[bot] avatar May 28 '24 09:05 github-actions[bot]