doris
doris copied to clipboard
apache
[enhance](hdfs) Add hdfs file writer benchmark and flush hdfs writer at some threshold
Proposed changes
Add hdfs file writer benchmark and flush hdfs writer at some threshold
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.
TPC-H: Total hot run time: 38721 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3e3bef918dd8cd4a13bd44bd109aba3fa11cf3d9, data reload: false
------ Round 1 ----------------------------------
q1 18982 4562 4413 4413
q2 2898 182 168 168
q3 11955 1135 1124 1124
q4 10680 841 865 841
q5 8123 2855 2756 2756
q6 221 137 134 134
q7 965 605 605 605
q8 9349 2221 2247 2221
q9 7265 6544 6571 6544
q10 7143 2491 2603 2491
q11 459 237 236 236
q12 409 225 223 223
q13 18782 2968 2983 2968
q14 287 239 237 237
q15 535 478 474 474
q16 501 385 376 376
q17 983 655 718 655
q18 7543 6897 6719 6719
q19 6328 1375 1406 1375
q20 680 320 326 320
q21 3684 2847 2882 2847
q22 1104 1039 994 994
Total cold run time: 118876 ms
Total hot run time: 38721 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4355 4286 4293 4286
q2 378 274 267 267
q3 2862 2648 2634 2634
q4 1928 1627 1588 1588
q5 5349 5391 5385 5385
q6 225 128 131 128
q7 2032 1684 1690 1684
q8 3172 3342 3341 3341
q9 8490 8418 8379 8379
q10 3348 3119 3178 3119
q11 608 519 490 490
q12 777 595 580 580
q13 17698 2993 2981 2981
q14 313 256 272 256
q15 521 482 481 481
q16 487 414 415 414
q17 1789 1476 1471 1471
q18 7572 7565 7321 7321
q19 1656 1449 1521 1449
q20 1982 1759 1785 1759
q21 5081 5132 5058 5058
q22 1081 990 978 978
Total cold run time: 71704 ms
Total hot run time: 54049 ms
TPC-DS: Total hot run time: 185020 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3e3bef918dd8cd4a13bd44bd109aba3fa11cf3d9, data reload: false
query1 903 379 382 379
query2 6455 1823 1811 1811
query3 6655 206 211 206
query4 34191 23180 23039 23039
query5 4148 484 473 473
query6 263 169 172 169
query7 4603 293 292 292
query8 257 203 209 203
query9 8761 2420 2417 2417
query10 429 279 267 267
query11 16492 14982 15007 14982
query12 150 98 97 97
query13 1624 356 373 356
query14 9516 6923 6978 6923
query15 254 176 177 176
query16 7947 475 478 475
query17 1576 556 545 545
query18 2023 293 286 286
query19 204 162 155 155
query20 115 111 108 108
query21 216 135 133 133
query22 4273 4028 4039 4028
query23 33871 33105 33196 33105
query24 12102 2771 2911 2771
query25 702 400 398 398
query26 1825 160 157 157
query27 2900 284 278 278
query28 7487 2076 2035 2035
query29 1156 664 635 635
query30 305 154 144 144
query31 979 773 759 759
query32 99 56 57 56
query33 751 302 294 294
query34 1002 499 466 466
query35 835 718 714 714
query36 1066 900 887 887
query37 290 81 85 81
query38 3900 3783 3914 3783
query39 1439 1400 1421 1400
query40 279 125 123 123
query41 49 46 46 46
query42 119 100 101 100
query43 508 454 468 454
query44 1168 723 750 723
query45 197 165 167 165
query46 1089 742 732 732
query47 1875 1741 1754 1741
query48 371 302 294 294
query49 1205 419 422 419
query50 807 395 393 393
query51 6798 6687 6601 6601
query52 109 96 98 96
query53 364 302 288 288
query54 1099 468 466 466
query55 78 75 75 75
query56 356 247 244 244
query57 1141 1065 1019 1019
query58 248 221 225 221
query59 2924 2777 2696 2696
query60 295 266 258 258
query61 97 98 100 98
query62 814 633 654 633
query63 326 287 283 283
query64 6223 2196 1646 1646
query65 3361 3143 3153 3143
query66 1381 322 360 322
query67 15392 14936 14892 14892
query68 7177 548 548 548
query69 647 427 321 321
query70 1125 1069 1112 1069
query71 500 277 273 273
query72 6549 2264 2000 2000
query73 841 318 318 318
query74 9136 8750 8723 8723
query75 3992 2717 2728 2717
query76 4515 1049 975 975
query77 669 299 302 299
query78 9504 9208 8954 8954
query79 7493 515 513 513
query80 2071 480 468 468
query81 594 227 221 221
query82 781 135 135 135
query83 263 142 157 142
query84 281 84 81 81
query85 1454 316 318 316
query86 409 307 281 281
query87 4328 4148 4203 4148
query88 4815 2307 2299 2299
query89 485 390 397 390
query90 1934 198 201 198
query91 132 99 102 99
query92 66 52 54 52
query93 5831 521 506 506
query94 1150 288 293 288
query95 409 318 315 315
query96 621 279 270 270
query97 3223 3017 3036 3017
query98 231 199 196 196
query99 1594 1234 1255 1234
Total cold run time: 313257 ms
Total hot run time: 185020 ms
PR approved by at least one committer and no changes requested.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!