apache
[Configurations](multi-catalog) Add `enable_parquet_filter_by_min_max` and `enable_orc_filter_by_min_max` Session variables.
Proposed changes
backport #35012 #35320
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.
TPC-H: Total hot run time: 50219 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 320ccbc356850a049d04e572961627f169e8b883, data reload: false
------ Round 1 ----------------------------------
q1 17672 4550 4357 4357
q2 2067 153 142 142
q3 10452 1975 1978 1975
q4 10367 1303 1366 1303
q5 8652 3967 3930 3930
q6 235 127 129 127
q7 2048 1594 1628 1594
q8 9552 2761 2739 2739
q9 14321 10699 10574 10574
q10 8660 3545 3525 3525
q11 414 237 246 237
q12 479 298 298 298
q13 18325 3925 4022 3925
q14 350 331 329 329
q15 524 473 458 458
q16 671 575 571 571
q17 1166 952 908 908
q18 7170 6858 6844 6844
q19 1707 1648 1561 1561
q20 537 312 310 310
q21 4451 4203 4130 4130
q22 497 394 382 382
Total cold run time: 120317 ms
Total hot run time: 50219 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4324 4282 4259 4259
q2 324 223 222 222
q3 4148 4164 4163 4163
q4 2764 2741 2744 2741
q5 7244 7202 7203 7202
q6 242 122 120 120
q7 3264 2871 2850 2850
q8 4370 4498 4481 4481
q9 17440 17128 16999 16999
q10 4231 4259 4281 4259
q11 752 693 736 693
q12 1031 856 858 856
q13 7038 3725 3722 3722
q14 455 412 422 412
q15 514 453 449 449
q16 732 676 674 674
q17 3802 3861 3781 3781
q18 8711 8679 8665 8665
q19 1760 1688 1698 1688
q20 2407 2126 2088 2088
q21 8467 8493 8487 8487
q22 1035 911 934 911
Total cold run time: 85055 ms
Total hot run time: 79722 ms
TPC-DS: Total hot run time: 204355 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 320ccbc356850a049d04e572961627f169e8b883, data reload: false
query1 941 387 372 372
query2 6543 2857 2645 2645
query3 6925 208 208 208
query4 20113 18079 18090 18079
query5 19715 6519 6533 6519
query6 279 218 226 218
query7 4150 289 302 289
query8 267 242 276 242
query9 3129 2678 2599 2599
query10 409 308 299 299
query11 11472 10804 10844 10804
query12 120 78 73 73
query13 5577 692 668 668
query14 17798 13148 13393 13148
query15 360 227 221 221
query16 6468 273 258 258
query17 1766 1477 863 863
query18 2310 403 402 402
query19 198 145 145 145
query20 78 77 81 77
query21 186 100 92 92
query22 5268 4873 5042 4873
query23 32467 31837 31794 31794
query24 6978 6584 6512 6512
query25 524 433 429 429
query26 624 157 157 157
query27 2052 294 296 294
query28 6157 2363 2319 2319
query29 2817 2690 2732 2690
query30 241 162 162 162
query31 898 737 725 725
query32 72 62 60 60
query33 403 256 255 255
query34 839 465 488 465
query35 1133 924 937 924
query36 1656 1157 1331 1157
query37 93 62 60 60
query38 3050 2944 2927 2927
query39 1388 1315 1315 1315
query40 249 94 86 86
query41 49 44 35 35
query42 85 80 81 80
query43 700 634 666 634
query44 1127 707 722 707
query45 244 234 226 226
query46 1249 968 993 968
query47 1900 1659 1692 1659
query48 1014 720 699 699
query49 623 359 380 359
query50 871 626 603 603
query51 4730 4664 4684 4664
query52 92 78 80 78
query53 448 314 318 314
query54 2641 2465 2495 2465
query55 89 83 84 83
query56 224 240 229 229
query57 1208 1113 1082 1082
query58 210 210 206 206
query59 4146 3992 3924 3924
query60 203 183 187 183
query61 91 84 83 83
query62 782 503 459 459
query63 479 342 333 333
query64 2625 1489 1534 1489
query65 3605 3546 3534 3534
query66 800 376 393 376
query67 16066 15484 15933 15484
query68 9311 670 660 660
query69 559 336 354 336
query70 1837 1449 1562 1449
query71 420 301 320 301
query72 6381 3439 3427 3427
query73 740 330 335 330
query74 6295 5934 5815 5815
query75 5411 3802 3764 3764
query76 5901 1145 1159 1145
query77 983 256 268 256
query78 12591 12838 12611 12611
query79 11980 668 673 668
query80 778 393 422 393
query81 489 235 233 233
query82 536 94 96 94
query83 165 137 133 133
query84 255 70 69 69
query85 750 309 297 297
query86 330 296 340 296
query87 3246 3020 3053 3020
query88 5120 2366 2396 2366
query89 399 296 296 296
query90 1934 216 206 206
query91 193 136 145 136
query92 58 53 54 53
query93 4234 538 585 538
query94 710 206 201 201
query95 1098 1058 1046 1046
query96 659 334 344 334
query97 6571 6464 6473 6464
query98 195 189 177 177
query99 2909 862 860 860
Total cold run time: 315052 ms
Total hot run time: 204355 ms
ClickBench: Total hot run time: 30.46 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 320ccbc356850a049d04e572961627f169e8b883, data reload: false
query1 0.02 0.02 0.02
query2 0.07 0.02 0.02
query3 0.24 0.05 0.04
query4 1.80 0.06 0.06
query5 0.55 0.52 0.52
query6 1.27 0.62 0.67
query7 0.02 0.01 0.00
query8 0.03 0.02 0.02
query9 0.52 0.47 0.48
query10 0.54 0.53 0.55
query11 0.11 0.08 0.08
query12 0.11 0.09 0.09
query13 0.62 0.62 0.60
query14 0.78 0.79 0.77
query15 0.78 0.76 0.76
query16 0.37 0.39 0.37
query17 1.00 0.97 1.03
query18 0.22 0.26 0.23
query19 1.95 1.79 1.88
query20 0.01 0.01 0.01
query21 15.50 0.56 0.56
query22 2.03 2.07 1.54
query23 17.51 1.09 1.02
query24 3.41 0.89 1.62
query25 0.36 0.15 0.04
query26 0.44 0.15 0.15
query27 0.04 0.04 0.04
query28 9.09 0.71 0.72
query29 12.79 2.25 2.31
query30 0.56 0.58 0.52
query31 2.81 0.38 0.37
query32 3.38 0.49 0.50
query33 3.08 3.03 3.08
query34 15.22 4.81 4.79
query35 4.87 4.84 4.84
query36 1.06 1.01 1.00
query37 0.06 0.05 0.04
query38 0.03 0.02 0.02
query39 0.02 0.02 0.01
query40 0.16 0.14 0.14
query41 0.06 0.01 0.02
query42 0.02 0.02 0.01
query43 0.02 0.02 0.02
Total cold run time: 103.53 s
Total hot run time: 30.46 s
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Load test result on commit 320ccbc356850a049d04e572961627f169e8b883 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.2 seconds inserted 10000000 Rows, about 471K ops/s
TeamCity be ut coverage result: Function Coverage: 37.79% (8076/21369) Line Coverage: 29.45% (65942/223928) Region Coverage: 28.92% (33954/117409) Branch Coverage: 24.77% (17424/70332) Coverage Report: http://coverage.selectdb-in.cc/coverage/320ccbc356850a049d04e572961627f169e8b883_320ccbc356850a049d04e572961627f169e8b883/report/index.html
TeamCity be ut coverage result: Function Coverage: 37.79% (8078/21378) Line Coverage: 29.44% (65956/224014) Region Coverage: 28.93% (33995/117491) Branch Coverage: 24.80% (17456/70394) Coverage Report: http://coverage.selectdb-in.cc/coverage/b77d820fc52d922617a30038b9f8a2a55675ea84_b77d820fc52d922617a30038b9f8a2a55675ea84/report/index.html
TeamCity be ut coverage result: Function Coverage: 37.80% (8080/21378) Line Coverage: 29.45% (65975/224014) Region Coverage: 28.95% (34010/117491) Branch Coverage: 24.81% (17464/70394) Coverage Report: http://coverage.selectdb-in.cc/coverage/9076f9d9b5640886f380179e12b59c6a6a5c040e_9076f9d9b5640886f380179e12b59c6a6a5c040e/report/index.html
TeamCity be ut coverage result: Function Coverage: 37.78% (8078/21379) Line Coverage: 29.44% (65953/224028) Region Coverage: 28.93% (33994/117500) Branch Coverage: 24.79% (17453/70398) Coverage Report: http://coverage.selectdb-in.cc/coverage/11a859af14747f485712fea3780c26381b34d7a2_11a859af14747f485712fea3780c26381b34d7a2/report/index.html