doris
doris copied to clipboard
apache
[opt](scan) Use lazy-init for segment iterators and avoid caching all segments in the rowset reader
Proposed changes
Issue Number: close #xxx
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
TPC-H: Total hot run time: 41019 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5bf6523ec583f8ea240dac4676421c5086be8b97, data reload: false
------ Round 1 ----------------------------------
q1 17598 4366 4258 4258
q2 2011 189 193 189
q3 10500 1272 1196 1196
q4 10197 846 767 767
q5 7536 2789 2729 2729
q6 222 139 133 133
q7 980 605 610 605
q8 9228 2137 2121 2121
q9 9304 6761 6664 6664
q10 9631 3941 3847 3847
q11 449 247 237 237
q12 485 236 219 219
q13 17223 3192 3252 3192
q14 283 236 230 230
q15 507 467 474 467
q16 506 375 383 375
q17 994 729 708 708
q18 8473 7830 7696 7696
q19 1687 1569 1580 1569
q20 647 309 297 297
q21 5211 3235 3361 3235
q22 358 288 285 285
Total cold run time: 114030 ms
Total hot run time: 41019 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4482 4406 4401 4401
q2 382 273 263 263
q3 3147 2975 2859 2859
q4 2016 1749 1678 1678
q5 5305 5532 5463 5463
q6 213 123 125 123
q7 2173 1848 1807 1807
q8 3270 3729 3431 3431
q9 8638 8643 8698 8643
q10 4127 3844 3690 3690
q11 602 497 481 481
q12 768 597 640 597
q13 17078 3134 3128 3128
q14 316 274 257 257
q15 547 473 479 473
q16 497 447 432 432
q17 1811 1528 1517 1517
q18 8015 7582 7265 7265
q19 5359 1521 1605 1521
q20 2016 1791 1792 1791
q21 4984 4595 4773 4595
q22 577 499 482 482
Total cold run time: 76323 ms
Total hot run time: 54897 ms
TeamCity be ut coverage result: Function Coverage: 35.74% (9008/25205) Line Coverage: 27.37% (74558/272448) Region Coverage: 26.58% (38570/145093) Branch Coverage: 23.44% (19663/83896) Coverage Report: http://coverage.selectdb-in.cc/coverage/5bf6523ec583f8ea240dac4676421c5086be8b97_5bf6523ec583f8ea240dac4676421c5086be8b97/report/index.html
TPC-DS: Total hot run time: 168376 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5bf6523ec583f8ea240dac4676421c5086be8b97, data reload: false
query1 924 385 370 370
query2 6565 2360 2381 2360
query3 6643 201 198 198
query4 19690 17327 17227 17227
query5 4197 416 431 416
query6 251 158 147 147
query7 4575 295 294 294
query8 241 190 179 179
query9 8566 2310 2331 2310
query10 464 279 259 259
query11 10475 10231 10177 10177
query12 136 86 87 86
query13 1640 365 357 357
query14 10422 6339 7143 6339
query15 216 167 166 166
query16 7577 257 250 250
query17 1663 522 524 522
query18 1902 288 265 265
query19 191 156 159 156
query20 90 85 87 85
query21 199 139 129 129
query22 4232 3906 3878 3878
query23 33589 32950 32850 32850
query24 10549 2838 2756 2756
query25 590 355 364 355
query26 704 154 151 151
query27 2220 314 316 314
query28 5741 2007 2000 2000
query29 850 604 592 592
query30 242 148 151 148
query31 954 739 741 739
query32 92 55 51 51
query33 648 291 259 259
query34 897 474 468 468
query35 719 592 595 592
query36 1075 917 870 870
query37 101 66 64 64
query38 2935 2748 2793 2748
query39 841 796 788 788
query40 199 125 123 123
query41 47 45 46 45
query42 103 95 96 95
query43 579 533 541 533
query44 1118 725 738 725
query45 174 160 162 160
query46 1074 720 713 713
query47 1885 1789 1808 1789
query48 376 301 292 292
query49 847 370 375 370
query50 781 391 405 391
query51 6832 6656 6550 6550
query52 104 86 90 86
query53 352 274 287 274
query54 842 429 415 415
query55 71 70 68 68
query56 260 245 244 244
query57 1133 1028 1084 1028
query58 225 217 209 209
query59 3332 3208 3191 3191
query60 303 260 258 258
query61 115 93 90 90
query62 628 448 453 448
query63 308 277 281 277
query64 8539 2214 1714 1714
query65 3180 3067 3110 3067
query66 792 328 320 320
query67 15212 14916 14833 14833
query68 4490 531 525 525
query69 443 264 304 264
query70 1195 1147 1161 1147
query71 366 265 265 265
query72 7206 5780 2744 2744
query73 717 328 318 318
query74 6079 5635 5602 5602
query75 3288 2641 2601 2601
query76 2309 1042 994 994
query77 401 260 263 260
query78 10220 9790 9858 9790
query79 1572 507 500 500
query80 1071 436 426 426
query81 544 224 221 221
query82 905 89 91 89
query83 242 171 166 166
query84 236 88 87 87
query85 1227 329 353 329
query86 438 296 326 296
query87 3338 3123 3146 3123
query88 3233 2414 2400 2400
query89 482 383 375 375
query90 1962 187 187 187
query91 121 95 97 95
query92 56 52 47 47
query93 1490 506 488 488
query94 1262 181 179 179
query95 403 305 300 300
query96 590 267 273 267
query97 3204 3031 3000 3000
query98 242 222 226 222
query99 1209 846 850 846
Total cold run time: 263558 ms
Total hot run time: 168376 ms
ClickBench: Total hot run time: 30.37 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5bf6523ec583f8ea240dac4676421c5086be8b97, data reload: false
query1 0.04 0.03 0.03
query2 0.08 0.04 0.04
query3 0.23 0.05 0.05
query4 1.69 0.07 0.07
query5 0.50 0.50 0.50
query6 1.13 0.72 0.72
query7 0.02 0.01 0.02
query8 0.05 0.04 0.04
query9 0.54 0.48 0.48
query10 0.55 0.55 0.51
query11 0.16 0.11 0.12
query12 0.15 0.12 0.12
query13 0.60 0.61 0.60
query14 0.75 0.79 0.77
query15 0.83 0.80 0.81
query16 0.36 0.37 0.34
query17 1.02 0.94 0.97
query18 0.24 0.21 0.26
query19 1.74 1.71 1.67
query20 0.02 0.01 0.01
query21 15.72 0.65 0.65
query22 5.06 6.11 1.93
query23 18.33 1.32 1.28
query24 1.90 0.24 0.20
query25 0.14 0.08 0.08
query26 0.27 0.17 0.17
query27 0.08 0.08 0.09
query28 13.30 1.02 1.00
query29 13.78 3.34 3.28
query30 0.24 0.06 0.06
query31 2.86 0.38 0.38
query32 3.30 0.46 0.47
query33 2.88 2.86 2.87
query34 17.01 4.39 4.43
query35 4.47 4.49 4.62
query36 0.66 0.46 0.46
query37 0.18 0.15 0.15
query38 0.16 0.14 0.14
query39 0.05 0.04 0.03
query40 0.15 0.14 0.14
query41 0.09 0.06 0.04
query42 0.05 0.05 0.04
query43 0.04 0.04 0.04
Total cold run time: 111.42 s
Total hot run time: 30.37 s
TeamCity be ut coverage result: Function Coverage: 35.77% (9014/25198) Line Coverage: 27.37% (74592/272513) Region Coverage: 26.59% (38590/145125) Branch Coverage: 23.45% (19674/83894) Coverage Report: http://coverage.selectdb-in.cc/coverage/545fd639f58a6f730ef5fece1fff8331a831394d_545fd639f58a6f730ef5fece1fff8331a831394d/report/index.html
TPC-H: Total hot run time: 40625 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 545fd639f58a6f730ef5fece1fff8331a831394d, data reload: false
------ Round 1 ----------------------------------
q1 18027 4369 4373 4369
q2 2587 212 210 210
q3 12418 1198 1136 1136
q4 11206 904 805 805
q5 7623 2803 2606 2606
q6 214 131 133 131
q7 940 610 599 599
q8 9220 2052 2065 2052
q9 8692 6491 6473 6473
q10 8950 3687 3664 3664
q11 475 243 244 243
q12 412 219 221 219
q13 17767 2955 2976 2955
q14 268 222 215 215
q15 503 468 460 460
q16 537 381 372 372
q17 966 682 725 682
q18 8004 7558 7478 7478
q19 2729 1559 1505 1505
q20 654 301 306 301
q21 4890 3875 3948 3875
q22 337 284 275 275
Total cold run time: 117419 ms
Total hot run time: 40625 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4302 4206 4146 4146
q2 371 274 284 274
q3 2987 2772 2724 2724
q4 1883 1521 1612 1521
q5 5268 5270 5261 5261
q6 215 123 126 123
q7 2153 1740 1715 1715
q8 3179 3286 3288 3286
q9 8331 8288 8336 8288
q10 3911 3657 3641 3641
q11 588 505 480 480
q12 773 621 593 593
q13 16986 2972 2971 2971
q14 302 256 289 256
q15 513 497 497 497
q16 463 418 418 418
q17 1754 1486 1484 1484
q18 7755 7618 7435 7435
q19 1652 1564 1552 1552
q20 1978 1771 1812 1771
q21 4885 4691 4877 4691
q22 566 501 478 478
Total cold run time: 70815 ms
Total hot run time: 53605 ms
TPC-DS: Total hot run time: 169491 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 545fd639f58a6f730ef5fece1fff8331a831394d, data reload: false
query1 910 373 372 372
query2 6464 2275 2259 2259
query3 6648 203 205 203
query4 19381 17597 17410 17410
query5 4133 437 413 413
query6 258 156 153 153
query7 4589 299 295 295
query8 255 195 194 194
query9 8443 2381 2351 2351
query10 462 282 260 260
query11 10579 10151 10139 10139
query12 145 94 89 89
query13 1654 379 384 379
query14 10138 6819 7248 6819
query15 211 171 171 171
query16 7596 272 276 272
query17 1823 546 531 531
query18 1509 301 288 288
query19 201 161 155 155
query20 97 88 88 88
query21 200 136 131 131
query22 4210 4263 4023 4023
query23 33852 33142 33232 33142
query24 11807 2870 2874 2870
query25 673 353 356 353
query26 1685 153 161 153
query27 2786 325 317 317
query28 7125 2035 2037 2035
query29 1031 617 598 598
query30 283 145 155 145
query31 965 776 760 760
query32 112 53 53 53
query33 758 264 263 263
query34 1059 474 468 468
query35 747 611 578 578
query36 1056 926 931 926
query37 267 64 65 64
query38 2944 2823 2809 2809
query39 863 776 785 776
query40 273 125 124 124
query41 48 45 44 44
query42 105 94 97 94
query43 589 546 562 546
query44 1286 740 741 740
query45 178 163 170 163
query46 1068 733 744 733
query47 1853 1778 1778 1778
query48 385 293 293 293
query49 1204 381 380 380
query50 765 377 393 377
query51 6866 6635 6741 6635
query52 100 92 86 86
query53 346 290 286 286
query54 1011 426 415 415
query55 80 71 72 71
query56 263 240 237 237
query57 1123 1033 1030 1030
query58 242 216 225 216
query59 3395 3087 3122 3087
query60 278 285 261 261
query61 92 84 86 84
query62 668 447 461 447
query63 306 280 276 276
query64 9789 2181 1723 1723
query65 3209 3119 3124 3119
query66 1299 327 381 327
query67 15341 14911 14717 14717
query68 4568 532 532 532
query69 445 262 262 262
query70 1093 1090 1046 1046
query71 394 264 270 264
query72 7542 2734 2561 2561
query73 711 320 322 320
query74 6069 5649 5546 5546
query75 3376 2675 2635 2635
query76 2898 1040 948 948
query77 406 263 263 263
query78 10271 10039 9815 9815
query79 2156 505 500 500
query80 1213 457 469 457
query81 506 222 215 215
query82 685 94 89 89
query83 232 170 171 170
query84 231 86 96 86
query85 1925 280 265 265
query86 496 298 310 298
query87 3342 3169 3102 3102
query88 4172 2443 2439 2439
query89 484 388 385 385
query90 1996 189 189 189
query91 125 100 102 100
query92 61 49 50 49
query93 2182 511 494 494
query94 1226 193 191 191
query95 405 312 310 310
query96 590 277 270 270
query97 3184 3007 3015 3007
query98 239 223 218 218
query99 1163 880 871 871
Total cold run time: 274467 ms
Total hot run time: 169491 ms
PR approved by at least one committer and no changes requested.
TPC-H: Total hot run time: 40107 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4d61775e3369777d84497fa8fc3f04d5a8aa8776, data reload: false
------ Round 1 ----------------------------------
q1 17611 5135 4268 4268
q2 2025 190 198 190
q3 10453 1193 1191 1191
q4 10151 802 854 802
q5 7474 2715 2660 2660
q6 214 130 129 129
q7 935 586 594 586
q8 9217 2092 2103 2092
q9 8710 6547 6496 6496
q10 8974 3773 3738 3738
q11 476 253 232 232
q12 457 218 217 217
q13 17783 3022 2989 2989
q14 268 219 220 219
q15 507 470 477 470
q16 537 382 384 382
q17 984 661 783 661
q18 8063 7481 7492 7481
q19 5743 1555 1549 1549
q20 660 313 309 309
q21 4966 3927 3163 3163
q22 351 286 283 283
Total cold run time: 116559 ms
Total hot run time: 40107 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4335 4207 4194 4194
q2 381 297 266 266
q3 3002 2736 2775 2736
q4 1882 1636 1595 1595
q5 5257 5282 5282 5282
q6 214 124 122 122
q7 2096 1786 1739 1739
q8 3169 3315 3319 3315
q9 8325 8373 8346 8346
q10 3898 3702 3667 3667
q11 650 507 487 487
q12 745 590 593 590
q13 17398 2985 3002 2985
q14 285 252 262 252
q15 517 466 470 466
q16 479 429 417 417
q17 1766 1484 1472 1472
q18 7766 7604 7485 7485
q19 1641 1476 1593 1476
q20 2000 1791 1777 1777
q21 4829 4741 4900 4741
q22 550 500 481 481
Total cold run time: 71185 ms
Total hot run time: 53891 ms
TPC-DS: Total hot run time: 169908 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4d61775e3369777d84497fa8fc3f04d5a8aa8776, data reload: false
query1 920 391 373 373
query2 6460 2490 2480 2480
query3 6647 202 213 202
query4 19269 17509 17405 17405
query5 4168 411 409 409
query6 249 167 157 157
query7 4587 303 299 299
query8 261 181 178 178
query9 8543 2338 2304 2304
query10 449 281 257 257
query11 10596 10323 10137 10137
query12 135 96 86 86
query13 1661 370 378 370
query14 8704 7681 7359 7359
query15 258 169 164 164
query16 8184 261 260 260
query17 1837 522 507 507
query18 2102 276 273 273
query19 200 160 156 156
query20 93 86 82 82
query21 198 127 131 127
query22 4316 4007 3908 3908
query23 33967 32995 33259 32995
query24 11655 2834 2838 2834
query25 660 361 361 361
query26 1753 155 156 155
query27 3000 320 315 315
query28 7721 2011 2011 2011
query29 1063 602 590 590
query30 304 150 158 150
query31 957 754 752 752
query32 97 50 58 50
query33 762 268 260 260
query34 1014 474 473 473
query35 741 598 599 598
query36 1071 931 922 922
query37 165 66 74 66
query38 2945 2800 2798 2798
query39 841 810 790 790
query40 274 123 128 123
query41 46 44 43 43
query42 104 95 101 95
query43 612 541 543 541
query44 1207 726 779 726
query45 185 164 162 162
query46 1062 727 700 700
query47 1892 1792 1773 1773
query48 375 292 299 292
query49 1170 383 383 383
query50 773 396 395 395
query51 6856 6621 6669 6621
query52 105 88 94 88
query53 352 285 281 281
query54 902 436 426 426
query55 73 74 71 71
query56 266 244 247 244
query57 1130 1058 1039 1039
query58 235 215 206 206
query59 3347 3218 3048 3048
query60 269 252 249 249
query61 91 94 90 90
query62 661 467 448 448
query63 308 282 281 281
query64 9732 2209 1804 1804
query65 3180 3089 3137 3089
query66 1392 323 325 323
query67 15230 15173 14753 14753
query68 4550 514 530 514
query69 437 288 270 270
query70 1152 1065 1138 1065
query71 403 261 267 261
query72 7437 2749 2580 2580
query73 704 323 319 319
query74 5948 5704 5601 5601
query75 3491 2626 2632 2626
query76 2880 1029 929 929
query77 424 264 268 264
query78 10222 9889 9874 9874
query79 1175 512 500 500
query80 856 437 439 437
query81 510 219 220 219
query82 534 93 91 91
query83 249 169 169 169
query84 242 85 88 85
query85 1323 266 276 266
query86 448 287 280 280
query87 3372 3197 3179 3179
query88 3981 2406 2420 2406
query89 455 383 381 381
query90 2005 189 187 187
query91 121 102 97 97
query92 59 49 50 49
query93 1586 505 492 492
query94 1192 233 186 186
query95 403 309 312 309
query96 591 269 264 264
query97 3217 3031 3054 3031
query98 223 226 213 213
query99 1105 857 872 857
Total cold run time: 271677 ms
Total hot run time: 169908 ms
ClickBench: Total hot run time: 30.96 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4d61775e3369777d84497fa8fc3f04d5a8aa8776, data reload: false
query1 0.04 0.03 0.04
query2 0.08 0.04 0.04
query3 0.23 0.04 0.05
query4 1.67 0.07 0.07
query5 0.49 0.49 0.50
query6 1.13 0.73 0.72
query7 0.02 0.01 0.01
query8 0.05 0.04 0.04
query9 0.53 0.48 0.47
query10 0.53 0.54 0.53
query11 0.16 0.11 0.11
query12 0.14 0.12 0.12
query13 0.59 0.58 0.60
query14 0.79 0.77 0.76
query15 0.82 0.81 0.82
query16 0.36 0.37 0.36
query17 0.96 0.97 1.01
query18 0.22 0.25 0.23
query19 1.74 1.72 1.65
query20 0.02 0.01 0.02
query21 15.75 0.66 0.65
query22 4.26 5.82 2.33
query23 18.26 1.39 1.38
query24 1.43 0.33 0.25
query25 0.14 0.08 0.09
query26 0.27 0.17 0.17
query27 0.08 0.08 0.08
query28 13.42 1.03 1.00
query29 13.19 3.28 3.30
query30 0.25 0.05 0.05
query31 2.85 0.38 0.38
query32 3.29 0.48 0.49
query33 2.87 2.89 2.88
query34 17.16 4.42 4.41
query35 4.53 4.48 4.52
query36 0.65 0.49 0.46
query37 0.18 0.16 0.15
query38 0.16 0.14 0.15
query39 0.04 0.04 0.03
query40 0.18 0.17 0.13
query41 0.09 0.04 0.04
query42 0.05 0.05 0.04
query43 0.04 0.03 0.03
Total cold run time: 109.71 s
Total hot run time: 30.96 s
TeamCity be ut coverage result: Function Coverage: 35.77% (9017/25206) Line Coverage: 27.37% (74600/272573) Region Coverage: 26.59% (38593/145149) Branch Coverage: 23.44% (19670/83904) Coverage Report: http://coverage.selectdb-in.cc/coverage/4d61775e3369777d84497fa8fc3f04d5a8aa8776_4d61775e3369777d84497fa8fc3f04d5a8aa8776/report/index.html