doris
doris copied to clipboard
apache
[feature](rowset) To support rowset on remote FS using paths of different format
Proposed changes
The current storage path format for remote segment files in Doris is:
${prefix}/data/${tablet_id}/${rowset_id}_${seg_id}.dat
There are the following issues with this storage path on HDFS:
- HDFS has a limitation on the number of directory entries in a directory, typically not exceeding 1 million (dfs.namenode.fs-limits.max-directory-items).
- The HDFS list interface does not have pagination functionality, resulting in high overhead when listing a large number of files.
- The Recycler needs to support recycling rowsets with an unknown number of segments, but efficiently deleting all files with the
rowset_idprefix is challenging in HDFS.
A more HDFS-friendly path format would be:
${prefix}/data/${shard_id}/${tablet_id}/${rowset_id}/${seg_id}.dat
To support the aforementioned path format while maintaining compatibility with existing data, all existing StorageResource / StorageVaults are assumed to use path v0 for reading and writing segment files. When reading or writing remote segment files, the corresponding file paths need to be obtained from the StorageResource.
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.
TeamCity be ut coverage result: Function Coverage: 36.35% (9248/25445) Line Coverage: 27.66% (75783/274001) Region Coverage: 26.88% (39239/145992) Branch Coverage: 23.62% (19906/84268) Coverage Report: http://coverage.selectdb-in.cc/coverage/8362d686f173745737ef596cb283bfe918c3b75d_8362d686f173745737ef596cb283bfe918c3b75d/report/index.html
TPC-H: Total hot run time: 40123 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8362d686f173745737ef596cb283bfe918c3b75d, data reload: false
------ Round 1 ----------------------------------
q1 17862 4487 4357 4357
q2 2648 197 197 197
q3 11498 1198 1138 1138
q4 10474 791 806 791
q5 7871 2749 2694 2694
q6 224 134 139 134
q7 968 627 600 600
q8 9585 2120 2096 2096
q9 8881 6526 6466 6466
q10 8826 3695 3750 3695
q11 444 258 235 235
q12 486 218 214 214
q13 17782 2957 2965 2957
q14 254 231 216 216
q15 512 485 479 479
q16 530 384 374 374
q17 973 782 761 761
q18 8039 7500 7357 7357
q19 7626 1561 1551 1551
q20 646 305 300 300
q21 4864 3187 3772 3187
q22 374 324 330 324
Total cold run time: 121367 ms
Total hot run time: 40123 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4341 4182 4187 4182
q2 383 257 275 257
q3 2953 2780 2703 2703
q4 1883 1557 1581 1557
q5 5256 5245 5239 5239
q6 215 125 126 125
q7 2118 1715 1700 1700
q8 3206 3301 3283 3283
q9 8308 8327 8240 8240
q10 3899 3722 3675 3675
q11 590 483 485 483
q12 779 574 612 574
q13 16451 2941 2989 2941
q14 288 259 251 251
q15 501 470 466 466
q16 482 410 430 410
q17 1776 1473 1478 1473
q18 7690 7441 7388 7388
q19 1674 1533 1534 1533
q20 1965 1767 1806 1767
q21 4843 4756 4646 4646
q22 621 537 523 523
Total cold run time: 70222 ms
Total hot run time: 53416 ms
TPC-DS: Total hot run time: 171640 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8362d686f173745737ef596cb283bfe918c3b75d, data reload: false
query1 927 380 364 364
query2 6461 2546 2242 2242
query3 6671 205 208 205
query4 19711 17273 17164 17164
query5 4180 407 410 407
query6 246 158 150 150
query7 4594 295 294 294
query8 311 283 284 283
query9 8594 2401 2361 2361
query10 459 304 259 259
query11 10833 10033 10044 10033
query12 132 93 87 87
query13 1640 378 364 364
query14 8433 7841 7001 7001
query15 262 189 201 189
query16 8083 263 278 263
query17 1842 511 502 502
query18 2071 267 262 262
query19 191 153 147 147
query20 89 86 83 83
query21 189 130 129 129
query22 4277 3998 4012 3998
query23 33865 32807 33273 32807
query24 11579 2824 2837 2824
query25 642 360 352 352
query26 1776 152 150 150
query27 3010 317 316 316
query28 7555 2047 2047 2047
query29 991 612 588 588
query30 293 147 148 147
query31 1010 756 769 756
query32 87 51 53 51
query33 770 263 263 263
query34 1074 485 469 469
query35 735 594 595 594
query36 1097 935 880 880
query37 165 65 67 65
query38 2945 2839 2768 2768
query39 845 768 792 768
query40 273 124 127 124
query41 55 52 48 48
query42 102 98 94 94
query43 580 543 557 543
query44 1224 720 728 720
query45 183 173 175 173
query46 1071 696 686 686
query47 1883 1791 1795 1791
query48 358 291 293 291
query49 1198 377 389 377
query50 760 377 387 377
query51 6859 6762 6862 6762
query52 107 91 93 91
query53 352 280 291 280
query54 981 444 430 430
query55 77 71 72 71
query56 266 234 271 234
query57 1130 1024 1049 1024
query58 238 209 209 209
query59 3479 3132 3228 3132
query60 270 251 274 251
query61 88 83 90 83
query62 642 465 460 460
query63 310 299 295 295
query64 9776 2238 1690 1690
query65 3175 3086 3086 3086
query66 1375 339 329 329
query67 15281 14879 14738 14738
query68 4648 540 526 526
query69 457 265 266 265
query70 1116 1129 1107 1107
query71 396 280 258 258
query72 7697 5345 5874 5345
query73 712 330 323 323
query74 6083 5667 5619 5619
query75 3564 2640 2731 2640
query76 2896 1005 1017 1005
query77 402 259 299 259
query78 10651 9805 9768 9768
query79 2410 507 514 507
query80 1782 433 427 427
query81 542 221 219 219
query82 682 88 91 88
query83 271 166 161 161
query84 261 87 83 83
query85 2125 267 269 267
query86 509 299 312 299
query87 3259 3096 3101 3096
query88 4051 2351 2378 2351
query89 480 383 390 383
query90 2006 183 190 183
query91 129 97 97 97
query92 61 49 48 48
query93 2498 504 493 493
query94 1231 186 188 186
query95 413 308 305 305
query96 599 269 265 265
query97 3183 2984 3015 2984
query98 243 218 217 217
query99 1215 845 883 845
Total cold run time: 277300 ms
Total hot run time: 171640 ms
ClickBench: Total hot run time: 30.43 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8362d686f173745737ef596cb283bfe918c3b75d, data reload: false
query1 0.04 0.03 0.04
query2 0.08 0.04 0.04
query3 0.23 0.05 0.05
query4 1.68 0.07 0.06
query5 0.50 0.49 0.49
query6 1.13 0.72 0.72
query7 0.02 0.01 0.02
query8 0.05 0.04 0.04
query9 0.55 0.49 0.49
query10 0.54 0.55 0.54
query11 0.15 0.12 0.10
query12 0.14 0.12 0.12
query13 0.59 0.58 0.58
query14 0.75 0.77 0.76
query15 0.82 0.81 0.81
query16 0.35 0.36 0.35
query17 1.01 0.96 0.93
query18 0.22 0.23 0.25
query19 1.87 1.69 1.64
query20 0.01 0.01 0.01
query21 15.44 0.71 0.67
query22 4.35 6.90 1.93
query23 18.29 1.43 1.33
query24 1.70 0.30 0.21
query25 0.14 0.09 0.08
query26 0.26 0.17 0.16
query27 0.07 0.08 0.08
query28 13.30 1.01 1.00
query29 13.25 3.26 3.27
query30 0.24 0.05 0.05
query31 2.87 0.39 0.39
query32 3.31 0.49 0.46
query33 2.87 2.85 2.88
query34 17.20 4.42 4.45
query35 4.47 4.48 4.54
query36 0.67 0.48 0.45
query37 0.18 0.16 0.15
query38 0.15 0.15 0.15
query39 0.04 0.04 0.04
query40 0.16 0.14 0.16
query41 0.09 0.05 0.04
query42 0.05 0.06 0.06
query43 0.04 0.03 0.04
Total cold run time: 109.87 s
Total hot run time: 30.43 s
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
PR approved by at least one committer and no changes requested.
TPC-H: Total hot run time: 41358 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ecd95a946bc23b7c610ae3bbb4d5c29d1ba962c5, data reload: false
------ Round 1 ----------------------------------
q1 17616 4380 4300 4300
q2 2019 194 200 194
q3 10463 1323 1111 1111
q4 10189 874 766 766
q5 7483 2792 2738 2738
q6 224 132 136 132
q7 963 633 631 631
q8 9221 2173 2122 2122
q9 9309 6728 6702 6702
q10 9232 3940 3904 3904
q11 430 247 241 241
q12 521 241 238 238
q13 17238 3202 3279 3202
q14 280 232 241 232
q15 541 458 470 458
q16 474 388 379 379
q17 984 633 628 628
q18 8554 7967 7995 7967
q19 7601 1354 1387 1354
q20 664 337 327 327
q21 5197 3379 4112 3379
q22 414 359 353 353
Total cold run time: 119617 ms
Total hot run time: 41358 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4526 4498 4426 4426
q2 392 272 262 262
q3 3194 2938 2927 2927
q4 2012 1738 1636 1636
q5 5369 5538 5540 5538
q6 222 126 126 126
q7 2187 1868 1845 1845
q8 3231 3376 3426 3376
q9 8818 8706 8720 8706
q10 4136 3986 3865 3865
q11 569 489 472 472
q12 773 579 587 579
q13 16070 3133 3195 3133
q14 301 280 277 277
q15 520 480 479 479
q16 477 455 429 429
q17 1825 1530 1489 1489
q18 8157 7733 7403 7403
q19 1697 1614 1578 1578
q20 3042 1807 1781 1781
q21 4961 4709 4793 4709
q22 776 542 552 542
Total cold run time: 73255 ms
Total hot run time: 55578 ms
TeamCity be ut coverage result: Function Coverage: 36.35% (9257/25466) Line Coverage: 27.68% (75934/274331) Region Coverage: 26.91% (39336/146178) Branch Coverage: 23.63% (19927/84344) Coverage Report: http://coverage.selectdb-in.cc/coverage/ecd95a946bc23b7c610ae3bbb4d5c29d1ba962c5_ecd95a946bc23b7c610ae3bbb4d5c29d1ba962c5/report/index.html
TPC-DS: Total hot run time: 171991 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ecd95a946bc23b7c610ae3bbb4d5c29d1ba962c5, data reload: false
query1 949 389 372 372
query2 6485 2553 2402 2402
query3 6648 203 220 203
query4 19826 17516 17228 17228
query5 4150 456 454 454
query6 278 163 160 160
query7 4585 305 289 289
query8 330 290 287 287
query9 8426 2380 2411 2380
query10 442 287 271 271
query11 10499 10122 9996 9996
query12 138 91 86 86
query13 1629 367 356 356
query14 9509 6099 7566 6099
query15 233 185 187 185
query16 7725 265 258 258
query17 1343 521 504 504
query18 1940 271 271 271
query19 204 168 151 151
query20 93 84 88 84
query21 206 132 133 132
query22 4278 4235 4140 4140
query23 33484 32918 33031 32918
query24 11293 2855 2779 2779
query25 616 345 358 345
query26 1254 178 161 161
query27 3023 325 318 318
query28 7570 2081 2079 2079
query29 883 598 590 590
query30 286 155 151 151
query31 939 737 744 737
query32 86 54 54 54
query33 764 273 272 272
query34 956 466 474 466
query35 732 643 611 611
query36 1071 934 926 926
query37 155 67 70 67
query38 2861 2744 2733 2733
query39 833 778 784 778
query40 210 124 123 123
query41 59 51 50 50
query42 121 96 96 96
query43 603 548 536 536
query44 1293 728 743 728
query45 200 169 176 169
query46 1082 732 717 717
query47 1906 1792 1814 1792
query48 378 299 291 291
query49 1088 402 411 402
query50 800 378 392 378
query51 7021 6779 6871 6779
query52 106 89 98 89
query53 357 291 289 289
query54 833 454 439 439
query55 72 74 70 70
query56 273 253 261 253
query57 1126 1064 1039 1039
query58 259 278 254 254
query59 3653 3369 3282 3282
query60 289 275 268 268
query61 110 107 130 107
query62 635 448 431 431
query63 315 289 282 282
query64 8864 2188 1750 1750
query65 3148 3120 3166 3120
query66 819 321 330 321
query67 15655 14979 15151 14979
query68 4479 540 558 540
query69 464 361 307 307
query70 1124 1076 1100 1076
query71 384 284 302 284
query72 7011 5900 5483 5483
query73 738 327 323 323
query74 5838 5504 5543 5504
query75 3339 2659 2673 2659
query76 2345 894 968 894
query77 473 294 288 288
query78 10142 9806 9727 9727
query79 2215 514 514 514
query80 2268 467 454 454
query81 573 222 231 222
query82 1050 103 100 100
query83 299 175 170 170
query84 266 93 87 87
query85 1211 333 259 259
query86 461 310 297 297
query87 3307 3081 3099 3081
query88 3292 2346 2363 2346
query89 474 383 371 371
query90 1744 185 182 182
query91 126 95 97 95
query92 64 49 48 48
query93 2048 519 510 510
query94 1248 180 183 180
query95 403 311 307 307
query96 581 265 263 263
query97 3211 3006 3015 3006
query98 264 228 220 220
query99 1189 873 867 867
Total cold run time: 270628 ms
Total hot run time: 171991 ms
ClickBench: Total hot run time: 30.85 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ecd95a946bc23b7c610ae3bbb4d5c29d1ba962c5, data reload: false
query1 0.04 0.03 0.04
query2 0.08 0.04 0.04
query3 0.23 0.04 0.05
query4 1.67 0.07 0.07
query5 0.48 0.48 0.48
query6 1.12 0.73 0.72
query7 0.02 0.02 0.02
query8 0.05 0.04 0.04
query9 0.53 0.48 0.49
query10 0.54 0.54 0.56
query11 0.15 0.11 0.11
query12 0.14 0.12 0.12
query13 0.58 0.59 0.60
query14 0.78 0.77 0.78
query15 0.82 0.82 0.81
query16 0.35 0.35 0.37
query17 0.99 1.03 0.98
query18 0.23 0.24 0.23
query19 1.85 1.76 1.74
query20 0.02 0.01 0.02
query21 15.45 0.70 0.67
query22 5.13 5.87 2.15
query23 18.32 1.34 1.26
query24 2.04 0.20 0.21
query25 0.14 0.09 0.08
query26 0.26 0.18 0.17
query27 0.08 0.07 0.08
query28 13.30 1.05 1.01
query29 12.77 3.35 3.29
query30 0.24 0.06 0.05
query31 2.90 0.40 0.39
query32 3.26 0.48 0.47
query33 2.93 2.88 2.90
query34 17.23 4.41 4.43
query35 4.49 4.53 4.52
query36 0.69 0.48 0.48
query37 0.18 0.16 0.16
query38 0.15 0.15 0.14
query39 0.04 0.03 0.04
query40 0.15 0.14 0.14
query41 0.09 0.05 0.04
query42 0.05 0.04 0.04
query43 0.04 0.03 0.04
Total cold run time: 110.6 s
Total hot run time: 30.85 s
PR approved by at least one committer and no changes requested.