doris
doris copied to clipboard
apache
[Fix](hive-writer) Fix crash when hive partition writer building partition update.
Proposed changes
Issue:
/home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
3# 0x00007F963FA9D090 in /lib/x86_64-linux-gnu/libc.so.6
4# doris::vectorized::VHivePartitionWriter::_build_partition_update() at /home/zcp/repo_center/doris_master/doris/be/src/vec/sink/writer/vhive_partition_writer.cpp:215
5# doris::vectorized::VHivePartitionWriter::close(doris::Status const&) at /home/zcp/repo_center/doris_master/doris/be/src/vec/sink/writer/vhive_partition_writer.cpp:164
6# doris::vectorized::VHiveTableWriter::close(doris::Status) at /home/zcp/repo_center/doris_master/doris/be/src/vec/sink/writer/vhive_table_writer.cpp:209
7# doris::vectorized::AsyncResultWriter::process_block(doris::RuntimeState*, doris::RuntimeProfile*) at /home/zcp/repo_center/doris_master/doris/be/src/vec/sink/writer/async_result_writer.cpp:184
8# doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0::operator()() const at
Root cause
Building hive partition update information use _file_format_transformer->written_len(), but _file_format_transformer will be nullptr when hivePartitionWriter open() failed (return error status).
Solution
- Build hive partition update information only when status is ok.
Further comments
If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...
Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR
Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.
TPC-H: Total hot run time: 39869 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e24fd301b23776b050a29e623fa33b4c293f3453, data reload: false
------ Round 1 ----------------------------------
q1 17621 4268 4220 4220
q2 2031 202 197 197
q3 10535 1241 1179 1179
q4 10332 757 809 757
q5 7528 2687 2680 2680
q6 229 132 136 132
q7 957 614 607 607
q8 9256 2091 2074 2074
q9 9362 6606 6567 6567
q10 9042 3709 3705 3705
q11 458 238 233 233
q12 446 221 218 218
q13 17754 3010 2993 2993
q14 264 223 221 221
q15 509 474 471 471
q16 518 375 374 374
q17 966 652 746 652
q18 8173 7401 7461 7401
q19 6776 1552 1511 1511
q20 644 315 299 299
q21 4930 3108 3895 3108
q22 359 272 270 270
Total cold run time: 118690 ms
Total hot run time: 39869 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4330 4211 4221 4211
q2 389 270 279 270
q3 3006 2758 2727 2727
q4 1847 1546 1584 1546
q5 5242 5270 5251 5251
q6 210 124 125 124
q7 2153 1720 1724 1720
q8 3160 3306 3302 3302
q9 8316 8302 8358 8302
q10 3901 3676 3672 3672
q11 576 483 498 483
q12 807 602 609 602
q13 17230 2958 3013 2958
q14 298 273 256 256
q15 515 483 467 467
q16 481 425 427 425
q17 1753 1492 1445 1445
q18 7684 7608 7449 7449
q19 1645 1586 1557 1557
q20 1974 1778 1769 1769
q21 4836 4713 4731 4713
q22 571 495 485 485
Total cold run time: 70924 ms
Total hot run time: 53734 ms
TeamCity be ut coverage result: Function Coverage: 35.65% (9018/25294) Line Coverage: 27.31% (74581/273046) Region Coverage: 26.55% (38607/145423) Branch Coverage: 23.41% (19694/84130) Coverage Report: http://coverage.selectdb-in.cc/coverage/e24fd301b23776b050a29e623fa33b4c293f3453_e24fd301b23776b050a29e623fa33b4c293f3453/report/index.html
TPC-DS: Total hot run time: 169482 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e24fd301b23776b050a29e623fa33b4c293f3453, data reload: false
query1 928 388 377 377
query2 6445 2529 2437 2437
query3 6655 210 209 209
query4 18938 17553 17429 17429
query5 4180 425 420 420
query6 245 153 152 152
query7 4599 295 294 294
query8 255 192 181 181
query9 8451 2373 2345 2345
query10 478 276 272 272
query11 10742 10194 10132 10132
query12 139 92 91 91
query13 1650 373 370 370
query14 10104 7713 6698 6698
query15 201 179 175 175
query16 7640 266 260 260
query17 1858 547 505 505
query18 1452 274 260 260
query19 202 151 152 151
query20 91 89 92 89
query21 195 125 126 125
query22 4133 4005 3862 3862
query23 33662 33091 32982 32982
query24 12132 2887 2863 2863
query25 665 357 372 357
query26 1700 161 157 157
query27 2828 327 314 314
query28 7459 2031 2022 2022
query29 1031 617 597 597
query30 287 173 177 173
query31 961 766 739 739
query32 98 53 53 53
query33 764 258 261 258
query34 1062 466 472 466
query35 743 627 595 595
query36 1036 918 914 914
query37 281 67 71 67
query38 2914 2819 2772 2772
query39 850 799 798 798
query40 270 124 126 124
query41 46 45 43 43
query42 103 95 97 95
query43 572 582 581 581
query44 1194 716 736 716
query45 182 164 165 164
query46 1071 727 744 727
query47 1860 1773 1791 1773
query48 377 296 308 296
query49 1179 377 375 375
query50 764 387 378 378
query51 6969 6824 6751 6751
query52 102 89 90 89
query53 347 284 280 280
query54 966 470 417 417
query55 76 74 75 74
query56 257 234 246 234
query57 1160 1056 1050 1050
query58 230 206 229 206
query59 3327 3491 3198 3198
query60 267 241 253 241
query61 89 86 85 85
query62 657 444 457 444
query63 305 283 282 282
query64 9827 2250 1700 1700
query65 3224 3094 3131 3094
query66 1291 331 330 330
query67 15421 15000 14820 14820
query68 4534 535 539 535
query69 457 272 270 270
query70 1122 1165 1084 1084
query71 387 266 268 266
query72 7980 5162 2758 2758
query73 712 328 321 321
query74 6063 5731 5592 5592
query75 3295 2620 2669 2620
query76 2741 981 960 960
query77 409 266 277 266
query78 10236 9769 9751 9751
query79 2117 512 518 512
query80 955 421 430 421
query81 518 249 243 243
query82 645 96 92 92
query83 245 166 170 166
query84 234 86 84 84
query85 1904 270 264 264
query86 509 321 295 295
query87 3301 3130 3142 3130
query88 4133 2336 2339 2336
query89 477 400 381 381
query90 1947 257 187 187
query91 123 98 98 98
query92 64 47 50 47
query93 2344 500 494 494
query94 1253 190 184 184
query95 399 306 307 306
query96 603 270 259 259
query97 3171 2983 2999 2983
query98 244 217 210 210
query99 1108 859 839 839
Total cold run time: 274417 ms
Total hot run time: 169482 ms
ClickBench: Total hot run time: 31.11 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e24fd301b23776b050a29e623fa33b4c293f3453, data reload: false
query1 0.04 0.04 0.04
query2 0.08 0.04 0.04
query3 0.23 0.05 0.05
query4 1.67 0.08 0.10
query5 0.50 0.49 0.52
query6 1.12 0.73 0.72
query7 0.02 0.02 0.01
query8 0.06 0.05 0.05
query9 0.55 0.49 0.49
query10 0.54 0.54 0.53
query11 0.16 0.11 0.11
query12 0.15 0.12 0.12
query13 0.59 0.58 0.59
query14 0.77 0.77 0.79
query15 0.83 0.81 0.80
query16 0.36 0.37 0.37
query17 0.99 1.01 1.01
query18 0.22 0.24 0.22
query19 1.74 1.76 1.69
query20 0.01 0.01 0.01
query21 15.72 0.68 0.66
query22 4.20 6.34 2.58
query23 18.32 1.48 1.21
query24 1.73 0.28 0.21
query25 0.16 0.08 0.09
query26 0.27 0.17 0.16
query27 0.08 0.09 0.08
query28 13.38 1.02 1.00
query29 13.21 3.27 3.24
query30 0.24 0.06 0.05
query31 2.87 0.40 0.39
query32 3.26 0.47 0.46
query33 2.92 2.88 2.88
query34 17.32 4.40 4.42
query35 4.55 4.47 4.51
query36 0.66 0.47 0.47
query37 0.18 0.17 0.15
query38 0.15 0.15 0.14
query39 0.05 0.04 0.04
query40 0.16 0.14 0.16
query41 0.10 0.05 0.04
query42 0.05 0.05 0.04
query43 0.04 0.04 0.04
Total cold run time: 110.25 s
Total hot run time: 31.11 s
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
TeamCity be ut coverage result: Function Coverage: 35.66% (9019/25295) Line Coverage: 27.31% (74589/273072) Region Coverage: 26.54% (38602/145441) Branch Coverage: 23.41% (19698/84144) Coverage Report: http://coverage.selectdb-in.cc/coverage/f747df6d5f14b38f176a6081ec1cbd186f3daefc_f747df6d5f14b38f176a6081ec1cbd186f3daefc/report/index.html
TPC-H: Total hot run time: 39987 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f747df6d5f14b38f176a6081ec1cbd186f3daefc, data reload: false
------ Round 1 ----------------------------------
q1 17907 4399 4367 4367
q2 2482 199 203 199
q3 11341 1126 1150 1126
q4 10206 752 757 752
q5 7517 2678 2678 2678
q6 222 137 139 137
q7 959 600 598 598
q8 9360 2102 2074 2074
q9 8723 6534 6445 6445
q10 9006 3691 3694 3691
q11 472 255 238 238
q12 511 218 219 218
q13 18789 3003 2962 2962
q14 266 221 242 221
q15 506 477 480 477
q16 551 390 385 385
q17 970 647 703 647
q18 8135 7486 7490 7486
q19 6326 1562 1498 1498
q20 656 322 313 313
q21 4944 3201 3376 3201
q22 338 274 285 274
Total cold run time: 120187 ms
Total hot run time: 39987 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4352 4234 4194 4194
q2 379 274 270 270
q3 2974 2816 2742 2742
q4 1849 1605 1546 1546
q5 5248 5433 5284 5284
q6 212 128 125 125
q7 2106 1720 1765 1720
q8 3166 3292 3295 3292
q9 8298 8302 8330 8302
q10 3929 3725 3712 3712
q11 580 491 512 491
q12 774 580 591 580
q13 17148 2994 2955 2955
q14 285 257 273 257
q15 509 483 490 483
q16 492 417 425 417
q17 1759 1514 1459 1459
q18 7648 7543 7597 7543
q19 1658 1538 1553 1538
q20 1995 1785 1802 1785
q21 4979 4758 4736 4736
q22 553 491 493 491
Total cold run time: 70893 ms
Total hot run time: 53922 ms
TPC-DS: Total hot run time: 171542 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f747df6d5f14b38f176a6081ec1cbd186f3daefc, data reload: false
query1 909 396 382 382
query2 6467 2345 2320 2320
query3 6650 209 206 206
query4 20460 17341 17443 17341
query5 4144 418 421 418
query6 245 151 149 149
query7 4586 298 291 291
query8 236 182 192 182
query9 8667 2369 2361 2361
query10 455 281 257 257
query11 10635 10030 10091 10030
query12 136 91 89 89
query13 1642 360 353 353
query14 9301 7530 6875 6875
query15 206 172 172 172
query16 7773 266 262 262
query17 1351 521 525 521
query18 1925 276 269 269
query19 198 163 164 163
query20 95 93 85 85
query21 196 127 130 127
query22 4199 4104 3942 3942
query23 33569 32968 32889 32889
query24 12271 2901 2781 2781
query25 662 353 363 353
query26 1728 164 158 158
query27 2940 320 332 320
query28 7340 2030 2026 2026
query29 1047 628 613 613
query30 309 172 178 172
query31 966 795 735 735
query32 89 53 58 53
query33 763 262 257 257
query34 1050 484 479 479
query35 739 604 583 583
query36 1060 906 924 906
query37 142 69 71 69
query38 2900 2783 2761 2761
query39 845 777 797 777
query40 271 128 124 124
query41 47 44 46 44
query42 103 96 97 96
query43 571 566 530 530
query44 1221 714 732 714
query45 180 164 171 164
query46 1074 720 708 708
query47 1819 1765 1790 1765
query48 407 290 293 290
query49 1158 372 386 372
query50 760 378 384 378
query51 6876 6828 6684 6684
query52 99 89 91 89
query53 347 289 278 278
query54 1009 420 417 417
query55 72 71 73 71
query56 262 241 251 241
query57 1114 1032 1069 1032
query58 237 212 207 207
query59 3376 3203 3132 3132
query60 280 268 282 268
query61 111 104 107 104
query62 648 461 452 452
query63 314 288 286 286
query64 9903 2202 1767 1767
query65 3192 3079 3150 3079
query66 1381 323 330 323
query67 15383 14743 15057 14743
query68 4643 527 534 527
query69 483 339 278 278
query70 1128 1148 1137 1137
query71 403 270 266 266
query72 7627 5777 5353 5353
query73 713 321 321 321
query74 5999 5610 5685 5610
query75 3557 2688 2577 2577
query76 2798 1013 933 933
query77 632 282 259 259
query78 10390 9770 9610 9610
query79 2962 510 516 510
query80 2126 463 433 433
query81 534 246 244 244
query82 1582 99 98 98
query83 351 180 176 176
query84 269 89 87 87
query85 1558 262 290 262
query86 474 303 316 303
query87 3363 3119 3181 3119
query88 4284 2314 2319 2314
query89 478 413 375 375
query90 1968 190 187 187
query91 131 107 178 107
query92 57 48 48 48
query93 3564 510 487 487
query94 1219 185 185 185
query95 399 316 314 314
query96 602 270 262 262
query97 3217 3006 3031 3006
query98 247 224 208 208
query99 1202 876 861 861
Total cold run time: 280071 ms
Total hot run time: 171542 ms
ClickBench: Total hot run time: 31.7 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f747df6d5f14b38f176a6081ec1cbd186f3daefc, data reload: false
query1 0.04 0.04 0.04
query2 0.08 0.03 0.04
query3 0.22 0.05 0.05
query4 1.69 0.06 0.07
query5 0.50 0.49 0.50
query6 1.12 0.72 0.73
query7 0.02 0.01 0.02
query8 0.05 0.04 0.05
query9 0.53 0.49 0.51
query10 0.53 0.55 0.55
query11 0.15 0.11 0.12
query12 0.15 0.13 0.12
query13 0.60 0.58 0.59
query14 0.77 0.78 0.79
query15 0.82 0.82 0.81
query16 0.36 0.37 0.37
query17 1.02 1.01 1.04
query18 0.21 0.23 0.24
query19 1.89 1.74 1.78
query20 0.02 0.01 0.01
query21 15.65 0.66 0.65
query22 4.77 5.03 3.08
query23 18.30 1.32 1.24
query24 1.74 0.26 0.21
query25 0.15 0.09 0.07
query26 0.25 0.16 0.17
query27 0.08 0.08 0.07
query28 13.35 1.02 1.00
query29 13.15 3.24 3.24
query30 0.24 0.07 0.06
query31 2.88 0.39 0.38
query32 3.30 0.48 0.47
query33 2.83 2.89 2.93
query34 17.19 4.41 4.42
query35 4.46 4.48 4.49
query36 0.65 0.50 0.46
query37 0.18 0.16 0.15
query38 0.16 0.14 0.14
query39 0.05 0.04 0.04
query40 0.16 0.15 0.16
query41 0.10 0.04 0.05
query42 0.06 0.04 0.05
query43 0.04 0.04 0.03
Total cold run time: 110.51 s
Total hot run time: 31.7 s
PR approved by at least one committer and no changes requested.