[Bug] be asan stack-buffer-overflow in thrift rpc doris::FrontendServiceClient::send_report

Open xiaokang opened this issue 2 years ago • 0 comments
Search before asking

[X] I had searched in the issues and found no similar issues.
Version

maser
What's Wrong?


start time: Thu 02 Mar 2023 07:10:20 PM CST
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0302 19:10:20.475816 1482042 doris_main.cpp:324] enable_fuzzy_mode is true, set fuzzy configs
=================================================================
==1482042==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fbb5a9cb848 at pc 0x557378f79b17 bp 0x7fbb5a9cb810 sp 0x7fbb5a9cafb8
WRITE of size 24 at 0x7fbb5a9cb848 thread T741 (TaskWorkerPool.)
    #0 0x557378f79b16 in __interceptor_sigaltstack.part.0 (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster7/be/lib/doris_be+0xa252b16)
    #1 0x557378fdf8bf in __asan::PlatformUnpoisonStacks() (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster7/be/lib/doris_be+0xa2b88bf)
    #2 0x557378fe5224 in __asan_handle_no_return (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster7/be/lib/doris_be+0xa2be224)
    #3 0x55737bb5894d in apache::thrift::protocol::TProtocol::decrementOutputRecursionDepth() /var/local/thirdparty/installed/include/thrift/protocol/TProtocol.h:576
    #4 0x55737bb5894d in apache::thrift::protocol::TOutputRecursionTracker::~TOutputRecursionTracker() /var/local/thirdparty/installed/include/thrift/protocol/TProtocol.h:648
    #5 0x55737bb5894d in doris::TTabletInfo::write(apache::thrift::protocol::TProtocol*) const /root/doris/gensrc/build/gen_cpp/MasterService_types.cpp:483
    #6 0x55737bb4ed88 in doris::TTablet::write(apache::thrift::protocol::TProtocol*) const /root/doris/gensrc/build/gen_cpp/MasterService_types.cpp:1215
    #7 0x55737bb553d6 in doris::TReportRequest::write(apache::thrift::protocol::TProtocol*) const /root/doris/gensrc/build/gen_cpp/MasterService_types.cpp:1930
    #8 0x55737b7a6c83 in doris::FrontendService_report_pargs::write(apache::thrift::protocol::TProtocol*) const /root/doris/gensrc/build/gen_cpp/FrontendService.cpp:1204
    #9 0x55737b7e0174 in doris::FrontendServiceClient::send_report(doris::TReportRequest const&) /root/doris/gensrc/build/gen_cpp/FrontendService.cpp:5566
    #10 0x55737b806e68 in doris::FrontendServiceClient::report(doris::TMasterResult&, doris::TReportRequest const&) /root/doris/gensrc/build/gen_cpp/FrontendService.cpp:5555
    #11 0x557379ce124f in doris::MasterServerClient::report(doris::TReportRequest const&, doris::TMasterResult*) /root/doris/be/src/agent/utils.cpp:109
    #12 0x557379c6bad4 in doris::TaskWorkerPool::_handle_report(doris::TReportRequest&, doris::TaskWorkerPool::ReportType) /root/doris/be/src/agent/task_worker_pool.cpp:1651
    #13 0x557379c72675 in doris::TaskWorkerPool::_report_tablet_worker_thread_callback() /root/doris/be/src/agent/task_worker_pool.cpp:1376
    #14 0x557379cc0e01 in void std::__invoke_impl<void, void (doris::TaskWorkerPool::*&)(), doris::TaskWorkerPool*&>(std::__invoke_memfun_deref, void (doris::TaskWorkerPool::*&)(), doris::TaskWorkerPool*&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:74
    #15 0x557379cc0e01 in std::enable_if<is_invocable_r_v<void, void (doris::TaskWorkerPool::*&)(), doris::TaskWorkerPool*&>, void>::type std::__invoke_r<void, void (doris::TaskWorkerPool::*&)(), doris::TaskWorkerPool*&>(void (doris::TaskWorkerPool::*&)(), doris::TaskWorkerPool*&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:111
    #16 0x557379cc0e01 in void std::_Bind_result<void, void (doris::TaskWorkerPool::*(doris::TaskWorkerPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /var/local/ldb-toolchain/include/c++/11/functional:570
    #17 0x557379cc0e01 in void std::_Bind_result<void, void (doris::TaskWorkerPool::*(doris::TaskWorkerPool*))()>::operator()<>() /var/local/ldb-toolchain/include/c++/11/functional:629
    #18 0x557379cc0e01 in void std::__invoke_impl<void, std::_Bind_result<void, void (doris::TaskWorkerPool::*(doris::TaskWorkerPool*))()>&>(std::__invoke_other, std::_Bind_result<void, void (doris::TaskWorkerPool::*(doris::TaskWorkerPool*))()>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:61
    #19 0x557379cc0e01 in std::enable_if<is_invocable_r_v<void, std::_Bind_result<void, void (doris::TaskWorkerPool::*(doris::TaskWorkerPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind_result<void, void (doris::TaskWorkerPool::*(doris::TaskWorkerPool*))()>&>(std::_Bind_result<void, void (doris::TaskWorkerPool::*(doris::TaskWorkerPool*))()>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:111
    #20 0x557379cc0e01 in std::_Function_handler<void (), std::_Bind_result<void, void (doris::TaskWorkerPool::*(doris::TaskWorkerPool*))()> >::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291
    #21 0x55737b2cd3b6 in std::function<void ()>::operator()() const /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:560
    #22 0x55737b2cd3b6 in doris::FunctionRunnable::run() /root/doris/be/src/util/threadpool.cpp:46
    #23 0x55737b2cb290 in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:529
    #24 0x55737b2ccf6a in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:74
    #25 0x55737b2ccf6a in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:96
    #26 0x55737b2ccf6a in void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /var/local/ldb-toolchain/include/c++/11/functional:420
    #27 0x55737b2ccf6a in void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::operator()<, void>() /var/local/ldb-toolchain/include/c++/11/functional:503
    #28 0x55737b2ccf6a in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:61
    #29 0x55737b2ccf6a in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:111
    #30 0x55737b2ccf6a in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291
    #31 0x55737b29d556 in std::function<void ()>::operator()() const /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:560
    #32 0x55737b29d556 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:453
    #33 0x7fbd446e9608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477
    #34 0x7fbd444bf132 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x11f132)

Address 0x7fbb5a9cb848 is located in stack of thread T741 (TaskWorkerPool.) at offset 40 in frame
    #0 0x557379ce360f in apache::thrift::protocol::TVirtualProtocol<apache::thrift::protocol::TBinaryProtocolT<apache::thrift::transport::TTransport, apache::thrift::protocol::TNetworkBigEndian>, apache::thrift::protocol::TProtocolDefaults>::writeI64_virt(long) /var/local/thirdparty/installed/include/thrift/protocol/TVirtualProtocol.h:380

  This frame has 1 object(s):
    [32, 40) 'net' <== Memory access at offset 40 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
Thread T741 (TaskWorkerPool.) created by T0 here:
    #0 0x557378f7e061 in pthread_create (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster7/be/lib/doris_be+0xa257061)
    #1 0x55737b299312 in doris::Thread::start_thread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&, unsigned long, scoped_refptr<doris::Thread>*) /root/doris/be/src/util/thread.cpp:407
    #2 0x55737b2b7302 in doris::Status doris::Thread::create<void (doris::ThreadPool::*)(), doris::ThreadPool*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void (doris::ThreadPool::* const&)(), doris::ThreadPool* const&, scoped_refptr<doris::Thread>*) /root/doris/be/src/util/thread.h:57
    #3 0x55737b2b7302 in doris::ThreadPool::create_thread() /root/doris/be/src/util/threadpool.cpp:598
    #4 0x55737b2c2591 in doris::ThreadPool::init() /root/doris/be/src/util/threadpool.cpp:257
    #5 0x557379c6a1af in doris::Status doris::ThreadPoolBuilder::build<doris::ThreadPool>(std::unique_ptr<doris::ThreadPool, std::default_delete<doris::ThreadPool> >*) const /root/doris/be/src/util/threadpool.h:114
    #6 0x557379c6a1af in doris::TaskWorkerPool::start() /root/doris/be/src/agent/task_worker_pool.cpp:223
    #7 0x55737ade441a in doris::AgentServer::AgentServer(doris::ExecEnv*, doris::TMasterInfo const&) /root/doris/be/src/agent/agent_server.cpp:96
    #8 0x55737adbd5bb in doris::BackendService::BackendService(doris::ExecEnv*) /root/doris/be/src/service/backend_service.cpp:68
    #9 0x55737adc866b in doris::BackendService::create_service(doris::ExecEnv*, int, doris::ThriftServer**) /root/doris/be/src/service/backend_service.cpp:71
    #10 0x55737902ba87 in main /root/doris/be/src/service/doris_main.cpp:464
    #11 0x7fbd443c4082 in __libc_start_main ../csu/libc-start.c:308

SUMMARY: AddressSanitizer: stack-buffer-overflow (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster7/be/lib/doris_be+0xa252b16) in __interceptor_sigaltstack.part.0
Shadow bytes around the buggy address:
  0x0ff7eb5316b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff7eb5316c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff7eb5316d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff7eb5316e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff7eb5316f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0ff7eb531700: 00 00 00 00 f1 f1 f1 f1 00[f3]f3 f3 00 00 00 00
  0x0ff7eb531710: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff7eb531720: f1 f1 f1 f1 f1 f1 01 f2 00 f2 f2 f2 00 f2 f2 f2
  0x0ff7eb531730: 00 f2 f2 f2 00 f2 f2 f2 00 f3 f3 f3 00 00 00 00
  0x0ff7eb531740: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff7eb531750: f1 f1 f1 f1 f1 f1 01 f2 00 f2 f2 f2 00 f2 f2 f2
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==1482042==ABORTING
What You Expected?

be works normally
How to Reproduce?

p0 regression test
Anything Else?

No response
Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!
Code of Conduct

[X] I agree to follow this project's Code of Conduct
Mar 03 '23 06:03 xiaokang