beast coredump about http BIO

I meet a coredump, I dont know what happend, is there some suggestions to solve this problem, please give me some clues, thanks

#0  0x00007ff9500000f0 in ?? ()
#1  0x00007ff9729c381e in bio_write_intern () from /lib64/libcrypto.so.1.1
#2  0x00007ff9729c3ce7 in BIO_write () from /lib64/libcrypto.so.1.1
#3  0x00007ff97247ce5f in boost::asio::ssl::detail::engine::put_input (data=..., this=<optimized out>, this=<optimized out>) at /usr/local/include/boost/asio/ssl/detail/impl/engine.ipp:229
#4  0x00007ff9724f5678 in boost::asio::ssl::detail::io_op<boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::any_io_executor, boost::beast::unlimited_rate_policy>, boost::asio::ssl::detail::read_op<boost::asio::mutable_buffer>, boost::asio::detail::composed_op<boost::beast::http::detail::read_some_op<boost::asio::ssl::stream<boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::any_io_executor, boost::beast::unlimited_rate_policy> >, boost::beast::basic_flat_buffer<std::allocator<char> >, false>, boost::asio::detail::composed_work<void (boost::asio::any_io_executor)>, boost::asio::detail::composed_op<boost::beast::http::detail::read_op<boost::asio::ssl::stream<boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::any_io_executor, boost::beast::unlimited_rate_policy> >, boost::beast::basic_flat_buffer<std::allocator<char> >, false, boost::beast::http::detail::parser_is_done>, boost::asio::detail::composed_work<void (boost::asio::any_io_executor)>, boost::beast::http::detail::read_msg_op<boost::asio::ssl::stream<boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::any_io_executor, boost::beast::unlimited_rate_policy> >, boost::beast::basic_flat_buffer<std::allocator<char> >, false, boost::beast::http::basic_string_body<char, std::char_traits<char>, std::allocator<char> >, std::allocator<char>, boost::beast::detail::bind_front_wrapper<void (cpp_frame::http::http_connection::*)(boost::system::error_code, unsigned long), cpp_frame::http::http_connection*> >, void (boost::system::error_code, unsigned long)>, void (boost::system::error_code, unsigned long)> >::operator()(boost::system::error_code, unsigned long, int) (this=0x7ff965221388, ec=..., bytes_transferred=<optimized out>, start=0) at /usr/local/include/boost/asio/ssl/detail/io.hpp:269
#5  0x00007ff9724f8d36 in boost::beast::async_base<boost::asio::ssl::detail::io_op<boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::any_io_executor, boost::beast::unlimited_rate_policy>, boost::asio::ssl::detail::read_op<boost::asio::mutable_buffer>, boost::asio::detail::composed_op<boost::beast::http::detail::read_some_op<boost::asio::ssl::stream<boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::any_io_executor, boost::beast::unlimited_rate_policy> >, boost::beast::basic_flat_buffer<std::allocator<char> >, false>, boost::asio::detail::composed_work<void (boost::asio::any_io_executor)>, boost::asio::detail::composed_op<boost::beast::http::detail::read_op<boost::asio::ssl::stream<boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::any_io_executor, boost::beast::unlimited_rate_policy> >, boost::beast::basic_flat_buffer<std::allocator<char> >, false, boost::beast::http::detail::parser_is_done>, boost::asio::detail::composed_work<void (boost::asio::any_io_executor)>, boost::beast::http::detail::read_msg_op<boost::asio::ssl::stream<boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::any_io_executor, boost::beast::unlimited_rate_policy> >, boost::beast::basic_flat_buffer<std::allocator<char> >, false, boost::beast::http::basic_string_body<char, std::char_traits<char>, std::allocator<char> >, std::allocator<char>, boost::beast::detail::bind_front_wrapper<void (cpp_frame::http::http_connection::*)(boost::system::error_code, unsigned long), cpp_frame::http::http_connection*> >, void (boost::system::error_code, unsigned long)>, void (boost::system::error_code, unsigned long)> >, boost::asio::any_io_executor, std::allocator<void> >::complete_now<boost::system::error_code&, unsigned long&>(boost::system::error_code&, unsigned long&) (this=0x7ff965221380)
    at /usr/local/include/boost/beast/core/async_base.hpp:392
#6  boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::any_io_executor, boost::beast::unlimited_rate_policy>::ops::transfer_op<true, boost::asio::mutable_buffers_1, boost::asio::ssl::detail::io_op<boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::any_io_executor, boost::beast::unlimited_rate_policy>, boost::asio::ssl::detail::read_op<boost::asio::mutable_buffer>, boost::asio::detail::composed_op<boost::beast::http::detail::read_some_op<boost::asio::ssl::stream<boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::any_io_executor, boost::beast::unlimited_rate_policy> >, boost::beast::basic_flat_buffer<std::allocator<char> >, false>, boost::asio::detail::composed_work<void (boost::asio::any_io_executor)>, boost::asio::detail::composed_op<boost::beast::http::detail::read_op<boost::asio::ssl::stream<boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::any_io_executor, boost::beast::unlimited_rate_policy> >, boost::beast::basic_flat_buffer<std::allocator<char> >, false, boost::beast::http::detail::parser_is_done>, boost::asio::detail::composed_work<void (boost::asio::any_io_executor)>, boost::beast::http::detail::read_msg_op<boost::asio::ssl::stream<boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::any_io_executor, boost::beast::unlimited_rate_policy> >, boost::beast::basic_flat_buffer<std::allocator<char> >, false, boost::beast::http::basic_string_body<char, std::char_traits<char>, std::allocator<char> >, std::allocator<char>, boost::beast::detail::bind_front_wrapper<void (cpp_frame::http::http_connection::*)(boost::system::error_code, unsigned long), cpp_frame::http::http_connection*> >, void (boost::system::error_code, unsigned long)>, void (boost::system::error_code, unsigned long)> > >::operator()(boost::system::error_code, unsigned long) (this=this@entry=0x7ff965221380, ec=..., 
    bytes_transferred=0) at /usr/local/include/boost/beast/core/impl/basic_stream.hpp:424

......
#30 boost::asio::detail::reactive_socket_recv_op<boost::beast::buffers_prefix_view<boost::asio::mutable_buffers_1>, boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::any_io_executor, boost::beast::unlimited_rate_policy>::ops::transfer_op<true, boost::asio::mutable_buffers_1, boost::asio::ssl::detail::io_op<boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::any_io_executor, boost::beast::unlimited_rate_policy>, boost::asio::ssl::detail::read_op<boost::asio::mutable_buffer>, boost::asio::detail::composed_op<boost::beast::http::detail::read_some_op<boost::asio::ssl::stream<boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::any_io_executor, boost::beast::unlimited_rate_policy> >, boost::beast::basic_flat_buffer<std::allocator<char> >, false>, boost::asio::detail::composed_work<void (boost::asio::any_io_executor)>, boost::asio::detail::composed_op<boost::beast::http::detail::read_op<boost::asio::ssl::stream<boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::any_io_executor, boost::beast::unlimited_rate_policy> >, boost::beast::basic_flat_buffer<std::allocator<char> >, false, boost::beast::http::detail::parser_is_done>, boost::asio::detail::composed_work<void (boost::asio::any_io_executor)>, boost::beast::http::detail::read_msg_op<boost::asio::ssl::stream<boost::beast::basic_stream<boost::asio::ip::tcp, boost::asio::any_io_executor, boost::beast::unlimited_rate_policy> >, boost::beast::basic_flat_buffer<std::allocator<char> >, false, boost::beast::http::basic_string_body<char, std::char_traits<char>, std::allocator<char> >, std::allocator<char>, boost::beast::detail::bind_front_wrapper<void (cpp_frame::http::http_connection::*)(boost::system::error_code, unsigned long), cpp_frame::http::http_connection*> >, void (boost::system::error_code, unsigned long)>, void (boost::system::error_code, unsigned long)> > >, boost::asio::any_io_executor>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) (owner=0x17033a0, base=<optimized out>)
    at /usr/local/include/boost/asio/detail/reactive_socket_recv_op.hpp:147
#31 0x000000000043a9cc in boost::asio::detail::scheduler_operation::complete (bytes_transferred=0, ec=..., owner=0x17033a0, this=0x7ff95018eb50) at /usr/local/include/boost/asio/detail/scheduler_operation.hpp:40
#32 boost::asio::detail::scheduler::do_run_one (ec=..., this_thread=..., lock=..., this=0x17033a0) at /usr/local/include/boost/asio/detail/impl/scheduler.ipp:492
#33 boost::asio::detail::scheduler::run (this=0x17033a0, ec=...) at /usr/local/include/boost/asio/detail/impl/scheduler.ipp:210
#34 0x00000000004308e6 in boost::asio::io_context::run (this=<optimized out>, this=<optimized out>) at /usr/local/include/boost/asio/impl/io_context.ipp:63

May 29 '24 01:05 ahao1995

Are you using a multi-threaded executor? There might be a concurrency bug in your code. It is hard to tell what's wrong solely from that stack trace; a minimal example would help.

May 29 '24 06:05 ashtum

thanks, I will try to review my code, there is 2 thread in my code maybe is that bug

May 29 '24 08:05 ahao1995

there are many pthread_cond_wait in stack, I dont know pthread_cond_wait from where, I only use a thead with io_context

Thread 5 (Thread 0x7f26ea951700 (LWP 1010747) "td_proxy"):
#0  0x00007f26fa1dd3fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000043a7dc in boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock> (lock=..., this=<optimized out>) at /usr/local/include/boost/asio/detail/conditionally_enabled_mutex.hpp:98
#2  boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock> (lock=..., this=0x162e310) at /usr/local/include/boost/asio/detail/posix_event.hpp:113
#3  boost::asio::detail::conditionally_enabled_event::wait (lock=..., this=0x162e308) at /usr/local/include/boost/asio/detail/conditionally_enabled_event.hpp:97
#4  boost::asio::detail::scheduler::do_run_one (ec=..., this_thread=..., lock=..., this=0x162e2a0) at /usr/local/include/boost/asio/detail/impl/scheduler.ipp:501
#5  boost::asio::detail::scheduler::run (this=0x162e2a0, ec=...) at /usr/local/include/boost/asio/detail/impl/scheduler.ipp:210
#6  0x00007f26f94bf9d9 in boost::asio::system_context::thread_function::operator() (this=<optimized out>) at /usr/local/include/boost/asio/impl/system_context.ipp:37
#7  boost::asio::detail::posix_thread::func<boost::asio::system_context::thread_function>::run (this=<optimized out>) at /usr/local/include/boost/asio/detail/posix_thread.hpp:86
#8  0x0000000000435510 in boost::asio::detail::boost_asio_detail_posix_thread_function (arg=0x1612120) at /usr/local/include/boost/asio/detail/impl/posix_thread.ipp:74
#9  0x00007f26fa1d717a in start_thread () from /lib64/libpthread.so.0
#10 0x00007f26f852ddc3 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f26eb152700 (LWP 1010746) "td_proxy"):
#0  0x00007f26fa1dd3fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000043a7dc in boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock> (lock=..., this=<optimized out>) at /usr/local/include/boost/asio/detail/conditionally_enabled_mutex.hpp:98
#2  boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock> (lock=..., this=0x162e310) at /usr/local/include/boost/asio/detail/posix_event.hpp:113
#3  boost::asio::detail::conditionally_enabled_event::wait (lock=..., this=0x162e308) at /usr/local/include/boost/asio/detail/conditionally_enabled_event.hpp:97
#4  boost::asio::detail::scheduler::do_run_one (ec=..., this_thread=..., lock=..., this=0x162e2a0) at /usr/local/include/boost/asio/detail/impl/scheduler.ipp:501
#5  boost::asio::detail::scheduler::run (this=0x162e2a0, ec=...) at /usr/local/include/boost/asio/detail/impl/scheduler.ipp:210
#6  0x00007f26f94bf9d9 in boost::asio::system_context::thread_function::operator() (this=<optimized out>) at /usr/local/include/boost/asio/impl/system_context.ipp:37
#7  boost::asio::detail::posix_thread::func<boost::asio::system_context::thread_function>::run (this=<optimized out>) at /usr/local/include/boost/asio/detail/posix_thread.hpp:86
#8  0x0000000000435510 in boost::asio::detail::boost_asio_detail_posix_thread_function (arg=0x15a8610) at /usr/local/include/boost/asio/detail/impl/posix_thread.ipp:74
#9  0x00007f26fa1d717a in start_thread () from /lib64/libpthread.so.0
#10 0x00007f26f852ddc3 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f26eb953700 (LWP 1010745) "td_proxy"):
#0  0x00007f26fa1dd3fc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000043a7dc in boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock> (lock=..., this=<optimized out>) at /usr/local/include/boost/asio/detail/conditionally_enabled_mutex.hpp:98
#2  boost::asio::detail::posix_event::wait<boost::asio::detail::conditionally_enabled_mutex::scoped_lock> (lock=..., this=0x162e310) at /usr/local/include/boost/asio/detail/posix_event.hpp:113
#3  boost::asio::detail::conditionally_enabled_event::wait (lock=..., this=0x162e308) at /usr/local/include/boost/asio/detail/conditionally_enabled_event.hpp:97
#4  boost::asio::detail::scheduler::do_run_one (ec=..., this_thread=..., lock=..., this=0x162e2a0) at /usr/local/include/boost/asio/detail/impl/scheduler.ipp:501
#5  boost::asio::detail::scheduler::run (this=0x162e2a0, ec=...) at /usr/local/include/boost/asio/detail/impl/scheduler.ipp:210
#6  0x00007f26f94bf9d9 in boost::asio::system_context::thread_function::operator() (this=<optimized out>) at /usr/local/include/boost/asio/impl/system_context.ipp:37
#7  boost::asio::detail::posix_thread::func<boost::asio::system_context::thread_function>::run (this=<optimized out>) at /usr/local/include/boost/asio/detail/posix_thread.hpp:86
#8  0x0000000000435510 in boost::asio::detail::boost_asio_detail_posix_thread_function (arg=0x15a8650) at /usr/local/include/boost/asio/detail/impl/posix_thread.ipp:74
#9  0x00007f26fa1d717a in start_thread () from /lib64/libpthread.so.0
#10 0x00007f26f852ddc3 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f26ec154700 (LWP 1010744) "td_proxy"):
#0  0x00007f26f852e0f7 in epoll_wait () from /lib64/libc.so.6
#1  0x0000000000437a36 in boost::asio::detail::epoll_reactor::run (this=0x159db30, usec=<optimized out>, ops=...) at /usr/local/include/boost/asio/detail/impl/epoll_reactor.ipp:501
#2  0x000000000043a64b in boost::asio::detail::scheduler::do_run_one (ec=..., this_thread=..., lock=..., this=0x159d3a0) at /usr/local/include/boost/asio/detail/impl/scheduler.ipp:476
#3  boost::asio::detail::scheduler::run (this=0x159d3a0, ec=...) at /usr/local/include/boost/asio/detail/impl/scheduler.ipp:210
#4  0x00000000004308e6 in boost::asio::io_context::run (this=<optimized out>, this=<optimized out>) at /usr/local/include/boost/asio/impl/io_context.ipp:63
#5  operator() (__closure=<optimized out>) at /home/FastTrader/td_proxy/src/trader_engine.cc:93
#6  std::__invoke_impl<void, fast_trader::td::trader_engine::start_net()::<lambda()> > (__f=...) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/invoke.h:61
#7  std::__invoke<fast_trader::td::trader_engine::start_net()::<lambda()> > (__fn=...) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/invoke.h:96
#8  std::thread::_Invoker<std::tuple<fast_trader::td::trader_engine::start_net()::<lambda()> > >::_M_invoke<0> (this=<optimized out>) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/std_thread.h:253
#9  std::thread::_Invoker<std::tuple<fast_trader::td::trader_engine::start_net()::<lambda()> > >::operator() (this=<optimized out>) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/std_thread.h:260
#10 std::thread::_State_impl<std::thread::_Invoker<std::tuple<fast_trader::td::trader_engine::start_net()::<lambda()> > > >::_M_run(void) (this=<optimized out>) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/std_thread.h:211
#11 0x00007f26f8e52ba3 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#12 0x00007f26fa1d717a in start_thread () from /lib64/libpthread.so.0
#13 0x00007f26f852ddc3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f26fa5e8380 (LWP 1010743) "td_proxy"):
#0  0x00007f26fa1d866d in __pthread_timedjoin_ex () from /lib64/libpthread.so.0
#1  0x00007f26f8e52e27 in std::thread::join() () from /lib64/libstdc++.so.6
#2  0x00000000004201dd in main () at /home/FastTrader/td_proxy/src/main.cc:26

Jun 03 '24 01:06 ahao1995

In these cases I find stack dumps to be less helpful. Instead, I prefer to have a detailed explanation of how the program manages the lifetime of its connections. For example, this is how I would explain it for the example async servers in beast:

The completion handler for every asynchronous operation acquires shared ownership of the connection object by retaining a copy of its shared pointer
The lifetime of the connection object is extended when a completion handler initiates another asynchronous operation for the same connection before returning
The connection object is destroyed when the last outstanding completion handler exits

and

completion handlers for asynchronous operations on connections associated with multi-threaded I/O contexts is posted through a strand

What are the rules for your application?

Jun 03 '24 18:06 vinniefalco

In these cases I find stack dumps to be less helpful. Instead, I prefer to have a detailed explanation of how the program manages the lifetime of its connections. For example, this is how I would explain it for the example async servers in beast:

The completion handler for every asynchronous operation acquires shared ownership of the connection object by retaining a copy of its shared pointer

The lifetime of the connection object is extended when a completion handler initiates another asynchronous operation for the same connection before returning

The connection object is destroyed when the last outstanding completion handler exits

and

completion handlers for asynchronous operations on connections associated with multi-threaded I/O contexts is posted through a strand

What are the rules for your application?

maybe it is life time issue, I use raw pointer and delete it when it close

Jun 04 '24 00:06 ahao1995

I use raw pointer and delete it when it close

What happens when there are concurrent asynchronous operations for the same connection? Instead of using a raw pointer, try using std::unique_ptr<connection>. This will guarantee that you do not have more than one operation active at the same time, as any attempt to launch a second operation while one is pending will immediately throw.

Jun 04 '24 15:06 vinniefalco

I use raw pointer and delete it when it close

What happens when there are concurrent asynchronous operations for the same connection? Instead of using a raw pointer, try using std::unique_ptr<connection>. This will guarantee that you do not have more than one operation active at the same time, as any attempt to launch a second operation while one is pending will immediately throw.

thanks

Jun 06 '24 01:06 ahao1995

coredump about http BIO_write