bugtracker
bugtracker copied to clipboard
Kurento
Kurento Crash With Segmentation Fault in kmsagnosticbin
Prerequisites
[X ] I have read the SUPPORT document
[X ] I have checked the Troubleshooting Guide
[X ] I have tested with the latest version of Kurento
Issue description
Kurento randomly crashes with a segmentation fault when we are trying to composite several WebRtcEndpoint audio streams into a RtpEndpoint.
We use several WebRtcEndpoint passed through a composite then the composite is connected to the RtpEndpoint.
The crash happened when the connect call is made.
Depending on usage, this crash can happen several times a day with about 50-100 users on a single kurento server with mixed WebRtc and Rtp dial in.
Any help is appreciated.
Stacktrace:
Segmentation fault (thread 139775553021696, pid 19238)
Stack trace:
[gst_caps_set_features]
/workdir/gst/gstcaps.c:902
[check_bin]
/workdir/src/gst-plugins/kmsagnosticbin.c:503
[kms_agnostic_bin2_find_bin_for_caps]
/workdir/src/gst-plugins/kmsagnosticbin.c:537
[kms_agnostic_bin2_find_or_create_bin_for_caps]
/workdir/src/gst-plugins/kmsagnosticbin.c:718
[kms_agnostic_bin2_link_pad]
/workdir/src/gst-plugins/kmsagnosticbin.c:787
[kms_agnostic_bin2_src_reconfigure_probe]
/workdir/src/gst-plugins/kmsagnosticbin.c:1042
[probe_hook_marshal]
/workdir/gst/gstpad.c:3450
[g_hook_list_marshal]
/build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/ghook.c:673
[do_probe_callbacks]
/workdir/gst/gstpad.c:3605
[gst_pad_send_event_unchecked]
/workdir/gst/gstpad.c:5563
[gst_pad_push_event_unchecked]
/workdir/gst/gstpad.c:5244
[gst_pad_push_event]
/workdir/gst/gstpad.c:5380
[event_forward_func]
/workdir/gst/gstpad.c:2990
[gst_pad_forward]
/workdir/gst/gstpad.c:2944
[gst_pad_event_default]
/workdir/gst/gstpad.c:3044
[gst_pad_send_event_unchecked]
/workdir/gst/gstpad.c:5582
GDB bt from a core dump
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/kurento-media-server'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fe05d2fb428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7fe04d518700 (LWP 30284))]
(gdb) bt
#0 0x00007fe05d2fb428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007fe05d2fd02a in __GI_abort () at abort.c:89
#2 0x0000563fdb8bc979 in Debug::DeathHandler::SignalHandler(int, void*, void*) () at /workdir/server/death_handler.cpp:299
#3 0x0000563fdb8bc979 in Debug::DeathHandler::SignalHandler(int, void*, void*) (sig=<optimized out>, secret=<optimized out>) at /workdir/server/death_handler.cpp:600
#4 0x00007fe05fa1e390 in <signal handler called> () at /lib/x86_64-linux-gnu/libpthread.so.0
#5 0x00007fe05ee74411 in gst_caps_set_features (caps=0x7fe038408630, index=<optimized out>, features=0x7fe0386a4b40) at gstcaps.c:902
#6 0x00007fe010031d1b in check_bin (tree_bin=tree_bin@entry=0x7fdddc32cbc0 [KmsEncTreeBin], caps=caps@entry=0x7fdfe416f280) at /workdir/src/gst-plugins/kmsagnosticbin.c:501
#7 0x00007fe010032338 in kms_agnostic_bin2_find_bin_for_caps (self=0x7fdd4c2260c0 [KmsAgnosticBin2], caps=0x7fdfe416f280) at /workdir/src/gst-plugins/kmsagnosticbin.c:537
#8 0x00007fe010032406 in kms_agnostic_bin2_find_or_create_bin_for_caps (self=self@entry=0x7fdd4c2260c0 [KmsAgnosticBin2], caps=caps@entry=0x7fdfe416f280) at /workdir/src/gst-plugins/kmsagnosticbin.c:716
#9 0x00007fe010032e56 in kms_agnostic_bin2_process_pad (peer=0x7fdb5c0d9650 [GstProxyPad], pad=0x7fdd58b77650 [GstGhostPad], self=0x7fdd4c2260c0 [KmsAgnosticBin2])
at /workdir/src/gst-plugins/kmsagnosticbin.c:785
#10 0x00007fe010032e56 in kms_agnostic_bin2_process_pad (self=0x7fdd4c2260c0 [KmsAgnosticBin2], pad=0x7fdd58b77650 [GstGhostPad]) at /workdir/src/gst-plugins/kmsagnosticbin.c:856
#11 0x00007fe01003475f in kms_agnostic_bin2_src_reconfigure_probe (pad=pad@entry=0x7fdd58b77650 [GstGhostPad], info=info@entry=0x7fe04d514ad0, user_data=0x7fdd4c2260c0)
at /workdir/src/gst-plugins/kmsagnosticbin.c:1041
#12 0x00007fe05ee9d29b in probe_hook_marshal (hook=0x7fdda00eb630, data=0x7fe04d514a30) at gstpad.c:3448
#13 0x00007fe05e907904 in g_hook_list_marshal (hook_list=hook_list@entry=0x7fdd58b776e8, may_recurse=may_recurse@entry=1, marshaller=marshaller@entry=0x7fe05ee9d140 <probe_hook_marshal>, data=data@entry=0x7fe04d514a30) at /build/glib2.0-xkQkqE/glib2.0-2.48.2/./glib/ghook.c:672
#14 0x00007fe05ee9bafb in do_probe_callbacks (pad=pad@entry=0x7fdd58b77650 [GstGhostPad], info=info@entry=0x7fe04d514ad0, defaultval=defaultval@entry=GST_FLOW_OK) at gstpad.c:3600
#15 0x00007fe05ee9e0d7 in gst_pad_send_event_unchecked (pad=pad@entry=0x7fdd58b77650 [GstGhostPad], event=event@entry=0x7fe0143441d0, type=<optimized out>) at gstpad.c:5563
#16 0x00007fe05eea8ba9 in gst_pad_send_event (pad=pad@entry=0x7fdd58b77650 [GstGhostPad], event=0x7fe0143441d0) at gstpad.c:5752
#17 0x00007fe05eea8fd2 in gst_pad_link_full (srcpad=srcpad@entry=0x7fdd58b77650 [GstGhostPad], sinkpad=sinkpad@entry=0x7fdb5c0d9650 [GstProxyPad], flags=flags@entry=GST_PAD_LINK_CHECK_NOTHING) at gstpad.c:2521
#18 0x00007fe05ee91903 in gst_ghost_pad_set_target (gpad=0x7fdd58b773e0 [GstGhostPad], newtarget=newtarget@entry=0x7fdd58b77650 [GstGhostPad]) at gstghostpad.c:880
#19 0x00007fe05c2fd7d6 in kms_element_set_target_on_linked (pad=0x7fdd58b773e0 [GstGhostPad], peer=<optimized out>, element=<optimized out>) at /workdir/src/gst-plugins/commons/kmselement.c:426
#20 0x00007fe05ebf10a7 in g_cclosure_marshal_VOID__OBJECTv (closure=0x7fe03882df30, return_value=<optimized out>, instance=<optimized out>, args=<optimized out>, marshal_data=0x0, n_params=<optimized out>, param_types=0x563fdbd7ad30) at /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gmarshal.c:2102
#21 0x00007fe05ebee1d4 in _g_closure_invoke_va (closure=closure@entry=0x7fe03882df30, return_value=return_value@entry=0x0, instance=instance@entry=0x7fdd58b773e0, args=args@entry=0x7fe04d514ea0, n_params=<optimized out>, param_types=0x563fdbd7ad30) at /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gclosure.c:867
#22 0x00007fe05ec089a6 in g_signal_emit_valist (instance=0x7fdd58b773e0, signal_id=<optimized out>, detail=0, var_args=var_args@entry=0x7fe04d514ea0)
at /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gsignal.c:3294
#23 0x00007fe05ec0908f in g_signal_emit (instance=instance@entry=0x7fdd58b773e0, signal_id=<optimized out>, detail=detail@entry=0) at /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gsignal.c:3441
#24 0x00007fe05eea8eff in gst_pad_link_full (srcpad=srcpad@entry=0x7fdd58b773e0 [GstGhostPad], sinkpad=sinkpad@entry=0x7fdc9424e7a0 [GstGhostPad], flags=flags@entry=GST_PAD_LINK_CHECK_NOTHING) at gstpad.c:2515
#25 0x00007fe05e5e2bc5 in kurento::MediaElementImpl::performConnection(std::shared_ptr<kurento::ElementConnectionDataInternal>) (this=this@entry=0x7fe0383b1d70, data=std::shared_ptr (count 4, weak 0) 0x7fe03802ff60) at /workdir/src/server/implementation/objects/MediaElementImpl.cpp:996
#26 0x00007fe05e5ecc03 in kurento::MediaElementImpl::connect(std::shared_ptr<kurento::MediaElement>, std::shared_ptr<kurento::MediaType>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (this=<optimized out>, sink=..., mediaType=..., sourceMediaDescription=..., sinkMediaDescription=...)
at /workdir/src/server/implementation/objects/MediaElementImpl.cpp:956
#27 0x00007fe05e5e17e6 in kurento::MediaElementImpl::connect(std::shared_ptr<kurento::MediaElement>, std::shared_ptr<kurento::MediaType>) (this=0x7fe0383b1d70, sink=warning: RTTI symbol not found for class 'std::_Sp_counted_deleter<kurento::MediaObjectImpl*, kurento::MediaSet::ref(kurento::MediaObjectImpl*)::{lambda(kurento::MediaObjectImpl*)#1}, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_deleter<kurento::MediaObjectImpl*, kurento::MediaSet::ref(kurento::MediaObjectImpl*)::{lambda(kurento::MediaObjectImpl*)#1}, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 7, weak 9) 0x7fe0242bac48, mediaType=...) at /workdir/src/server/implementation/objects/MediaElementImpl.cpp:881
#28 0x00007fe05e633b4c in kurento::MediaElementMethodConnect::invoke(std::shared_ptr<kurento::MediaElement>) (this=this@entry=0x7fe04d5157e0, obj=warning: RTTI symbol not found for class 'std::_Sp_counted_deleter<kurento::MediaObjectImpl*, kurento::MediaSet::ref(kurento::MediaObjectImpl*)::{lambda(kurento::MediaObjectImpl*)#1}, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_deleter<kurento::MediaObjectImpl*, kurento::MediaSet::ref(kurento::MediaObjectImpl*)::{lambda(kurento::MediaObjectImpl*)#1}, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 7, weak 6) 0x7fe0383b2078)
at /workdir/obj-x86_64-linux-gnu/src/server/interface/generated-cpp/MediaElementInternal.cpp:147
#29 0x00007fe05e623e03 in kurento::MediaElementImpl::invoke(std::shared_ptr<kurento::MediaObjectImpl>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Json::Value const&,Json::Value&) (this=<optimized out>, obj=warning: RTTI symbol not found for class 'std::_Sp_counted_deleter<kurento::MediaObjectImpl*, kurento::MediaSet::ref(kurento::MediaObjectImpl*)::{lambda(kurento::MediaObjectImpl*)#1}, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_deleter<kurento::MediaObjectImpl*, kurento::MediaSet::ref(kurento::MediaObjectImpl*)::{lambda(kurento::MediaObjectImpl*)#1}, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>'
std::shared_ptr (count 7, weak 6) 0x7fe0383b1d70, methodName=..., params=..., response=...)
at /workdir/obj-x86_64-linux-gnu/src/server/implementation/generated-cpp/MediaElementImplInternal.cpp:62
#30 0x00007fe05e610122 in kurento::HubPortImpl::invoke(std::shared_ptr<kurento::MediaObjectImpl>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Json::Value const&, Json::Value&) (this=<optimized out>, obj=..., methodName=..., params=..., response=...) at /workdir/obj-x86_64-linux-gnu/src/server/implementation/generated-cpp/HubPortImplInternal.cpp:30
#31 0x0000563fdb886d78 in kurento::ServerMethods::invoke(Json::Value const&, Json::Value&) (this=<optimized out>, params=..., response=...) at /workdir/server/ServerMethods.cpp:603
#32 0x00007fe05e2dd7fa in kurento::JsonRpc::Handler::process(Json::Value const&, Json::Value&) (__args#1=..., __args#0=..., this=<optimized out>) at /usr/include/c++/5/functional:2267
#33 0x00007fe05e2dd7fa in kurento::JsonRpc::Handler::process(Json::Value const&, Json::Value&) (this=this@entry=0x563fdc114a30, msg=..., _response=...) at /workdir/src/JsonRpcHandler.cpp:122
#34 0x0000563fdb88b0ba in kurento::ServerMethods::process(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (this=0x563fdc114a20, requestStr=..., responseStr="", sessionId="085c296e-c56b-42df-ad39-33725ccf5491")
at /workdir/server/ServerMethods.cpp:277
#35 0x0000563fdb8f6f9e in kurento::WebSocketTransport::processMessage<websocketpp::server<websocketpp::config::asio> >(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, websocketpp::server<websocketpp::config::asio>::message_ptr) (this=0x563fdc072000, s=0x563fdc072118, hdl=std::weak_ptr (count 3, weak 7) 0x563fdc16e030, msg=...) at /workdir/server/transport/websocket/WebSocketTransport.cpp:461
---Type <return> to continue, or q <return> to quit---
#36 0x0000563fdb8dc274 in std::_Function_handler<void (std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >), std::_Bind<std::_Mem_fn<void (kurento::WebSocketTransport::*)(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >)> (kurento::WebSocketTransport*, websocketpp::server<websocketpp::config::asio>*, std::_Placeholder<1>, std::_Placeholder<2>)> >::_M_invoke(std::_Any_data const&, std::weak_ptr<void>&&, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >&&) (__object=<optimized out>, this=<optimized out>) at /usr/include/c++/5/functional:600
#37 0x0000563fdb8dc274 in std::_Function_handler<void (std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >), std::_Bind<std::_Mem_fn<void (kurento::WebSocketTransport::*)(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >)> (kurento::WebSocketTransport*, websocketpp::server<websocketpp::config::asio>*, std::_Placeholder<1>, std::_Placeholder<2>)> >::_M_invoke(std::_Any_data const&, std::weak_ptr<void>&&, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >&&) (__args=<optimized out>, this=<optimized out>) at /usr/include/c++/5/functional:1074
#38 0x0000563fdb8dc274 in std::_Function_handler<void (std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >), std::_Bind<std::_Mem_fn<void (kurento::WebSocketTransport::*)(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >)> (kurento::WebSocketTransport*, websocketpp::server<websocketpp::config::asio>*, std::_Placeholder<1>, std::_Placeholder<2>)> >::_M_invoke(std::_Any_data const&, std::weak_ptr<void>&&, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >&&) (this=<optimized out>) at /usr/include/c++/5/functional:1133
#39 0x0000563fdb8dc274 in std::_Function_handler<void (std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >), std::_Bind<std::_Mem_fn<void (kurento::WebSocketTransport::*)(websocketpp::server<websocketpp::config::asio>*, std::weak_ptr<void>, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >)> (kurento::WebSocketTransport*, websocketpp::server<websocketpp::config::asio>*, std::_Placeholder<1>, std::_Placeholder<2>)> >::_M_invoke(std::_Any_data const&, std::weak_ptr<void>&&, std::shared_ptr<websocketpp::message_buffer::message<websocketpp::message_buffer::alloc::con_msg_manager> >&&) (__functor=..., __args#0=<optimized out>, __args#1=<optimized out>) at /usr/include/c++/5/functional:1871
#40 0x0000563fdb90f4e6 in websocketpp::connection<websocketpp::config::asio>::handle_read_frame(std::error_code const&, unsigned long) (__args#1=std::shared_ptr (count -602461136, weak 22078) 0x7fe04d5167b0, __args#0=<error reading variable: Cannot access memory at address 0x28>, this=0x563fdc16eb10) at /usr/include/c++/5/functional:2267
#41 0x0000563fdb90f4e6 in websocketpp::connection<websocketpp::config::asio>::handle_read_frame(std::error_code const&, unsigned long) (this=0x563fdc16e030, ec=..., bytes_transferred=481)
at /workdir/server/transport/websocket/websocketpp/impl/connection_impl.hpp:1164
#42 0x0000563fdb8ed49a in websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::handle_async_read(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long) (__args#1=481, __args#0=..., this=0x7fe04d516ab0) at /usr/include/c++/5/functional:2267
#43 0x0000563fdb8ed49a in websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::handle_async_read(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long) (this=0x563fdc16e030, handler=..., ec=..., bytes_transferred=481) at /workdir/server/transport/websocket/websocketpp/transport/asio/connection.hpp:931
#44 0x0000563fdb8ff2f7 in boost::asio::detail::strand_service::dispatch<boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>)> >, boost::system::error_code, unsigned long> >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>)> >, boost::system::error_code, unsigned long>&) (__ptr=std::shared_ptr (count 3, weak 7) 0x563fdc16e030, this=0x7fe04d516bc8) at /usr/include/c++/5/functional:634
#45 0x0000563fdb8ff2f7 in boost::asio::detail::strand_service::dispatch<boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>)> >, boost::system::error_code, unsigned long> >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>)> >, boost::system::error_code, unsigned long>&) (__object=..., this=0x7fe04d516bc8) at /usr/include/c++/5/functional:610
#46 0x0000563fdb8ff2f7 in boost::asio::detail::strand_service::dispatch<boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>)> >, boost::system::error_code, unsigned long> >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>)> >, boost::system::error_code, unsigned long>&) (__args=<optimized out>, this=0x7fe04d516bc8) at /usr/include/c++/5/functional:1074
#47 0x0000563fdb8ff2f7 in boost::asio::detail::strand_service::dispatch<boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>)> >, boost::system::error_code, unsigned long> >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&, unsigned long)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&, unsigned long)>, std::_Placeholder<1>, std::_Placeholder<2>)> >, boost::system::error_code, unsigned long>&) (this=0x7fe04d516bc8) at /usr/include/c++/5/functional:1133
#48 0x0000563fdb8ff2f7 in boost::asio::detail::strand_service::dispatch<boost::asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio:
Context
We are trying to get Audio Dial-in feature to work with Kurento. This involves using the composite to be able to merge the audio from different participants to send to the rtp endpoint. The feature works when in testing but it crashes the Kurento Media Server in production. There isn't a reliable way to reproduce this bug atm.
How to reproduce?
I am not able to reproduce it aside from it is crashing when used in production.
Expected & current behavior
Kurento should composite the audio and send it to the rtp endpoint. Currently, it sometimes crashes with a segmentation fault.
Environment info
We are using WebRtcEndpoints, Composite, and RtpEndpoint.
INFO: Kurento Media Server
- Kurento version: 6.13.0, nightly
- Server OS: Ubuntu 16.04 (Xenial)
- Installation method:
- [X] AWS Cloudformation
INFO: Application Server
- Language: Java
- Kurento Client version: 6.13.1
Version details
No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 16.04.6 LTS Release: 16.04 Codename: xenial
Kurento Media Server version: 6.13.0 Found modules: 'core' version 6.13.0 'elements' version 6.13.0 'filters' version 6.13.0
ii gstreamer1.5-alsa:amd64 1.8.1-1kurento2.16.04 amd64 GStreamer plugin for ALSA
ii gstreamer1.5-libav:amd64 1.8.1-1kurento1.16.04 amd64 libav plugin for GStreamer
ii gstreamer1.5-libav-dbg:amd64 1.8.1-1kurento1.16.04 amd64 libav plugin for GStreamer (debug symbols)
ii gstreamer1.5-nice:amd64 0.1.15-1kurento3.16.04 amd64 ICE library (GStreamer 1.5 plugin)
ii gstreamer1.5-nice-dbgsym:amd64 0.1.15-1kurento3.16.04 amd64 debug symbols for package gstreamer1.5-nice
ii gstreamer1.5-plugins-bad:amd64 1.8.1-1kurento4.16.04 amd64 GStreamer plugins from the "bad" set
ii gstreamer1.5-plugins-bad-dbg:amd64 1.8.1-1kurento4.16.04 amd64 GStreamer plugins from the "bad" set (debug symbols)
ii gstreamer1.5-plugins-base:amd64 1.8.1-1kurento2.16.04 amd64 GStreamer plugins from the "base" set
ii gstreamer1.5-plugins-base-dbg:amd64 1.8.1-1kurento2.16.04 amd64 GStreamer plugins from the "base" set
ii gstreamer1.5-plugins-good:amd64 1.8.1-1kurento3.16.04 amd64 GStreamer plugins from the "good" set
ii gstreamer1.5-plugins-good-dbg:amd64 1.8.1-1kurento3.16.04 amd64 GStreamer plugins from the "good" set
ii gstreamer1.5-plugins-ugly:amd64 1.8.1-1kurento1.16.04 amd64 GStreamer plugins from the "ugly" set
ii gstreamer1.5-plugins-ugly-dbg:amd64 1.8.1-1kurento1.16.04 amd64 GStreamer plugins from the "ugly" set (debug symbols)
ii gstreamer1.5-pulseaudio:amd64 1.8.1-1kurento3.16.04 amd64 GStreamer plugin for PulseAudio
ii gstreamer1.5-x:amd64 1.8.1-1kurento2.16.04 amd64 GStreamer plugins for X11 and Pango
ii kms-core 6.13.0-0kurento1.16.04 amd64 Kurento Core module
ii kms-core-dbg 6.13.0-0kurento1.16.04 amd64 Kurento Core module
ii kms-elements 6.13.0-0kurento1.16.04 amd64 Kurento Elements module
ii kms-elements-dbg 6.13.0-0kurento1.16.04 amd64 Kurento Elements module
ii kms-filters 6.13.0-0kurento1.16.04 amd64 Kurento Filters module
ii kms-filters-dbg 6.13.0-0kurento1.16.04 amd64 Kurento Filters module
ii kms-jsonrpc 6.13.0-0kurento1.16.04 amd64 Kurento JSON-RPC library
ii kms-jsonrpc-dbg 6.13.0-0kurento1.16.04 amd64 Kurento JSON-RPC library
ii kmsjsoncpp 1.6.3-1kurento1.16.04 amd64 Kurento jsoncpp library
ii kmsjsoncpp-dbg 1.6.3-1kurento1.16.04 amd64 Kurento jsoncpp library
ii kurento-dbg 6.13.0-0kurento1.16.04 amd64 Meta-package that installs debug symbols
ii kurento-media-server 6.13.0-0kurento1.16.04 amd64 Kurento Media Server
ii kurento-media-server-dbg 6.13.0-0kurento1.16.04 amd64 Kurento Media Server
ii libgstreamer-plugins-bad1.5-0:amd64 1.8.1-1kurento4.16.04 amd64 GStreamer development files for libraries from the "bad" set
ii libgstreamer-plugins-base1.5-0:amd64 1.8.1-1kurento2.16.04 amd64 GStreamer libraries from the "base" set
ii libgstreamer1.5-0:amd64 1.8.1-1kurento2.16.04 amd64 Core GStreamer libraries and elements
ii libgstreamer1.5-0-dbg:amd64 1.8.1-1kurento2.16.04 amd64 Core GStreamer libraries and elements
ii libnice10:amd64 0.1.15-1kurento3.16.04 amd64 ICE library (shared library)
ii libnice10-dbgsym:amd64 0.1.15-1kurento3.16.04 amd64 debug symbols for package libnice10
ii libsrtp0:amd64 1.6.0-0kurento1.16.04 amd64 Secure RTP (SRTP) and UST Reference Implementations - shared library
ii libusrsctp 0.9.2-1kurento1.16.04 amd64 sctp-refimpl library
ii openh264 1.4.0-1kurento1.16.04 amd64 OpenH264 library
ii openh264-gst-plugins-bad-1.5:amd64 1.8.1-1kurento4.16.04 amd64 GStreamer plugins from openh264
ii openwebrtc-gst-plugins 0.10.0-1kurento1.16.04 amd64 OpenWebRTC specific GStreamer plugins
ii openwebrtc-gst-plugins-dbg 0.10.0-1kurento1.16.04 amd64 OpenWebRTC specific GStreamer plugins
Hi, do you still have the core file, and can you send it? I'd like to try and see if I can get some more information between backtrace frames #4 and #5.
Or if you are able to, add the Ubuntu Debug repository to install both libc6-dbgsym and libc6-dbg, and hopefully that should show more details between those two frames.
Maybe this is caused by having lots of sessions? That would maybe explain why it is not possible to reproduce with a simpler test, but then in production with more people trying to join, the crash is able to happen. Try some test with a higher number of participants to see if you can make it crash.
Hi,
I have attached the core dump in Google Drive. https://drive.google.com/file/d/1mZw739BCIrI_5ydn7QXlKxbPHrmYoyeH/view?usp=sharing
I tried to install the debug symbols and gdb the core dump again but it didn't look like it has more information. Maybe I am doing it incorrectly.
(gdb) bt
#0 0x00007fe05d2fb428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007fe05d2fd02a in __GI_abort () at abort.c:89
#2 0x0000563fdb8bc979 in Debug::DeathHandler::SignalHandler(int, void*, void*) () at /workdir/server/death_handler.cpp:299
#3 0x0000563fdb8bc979 in Debug::DeathHandler::SignalHandler(int, void*, void*) (sig=
I will look into reproducing the issue but do you have any insight on the possible cause? There are 50-200 users connected when the crash happens but it is not clear whether there is a direct correlation between load and crash.
Thank you for your help.
do you have any insight on the possible cause?
I've been analyzing the code and it seems a bug caused by out of bounds index in an array access. A tentative fix is here: https://github.com/Kurento/kms-core/commit/4c61821c2d21ddd74e9ff493215f0d920a71616e
I've published it in the nightly repos, if you wish to test the fix please change your kurento.list file and change the version to /dev, as explained in Installing Nightly,
then upgrade your KMS installation to nightly packages, as explained in Local Upgrade.
Make sure your version was correctly updated by checking with kurento-media-server --version. Nightly version should be 6.13.2, CI will finish deploying it a couple hours from now.
That's great news. Thanks for the quick turnaround. I will look into reproducing the issue on our side and use the new build to test that it is fixed.
Thanks!