LogDev hang

[xiaoxichen]

[03/10/24 21:06:21-07:00] [I] [508688] [replication_state_machine.cpp:7:on_commit] applying raft log commit with lsn:232
[03/10/24 21:06:21-07:00] [I] [508141] [hs_blob_tests.cpp:73:TestBody] delete blob shard 1125899906842625 blob 51
[03/10/24 21:06:21-07:00] [I] [508141] [replication_state_machine.cpp:35:on_pre_commit] on_pre_commit with lsn:233
[03/10/24 21:06:21-07:00] [I] [508688] [replication_state_machine.cpp:7:on_commit] applying raft log commit with lsn:233
[03/10/24 21:06:21-07:00] [I] [508141] [hs_blob_tests.cpp:73:TestBody] delete blob shard 1125899906842625 blob 52
[03/10/24 21:06:21-07:00] [I] [508141] [replication_state_machine.cpp:35:on_pre_commit] on_pre_commit with lsn:234
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^[^@^@^@^[^@^@^@^@^@

hanging after pre_commit but before commit. The corresponding thread backtrack is below

Thread 59 (Thread 0x7ff9ecff9600 (LWP 508688) "nuraft_commit"):
#0  __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x562de81f7f74) at ./nptl/futex-internal.c:57
#1  __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x0, clockid=0, expected=0, futex_word=0x562de81f7f74) at ./nptl/futex-internal.c:87
#2  __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x562de81f7f74, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=0) at ./nptl/futex-internal.c:139
#3  0x00007ffaa2955a41 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x562de81f7f20, cond=0x562de81f7f48) at ./nptl/pthread_cond_wait.c:503
#4  ___pthread_cond_wait (cond=0x562de81f7f48, mutex=0x562de81f7f20) at ./nptl/pthread_cond_wait.c:627
#5  0x0000562de1636f3b in std::condition_variable::wait<homestore::HomeLogStore::flush_sync(homestore::logstore_seq_num_t)::<lambda()> >(std::unique_lock<std::mutex> &, struct {...}) (this=0x562de81f7f48, __lock=..., __p=...) at /usr/include/c++/11/condition_variable:103
#6  0x0000562de1632ad1 in homestore::HomeLogStore::flush_sync (this=0x562de81f7db0, upto_seq_num=233) at /var/home/centos/.conan/data/homestore/5.1.12-35/oss/master/build/184931f8962d502a27b6183756ace784abafe634/src/lib/logstore/log_store.cpp:417
#7  0x0000562de162f1f8 in homestore::HomeLogStore::read_sync (this=0x562de81f7db0, seq_num=233) at /var/home/centos/.conan/data/homestore/5.1.12-35/oss/master/build/184931f8962d502a27b6183756ace784abafe634/src/lib/logstore/log_store.cpp:128
#8  0x0000562de14f2c66 in homestore::HomeRaftLogStore::entry_at (this=0x562de8cf3f90, index=234) at /var/home/centos/.conan/data/homestore/5.1.12-35/oss/master/build/184931f8962d502a27b6183756ace784abafe634/src/lib/replication/log_store/home_raft_log_store.cpp:160
#9  0x0000562de26d34fa in nuraft::raft_server::commit_in_bg_exec (this=0x562de866e6b0, timeout_ms=0) at /var/home/centos/.conan/data/nuraft/2.3.0/_/_/build/02c66fad048eac12445a1f37cedcfa3c59139f1c/src/src/handle_commit.cxx:225
#10 0x0000562de26d2b30 in nuraft::raft_server::commit_in_bg (this=0x562de866e6b0) at /var/home/centos/.conan/data/nuraft/2.3.0/_/_/build/02c66fad048eac12445a1f37cedcfa3c59139f1c/src/src/handle_commit.cxx:144
#11 0x0000562de2729401 in std::__invoke_impl<void, void (nuraft::raft_server::*&)(), nuraft::raft_server*&> (__f=@0x562de866e608: (void (nuraft::raft_server::*)(nuraft::raft_server * const)) 0x562de26d2762 <nuraft::raft_server::commit_in_bg()>, __t=@0x562de866e618: 0x562de866e6b0) at /usr/include/c++/11/bits/invoke.h:74
#12 0x0000562de27289ea in std::__invoke<void (nuraft::raft_server::*&)(), nuraft::raft_server*&> (__fn=@0x562de866e608: (void (nuraft::raft_server::*)(nuraft::raft_server * const)) 0x562de26d2762 <nuraft::raft_server::commit_in_bg()>) at /usr/include/c++/11/bits/invoke.h:96
#13 0x0000562de2727cad in std::_Bind<void (nuraft::raft_server::*(nuraft::raft_server*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x562de866e608, __args=...) at /usr/include/c++/11/functional:420
#14 0x0000562de27272ef in std::_Bind<void (nuraft::raft_server::*(nuraft::raft_server*))()>::operator()<, void>() (this=0x562de866e608) at /usr/include/c++/11/functional:503
#15 0x0000562de2729f4e in std::__invoke_impl<void, std::_Bind<void (nuraft::raft_server::*(nuraft::raft_server*))()>>(std::__invoke_other, std::_Bind<void (nuraft::raft_server::*(nuraft::raft_server*))()>&&) (__f=...) at /usr/include/c++/11/bits/invoke.h:61
#16 0x0000562de2729f09 in std::__invoke<std::_Bind<void (nuraft::raft_server::*(nuraft::raft_server*))()>>(std::_Bind<void (nuraft::raft_server::*(nuraft::raft_server*))()>&&) (__fn=...) at /usr/include/c++/11/bits/invoke.h:96
#17 0x0000562de2729eaa in std::thread::_Invoker<std::tuple<std::_Bind<void (nuraft::raft_server::*(nuraft::raft_server*))()> > >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x562de866e608) at /usr/include/c++/11/bits/std_thread.h:259
#18 0x0000562de2729e56 in std::thread::_Invoker<std::tuple<std::_Bind<void (nuraft::raft_server::*(nuraft::raft_server*))()> > >::operator()() (this=0x562de866e608) at /usr/include/c++/11/bits/std_thread.h:266
#19 0x0000562de2729e36 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind<void (nuraft::raft_server::*(nuraft::raft_server*))()> > > >::_M_run() (this=0x562de866e600) at /usr/include/c++/11/bits/std_thread.h:211
#20 0x00007ffaa2cce253 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#21 0x00007ffaa2956ac3 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#22 0x00007ffaa29e8850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81