oneTBB icon indicating copy to clipboard operation
oneTBB copied to clipboard
verified publisher icon oneapi-src

Possible data race reported by clang thread sanitizer

Open emmenlau opened this issue 2 years ago • 11 comments

We have found a possible data race reported when using OpenCV resize method, and reported it in https://github.com/opencv/opencv/issues/24063. It seems that the origin is in oneTBB, so copying the report here:

As far as we can see there is nothing specific in our build that would explain the issue. We have just reproduced the issue again with OpenCV 4.9.0 (current latest), oneTBB 2021.10.0 and clang 17.x on Linux. Previously the issue could be experienced with lower version numbers of all tools as well.

Here is what we did:

  • Build OpenCV against oneTBB, AFAIK without any super special parameters
  • Build a simple OpenCV executable that calls resize(), with clang on Linux in debug mode, and enable thread sanitizer in the build:
    • Add option -fsanitize=thread to compile and link arguments list of clang
  • Run the executable, and observe the below info printed

The executable will print:

[ INFO:[email protected]] global registry_parallel.impl.hpp:96 ParallelBackendRegistry core(parallel): Enabled backends(2, sorted by priority): TBB(1000); OPENMP(990)
[ INFO:[email protected]] global parallel_for.tbb.hpp:54 ParallelForBackend Initializing TBB parallel backend: TBB_INTERFACE_VERSION=12100
[ INFO:[email protected]] global parallel.cpp:77 createParallelForAPI core(parallel): using backend: TBB (priority=1000)
==================
WARNING: ThreadSanitizer: data race (pid=908834)
  Write of size 8 at 0x7b2000000000 by thread T4:
    #0 operator delete[](void*) <null> (OpenCVResizeDataRaceReproducer+0x4b00af) (BuildId: 2827f2d96c42af79)
    #1 tbb::detail::r1::destroy_process_mask() /home/user/Src/oneTBB/src/tbb/misc_ex.cpp:93:5 (libtbb_debug.so.12+0x7f2d5) (BuildId: c02118eec6345e66)

  Previous read of size 8 at 0x7b2000000000 by thread T2:
    #0 memcmp <null> (OpenCVResizeDataRaceReproducer+0x4336f0) (BuildId: 2827f2d96c42af79)
    #1 tbb::detail::r1::affinity_helper::protect_affinity_mask(bool) /home/user/Src/oneTBB/src/tbb/misc_ex.cpp:113:26 (libtbb_debug.so.12+0x7f49a) (BuildId: c02118eec6345e66)

  Thread T4 (tid=908839, running) created by thread T1 at:
    #0 pthread_create <null> (OpenCVResizeDataRaceReproducer+0x42786b) (BuildId: 2827f2d96c42af79)
    #1 tbb::detail::r1::rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) /home/user/Src/oneTBB/src/tbb/rml_thread_monitor.h:208:15 (libtbb_debug.so.12+0x877c5) (BuildId: c02118eec6345e66)

  Thread T2 (tid=908838, finished) created by main thread at:
    #0 pthread_create <null> (OpenCVResizeDataRaceReproducer+0x42786b) (BuildId: 2827f2d96c42af79)
    #1 tbb::detail::r1::rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) /home/user/Src/oneTBB/src/tbb/rml_thread_monitor.h:208:15 (libtbb_debug.so.12+0x877c5) (BuildId: c02118eec6345e66)
    #2 __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 (libc.so.6+0x29d8f) (BuildId: 203de0ae33b53fee1578b117cb4123e85d0534f0)

SUMMARY: ThreadSanitizer: data race (/data/user/Debug/OpenCVResizeDataRaceReproducer+0x4b00af) (BuildId: 2827f2d96c42af79) in operator delete[](void*)
==================
ThreadSanitizer: reported 1 warnings

emmenlau avatar Jan 10 '24 15:01 emmenlau

The following minimal example also produces a data race detected by ThreadSanitizer:

std::vector<std::uint64_t> v(1'000'000);
std::mt19937 gen {std::random_device{}()};
std::ranges::generate(v, gen);
std::sort(std::execution::par, v.begin(), v.end());

Error produced:

2:   Write of size 8 at 0x7f76295f7848 by thread T2:
2:     #0 __pstl::__tbb_backend::__func_task<__pstl::__tbb_backend::__stable_sort_func<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long*, std::less<unsigned long>, __pstl::__internal::__pattern_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false>, std::integral_constant<bool, true>, std::integral_constant<bool, true>)::{lambda()#1}::operator()() const::{lambda(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>)#1}> >::execute(tbb::detail::d1::execution_data&) /usr/include/c++/13/pstl/parallel_backend_tbb.h:618 (main.out+0x443c9f) (BuildId: 5e0a4b105b08ad76466044248f6318145de6ec50)
2:     #1 tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::outermost_worker_waiter>(tbb::detail::d1::task*, tbb::detail::r1::outermost_worker_waiter&) /localdisk/ci/runner010/intel-innersource/001/_work/libraries.threading.infrastructure.onetbb-ci/libraries.threading.infrastructure.onetbb-ci/onetbb_source_code/src/tbb/task_dispatcher.h:323 (libtbb_debug.so.12+0x1a365) (BuildId: 209272c75536bf0182a8275d42c8148347e6b5e5)
2: 
2:   Previous write of size 8 at 0x7f76295f7848 by main thread:
2:     #0 __pstl::__tbb_backend::__task::__task() /usr/include/c++/13/pstl/parallel_backend_tbb.h:515 (main.out+0x443848) (BuildId: 5e0a4b105b08ad76466044248f6318145de6ec50)
2:     #1 __pstl::__tbb_backend::__func_task<__pstl::__tbb_backend::__stable_sort_func<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long*, std::less<unsigned long>, __pstl::__internal::__pattern_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false>, std::integral_constant<bool, true>, std::integral_constant<bool, true>)::{lambda()#1}::operator()() const::{lambda(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>)#1}> >::__func_task<__pstl::__tbb_backend::__stable_sort_func<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long*, std::less<unsigned long>, __pstl::__internal::__pattern_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false>, std::integral_constant<bool, true>, std::integral_constant<bool, true>)::{lambda()#1}::operator()() const::{lambda(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>)#1}> >(__pstl::execution::v1::parallel_policy const&) /usr/include/c++/13/pstl/parallel_backend_tbb.h:663 (main.out+0x443848)
2:     #2 __pstl::__tbb_backend::__func_task<__pstl::__tbb_backend::__stable_sort_func<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long*, std::less<unsigned long>, __pstl::__internal::__pattern_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false>, std::integral_constant<bool, true>, std::integral_constant<bool, true>)::{lambda()#1}::operator()() const::{lambda(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>)#1}> >* tbb::detail::d1::small_object_allocator::new_object<__pstl::__tbb_backend::__func_task<__pstl::__tbb_backend::__stable_sort_func<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long*, std::less<unsigned long>, __pstl::__internal::__pattern_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false>, std::integral_constant<bool, true>, std::integral_constant<bool, true>)::{lambda()#1}::operator()() const::{lambda(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>)#1}> >, __pstl::__tbb_backend::__stable_sort_func<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long*, std::less<unsigned long>, __pstl::__internal::__pattern_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false>, std::integral_constant<bool, true>, std::integral_constant<bool, true>)::{lambda()#1}::operator()() const::{lambda(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>)#1}> >(tbb::detail::d1::execution_data&, __pstl::__tbb_backend::__stable_sort_func<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long*, std::less<unsigned long>, __pstl::__internal::__pattern_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false>, std::integral_constant<bool, true>, std::integral_constant<bool, true>)::{lambda()#1}::operator()() const::{lambda(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>)#1}>&&) /opt/intel/oneapi/tbb/2021.11/include/oneapi/tbb/detail/_small_object_pool.h:55 (main.out+0x443848)
2:     #3 __pstl::__tbb_backend::__task* __pstl::__tbb_backend::__task::allocate_func_task<__pstl::__tbb_backend::__stable_sort_func<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long*, std::less<unsigned long>, __pstl::__internal::__pattern_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false>, std::integral_constant<bool, true>, std::integral_constant<bool, true>)::{lambda()#1}::operator()() const::{lambda(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>)#1}> >(__pstl::execution::v1::parallel_policy const&) /usr/include/c++/13/pstl/parallel_backend_tbb.h:531 (main.out+0x443848)
2:     #4 __pstl::__tbb_backend::__task* __pstl::__tbb_backend::__task::make_child_of<__pstl::__tbb_backend::__stable_sort_func<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long*, std::less<unsigned long>, __pstl::__internal::__pattern_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false>, std::integral_constant<bool, true>, std::integral_constant<bool, true>)::{lambda()#1}::operator()() const::{lambda(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>)#1}> >(__pstl::__tbb_backend::__task*, __pstl::execution::v1::parallel_policy const&) /usr/include/c++/13/pstl/parallel_backend_tbb.h:563 (main.out+0x443848)
2:     #5 __pstl::__tbb_backend::__stable_sort_func<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long*, std::less<unsigned long>, __pstl::__internal::__pattern_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false>, std::integral_constant<bool, true>, std::integral_constant<bool, true>)::{lambda()#1}::operator()() const::{lambda(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>)#1}>::operator()(__pstl::__tbb_backend::__task*) /usr/include/c++/13/pstl/parallel_backend_tbb.h:1145 (main.out+0x443848)
2:     #6 __pstl::__tbb_backend::__func_task<__pstl::__tbb_backend::__stable_sort_func<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long*, std::less<unsigned long>, __pstl::__internal::__pattern_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false>, std::integral_constant<bool, true>, std::integral_constant<bool, true>)::{lambda()#1}::operator()() const::{lambda(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>)#1}> >::execute(tbb::detail::d1::execution_data&) /usr/include/c++/13/pstl/parallel_backend_tbb.h:620 (main.out+0x443cbb) (BuildId: 5e0a4b105b08ad76466044248f6318145de6ec50)
2:     #7 tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter>(tbb::detail::d1::task*, tbb::detail::r1::external_waiter&) /localdisk/ci/runner010/intel-innersource/001/_work/libraries.threading.infrastructure.onetbb-ci/libraries.threading.infrastructure.onetbb-ci/onetbb_source_code/src/tbb/task_dispatcher.h:323 (libtbb_debug.so.12+0x53c35) (BuildId: 209272c75536bf0182a8275d42c8148347e6b5e5)
2:     #8 tbb::detail::d1::task_arena_function<__pstl::__tbb_backend::__parallel_stable_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, __pstl::__internal::__pattern_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false>, std::integral_constant<bool, true>, std::integral_constant<bool, true>)::{lambda()#1}::operator()() const::{lambda(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>)#1}>(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, __pstl::__internal::__pattern_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false>, std::integral_constant<bool, true>, std::integral_constant<bool, true>)::{lambda()#1}::operator()() const::{lambda(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>)#1}, unsigned long)::{lambda()#1}, void>::operator()() const /opt/intel/oneapi/tbb/2021.11/include/oneapi/tbb/task_arena.h:68 (main.out+0x437cb6) (BuildId: 5e0a4b105b08ad76466044248f6318145de6ec50)
2:     #9 tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, long)::{lambda()#1}::operator()() const /localdisk/ci/runner010/intel-innersource/001/_work/libraries.threading.infrastructure.onetbb-ci/libraries.threading.infrastructure.onetbb-ci/onetbb_source_code/src/tbb/arena.cpp:865 (libtbb_debug.so.12+0x113e6) (BuildId: 209272c75536bf0182a8275d42c8148347e6b5e5)
2:     #10 void __pstl::__internal::__pattern_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false>, std::integral_constant<bool, true>, std::integral_constant<bool, true>) /usr/include/c++/13/pstl/algorithm_impl.h:2118 (main.out+0x40e62a) (BuildId: 5e0a4b105b08ad76466044248f6318145de6ec50)
2:     #11 std::enable_if<__pstl::execution::v1::is_execution_policy<std::remove_cv<std::remove_reference<__pstl::execution::v1::parallel_policy const&>::type>::type>::value, void>::type std::sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>) /usr/include/c++/13/pstl/glue_algorithm_impl.h:633 (main.out+0x40e62a)
2:     #12 std::enable_if<__pstl::execution::v1::is_execution_policy<std::remove_cv<std::remove_reference<__pstl::execution::v1::parallel_policy const&>::type>::type>::value, void>::type std::sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > > >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >) /usr/include/c++/13/pstl/glue_algorithm_impl.h:645 (main.out+0x40e62a)
2:     #13 main /<edited>/src/main.cpp:131 (main.out+0x40e62a)
2: 
2:   Thread T2 (tid=40991, running) created by main thread at:
2:     #0 pthread_create <null> (libtsan.so.2+0x3e006) (BuildId: 732e44e7f1cd4f0f9ca7d27895a253bebdea6827)
2:     #1 tbb::detail::r1::rml::internal::thread_monitor::launch(void* (*)(void*), void*, unsigned long) /localdisk/ci/runner010/intel-innersource/001/_work/libraries.threading.infrastructure.onetbb-ci/libraries.threading.infrastructure.onetbb-ci/onetbb_source_code/src/tbb/rml_thread_monitor.h:208 (libtbb_debug.so.12+0x4a120) (BuildId: 209272c75536bf0182a8275d42c8148347e6b5e5)
2:     #2 __pstl::__tbb_backend::__func_task<__pstl::__tbb_backend::__stable_sort_func<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long*, std::less<unsigned long>, __pstl::__internal::__pattern_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false>, std::integral_constant<bool, true>, std::integral_constant<bool, true>)::{lambda()#1}::operator()() const::{lambda(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>)#1}> >::execute(tbb::detail::d1::execution_data&) /usr/include/c++/13/pstl/parallel_backend_tbb.h:620 (main.out+0x443cbb) (BuildId: 5e0a4b105b08ad76466044248f6318145de6ec50)
2:     #3 tbb::detail::d1::task* tbb::detail::r1::task_dispatcher::local_wait_for_all<false, tbb::detail::r1::external_waiter>(tbb::detail::d1::task*, tbb::detail::r1::external_waiter&) /localdisk/ci/runner010/intel-innersource/001/_work/libraries.threading.infrastructure.onetbb-ci/libraries.threading.infrastructure.onetbb-ci/onetbb_source_code/src/tbb/task_dispatcher.h:323 (libtbb_debug.so.12+0x53c35) (BuildId: 209272c75536bf0182a8275d42c8148347e6b5e5)
2:     #4 tbb::detail::d1::task_arena_function<__pstl::__tbb_backend::__parallel_stable_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, __pstl::__internal::__pattern_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false>, std::integral_constant<bool, true>, std::integral_constant<bool, true>)::{lambda()#1}::operator()() const::{lambda(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>)#1}>(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, __pstl::__internal::__pattern_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false>, std::integral_constant<bool, true>, std::integral_constant<bool, true>)::{lambda()#1}::operator()() const::{lambda(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>)#1}, unsigned long)::{lambda()#1}, void>::operator()() const /opt/intel/oneapi/tbb/2021.11/include/oneapi/tbb/task_arena.h:68 (main.out+0x437cb6) (BuildId: 5e0a4b105b08ad76466044248f6318145de6ec50)
2:     #5 tbb::detail::r1::isolate_within_arena(tbb::detail::d1::delegate_base&, long)::{lambda()#1}::operator()() const /localdisk/ci/runner010/intel-innersource/001/_work/libraries.threading.infrastructure.onetbb-ci/libraries.threading.infrastructure.onetbb-ci/onetbb_source_code/src/tbb/arena.cpp:865 (libtbb_debug.so.12+0x113e6) (BuildId: 209272c75536bf0182a8275d42c8148347e6b5e5)
2:     #6 void __pstl::__internal::__pattern_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false>, std::integral_constant<bool, true>, std::integral_constant<bool, true>) /usr/include/c++/13/pstl/algorithm_impl.h:2118 (main.out+0x40e62a) (BuildId: 5e0a4b105b08ad76466044248f6318145de6ec50)
2:     #7 std::enable_if<__pstl::execution::v1::is_execution_policy<std::remove_cv<std::remove_reference<__pstl::execution::v1::parallel_policy const&>::type>::type>::value, void>::type std::sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>) /usr/include/c++/13/pstl/glue_algorithm_impl.h:633 (main.out+0x40e62a)
2:     #8 std::enable_if<__pstl::execution::v1::is_execution_policy<std::remove_cv<std::remove_reference<__pstl::execution::v1::parallel_policy const&>::type>::type>::value, void>::type std::sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > > >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >) /usr/include/c++/13/pstl/glue_algorithm_impl.h:645 (main.out+0x40e62a)
2:     #9 main  /<edited>/src/main.cpp:131 (main.out+0x40e62a)
2: 
2: SUMMARY: ThreadSanitizer: data race /usr/include/c++/13/pstl/parallel_backend_tbb.h:618 in __pstl::__tbb_backend::__func_task<__pstl::__tbb_backend::__stable_sort_func<__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long*, std::less<unsigned long>, __pstl::__internal::__pattern_sort<__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false> >(__pstl::execution::v1::parallel_policy const&, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>, std::integral_constant<bool, false>, std::integral_constant<bool, true>, std::integral_constant<bool, true>)::{lambda()#1}::operator()() const::{lambda(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, __gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, std::less<unsigned long>)#1}> >::execute(tbb::detail::d1::execution_data&)
2: ==================

I am using gcc:

❯ gcc -v    
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/13/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,objc,obj-c++,ada,go,d,m2,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --enable-libstdcxx-backtrace --with-libstdcxx-zoneinfo=/usr/share/zoneinfo --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl=/builddir/build/BUILD/gcc-13.2.1-20231205/obj-x86_64-redhat-linux/isl-install --enable-offload-targets=nvptx-none --without-cuda-driver --enable-offload-defaulted --enable-gnu-indirect-function --enable-cet --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux --with-build-config=bootstrap-lto --enable-link-serialization=1
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 13.2.1 20231205 (Red Hat 13.2.1-6) (GCC)

Alejandro-FA avatar Feb 12 '24 20:02 Alejandro-FA

Hi @emmenlau, @Alejandro-FA, could you please try to rebuild TBB with ThreadSanitizer support .

pavelkumbrasev avatar Feb 13 '24 10:02 pavelkumbrasev

Hi @pavelkumbrasev, I have tried it but I get an error in the build process.

I have configured the project as follows:

cd /tmp
git clone https://github.com/oneapi-src/oneTBB.git
cd oneTBB
mkdir cmake-build && cd cmake-build
cmake -DCMAKE_INSTALL_PREFIX=/tmp/intel/oneTBB_tsan -DTBB_SANITIZE=thread ..

Here is the output of the configuration process:

-- The CXX compiler identification is GNU 13.2.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- CMAKE_BUILD_TYPE is not specified. Using default: RelWithDebInfo
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Checking for one of the modules 'hwloc'
-- Looking for getcontext
-- Looking for getcontext - found
-- Performing Test _fsanitize_thread
-- Performing Test _fsanitize_thread - Success
-- The C compiler identification is GNU 13.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- The tbbbind_2_5 target will be configured using the HWLOC 2.10.0
-- Performing Test LINKER_HAS_NO_AS_NEEDED
-- Performing Test LINKER_HAS_NO_AS_NEEDED - Success
-- Configuring done (0.7s)
-- Generating done (0.1s)
-- Build files have been written to: /tmp/oneTBB/cmake-build

And when doing make -j I get the following error:

[ 15%] Building CXX object test/CMakeFiles/test_malloc_lib_unload.dir/tbbmalloc/test_malloc_lib_unload.cpp.o
In file included from /usr/include/c++/13/atomic:41,
                 from /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_utils.h:22,
                 from /tmp/oneTBB/src/tbb/concurrent_bounded_queue.cpp:17:
In function ‘void std::atomic_thread_fence(memory_order)’,
    inlined from ‘void tbb::detail::d0::atomic_fence_seq_cst()’ at /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_machine.h:87:29,
    inlined from ‘void tbb::detail::r1::concurrent_monitor_base<Context>::prepare_wait(tbb::detail::r1::wait_node<Context>&) [with Context = long unsigned int]’ at /tmp/oneTBB/src/tbb/concurrent_monitor.h:223:29,
    inlined from ‘bool tbb::detail::r1::concurrent_monitor_base<Context>::wait(Pred&&, NodeType&&) [with NodeType = tbb::detail::r1::sleep_node<long unsigned int>; Pred = tbb::detail::r1::wait_bounded_queue_monitor(concurrent_monitor*, std::size_t, std::ptrdiff_t, tbb::detail::d1::delegate_base&)::<lambda()>; Context = long unsigned int]’ at /tmp/oneTBB/src/tbb/concurrent_monitor.h:260:21,
    inlined from ‘void tbb::detail::r1::wait_bounded_queue_monitor(concurrent_monitor*, std::size_t, std::ptrdiff_t, tbb::detail::d1::delegate_base&)’ at /tmp/oneTBB/src/tbb/concurrent_bounded_queue.cpp:57:53:
/usr/include/c++/13/bits/atomic_base.h:144:26: error: ‘atomic_thread_fence’ is not supported with ‘-fsanitize=thread’ [-Werror=tsan]
  144 |   { __atomic_thread_fence(int(__m)); }
      |     ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
compilation terminated due to -Wfatal-errors.
cc1plus: all warnings being treated as errors
make[2]: *** [src/tbb/CMakeFiles/tbb.dir/build.make:132: src/tbb/CMakeFiles/tbb.dir/concurrent_bounded_queue.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /usr/include/c++/13/atomic:41,
                 from /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_utils.h:22,
                 from /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/tbb_allocator.h:20,
                 from /tmp/oneTBB/src/tbb/task_group_context.cpp:18:
In function ‘void std::atomic_thread_fence(memory_order)’,
    inlined from ‘void tbb::detail::d0::atomic_fence_seq_cst()’ at /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_machine.h:87:29,
    inlined from ‘void tbb::detail::r1::concurrent_monitor_base<Context>::prepare_wait(tbb::detail::r1::wait_node<Context>&) [with Context = tbb::detail::r1::market_context]’ at /tmp/oneTBB/src/tbb/concurrent_monitor.h:223:29,
    inlined from ‘bool tbb::detail::r1::concurrent_monitor_base<Context>::wait(Pred&&, NodeType&&) [with NodeType = tbb::detail::r1::resume_node&; Pred = tbb::detail::r1::suspend_point_type::resume_task::execute(tbb::detail::d1::execution_data&)::<lambda()>; Context = tbb::detail::r1::market_context]’ at /tmp/oneTBB/src/tbb/concurrent_monitor.h:260:21,
    inlined from ‘virtual tbb::detail::d1::task* tbb::detail::r1::suspend_point_type::resume_task::execute(tbb::detail::d1::execution_data&)’ at /tmp/oneTBB/src/tbb/task_dispatcher.h:77:27:
/usr/include/c++/13/bits/atomic_base.h:144:26: error: ‘atomic_thread_fence’ is not supported with ‘-fsanitize=thread’ [-Werror=tsan]
  144 |   { __atomic_thread_fence(int(__m)); }
      |     ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
compilation terminated due to -Wfatal-errors.
cc1plus: all warnings being treated as errors
make[2]: *** [src/tbb/CMakeFiles/tbb.dir/build.make:440: src/tbb/CMakeFiles/tbb.dir/task_group_context.cpp.o] Error 1
In file included from /usr/include/c++/13/atomic:41,
                 from /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_utils.h:22,
                 from /tmp/oneTBB/src/tbb/arena_slot.h:21,
                 from /tmp/oneTBB/src/tbb/arena_slot.cpp:17:
In function ‘void std::atomic_thread_fence(memory_order)’,
    inlined from ‘void tbb::detail::d0::atomic_fence_seq_cst()’ at /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_machine.h:87:29,
    inlined from ‘void tbb::detail::r1::arena::advertise_new_work() [with new_work_type work_type = tbb::detail::r1::arena::wakeup]’ at /tmp/oneTBB/src/tbb/arena.h:408:29:
/usr/include/c++/13/bits/atomic_base.h:144:26: error: ‘atomic_thread_fence’ is not supported with ‘-fsanitize=thread’ [-Werror=tsan]
  144 |   { __atomic_thread_fence(int(__m)); }
      |     ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
compilation terminated due to -Wfatal-errors.
cc1plus: all warnings being treated as errors
[ 15%] Linking CXX shared library ../../gnu_13.2_cxx11_64_relwithdebinfo/libtbbbind_2_5.so
make[2]: *** [src/tbb/CMakeFiles/tbb.dir/build.make:118: src/tbb/CMakeFiles/tbb.dir/arena_slot.cpp.o] Error 1
[ 15%] Built target tbbbind_2_5
In file included from /usr/include/c++/13/atomic:41,
                 from /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_utils.h:22,
                 from /tmp/oneTBB/src/tbb/misc.h:22,
                 from /tmp/oneTBB/src/tbb/governor.h:22,
                 from /tmp/oneTBB/src/tbb/governor.cpp:17:
In function ‘void std::atomic_thread_fence(memory_order)’,
    inlined from ‘void tbb::detail::d0::atomic_fence_seq_cst()’ at /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_machine.h:87:29,
    inlined from ‘void tbb::detail::r1::concurrent_monitor_base<Context>::prepare_wait(tbb::detail::r1::wait_node<Context>&) [with Context = tbb::detail::r1::market_context]’ at /tmp/oneTBB/src/tbb/concurrent_monitor.h:223:29,
    inlined from ‘bool tbb::detail::r1::concurrent_monitor_base<Context>::wait(Pred&&, NodeType&&) [with NodeType = tbb::detail::r1::resume_node&; Pred = tbb::detail::r1::suspend_point_type::resume_task::execute(tbb::detail::d1::execution_data&)::<lambda()>; Context = tbb::detail::r1::market_context]’ at /tmp/oneTBB/src/tbb/concurrent_monitor.h:260:21,
    inlined from ‘virtual tbb::detail::d1::task* tbb::detail::r1::suspend_point_type::resume_task::execute(tbb::detail::d1::execution_data&)’ at /tmp/oneTBB/src/tbb/task_dispatcher.h:77:27:
/usr/include/c++/13/bits/atomic_base.h:144:26: error: ‘atomic_thread_fence’ is not supported with ‘-fsanitize=thread’ [-Werror=tsan]
  144 |   { __atomic_thread_fence(int(__m)); }
      |     ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
compilation terminated due to -Wfatal-errors.
cc1plus: all warnings being treated as errors
make[2]: *** [src/tbb/CMakeFiles/tbb.dir/build.make:174: src/tbb/CMakeFiles/tbb.dir/governor.cpp.o] Error 1
In file included from /usr/include/c++/13/atomic:41,
                 from /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_utils.h:22,
                 from /tmp/oneTBB/src/tbb/address_waiter.cpp:17:
In function ‘void std::atomic_thread_fence(memory_order)’,
    inlined from ‘void tbb::detail::d0::atomic_fence_seq_cst()’ at /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_machine.h:87:29,
    inlined from ‘void tbb::detail::r1::concurrent_monitor_base<Context>::prepare_wait(tbb::detail::r1::wait_node<Context>&) [with Context = tbb::detail::r1::address_context]’ at /tmp/oneTBB/src/tbb/concurrent_monitor.h:223:29,
    inlined from ‘bool tbb::detail::r1::concurrent_monitor_base<Context>::wait(Pred&&, NodeType&&) [with NodeType = tbb::detail::r1::sleep_node<tbb::detail::r1::address_context>; Pred = tbb::detail::d1::delegate_base&; Context = tbb::detail::r1::address_context]’ at /tmp/oneTBB/src/tbb/concurrent_monitor.h:260:21,
    inlined from ‘void tbb::detail::r1::wait_on_address(void*, tbb::detail::d1::delegate_base&, uintptr_t)’ at /tmp/oneTBB/src/tbb/address_waiter.cpp:71:48:
/usr/include/c++/13/bits/atomic_base.h:144:26: error: ‘atomic_thread_fence’ is not supported with ‘-fsanitize=thread’ [-Werror=tsan]
  144 |   { __atomic_thread_fence(int(__m)); }
      |     ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
compilation terminated due to -Wfatal-errors.
cc1plus: all warnings being treated as errors
make[2]: *** [src/tbb/CMakeFiles/tbb.dir/build.make:76: src/tbb/CMakeFiles/tbb.dir/address_waiter.cpp.o] Error 1
In file included from /usr/include/c++/13/atomic:41,
                 from /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_utils.h:22,
                 from /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/cache_aligned_allocator.h:20,
                 from /tmp/oneTBB/src/tbb/small_object_pool.cpp:17:
In function ‘void std::atomic_thread_fence(memory_order)’,
    inlined from ‘void tbb::detail::d0::atomic_fence_seq_cst()’ at /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_machine.h:87:29,
    inlined from ‘void tbb::detail::r1::concurrent_monitor_base<Context>::prepare_wait(tbb::detail::r1::wait_node<Context>&) [with Context = tbb::detail::r1::market_context]’ at /tmp/oneTBB/src/tbb/concurrent_monitor.h:223:29,
    inlined from ‘bool tbb::detail::r1::concurrent_monitor_base<Context>::wait(Pred&&, NodeType&&) [with NodeType = tbb::detail::r1::resume_node&; Pred = tbb::detail::r1::suspend_point_type::resume_task::execute(tbb::detail::d1::execution_data&)::<lambda()>; Context = tbb::detail::r1::market_context]’ at /tmp/oneTBB/src/tbb/concurrent_monitor.h:260:21,
    inlined from ‘virtual tbb::detail::d1::task* tbb::detail::r1::suspend_point_type::resume_task::execute(tbb::detail::d1::execution_data&)’ at /tmp/oneTBB/src/tbb/task_dispatcher.h:77:27:
/usr/include/c++/13/bits/atomic_base.h:144:26: error: ‘atomic_thread_fence’ is not supported with ‘-fsanitize=thread’ [-Werror=tsan]
  144 |   { __atomic_thread_fence(int(__m)); }
      |     ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
compilation terminated due to -Wfatal-errors.
cc1plus: all warnings being treated as errors
make[2]: *** [src/tbb/CMakeFiles/tbb.dir/build.make:398: src/tbb/CMakeFiles/tbb.dir/small_object_pool.cpp.o] Error 1
In file included from /usr/include/c++/13/atomic:41,
                 from /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_utils.h:22,
                 from /tmp/oneTBB/src/tbb/task_dispatcher.h:20,
                 from /tmp/oneTBB/src/tbb/arena.cpp:17:
In function ‘void std::atomic_thread_fence(memory_order)’,
    inlined from ‘void tbb::detail::d0::atomic_fence_seq_cst()’ at /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_machine.h:87:29,
    inlined from ‘void tbb::detail::r1::concurrent_monitor_base<Context>::notify(const P&) [with P = tbb::detail::r1::arena::request_workers(int, int, bool)::<lambda(tbb::detail::r1::market_context)>; Context = tbb::detail::r1::market_context]’ at /tmp/oneTBB/src/tbb/concurrent_monitor.h:350:29,
    inlined from ‘void tbb::detail::r1::arena::request_workers(int, int, bool)’ at /tmp/oneTBB/src/tbb/arena.cpp:352:45:
/usr/include/c++/13/bits/atomic_base.h:144:26: error: ‘atomic_thread_fence’ is not supported with ‘-fsanitize=thread’ [-Werror=tsan]
  144 |   { __atomic_thread_fence(int(__m)); }
      |     ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
compilation terminated due to -Wfatal-errors.
cc1plus: all warnings being treated as errors
make[2]: *** [src/tbb/CMakeFiles/tbb.dir/build.make:104: src/tbb/CMakeFiles/tbb.dir/arena.cpp.o] Error 1
In file included from /usr/include/c++/13/atomic:41,
                 from /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_utils.h:22,
                 from /tmp/oneTBB/src/tbb/scheduler_common.h:20,
                 from /tmp/oneTBB/src/tbb/task.cpp:18:
In function ‘void std::atomic_thread_fence(memory_order)’,
    inlined from ‘void tbb::detail::d0::atomic_fence_seq_cst()’ at /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_machine.h:87:29,
    inlined from ‘void tbb::detail::r1::concurrent_monitor_base<Context>::notify(const P&) [with P = tbb::detail::r1::notify_waiters(uintptr_t)::<lambda(tbb::detail::r1::market_context)>; Context = tbb::detail::r1::market_context]’ at /tmp/oneTBB/src/tbb/concurrent_monitor.h:350:29,
    inlined from ‘void tbb::detail::r1::notify_waiters(uintptr_t)’ at /tmp/oneTBB/src/tbb/task.cpp:221:80:
/usr/include/c++/13/bits/atomic_base.h:144:26: error: ‘atomic_thread_fence’ is not supported with ‘-fsanitize=thread’ [-Werror=tsan]
  144 |   { __atomic_thread_fence(int(__m)); }
      |     ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
compilation terminated due to -Wfatal-errors.
cc1plus: all warnings being treated as errors
make[2]: *** [src/tbb/CMakeFiles/tbb.dir/build.make:412: src/tbb/CMakeFiles/tbb.dir/task.cpp.o] Error 1
[ 15%] Linking C shared library ../../gnu_13.2_cxx11_64_relwithdebinfo/libtbbmalloc.so
In file included from /usr/include/c++/13/atomic:41,
                 from /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_utils.h:22,
                 from /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/mutex.h:21,
                 from /tmp/oneTBB/src/tbb/threading_control.h:20,
                 from /tmp/oneTBB/src/tbb/threading_control.cpp:17:
In function ‘void std::atomic_thread_fence(memory_order)’,
    inlined from ‘void tbb::detail::d0::atomic_fence_seq_cst()’ at /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_machine.h:87:29,
    inlined from ‘void tbb::detail::r1::concurrent_monitor_base<Context>::abort_all() [with Context = tbb::detail::r1::market_context]’ at /tmp/oneTBB/src/tbb/concurrent_monitor.h:422:29,
    inlined from ‘void tbb::detail::r1::concurrent_monitor_base<Context>::destroy() [with Context = tbb::detail::r1::market_context]’ at /tmp/oneTBB/src/tbb/concurrent_monitor.h:456:24,
    inlined from ‘tbb::detail::r1::thread_control_monitor::~thread_control_monitor()’ at /tmp/oneTBB/src/tbb/thread_control_monitor.h:102:16,
    inlined from ‘void tbb::detail::r1::cache_aligned_deleter::operator()(T*) const [with T = tbb::detail::r1::thread_control_monitor]’ at /tmp/oneTBB/src/tbb/scheduler_common.h:75:16,
    inlined from ‘std::unique_ptr<_Tp, _Dp>::~unique_ptr() [with _Tp = tbb::detail::r1::thread_control_monitor; _Dp = tbb::detail::r1::cache_aligned_deleter]’ at /usr/include/c++/13/bits/unique_ptr.h:404:17,
    inlined from ‘tbb::detail::r1::threading_control_impl::~threading_control_impl()’ at /tmp/oneTBB/src/tbb/threading_control.h:42:7,
    inlined from ‘void tbb::detail::r1::cache_aligned_deleter::operator()(T*) const [with T = tbb::detail::r1::threading_control_impl]’ at /tmp/oneTBB/src/tbb/scheduler_common.h:75:16,
    inlined from ‘std::unique_ptr<_Tp, _Dp>::~unique_ptr() [with _Tp = tbb::detail::r1::threading_control_impl; _Dp = tbb::detail::r1::cache_aligned_deleter]’ at /usr/include/c++/13/bits/unique_ptr.h:404:17:
/usr/include/c++/13/bits/atomic_base.h:144:26: error: ‘atomic_thread_fence’ is not supported with ‘-fsanitize=thread’ [-Werror=tsan]
  144 |   { __atomic_thread_fence(int(__m)); }
      |     ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
compilation terminated due to -Wfatal-errors.
cc1plus: all warnings being treated as errors
[ 15%] Built target tbbmalloc
make[2]: *** [src/tbb/CMakeFiles/tbb.dir/build.make:482: src/tbb/CMakeFiles/tbb.dir/threading_control.cpp.o] Error 1
[ 15%] Building CXX object test/CMakeFiles/_test_malloc_used_by_lib.dir/tbbmalloc/test_malloc_used_by_lib.cpp.o
[ 15%] Building CXX object test/CMakeFiles/test_malloc_init_shutdown.dir/tbbmalloc/test_malloc_init_shutdown.cpp.o
[ 15%] Building C object test/CMakeFiles/test_malloc_pure_c.dir/tbbmalloc/test_malloc_pure_c.c.o
[ 16%] Building CXX object src/tbbmalloc_proxy/CMakeFiles/tbbmalloc_proxy.dir/function_replacement.cpp.o
[ 16%] Building CXX object test/CMakeFiles/test_malloc_compliance.dir/tbbmalloc/test_malloc_compliance.cpp.o
[ 16%] Building CXX object src/tbbmalloc_proxy/CMakeFiles/tbbmalloc_proxy.dir/proxy.cpp.o
[ 16%] Building CXX object test/CMakeFiles/test_malloc_pools.dir/tbbmalloc/test_malloc_pools.cpp.o
[ 16%] Building CXX object test/CMakeFiles/test_scalable_allocator.dir/tbbmalloc/test_scalable_allocator.cpp.o
[ 17%] Building CXX object test/CMakeFiles/test_malloc_regression.dir/tbbmalloc/test_malloc_regression.cpp.o
In file included from /usr/include/c++/13/atomic:41,
                 from /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_utils.h:22,
                 from /tmp/oneTBB/src/tbb/task_dispatcher.h:20,
                 from /tmp/oneTBB/src/tbb/task_dispatcher.cpp:17:
In function ‘void std::atomic_thread_fence(memory_order)’,
    inlined from ‘void tbb::detail::d0::atomic_fence_seq_cst()’ at /tmp/oneTBB/src/tbb/../../include/oneapi/tbb/detail/_machine.h:87:29,
    inlined from ‘void tbb::detail::r1::concurrent_monitor_base<Context>::prepare_wait(tbb::detail::r1::wait_node<Context>&) [with Context = tbb::detail::r1::market_context]’ at /tmp/oneTBB/src/tbb/concurrent_monitor.h:223:29,
    inlined from ‘bool tbb::detail::r1::concurrent_monitor_base<Context>::wait(Pred&&, NodeType&&) [with NodeType = tbb::detail::r1::sleep_node<tbb::detail::r1::market_context>; Pred = tbb::detail::r1::coroutine_waiter::pause(tbb::detail::r1::arena_slot&)::<lambda()>&; Context = tbb::detail::r1::market_context]’ at /tmp/oneTBB/src/tbb/concurrent_monitor.h:260:21,
    inlined from ‘void tbb::detail::r1::sleep_waiter::sleep(uintptr_t, Pred) [with Pred = tbb::detail::r1::coroutine_waiter::pause(tbb::detail::r1::arena_slot&)::<lambda()>]’ at /tmp/oneTBB/src/tbb/waiters.h:115:92,
    inlined from ‘void tbb::detail::r1::coroutine_waiter::pause(tbb::detail::r1::arena_slot&)’ at /tmp/oneTBB/src/tbb/waiters.h:178:14:
/usr/include/c++/13/bits/atomic_base.h:144:26: error: ‘atomic_thread_fence’ is not supported with ‘-fsanitize=thread’ [-Werror=tsan]
  144 |   { __atomic_thread_fence(int(__m)); }
      |     ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
compilation terminated due to -Wfatal-errors.
cc1plus: all warnings being treated as errors
make[2]: *** [src/tbb/CMakeFiles/tbb.dir/build.make:426: src/tbb/CMakeFiles/tbb.dir/task_dispatcher.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:457: src/tbb/CMakeFiles/tbb.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 17%] Linking C executable ../gnu_13.2_cxx11_64_relwithdebinfo/test_malloc_pure_c
[ 17%] Built target test_malloc_pure_c
[ 18%] Linking CXX shared library ../gnu_13.2_cxx11_64_relwithdebinfo/lib_test_malloc_used_by_lib.so
[ 18%] Built target _test_malloc_used_by_lib
[ 18%] Linking CXX shared library ../../gnu_13.2_cxx11_64_relwithdebinfo/libtbbmalloc_proxy.so
[ 18%] Built target tbbmalloc_proxy
[ 18%] Linking CXX executable ../gnu_13.2_cxx11_64_relwithdebinfo/test_handle_perror
[ 18%] Built target test_handle_perror
[ 18%] Linking CXX executable ../gnu_13.2_cxx11_64_relwithdebinfo/test_malloc_lib_unload
[ 18%] Built target test_malloc_lib_unload
[ 18%] Linking CXX executable ../gnu_13.2_cxx11_64_relwithdebinfo/test_malloc_init_shutdown
[ 18%] Built target test_malloc_init_shutdown
[ 18%] Linking CXX executable ../gnu_13.2_cxx11_64_relwithdebinfo/test_malloc_regression
[ 18%] Built target test_malloc_regression
[ 19%] Linking CXX executable ../gnu_13.2_cxx11_64_relwithdebinfo/test_malloc_compliance
[ 19%] Built target test_malloc_compliance
[ 20%] Linking CXX executable ../gnu_13.2_cxx11_64_relwithdebinfo/test_malloc_pools
[ 20%] Built target test_malloc_pools
[ 20%] Linking CXX executable ../gnu_13.2_cxx11_64_relwithdebinfo/test_malloc_whitebox
[ 21%] Linking CXX executable ../gnu_13.2_cxx11_64_relwithdebinfo/test_scalable_allocator
[ 21%] Built target test_malloc_whitebox
[ 21%] Built target test_scalable_allocator
make: *** [Makefile:166: all] Error 2

Alejandro-FA avatar Feb 13 '24 11:02 Alejandro-FA

From your comment, do I understand correctly that if I want to use Thread Sanitizer with my application, I need to build oneTBB with Thread Sanitizer support? At the moment I simply installed oneTBB following these instructions: https://www.intel.com/content/www/us/en/docs/onetbb/get-started-guide/2021-6/install-onetbb-on-linux-os.html

Alejandro-FA avatar Feb 13 '24 11:02 Alejandro-FA

GCC and clang TSAN does not support atomic_thread_fence:

  • https://stackoverflow.com/questions/70542993/why-does-the-thread-sanitizer-complain-about-acquire-release-thread-fences
  • https://gcc.gnu.org/bugzilla/show_bug.cgi?id=97868
  • https://github.com/google/sanitizers/issues/1352

phprus avatar Feb 13 '24 13:02 phprus

@Alejandro-FA, it might lead to unexpected false positive reports that will lead into TBB. So to properly test application that use TBB it also should be built with TSan support.

The error you got probably related to GNU 13.2.1 (we didn't test TSan with this version yet) (problem). Could you please try to build it with different version (GNU < 11 should be fine).

pavelkumbrasev avatar Feb 13 '24 14:02 pavelkumbrasev

@emmenlau is this issue still relevant?

nofuturre avatar Jul 12 '24 14:07 nofuturre

Sorry for not getting back to this! I did not have time to look into it any further. It's well possible that there is no problem, but I would need to follow it up by building our whole tech stack with TSan, to be sure. Is it fine to leave the issue open a bit longer?

emmenlau avatar Jul 12 '24 14:07 emmenlau

Sure, no problem.

Please let us know if there still an issue we will look into that.

nofuturre avatar Jul 12 '24 14:07 nofuturre