wasm-micro-runtime AOT data race with pigz test app

Hello, I'm running the pigz app from https://github.com/bytecodealliance/wasm-parallel-gzip on Ubuntu and running into some TSAN warnings:

./iwasm -v=5 --dir=wasm-apps wasm-apps/pigz.aot -p 3 wasm-apps/test.txt
[00:44:04:764 - 7F7C90D8A380]: Load module success.

[00:44:04:821 - 7F7C90D8A380]: Memory instantiate:
[00:44:04:825 - 7F7C90D8A380]:   page bytes: 65536, init pages: 4, max pages: 1024
[00:44:04:827 - 7F7C90D8A380]:   data offset: 0, stack size: 0
[00:44:04:829 - 7F7C90D8A380]:   heap offset: 262144, heap size: 0

==================
WARNING: ThreadSanitizer: data race (pid=185402)
  Write of size 8 at 0x7f7a8dda6360 by thread T3:
    #0 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:810 (libtsan.so.0+0x6225e)
    #1 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:808 (libtsan.so.0+0x6225e)
    #2 aot_memmove wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 (iwasm+0x34f71)
    #3 <null> <null> (0x000041271755)
    #4 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32715)
    #5 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f71)
    #6 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2037 (iwasm+0x13af4)
    #7 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a5c)
    #8 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d2d)
    #9 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb0d)

  Previous read of size 8 at 0x7f7a8dda6360 by thread T1:
    #0 writev ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:1203 (libtsan.so.0+0x4ce32)
    #1 os_writev wasm-micro-runtime/core/shared/platform/common/posix/posix_file.c:578 (iwasm+0x391e2)
    #2 blocking_op_writev wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/blocking_op.c:56 (iwasm+0x86171)
    #3 wasmtime_ssp_fd_write wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c:1120 (iwasm+0x42b37)
    #4 wasi_fd_write wasm-micro-runtime/core/iwasm/libraries/libc-wasi/libc_wasi_wrapper.c:669 (iwasm+0x81ebd)
    #5 <null> <null> (0x000041280a02)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32715)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f71)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2037 (iwasm+0x13af4)
    #9 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a5c)
    #10 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d2d)
    #11 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb0d)

  Thread T3 (tid=185406, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd1e)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdb8)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37123)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d7a)
    #5 <null> <null> (0x000041281d65)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32715)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f71)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2037 (iwasm+0x13af4)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf00a)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf608)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:112 (iwasm+0xa69c)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:941 (iwasm+0xc177)

  Thread T1 (tid=185404, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd1e)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdb8)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37123)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d7a)
    #5 <null> <null> (0x000041281d65)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32715)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f71)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2037 (iwasm+0x13af4)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf00a)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf608)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:112 (iwasm+0xa69c)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:941 (iwasm+0xc177)

SUMMARY: ThreadSanitizer: data race wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 in aot_memmove
==================
==================
WARNING: ThreadSanitizer: data race (pid=185402)
  Write of size 8 at 0x7f7a8dd8e220 by main thread:
    #0 readv ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:1094 (libtsan.so.0+0x53e74)
    #1 os_readv wasm-micro-runtime/core/shared/platform/common/posix/posix_file.c:564 (iwasm+0x39156)
    #2 blocking_op_readv wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/blocking_op.c:30 (iwasm+0x86070)
    #3 wasmtime_ssp_fd_read wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c:884 (iwasm+0x41fc0)
    #4 wasi_fd_read wasm-micro-runtime/core/iwasm/libraries/libc-wasi/libc_wasi_wrapper.c:500 (iwasm+0x8174c)
    #5 <null> <null> (0x000041280882)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32715)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f71)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2037 (iwasm+0x13af4)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf00a)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf608)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:112 (iwasm+0xa69c)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:941 (iwasm+0xc177)

  Previous read of size 8 at 0x7f7a8dd8e220 by thread T3:
    #0 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:810 (libtsan.so.0+0x6225e)
    #1 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:808 (libtsan.so.0+0x6225e)
    #2 aot_memmove wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 (iwasm+0x34f71)
    #3 <null> <null> (0x00004126b22a)
    #4 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32715)
    #5 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f71)
    #6 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2037 (iwasm+0x13af4)
    #7 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a5c)
    #8 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d2d)
    #9 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb0d)

  Thread T3 (tid=185406, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd1e)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdb8)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37123)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d7a)
    #5 <null> <null> (0x000041281d65)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32715)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f71)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2037 (iwasm+0x13af4)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf00a)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf608)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:112 (iwasm+0xa69c)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:941 (iwasm+0xc177)

SUMMARY: ThreadSanitizer: data race wasm-micro-runtime/core/shared/platform/common/posix/posix_file.c:564 in os_readv
==================
==================
WARNING: ThreadSanitizer: data race (pid=185402)
  Write of size 8 at 0x7f7a8de4dc48 by thread T4:
    #0 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:810 (libtsan.so.0+0x6225e)
    #1 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:808 (libtsan.so.0+0x6225e)
    #2 aot_memmove wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 (iwasm+0x34f71)
    #3 <null> <null> (0x000041271c2b)
    #4 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32715)
    #5 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f71)
    #6 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2037 (iwasm+0x13af4)
    #7 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a5c)
    #8 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d2d)
    #9 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb0d)

  Previous read of size 8 at 0x7f7a8de4dc48 by thread T1:
    #0 writev ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:1203 (libtsan.so.0+0x4ce32)
    #1 os_writev wasm-micro-runtime/core/shared/platform/common/posix/posix_file.c:578 (iwasm+0x391e2)
    #2 blocking_op_writev wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/blocking_op.c:56 (iwasm+0x86171)
    #3 wasmtime_ssp_fd_write wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c:1120 (iwasm+0x42b37)
    #4 wasi_fd_write wasm-micro-runtime/core/iwasm/libraries/libc-wasi/libc_wasi_wrapper.c:669 (iwasm+0x81ebd)
    #5 <null> <null> (0x000041280a02)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32715)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f71)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2037 (iwasm+0x13af4)
    #9 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a5c)
    #10 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d2d)
    #11 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb0d)

  Thread T4 (tid=185407, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd1e)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdb8)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37123)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d7a)
    #5 <null> <null> (0x000041281d65)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32715)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f71)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2037 (iwasm+0x13af4)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf00a)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf608)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:112 (iwasm+0xa69c)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:941 (iwasm+0xc177)

  Thread T1 (tid=185404, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd1e)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdb8)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37123)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d7a)
    #5 <null> <null> (0x000041281d65)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32715)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f71)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2037 (iwasm+0x13af4)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf00a)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf608)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:112 (iwasm+0xa69c)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:941 (iwasm+0xc177)

SUMMARY: ThreadSanitizer: data race wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 in aot_memmove
==================
ThreadSanitizer: reported 3 warnings

The pigz.wasm file I'm using: pigz.zip, copied into wasm-micro-runtime/samples/wasi-threads/build/wasm-apps.

How to reproduce from the latest main branch (assuning iwasm and wamrc already been built):

$ cd wasm-micro-runtime/samples/wasi-threads/build
$ ../../../wamr-compiler/build/wamrc \
    --enable-multi-thread \
    -o wasm-apps/pigz.aot wasm-apps/pigz.wasm
Create AoT compiler with:
  target:        x86_64
  target cpu:    alderlake
  target triple: x86_64-unknown-linux-gnu
  cpu features:  
  opt level:     3
  size level:    3
  output format: AoT file
Compile success, file wasm-apps/pigz.aot was generated.
$ head -c 10M </dev/urandom >wasm-apps/test.txt  # Generate random file
$ rm -rf wasm-apps/test.txt.gz  # Remove previously compressed random file
$ ninja && ./iwasm -v=5 --dir=wasm-apps wasm-apps/pigz.aot  -p 3 wasm-apps/test.txt

How I added the sanitizer:

git diff
diff --git a/samples/wasi-threads/CMakeLists.txt b/samples/wasi-threads/CMakeLists.txt
index 467f5fd1..135dbd86 100644
--- a/samples/wasi-threads/CMakeLists.txt
+++ b/samples/wasi-threads/CMakeLists.txt
@@ -70,6 +70,9 @@ add_subdirectory(wasm-apps)
 ################ wamr runtime ################
 include (${SHARED_DIR}/utils/uncommon/shared_uncommon.cmake)
 
+set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fsanitize=thread")
+set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fsanitize=thread")
+
 set (RUNTIME_SOURCE_ALL
     ${CMAKE_CURRENT_LIST_DIR}/../../product-mini/platforms/linux/main.c
     ${UNCOMMON_SHARED_SOURCE}

Nov 20 '23 14:11 eloparco

@eloparco I have a try, the aot_memmove and aot_memset are called from the AOT code when bulk memory feature is enabled, in which the wasm opcode memory.copy and memory.fill are compiled into calling aot_memmove and aot_memset. Acquiring the global shared memory lock in aot_memove can make the TSAN warning disappear:

diff --git a/core/iwasm/aot/aot_runtime.c b/core/iwasm/aot/aot_runtime.c
index 076574cb..6b8e170a 100644
--- a/core/iwasm/aot/aot_runtime.c
+++ b/core/iwasm/aot/aot_runtime.c
@@ -2312,13 +2312,19 @@ aot_check_app_addr_and_convert(AOTModuleInstance *module_inst, bool is_str,
 void *
 aot_memmove(void *dest, const void *src, size_t n)
 {
-    return memmove(dest, src, n);
+    os_mutex_lock(&g_shared_memory_lock);
+    void *ret = memmove(dest, src, n);
+    os_mutex_unlock(&g_shared_memory_lock);
+    return ret;
 }

 void *
 aot_memset(void *s, int c, size_t n)
 {
-    return memset(s, c, n);
+    os_mutex_lock(&g_shared_memory_lock);
+    void *ret = memset(s, c, n);
+    os_mutex_unlock(&g_shared_memory_lock);
+    return ret;
 }

 double

I think we can enhance the implementation to call runtime APIs to acquire/release shared memory in AOT code before calling aot_memmove/aot_memset, e.g. call aot_shared_mem_lock/unlock. But not sure whether it is enough, should we also acquire the lock in the libc-wasi native, e.g. wasi_fd_read/wasi_fd_write/wasi_fd_pread/wasi_fd_pwrite, or even each time calling the native API?

Nov 23 '23 01:11 wenyongh

I think I saw that in wasmtime; every time the wasi function is called, it locks the memory mutex, to make sure the linear memory is protected. I don't think we need to do this for every wasi function, but at least the ones that use linear memory should have the lock acquired, otherwise there's a chance for concurrent unprotected memory access.

I don't think there's going to be a significant performance penalty (although I'm just speculating) as calling WASI (or any native) function is quite heavy anyway.

Nov 23 '23 08:11 loganek

Yes, we had better acquire the lock only for the native functions that use linear memory, but it is not easy to know whether a wasi function uses linear memory or not in compilation time and difficult to emit lock/unlock shared memory IR before/after calling the native function. I think we had better check the implementation in libc_wasi_wrapper.c, and add lock one by one if needed, how do you think?

Nov 23 '23 09:11 wenyongh

Completely agree, I don't think there's an automatic way to find all the WASI functions that access linear memory (the call of validate_native_addr gives us a hint, but what if we forgot to add the validation or the memory is allocated using wasm_runtime_malloc somewhere in the implementation). On the other hand, there's (only?) about 100 functions to check and for most of them it's very clear whether the lock is necessary or not.

I'll speak to my team today about that and let you know what'd the plan be.

About the memmove/memset functions - are you planning to open a PR with your fix?

Nov 23 '23 09:11 loganek

@wenyongh just one more thought on that; users can expose custom native functions to WASM; don't you think we should in that case expose a public API for them to acquire the lock in case the native function accesses linear memory? I think that's no different to WASI functions.

Nov 23 '23 09:11 loganek

Completely agree, I don't think there's an automatic way to find all the WASI functions that access linear memory (the call of validate_native_addr gives us a hint, but what if we forgot to add the validation or the memory is allocated using wasm_runtime_malloc somewhere in the implementation). On the other hand, there's (only?) about 100 functions to check and for most of them it's very clear whether the lock is necessary or not.

I'll speak to my team today about that and let you know what'd the plan be.

About the memmove/memset functions - are you planning to open a PR with your fix?

I have submitted PR #2815 to fix this issue, currently it should work for the pigz.wasm with AOT mode when tsan is enabled. I didn't add lock for the wasi function, could your team help do that after this PR is merged? Thanks.

Nov 23 '23 10:11 wenyongh

@wenyongh just one more thought on that; users can expose custom native functions to WASM; don't you think we should in that case expose a public API for them to acquire the lock in case the native function accesses linear memory? I think that's no different to WASI functions.

Yes, how about void wasm_runtime_shared_mem_lock(wasm_module_inst_t module_inst) and void wasm_runtime_shared_mem_unlock(wasm_module_inst_t module_inst)?

Nov 23 '23 10:11 wenyongh

I didn't add lock for the wasi function, could your team help do that after this PR is merged?

Quite likely, but I'll confirm that later today.

Yes, how about void wasm_runtime_shared_mem_lock(wasm_module_inst_t module_inst) and void wasm_runtime_shared_mem_unlock(wasm_module_inst_t module_inst)

Yes, that's more or less what I've had in mind.

Nov 23 '23 10:11 loganek

I didn't add lock for the wasi function

do you mean wasi functions that access linear memory?

Nov 23 '23 23:11 g0djan

I didn't add lock for the wasi function

do you mean wasi functions that access linear memory?

Yes.

Nov 24 '23 02:11 wenyongh

@wenyongh we've discussed the case with the team and came to a conclusion that it should rather be user's responsibility to add or remove a lock in the application itself - there's no need to lock the entire memory, but rather only a region that's being accessed by the specific function - and that can be achieved only by the application. When it comes to allocating the memory; I'd expect user in multithreaded application to use thread-safe malloc, so that shouldn't be an issue too.

Having said that, this unfortunately complicates TSAN validation a bit - from TSAN's perspective, there are no 'regions' in the linear memory and the whole memory is treated as one blob; I'm not sure what'd the best path forward to it would be - one way is that we'll have a special TSAN build in the CI that adds a lock for all the WASI functions, although I'd like to know others' opinion on that as there might be a better solution.

Nov 24 '23 09:11 loganek

@loganek Yes, sounds good to let user add or remove lock by himself, and we can provide the API to acquire and release lock. For the TSAN, I have no good idea also. If to add a lock for all the WASI functions, one way I can find is to call runtime API to acquire the lock in AOT code before calling native functions, but it may need to add an option for wamrc.

BTW, could you help review PR #2815? Should we add lock for aot_memmove and aot_memset?

Nov 24 '23 09:11 wenyongh

BTW, could you help review PR https://github.com/bytecodealliance/wasm-micro-runtime/pull/2815? Should we add lock for aot_memmove and aot_memset?

I think the problem with aot_memmove and aot_memset is similar to WASI functions - most likely the lock is already acquired in WASM but TSAN doesn't know about that. If that's the case, I don't think we need the PR, users should lock the memory instead in WASM. @eloparco - could you confirm that?

we can provide the API to acquire and release lock.

Hmm, not sure if that's actually needed - user can create a global lock in WASM and use that instead.

Nov 24 '23 11:11 loganek

BTW, could you help review PR #2815? Should we add lock for aot_memmove and aot_memset?

I think the problem with aot_memmove and aot_memset is similar to WASI functions - most likely the lock is already acquired in WASM but TSAN doesn't know about that. If that's the case, I don't think we need the PR, users should lock the memory instead in WASM. @eloparco - could you confirm that?

we can provide the API to acquire and release lock.

Hmm, not sure if that's actually needed - user can create a global lock in WASM and use that instead.

OK, let's wait for @eloparco's confirmation, please let me know whether to add lock for aot_memmove, or just keep the API exported.

Nov 24 '23 12:11 wenyongh

Agree that locking the entire shared memory is overkill. Trying to understand if the TSAN problem is a false alarm or if it's coming from the WASM app (wasm-parallel-gzip) not using locks properly when dealing with shared variables. But in the latter case I would expect a data race been detected in WASM and not only AOT.

Nov 27 '23 10:11 eloparco

@eloparco thanks for the clarification! So let's only export the API wasm_runtime_shared_mem_lock/unlcok. Could you help review PR #2815?

Nov 27 '23 11:11 wenyongh

Actually, I was running the .wasm file (no AOT) with the sanitizer for the latest main and getting another race:

WARNING: ThreadSanitizer: data race (pid=12392)
  Read of size 4 at 0x000108801354 by thread T1:
    #0 wasm_interp_call_func_bytecode wasm_interp_fast.c:3851 (iwasm:arm64+0x100078b34)
    #1 wasm_interp_call_func_bytecode wasm_interp_fast.c:1178 (iwasm:arm64+0x10005cb54)
    #2 call_wasm_with_hw_bound_check wasm_runtime.c:2319 (iwasm:arm64+0x1000a388c)
    #3 wasm_call_function wasm_runtime.c:2386 (iwasm:arm64+0x1000a3664)
    #4 wasm_runtime_call_wasm wasm_runtime_common.c:2037 (iwasm:arm64+0x100052420)
    #5 thread_start lib_wasi_threads_wrapper.c:60 (iwasm:arm64+0x1000c2a24)
    #6 thread_manager_start_routine thread_manager.c:602 (iwasm:arm64+0x1000c4800)
    #7 os_thread_wrapper posix_thread.c:46 (iwasm:arm64+0x10000ba9c)

  Previous write of size 4 at 0x000108801354 by main thread (mutexes: write M24):
    #0 wasm_enlarge_memory_internal wasm_memory.c:847 (iwasm:arm64+0x10004decc)
    #1 wasm_enlarge_memory wasm_memory.c:894 (iwasm:arm64+0x10004e270)
    #2 wasm_interp_call_func_bytecode wasm_interp_fast.c:1902 (iwasm:arm64+0x100061cd0)
    #3 wasm_interp_call_func_bytecode wasm_interp_fast.c:1178 (iwasm:arm64+0x10005cb54)
    #4 call_wasm_with_hw_bound_check wasm_runtime.c:2319 (iwasm:arm64+0x1000a388c)
    #5 wasm_call_function wasm_runtime.c:2386 (iwasm:arm64+0x1000a3664)
    #6 wasm_runtime_call_wasm wasm_runtime_common.c:2037 (iwasm:arm64+0x100052420)
    #7 execute_main wasm_application.c:112 (iwasm:arm64+0x100032980)
    #8 wasm_application_execute_main wasm_application.c:238 (iwasm:arm64+0x10003283c)
    #9 app_instance_main main.c:113 (iwasm:arm64+0x1000038b8)
    #10 main main.c:947 (iwasm:arm64+0x10000313c)

Nov 27 '23 13:11 eloparco

How can I understand what function is calling those aot_memmove and aot_memset and what function is trying to read from those locations? To understand if the problem is coming from the runtime, the wasm code or it's a false alarm from TSAN.

Nov 27 '23 13:11 eloparco

I added logs for every time a mutex is acquired and released in the wasm-parallel-gzip app, but I didn't discover anything useful

+ ./iwasm -v=5 --dir=wasm-apps wasm-apps/pigz.aot -p 3 wasm-apps/test.txt
pigz acquiring lock 0x28024
pigz releasing lock 0x28024
pigz acquiring lock 0x38cd0
pigz releasing lock 0x38cd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x38cd0
pigz releasing lock 0x38cd0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x28024
pigz releasing lock 0x28024
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz releasing lock 0x38cd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38d20
pigz releasing lock 0x38d20
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x28024
pigz releasing lock 0x28024
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz releasing lock 0x38cd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x28024
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38d20
pigz releasing lock 0x38d20
pigz releasing lock 0x28024
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz releasing lock 0x38cd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz releasing lock 0x38cd0
pigz acquiring lock ==================
WARNING: ThreadSanitizer: data race (pid=864277)
  Write of size 8 at 0x7fb3b6028718 by main thread:
    #0 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:810 (libtsan.so.0+0x6225e)
    #1 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:808 (libtsan.so.0+0x6225e)
    #2 aot_memmove wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 (iwasm+0x34f75)
    #3 <null> <null> (0x000040e35699)
    #4 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #5 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #6 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #7 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #8 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #9 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #10 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

  Previous read of size 8 at 0x7fb3b6028718 by thread T4:
    #0 writev ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:1203 (libtsan.so.0+0x4ce32)
    #1 os_writev wasm-micro-runtime/core/shared/platform/common/posix/posix_file.c:578 (iwasm+0x391e6)
    #2 blocking_op_writev wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/blocking_op.c:56 (iwasm+0x86173)
    #3 wasmtime_ssp_fd_write wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c:1120 (iwasm+0x42b3b)
    #4 wasi_fd_write wasm-micro-runtime/core/iwasm/libraries/libc-wasi/libc_wasi_wrapper.c:669 (iwasm+0x81ebf)
    #5 <null> <null> (0x000040e36682)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a3d)
    #10 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d31)
    #11 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb16)

  Thread T4 (tid=864282, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd27)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdc1)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37127)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d5b)
    #5 <null> <null> (0x000040e0c8d2)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

SUMMARY: ThreadSanitizer: data race wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 in aot_memmove
==================
0x68ff0
pigz acquiring lock 0x38d20
pigz releasing lock 0x38d20
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz releasing lock 0x38cd0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38c80
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x134a80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x68ff0
pigz acquiring lock 0xda840
pigz acquiring lock 0x38c30
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38d20
pigz releasing lock 0x38d20
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x68f80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38d20
pigz releasing lock 0x38d20
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0xfa930
pigz acquiring lock 0x38d20
pigz acquiring lock 0x68f80
pigz releasing lock 0x68f80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38d20
pigz acquiring lock 0xfa930
pigz acquiring lock 0x1a6340
pigz acquiring lock 0x38d20
pigz acquiring lock 0xda840
pigz releasing lock 0xda840
pigz acquiring lock 0x38c80
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x251cd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x134a80
pigz releasing lock 0x134a80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz releasing lock 0x38cd0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz releasing lock 0x38cd0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c30
pigz acquiring lock ==================
WARNING: ThreadSanitizer: data race (pid=864277)
  Read of size 8 at 0x7fb3b60287c0 by main thread:
    #0 writev ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:1203 (libtsan.so.0+0x4ce32)
    #1 os_writev wasm-micro-runtime/core/shared/platform/common/posix/posix_file.c:578 (iwasm+0x391e6)
    #2 blocking_op_writev wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/blocking_op.c:56 (iwasm+0x86173)
    #3 wasmtime_ssp_fd_write wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c:1120 (iwasm+0x42b3b)
    #4 wasi_fd_write wasm-micro-runtime/core/iwasm/libraries/libc-wasi/libc_wasi_wrapper.c:669 (iwasm+0x81ebf)
    #5 <null> <null> (0x000040e36682)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

  Previous write of size 1 at 0x7fb3b60287c7 by thread T2:
    #0 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:810 (libtsan.so.0+0x6225e)
    #1 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:808 (libtsan.so.0+0x6225e)
    #2 aot_memmove wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 (iwasm+0x34f75)
    #3 <null> <null> (0x000040e35699)
    #4 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #5 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #6 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #7 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a3d)
    #8 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d31)
    #9 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb16)

  Thread T2 (tid=864280, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd27)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdc1)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37127)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d5b)
    #5 <null> <null> (0x000040e0c8d2)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

SUMMARY: ThreadSanitizer: data race wasm-micro-runtime/core/shared/platform/common/posix/posix_file.c:578 in os_writev
==================
==================
WARNING: ThreadSanitizer: data race (pid=864277)
  Write of size 8 at 0x7fb3b60286e0 by thread T4:
    #0 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:810 (libtsan.so.0+0x6225e)
    #1 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:808 (libtsan.so.0+0x6225e)
    #2 aot_memmove wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 (iwasm+0x34f75)
    #3 <null> <null> (0x000040e35699)
    #4 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #5 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #6 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #7 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a3d)
    #8 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d31)
    #9 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb16)

  Previous write of size 8 at 0x7fb3b60286e0 by thread T1:
    #0 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:810 (libtsan.so.0+0x6225e)
    #1 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:808 (libtsan.so.0+0x6225e)
    #2 aot_memmove wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 (iwasm+0x34f75)
    #3 <null> <null> (0x000040e35699)
    #4 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #5 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #6 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #7 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a3d)
    #8 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d31)
    #9 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb16)

  Thread T4 (tid=864282, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd27)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdc1)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37127)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d5b)
    #5 <null> <null> (0x000040e0c8d2)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

  Thread T1 (tid=864279, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd27)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdc1)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37127)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d5b)
    #5 <null> <null> (0x000040e0c8d2)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

SUMMARY: ThreadSanitizer: data race wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 in aot_memmove
==================
0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x211ae0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x211b50
pigz acquiring lock 0x231be0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x251cd0
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38c80
pigz acquiring lock 0x231c50
pigz acquiring lock 0x27bd60
pigz acquiring lock 0x38c30
pigz acquiring lock 0x134af0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x1a6340
pigz acquiring lock 0x29be50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x211ae0
pigz releasing lock 0x211ae0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x29be50
pigz acquiring lock 0x3100d0
pigz acquiring lock ==================
WARNING: ThreadSanitizer: data race (pid=864277)
  Write of size 8 at 0x7fb3b629bea0 by thread T2:
    #0 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:810 (libtsan.so.0+0x6225e)
    #1 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:808 (libtsan.so.0+0x6225e)
    #2 aot_memmove wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 (iwasm+0x34f75)
    #3 <null> <null> (0x000040e28c00)
    #4 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #5 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #6 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #7 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a3d)
    #8 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d31)
    #9 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb16)

  Previous read of size 8 at 0x7fb3b629bea0 by thread T1:
    #0 writev ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:1203 (libtsan.so.0+0x4ce32)
    #1 os_writev wasm-micro-runtime/core/shared/platform/common/posix/posix_file.c:578 (iwasm+0x391e6)
    #2 blocking_op_writev wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/blocking_op.c:56 (iwasm+0x86173)
    #3 wasmtime_ssp_fd_write wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c:1120 (iwasm+0x42b3b)
    #4 wasi_fd_write wasm-micro-runtime/core/iwasm/libraries/libc-wasi/libc_wasi_wrapper.c:669 (iwasm+0x81ebf)
    #5 <null> <null> (0x000040e322c2)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a3d)
    #10 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d31)
    #11 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb16)

  Thread T2 (tid=864280, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd27)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdc1)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37127)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d5b)
    #5 <null> <null> (0x000040e0c8d2)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

  Thread T1 (tid=864279, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd27)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdc1)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37127)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d5b)
    #5 <null> <null> (0x000040e0c8d2)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

SUMMARY: ThreadSanitizer: data race wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 in aot_memmove
==================
0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x211b50
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x231be0
pigz releasing lock 0x231be0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x231c50
pigz acquiring lock 0xfa930
pigz acquiring lock 0x38d20
pigz acquiring lock 0x27bd60
pigz releasing lock 0x27bd60
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x2eff60
pigz acquiring lock 0x38c30
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x68f80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38d20
pigz acquiring lock 0xfa930
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2c5ed0==================
WARNING: ThreadSanitizer: data race (pid=864277)
  Write of size 8 at 0x7fb3b6249ca0 by main thread:
    #0 readv ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:1094 (libtsan.so.0+0x53e74)
    #1 os_readv wasm-micro-runtime/core/shared/platform/common/posix/posix_file.c:564 (iwasm+0x3915a)
    #2 blocking_op_readv wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/blocking_op.c:30 (iwasm+0x86072)
    #3 wasmtime_ssp_fd_read wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c:884 (iwasm+0x41fc4)
    #4 wasi_fd_read wasm-micro-runtime/core/iwasm/libraries/libc-wasi/libc_wasi_wrapper.c:500 (iwasm+0x8174e)
    #5 <null> <null> (0x000040e32152)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

  Previous read of size 8 at 0x7fb3b6249ca0 by thread T4:
    #0 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:810 (libtsan.so.0+0x6225e)
    #1 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:808 (libtsan.so.0+0x6225e)
    #2 aot_memmove wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 (iwasm+0x34f75)
    #3 <null> <null> (0x000040e22c8f)
    #4 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #5 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #6 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #7 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a3d)
    #8 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d31)
    #9 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb16)

  Thread T4 (tid=864282, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd27)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdc1)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37127)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d5b)
    #5 <null> <null> (0x000040e0c8d2)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

SUMMARY: ThreadSanitizer: data race wasm-micro-runtime/core/shared/platform/common/posix/posix_file.c:564 in os_readv
==================
==================
WARNING: ThreadSanitizer: data race (pid=864277)
  Read of size 8 at 0x7fb3b6028a30 by thread T3:
    #0 writev ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:1203 (libtsan.so.0+0x4ce32)
    #1 os_writev wasm-micro-runtime/core/shared/platform/common/posix/posix_file.c:578 (iwasm+0x391e6)
    #2 blocking_op_writev wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/blocking_op.c:56 (iwasm+0x86173)
    #3 wasmtime_ssp_fd_write wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c:1120 (iwasm+0x42b3b)
    #4 wasi_fd_write wasm-micro-runtime/core/iwasm/libraries/libc-wasi/libc_wasi_wrapper.c:669 (iwasm+0x81ebf)
    #5 <null> <null> (0x000040e36682)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a3d)
    #10 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d31)
    #11 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb16)

  Previous write of size 1 at 0x7fb3b6028a37 by thread T4:
    #0 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:810 (libtsan.so.0+0x6225e)
    #1 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:808 (libtsan.so.0+0x6225e)
    #2 aot_memmove wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 (iwasm+0x34f75)
    #3 <null> <null> (0x000040e35699)
    #4 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #5 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #6 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #7 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a3d)
    #8 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d31)
    #9 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb16)

  Thread T3 (tid=864281, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd27)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdc1)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37127)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d5b)
    #5 <null> <null> (0x000040e0c8d2)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

  Thread T4 (tid=864282, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd27)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdc1)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37127)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d5b)
    #5 <null> <null> (0x000040e0c8d2)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

SUMMARY: ThreadSanitizer: data race wasm-micro-runtime/core/shared/platform/common/posix/posix_file.c:578 in os_writev
==================

pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38c80
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x134a80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x251cd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2eff60
pigz releasing lock 0x2eff60
pigz acquiring lock 0x38c80
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x251cd0
pigz acquiring lock 0x1a6340
pigz acquiring lock 0x38d20
pigz acquiring lock 0x68f80
pigz releasing lock 0x68f80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x310060
pigz acquiring lock 0x38c30
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x1a6340
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x330160
pigz acquiring lock 0x38c30
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x211ae0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38d20
pigz releasing lock 0x38d20
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38c80
pigz acquiring lock 0x211b50
pigz acquiring lock 0x231be0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38d20
pigz releasing lock 0x38d20
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38d20
pigz releasing lock 0x38d20
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38c80
pigz acquiring lock 0x231c50
pigz acquiring lock 0x68f80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x27bd60
pigz acquiring lock ==================
WARNING: ThreadSanitizer: data race (pid=864277)
  Write of size 8 at 0x7fb3b62b7ef8 by thread T2:
    #0 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:810 (libtsan.so.0+0x6225e)
    #1 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:808 (libtsan.so.0+0x6225e)
    #2 aot_memmove wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 (iwasm+0x34f75)
    #3 <null> <null> (0x000040e290a5)
    #4 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #5 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #6 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #7 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a3d)
    #8 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d31)
    #9 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb16)

  Previous read of size 8 at 0x7fb3b62b7ef8 by thread T1:
    #0 writev ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:1203 (libtsan.so.0+0x4ce32)
    #1 os_writev wasm-micro-runtime/core/shared/platform/common/posix/posix_file.c:578 (iwasm+0x391e6)
    #2 blocking_op_writev wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/blocking_op.c:56 (iwasm+0x86173)
    #3 wasmtime_ssp_fd_write wasm-micro-runtime/core/iwasm/libraries/libc-wasi/sandboxed-system-primitives/src/posix.c:1120 (iwasm+0x42b3b)
    #4 wasi_fd_write wasm-micro-runtime/core/iwasm/libraries/libc-wasi/libc_wasi_wrapper.c:669 (iwasm+0x81ebf)
    #5 <null> <null> (0x000040e322c2)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a3d)
    #10 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d31)
    #11 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb16)

  Thread T2 (tid=864280, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd27)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdc1)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37127)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d5b)
    #5 <null> <null> (0x000040e0c8d2)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

  Thread T1 (tid=864279, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd27)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdc1)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37127)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d5b)
    #5 <null> <null> (0x000040e0c8d2)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

SUMMARY: ThreadSanitizer: data race wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 in aot_memmove
==================
0x38c30
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x68ff0
pigz acquiring lock 0xda840
pigz acquiring lock 0x38c30
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x29be50
pigz acquiring lock 0x38d20
pigz acquiring lock 0xda840
pigz releasing lock 0xda840
pigz acquiring lock 0x38c80
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x29be50
pigz acquiring lock 0xfa930
pigz acquiring lock 0x38d20
pigz acquiring lock 0x134a80
pigz releasing lock 0x134a80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0xfa930
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x310060
pigz releasing lock 0x310060
pigz acquiring lock 0x38c80
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x251cd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x330160
pigz releasing lock 0x330160
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x1a6340
pigz acquiring lock 0x38d20
pigz acquiring lock 0x211ae0
pigz releasing lock 0x211ae0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x211b50
pigz acquiring lock 0x2eff50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x231be0
pigz releasing lock 0x231be0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x231c50
pigz acquiring lock 0x35a260
pigz acquiring lock 0x38d20
pigz acquiring lock 0x68f80
pigz releasing lock 0x68f80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x27bd60
pigz releasing lock 0x27bd60
pigz acquiring lock 0x38c80
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38c80
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x3ae370
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38c80
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x3842e0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x134a80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x35a260
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x35a1f0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2eff50
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x29be50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x35a1f0
pigz releasing lock 0x35a1f0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0xfa930
pigz acquiring lock 0x38d20
pigz acquiring lock 0x3ae370
pigz releasing lock 0x3ae370
pigz acquiring lock 0x38c80
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x134a80
pigz releasing lock 0x134a80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2effd0
pigz acquiring lock 0xda840
pigz acquiring lock 0x38c30
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x310060
pigz acquiring lock 0x38c30
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38d20
pigz acquiring lock 0xfa930
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x38d20
pigz acquiring lock 0xda840
pigz releasing lock 0xda840
pigz acquiring lock 0x38c80
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x211ae0
pigz acquiring lock 0x38c30
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x35a260
pigz acquiring lock 0x38d20
pigz acquiring lock 0x310060
pigz releasing lock 0x310060
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x35a260
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x2eff50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x211b50
pigz acquiring lock 0x211ae0
pigz releasing lock 0x211ae0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38c80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x231c50
pigz acquiring lock 0x68f80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x211b50
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2eff50
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x231be0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x231be0
pigz releasing lock 0x231be0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0xfa930
pigz acquiring lock 0x38d20
pigz acquiring lock 0x68f80
pigz releasing lock 0x68f80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x35a1f0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38d20
pigz acquiring lock 0xfa930
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x330160
pigz acquiring lock 0x38c30
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x3842e0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38c80
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x134a80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2effd0
pigz acquiring lock 0xda840
pigz acquiring lock 0x38c30
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x231c50
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x29be50
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x310060
pigz acquiring lock 0x38c30
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x211b50
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x1a6340
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x251cd0
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38c80
pigz acquiring lock 0x231c50
pigz acquiring lock 0x231be0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x68f80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x27bd60
pigz acquiring lock 0x38c30
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x35a260
pigz acquiring lock 0x38d20
pigz acquiring lock 0x27bd60
pigz releasing lock 0x27bd60
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x35a260
pigz acquiring lock 0x2eff50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x35a1f0
pigz releasing lock 0x35a1f0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x330160
pigz releasing lock 0x330160
pigz acquiring lock 0x38c80
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0xfa930
pigz acquiring lock 0x38d20
pigz acquiring lock 0x134a80
pigz releasing lock 0x134a80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38c80
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x211b50
pigz acquiring lock 0x211ae0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38d20
pigz acquiring lock 0xda840
pigz releasing lock 0xda840
pigz acquiring lock 0x38c80
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x29be50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x310060
pigz releasing lock 0x310060
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x29be50
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x251cd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x68f80
pigz releasing lock 0x68f80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x231c50
pigz acquiring lock 0x1a6340
pigz acquiring lock 0x38d20
pigz acquiring lock 0x231be0
pigz releasing lock 0x231be0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x211b50
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x35a260
pigz acquiring lock 0x38d20
pigz acquiring lock 0x211ae0
pigz releasing lock 0x211ae0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x3ae370
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x3ae370
pigz releasing lock 0x3ae370
pigz acquiring lock 0x38c80
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x35a1f0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x29be50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x35a1f0
pigz acquiring lock 0x38c80
pigz releasing lock 0x35a1f0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134a80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x29be50
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x35a260
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x134a80
pigz releasing lock 0x134a80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2effd0
pigz acquiring lock 0xda840
pigz acquiring lock 0x38c30
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x38d20
pigz acquiring lock 0xda840
pigz releasing lock 0xda840
pigz acquiring lock 0x38c80
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x68f80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x310060
pigz acquiring lock 0x38c30
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x1a6340
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x29be50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x310060
pigz releasing lock 0x310060
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x35a260
pigz acquiring lock 0x38d20
pigz acquiring lock 0x68f80
pigz releasing lock 0x68f80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38c80
pigz acquiring lock 0x231c50
pigz acquiring lock 0x231be0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38cd0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x35a260
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x231be0
pigz releasing lock 0x231be0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38c80
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x211b50
pigz acquiring lock 0x330160
pigz acquiring lock 0x38c30
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x211ae0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x29be50
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x330160
pigz releasing lock 0x330160
pigz acquiring lock 0x38c80
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x27bd60
pigz acquiring lock 0x38c30
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x1a6340
pigz acquiring lock 0x38d20
pigz acquiring lock 0x211ae0
pigz releasing lock 0x211ae0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x35a260
pigz acquiring lock 0x38d20
pigz acquiring lock 0x27bd60
pigz releasing lock 0x27bd60
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x35a1f0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x134a80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0xda840
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38d20
pigz acquiring lock 0x35a1f0
pigz releasing lock 0x35a1f0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x29be50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x134a80
pigz releasing lock 0x134a80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x38d20
pigz acquiring lock 0xda840
pigz releasing lock 0xda840
pigz acquiring lock 0x38c80
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x29be50
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x310060
pigz acquiring lock 0x38c30
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x310060
pigz releasing lock 0x310060
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0xda840
pigz acquiring lock 0x38c30
pigz acquiring lock 0x29be50
pigz acquiring lock 0x38d20
pigz acquiring lock 0xda840
pigz releasing lock 0xda840
pigz acquiring lock 0x38c80
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x134a80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x134a80
pigz releasing lock 0x134a80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x29be50
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x35a1f0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0xda840
pigz acquiring lock 0x38c30
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x35a1f0
pigz releasing lock 0x35a1f0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134a80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x29be50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x134a80
pigz releasing lock 0x134a80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x38d20
pigz acquiring lock 0xda840
pigz releasing lock 0xda840
pigz acquiring lock 0x38c80
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x29be50
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x68ff0
pigz acquiring lock 0xda840
pigz acquiring lock 0x38c30
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x310060
pigz acquiring lock 0x38c30
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x310060
pigz releasing lock 0x310060
pigz acquiring lock 0x38c80
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x29be50
pigz acquiring lock 0x38d20
pigz acquiring lock 0xda840
pigz releasing lock 0xda840
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x134a80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x134a80
pigz releasing lock 0x134a80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x29be50
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x35a1f0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x134a80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x35a1f0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0xda840
pigz acquiring lock 0x38c30
pigz releasing lock 0x35a1f0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x29be50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x134a80
pigz releasing lock 0x134a80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x38d20
pigz acquiring lock 0xda840
pigz releasing lock 0xda840
pigz acquiring lock 0x38c80
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x29be50
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x68ff0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x211b50
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x231c50
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2effd0
pigz acquiring lock 0xda840
pigz acquiring lock 0x38c30
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x35a260
pigz acquiring lock 0x134af0
pigz acquiring lock 0x134a80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x1a6340
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x310060
pigz acquiring lock 0x38c30
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x251cd0
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x310060
pigz releasing lock 0x310060
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x38d20
pigz acquiring lock 0xda840
pigz releasing lock 0xda840
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x29be50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x134a80
pigz releasing lock 0x134a80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x35a1f0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x27bd60
pigz acquiring lock 0x38c30
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x29be50
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x35a260
pigz acquiring lock 0x38d20
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x35a1f0
pigz releasing lock 0x35a1f0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38c80
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x211ae0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x1a6340
pigz acquiring lock 0x38d20
pigz acquiring lock 0x27bd60
pigz releasing lock 0x27bd60
pigz acquiring lock 0x38c80
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x1a6340
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x330160
pigz acquiring lock 0x38c30
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38c80
pigz acquiring lock 0x211b50
pigz acquiring lock 0x231be0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x35a260==================
WARNING: ThreadSanitizer: data race (pid=864277)
  Write of size 8 at 0x7fb3b61a6390 by thread T4:
    #0 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:810 (libtsan.so.0+0x6225e)
    #1 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:808 (libtsan.so.0+0x6225e)
    #2 aot_memmove wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 (iwasm+0x34f75)
    #3 <null> <null> (0x000040e28c00)
    #4 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #5 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #6 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #7 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a3d)
    #8 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d31)
    #9 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb16)

  Previous write of size 8 at 0x7fb3b61a6390 by thread T3:
    #0 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:810 (libtsan.so.0+0x6225e)
    #1 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:808 (libtsan.so.0+0x6225e)
    #2 aot_memmove wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 (iwasm+0x34f75)
    #3 <null> <null> (0x000040e28c00)
    #4 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #5 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #6 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #7 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a3d)
    #8 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d31)
    #9 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb16)

  Thread T4 (tid=864282, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd27)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdc1)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37127)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d5b)
    #5 <null> <null> (0x000040e0c8d2)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

  Thread T3 (tid=864281, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd27)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdc1)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37127)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d5b)
    #5 <null> <null> (0x000040e0c8d2)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

SUMMARY: ThreadSanitizer: data race wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 in aot_memmove
==================
==================
WARNING: ThreadSanitizer: data race (pid=864277)
  Write of size 1 at 0x7fb3b61c23e0 by thread T4:
    #0 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:810 (libtsan.so.0+0x6225e)
    #1 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:808 (libtsan.so.0+0x6225e)
    #2 aot_memmove wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 (iwasm+0x34f75)
    #3 <null> <null> (0x000040e28c00)
    #4 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #5 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #6 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #7 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a3d)
    #8 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d31)
    #9 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb16)

  Previous write of size 8 at 0x7fb3b61c23e0 by thread T3:
    #0 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:810 (libtsan.so.0+0x6225e)
    #1 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:808 (libtsan.so.0+0x6225e)
    #2 aot_memmove wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 (iwasm+0x34f75)
    #3 <null> <null> (0x000040e290a5)
    #4 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #5 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #6 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #7 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a3d)
    #8 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d31)
    #9 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb16)

  Thread T4 (tid=864282, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd27)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdc1)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37127)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d5b)
    #5 <null> <null> (0x000040e0c8d2)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

  Thread T3 (tid=864281, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd27)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdc1)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37127)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d5b)
    #5 <null> <null> (0x000040e0c8d2)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

SUMMARY: ThreadSanitizer: data race wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 in aot_memmove
==================
==================
WARNING: ThreadSanitizer: data race (pid=864277)
  Write of size 8 at 0x7fb3b61c23e8 by thread T4:
    #0 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:810 (libtsan.so.0+0x6225e)
    #1 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:808 (libtsan.so.0+0x6225e)
    #2 aot_memmove wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 (iwasm+0x34f75)
    #3 <null> <null> (0x000040e290a5)
    #4 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #5 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #6 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #7 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a3d)
    #8 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d31)
    #9 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb16)

  Previous write of size 8 at 0x7fb3b61c23e8 by thread T3:
    #0 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:810 (libtsan.so.0+0x6225e)
    #1 memmove ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:808 (libtsan.so.0+0x6225e)
    #2 aot_memmove wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 (iwasm+0x34f75)
    #3 <null> <null> (0x000040e290a5)
    #4 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #5 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #6 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #7 thread_start wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:60 (iwasm+0x79a3d)
    #8 thread_manager_start_routine wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:602 (iwasm+0x36d31)
    #9 os_thread_wrapper wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:46 (iwasm+0xcb16)

  Thread T4 (tid=864282, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd27)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdc1)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37127)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d5b)
    #5 <null> <null> (0x000040e0c8d2)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

  Thread T3 (tid=864281, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:969 (libtsan.so.0+0x605b8)
    #1 os_thread_create_with_prio wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:86 (iwasm+0xcd27)
    #2 os_thread_create wasm-micro-runtime/core/shared/platform/common/posix/posix_thread.c:100 (iwasm+0xcdc1)
    #3 wasm_cluster_create_thread wasm-micro-runtime/core/iwasm/libraries/thread-mgr/thread_manager.c:704 (iwasm+0x37127)
    #4 thread_spawn_wrapper wasm-micro-runtime/core/iwasm/libraries/lib-wasi-threads/lib_wasi_threads_wrapper.c:122 (iwasm+0x79d5b)
    #5 <null> <null> (0x000040e0c8d2)
    #6 invoke_native_with_hw_bound_check wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1445 (iwasm+0x32719)
    #7 aot_call_function wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:1654 (iwasm+0x32f75)
    #8 wasm_runtime_call_wasm wasm-micro-runtime/core/iwasm/common/wasm_runtime_common.c:2042 (iwasm+0x13afd)
    #9 execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:112 (iwasm+0xf013)
    #10 wasm_application_execute_main wasm-micro-runtime/core/iwasm/common/wasm_application.c:238 (iwasm+0xf611)
    #11 app_instance_main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:113 (iwasm+0xa6a5)
    #12 main wasm-micro-runtime/product-mini/platforms/linux/../posix/main.c:947 (iwasm+0xc180)

SUMMARY: ThreadSanitizer: data race wasm-micro-runtime/core/iwasm/aot/aot_runtime.c:2315 in aot_memmove
==================

pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x3ae370
pigz acquiring lock 0x38c30
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0xfa930
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x310060
pigz acquiring lock 0x38c30
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x134a80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2eff50
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0xda840
pigz acquiring lock 0x38c30
pigz acquiring lock 0x251cd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x211ae0
pigz releasing lock 0x211ae0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x68ff0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x29be50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x330160
pigz releasing lock 0x330160
pigz acquiring lock 0x38c80
pigz acquiring lock 0x211b50
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x2c5ed0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x231be0
pigz releasing lock 0x231be0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38c80
pigz acquiring lock 0x231c50
pigz acquiring lock 0x68f80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x231c50
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x1a6340
pigz acquiring lock 0x38d20
pigz acquiring lock 0x68f80
pigz releasing lock 0x68f80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x3100d0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x35a260
pigz acquiring lock 0x38d20
pigz acquiring lock 0x3ae370
pigz releasing lock 0x3ae370
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x310060
pigz releasing lock 0x310060
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0xfa930
pigz acquiring lock 0x38d20
pigz acquiring lock 0x134a80
pigz releasing lock 0x134a80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x2eff50
pigz acquiring lock 0x38d20
pigz acquiring lock 0xda840
pigz releasing lock 0xda840
pigz acquiring lock 0x38c80
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2eff50
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38d20
pigz acquiring lock 0xfa930
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x27bd60
pigz acquiring lock 0x38c30
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x134a80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x2eff50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x27bd60
pigz releasing lock 0x27bd60
pigz acquiring lock 0x38c80
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38c80
pigz acquiring lock 0xda8b0
pigz acquiring lock 0xda840
pigz acquiring lock 0x38c30
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x38d20
pigz acquiring lock 0xda840
pigz releasing lock 0xda840
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0xfa930
pigz acquiring lock 0x38d20
pigz acquiring lock 0x134a80
pigz releasing lock 0x134a80
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0xfa930
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2eff50
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x310060
pigz acquiring lock 0x38c30
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38c80
pigz acquiring lock 0xda8b0
pigz acquiring lock 0xda840
pigz acquiring lock 0x38c30
pigz acquiring lock 0xfa930
pigz acquiring lock 0x38d20
pigz acquiring lock 0x310060
pigz releasing lock 0x310060
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134a80
pigz acquiring lock 0x38c30
pigz acquiring lock 0x2eff50
pigz acquiring lock 0x38d20
pigz acquiring lock 0x134a80
pigz releasing lock 0x134a80
pigz acquiring lock 0x38c80
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x38d20
pigz acquiring lock 0xda840
pigz releasing lock 0xda840
pigz acquiring lock 0x38c80
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38c30
pigz acquiring lock 0x2effd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0xda8b0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x2eff50
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38d20
pigz acquiring lock 0xfa930
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x27bd60
pigz acquiring lock 0x38c30
pigz acquiring lock 0x48ef0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134af0
pigz acquiring lock 0xda840
pigz acquiring lock 0x38c30
pigz acquiring lock 0x3842e0
pigz acquiring lock 0x38d20
pigz acquiring lock 0x27bd60
pigz releasing lock 0x27bd60
pigz acquiring lock 0x38c80
pigz acquiring lock 0x134af0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38c80
pigz acquiring lock 0x2eff50
pigz acquiring lock 0x38d20
pigz acquiring lock 0xda840
pigz acquiring lock 0x27bdd0
pigz releasing lock 0xda840
pigz acquiring lock 0x38c80
pigz acquiring lock 0x27bdd0
pigz acquiring lock 0x38cd0
pigz acquiring lock 0x134a80
pigz acquiring lock 0x38c30
pigz acquiring lock 0xfa930
pigz acquiring lock 0x38d20
pigz acquiring lock 0x134a80
pigz releasing lock 0x134a80
pigz acquiring lock 0x38c30
pigz releasing lock 0x38c30
pigz acquiring lock 0x38c80
pigz acquiring lock 0x28024
pigz acquiring lock 0x28024
pigz releasing lock 0x28024
pigz acquiring lock 0x38c30
pigz acquiring lock 0x28024
pigz releasing lock 0x38c30
pigz acquiring lock 0x28024
pigz releasing lock 0x38c30
pigz releasing lock 0x38c30
pigz acquiring lock 0x28024
pigz acquiring lock 0x28024
pigz releasing lock 0x28024
pigz acquiring lock 0x38dc0
pigz releasing lock 0x38dc0
pigz acquiring lock 0x38d70
pigz releasing lock 0x38d70
pigz acquiring lock 0x38d20
pigz releasing lock 0x38d20
pigz acquiring lock 0x38cd0
pigz releasing lock 0x38cd0
ThreadSanitizer: reported 10 warnings
[04:48:29:745 - 7FB5B83DE380]: Load module success.

[04:48:29:820 - 7FB5B83DE380]: Memory instantiate:
[04:48:29:823 - 7FB5B83DE380]:   page bytes: 65536, init pages: 4, max pages: 1024
[04:48:29:825 - 7FB5B83DE380]:   data offset: 0, stack size: 0
[04:48:29:827 - 7FB5B83DE380]:   heap offset: 262144, heap size: 0

Nov 27 '23 14:11 eloparco

The print was added here https://github.com/madler/pigz/blob/master/yarn.c#L136

void possess_(lock *bolt, char const *file, long line) {
  printf("pigz acquiring lock %p\n", &bolt->mutex); // <------ 
  int ret = pthread_mutex_lock(&(bolt->mutex));
  if (ret)
    fail(ret, file, line, "mutex_lock");
}

void release_(lock *bolt, char const *file, long line) {
  printf("pigz releasing lock %p\n", &bolt->mutex); // <------ 
  int ret = pthread_mutex_unlock(&(bolt->mutex));
  if (ret)
    fail(ret, file, line, "mutex_unlock");
}

Nov 27 '23 15:11 eloparco

Actually, I was running the .wasm file (no AOT) with the sanitizer for the latest main and getting another race:

Taking a closer look, those data races match the ones in AOT since I was able to patch them locally by adding some locks around WASM_OP_MEMORY_FILL and WASM_OP_MEMORY_COPY. They happen when there's an wasm_enlarge_memory event.

Nov 27 '23 15:11 eloparco

Since the race condition is not caused by WASM code but by memory.grow, we can't control that in user code.

So I guess we need to add the locks as it was done initially in #2815 (I was doing it locally for classic and fast interpreter). Not sure it would be possible to use atomics for the single memory fields since the enlarge memory function modifies most of them and we need to make all those updates atomic.

Nov 27 '23 17:11 eloparco

I was taking a second look, basically what's causing the race conditions is the access to memory->memory_data_size.

We can't just use a mutex to access it because when the HW bound check is disabled, memory->memory_data_size is accessed for every memory operation from CHECK_MEMORY_OVERFLOW and the performance impact is huge. So we probably want to use atomics for it.

Nov 28 '23 16:11 eloparco

The issue is not fully fixed, AOT is still reporting TSAN errors (#2834 only applies to classic and fast interpreter)

Nov 29 '23 13:11 eloparco

I couldn't re-open the issue.

The fact that it's only happening in AOT makes me suspicious that maybe it's a bug in AOT mode and not a problem with the wasm application not protecting access to memory. But I didn't find a way to verify it.

Nov 29 '23 16:11 eloparco

@eloparco this issue was closed automatically when PR #2834 was merged. I re-open it now.

Nov 30 '23 00:11 wenyongh

Any suggestions on how to debug the problem? Couldn't find anything useful.

Dec 04 '23 14:12 eloparco

Any suggestions on how to debug the problem? Couldn't find anything useful.

@eloparco any issue on the AOT?

Dec 05 '23 04:12 wenyongh

Yes, still looking into the TSAN problem with AOT, not sure how to debug it.

I added some prints in the pigz app to understand which app functions call memmove and wasi_fd_write (and see if they're acquiring the lock properly), but then the sanitizer error disappeared.

Dec 05 '23 09:12 eloparco

The aot_memmove should be called by memory.copy opcode, maybe you can check which function has the opcode? It is introduced by the bulk memory feature.

BTW, since most of the blocking issues have been resolved, I plan to do some tests and create the new release if things run well. Any concern from you?

Dec 05 '23 09:12 wenyongh