bcc icon indicating copy to clipboard operation
bcc copied to clipboard
verified publisher icon iovisor

Build error for v0.11.0-1192-g187baf18 in Slackware64-15.0

Open rnsanchez opened this issue 3 years ago • 14 comments

While trying to build a current checkout (v0.11.0-1192-g187baf18) on a freshly-installed Slackware64-15.0 (LLVM 13.0.0, glibc-2.33):

[ 38%] Built target bps
[ 38%] Building CXX object examples/cpp/CMakeFiles/CGroupTest.dir/CGroupTest.cc.o
[ 38%] Linking CXX executable CGroupTest
/usr/bin/ld: ../../src/cc/libbcc.a(bcc_debug.cc.o): undefined reference to symbol '_ZN4llvm12MemoryBuffer16getMemBufferCopyENS_9StringRefERKNS_5TwineE'
/usr/bin/ld: /usr/lib64/gcc/x86_64-slackware-linux/11.2.0/../../../../lib64/../lib64/libLLVMSupport.so.13: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
[ 38%] Built target CGroupTest
[ 38%] Building CXX object examples/cpp/CMakeFiles/UseExternalMap.dir/UseExternalMap.cc.o
[ 38%] Linking CXX executable UseExternalMap
/usr/bin/ld: ../../src/cc/libbcc.a(bcc_debug.cc.o): undefined reference to symbol '_ZN4llvm12MemoryBuffer16getMemBufferCopyENS_9StringRefERKNS_5TwineE'
/usr/bin/ld: /usr/lib64/gcc/x86_64-slackware-linux/11.2.0/../../../../lib64/../lib64/libLLVMSupport.so.13: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
[ 38%] Built target UseExternalMap
[ 38%] Building CXX object examples/cpp/CMakeFiles/FollyRequestContextSwitch.dir/FollyRequestContextSwitch.cc.o
[ 38%] Linking CXX executable FollyRequestContextSwitch
/usr/bin/ld: ../../src/cc/libbcc.a(bcc_debug.cc.o): undefined reference to symbol '_ZN4llvm12MemoryBuffer16getMemBufferCopyENS_9StringRefERKNS_5TwineE'
/usr/bin/ld: /usr/lib64/gcc/x86_64-slackware-linux/11.2.0/../../../../lib64/../lib64/libLLVMSupport.so.13: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
[ 38%] Built target FollyRequestContextSwitch
[ 38%] Building CXX object examples/cpp/CMakeFiles/SkLocalStorageIterator.dir/SkLocalStorageIterator.cc.o
[ 38%] Linking CXX executable SkLocalStorageIterator
/usr/bin/ld: ../../src/cc/libbcc.a(bcc_debug.cc.o): undefined reference to symbol '_ZN4llvm12MemoryBuffer16getMemBufferCopyENS_9StringRefERKNS_5TwineE'
/usr/bin/ld: /usr/lib64/gcc/x86_64-slackware-linux/11.2.0/../../../../lib64/../lib64/libLLVMSupport.so.13: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
[ 38%] Built target SkLocalStorageIterator
[ 39%] Building CXX object examples/cpp/CMakeFiles/HelloWorld.dir/HelloWorld.cc.o
[ 39%] Linking CXX executable HelloWorld
/usr/bin/ld: ../../src/cc/libbcc.a(bcc_debug.cc.o): undefined reference to symbol '_ZN4llvm12MemoryBuffer16getMemBufferCopyENS_9StringRefERKNS_5TwineE'
/usr/bin/ld: /usr/lib64/gcc/x86_64-slackware-linux/11.2.0/../../../../lib64/../lib64/libLLVMSupport.so.13: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
[ 39%] Built target HelloWorld
[ 39%] Building CXX object examples/cpp/CMakeFiles/RecordMySQLQuery.dir/RecordMySQLQuery.cc.o
[ 40%] Linking CXX executable RecordMySQLQuery
/usr/bin/ld: ../../src/cc/libbcc.a(bcc_debug.cc.o): undefined reference to symbol '_ZN4llvm12MemoryBuffer16getMemBufferCopyENS_9StringRefERKNS_5TwineE'
/usr/bin/ld: /usr/lib64/gcc/x86_64-slackware-linux/11.2.0/../../../../lib64/../lib64/libLLVMSupport.so.13: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
[ 40%] Built target RecordMySQLQuery
[ 40%] Building CXX object examples/cpp/CMakeFiles/CPUDistribution.dir/CPUDistribution.cc.o
[ 41%] Linking CXX executable CPUDistribution
/usr/bin/ld: ../../src/cc/libbcc.a(bcc_debug.cc.o): undefined reference to symbol '_ZN4llvm12MemoryBuffer16getMemBufferCopyENS_9StringRefERKNS_5TwineE'
/usr/bin/ld: /usr/lib64/gcc/x86_64-slackware-linux/11.2.0/../../../../lib64/../lib64/libLLVMSupport.so.13: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
[ 41%] Built target CPUDistribution
[ 41%] Building CXX object examples/cpp/CMakeFiles/TaskIterator.dir/TaskIterator.cc.o
[ 42%] Linking CXX executable TaskIterator
/usr/bin/ld: ../../src/cc/libbcc.a(bcc_debug.cc.o): undefined reference to symbol '_ZN4llvm12MemoryBuffer16getMemBufferCopyENS_9StringRefERKNS_5TwineE'
/usr/bin/ld: /usr/lib64/gcc/x86_64-slackware-linux/11.2.0/../../../../lib64/../lib64/libLLVMSupport.so.13: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
[ 42%] Built target TaskIterator
[ 42%] Building CXX object examples/cpp/CMakeFiles/KFuncExample.dir/KFuncExample.cc.o
[ 43%] Linking CXX executable KFuncExample
/usr/bin/ld: ../../src/cc/libbcc.a(bcc_debug.cc.o): undefined reference to symbol '_ZN4llvm12MemoryBuffer16getMemBufferCopyENS_9StringRefERKNS_5TwineE'
/usr/bin/ld: /usr/lib64/gcc/x86_64-slackware-linux/11.2.0/../../../../lib64/../lib64/libLLVMSupport.so.13: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
[ 43%] Built target KFuncExample
[ 44%] Building CXX object examples/cpp/CMakeFiles/TCPSendStack.dir/TCPSendStack.cc.o
[ 44%] Linking CXX executable TCPSendStack
/usr/bin/ld: ../../src/cc/libbcc.a(bcc_debug.cc.o): undefined reference to symbol '_ZN4llvm12MemoryBuffer16getMemBufferCopyENS_9StringRefERKNS_5TwineE'
/usr/bin/ld: /usr/lib64/gcc/x86_64-slackware-linux/11.2.0/../../../../lib64/../lib64/libLLVMSupport.so.13: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
[ 44%] Built target TCPSendStack  
[ 44%] Building CXX object examples/cpp/CMakeFiles/KModRetExample.dir/KModRetExample.cc.o
[ 44%] Linking CXX executable KModRetExample
/usr/bin/ld: ../../src/cc/libbcc.a(bcc_debug.cc.o): undefined reference to symbol '_ZN4llvm12MemoryBuffer16getMemBufferCopyENS_9StringRefERKNS_5TwineE'
/usr/bin/ld: /usr/lib64/gcc/x86_64-slackware-linux/11.2.0/../../../../lib64/../lib64/libLLVMSupport.so.13: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
[ 44%] Built target KModRetExample
[ 45%] Building CXX object examples/cpp/CMakeFiles/LLCStat.dir/LLCStat.cc.o
[ 45%] Linking CXX executable LLCStat
/usr/bin/ld: ../../src/cc/libbcc.a(bcc_debug.cc.o): undefined reference to symbol '_ZN4llvm12MemoryBuffer16getMemBufferCopyENS_9StringRefERKNS_5TwineE'
/usr/bin/ld: /usr/lib64/gcc/x86_64-slackware-linux/11.2.0/../../../../lib64/../lib64/libLLVMSupport.so.13: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
[ 45%] Built target LLCStat
[ 46%] Building CXX object examples/cpp/CMakeFiles/RandomRead.dir/RandomRead.cc.o
[ 46%] Linking CXX executable RandomRead
/usr/bin/ld: ../../src/cc/libbcc.a(bcc_debug.cc.o): undefined reference to symbol '_ZN4llvm12MemoryBuffer16getMemBufferCopyENS_9StringRefERKNS_5TwineE'
/usr/bin/ld: /usr/lib64/gcc/x86_64-slackware-linux/11.2.0/../../../../lib64/../lib64/libLLVMSupport.so.13: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status

Any more info / checks I could provide for helping out with this issue?

rnsanchez avatar Sep 04 '22 18:09 rnsanchez

I still have the output from cmake:

~/bcc/build$ cmake ..
CMake Deprecation Warning at CMakeLists.txt:3 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- The C compiler identification is GNU 11.2.0
-- The CXX compiler identification is GNU 11.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
Submodule 'libbpf-tools/bpftool' (https://github.com/libbpf/bpftool) registered for path 'libbpf-tools/bpftool'
Submodule 'src/cc/libbpf' (https://github.com/libbpf/libbpf.git) registered for path 'src/cc/libbpf'
Cloning into '/home/rnsanchez/bcc/libbpf-tools/bpftool'...
Cloning into '/home/rnsanchez/bcc/src/cc/libbpf'...
Submodule path 'libbpf-tools/bpftool': checked out '6eb3e20583da834da18ea3011dcefd08b3493f8d'
Submodule 'libbpf' (https://github.com/libbpf/libbpf.git) registered for path 'libbpf-tools/bpftool/libbpf'
Cloning into '/home/rnsanchez/bcc/libbpf-tools/bpftool/libbpf'...
Submodule path 'libbpf-tools/bpftool/libbpf': checked out '7984737fbf3b2a14a86321387bb62abb16cfc4ed'
Submodule path 'src/cc/libbpf': checked out 'b78c75fcb347b06c31996860353f40087ed02f48'
-- Latest recognized Git tag is v0.25.0
-- Git HEAD is 187baf1811115e997049278ec8586513f5e8bb4a
-- Revision is 0.25.0-187baf18 (major 0, minor 25, patch 0)
-- Performing Test HAVE_NO_PIE_FLAG
-- Performing Test HAVE_NO_PIE_FLAG - Success
-- Performing Test HAVE_REALLOCARRAY_SUPPORT
-- Performing Test HAVE_REALLOCARRAY_SUPPORT - Success
-- Kernel release: 5.15.64
-- Kernel headers: KERNELHEADERS_DIR-NOTFOUND
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.11")
-- Found LibXml2: /usr/lib64/libxml2.so (found version "2.9.12")
-- Found LLVM: /usr/include 13.0.0 (Use LLVM_ROOT envronment variable for another version of LLVM)
-- Found BISON: /usr/bin/bison (found version "3.8.2")
-- Found FLEX: /usr/bin/flex (found version "2.6.4")
-- Found LibElf: /usr/lib64/libelf.so
-- Performing Test ELF_GETSHDRSTRNDX
-- Performing Test ELF_GETSHDRSTRNDX - Success
-- Found LibDebuginfod: /usr/lib64/libdebuginfod.so
-- Found LLVM: /usr/include 13.0.0 (Use LLVM_ROOT envronment variable for another version of LLVM)
-- Found LuaJIT: /usr/lib64/libluajit-5.1.a;/usr/lib64/libdl.so;/usr/lib64/libm.so
CMake Warning at tests/python/CMakeLists.txt:10 (message):
  Recommended test program 'netperf' not found


CMake Warning at tests/python/CMakeLists.txt:16 (message):
  Recommended test program 'iperf' or 'iperf3' not found


-- Configuring done
-- Generating done
-- Build files have been written to: ...

rnsanchez avatar Sep 04 '22 18:09 rnsanchez

I've been able to build after a lot of tries, with this smallest diff:

diff --git a/src/cc/CMakeLists.txt b/src/cc/CMakeLists.txt
index ffe8feec..018afb9e 100644
--- a/src/cc/CMakeLists.txt
+++ b/src/cc/CMakeLists.txt
@@ -124,7 +124,7 @@ set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} ${llvm_lib_exclude_f
 # bcc_common_libs_for_a for archive libraries
 # bcc_common_libs_for_s for shared libraries
 set(bcc_common_libs clang_frontend
-  -Wl,--whole-archive ${clang_libs} ${llvm_libs} -Wl,--no-whole-archive
+  -Wl,--whole-archive,--copy-dt-needed-entries ${clang_libs} ${llvm_libs} -Wl,--no-whole-archive
   ${LIBELF_LIBRARIES})
 if (LIBDEBUGINFOD_FOUND)
   list(APPEND bcc_common_libs ${LIBDEBUGINFOD_LIBRARIES})
@@ -147,7 +147,7 @@ if(ENABLE_CPP_API)
   add_subdirectory(api)
   list(APPEND bcc_common_libs_for_a api-static)
   # Keep all API functions
-  list(APPEND bcc_common_libs_for_s -Wl,--whole-archive api-static -Wl,--no-whole-archive)
+  list(APPEND bcc_common_libs_for_s -Wl,--whole-archive,--copy-dt-needed-entries api-static -Wl,--no-whole-archive)
 endif()
 
 if(ENABLE_USDT)

Credit goes to Boyi: https://zhangboyi.gitlab.io/post/2020-09-14-resolve-dso-missing-from-command-line-error/

I have tried to manually reorder libs/objects and making only slightest progress, while running into all sorts of errors.

rnsanchez avatar Sep 07 '22 01:09 rnsanchez

Please paste the output of make VERBOSE=1 here.

chenhengqi avatar Sep 07 '22 11:09 chenhengqi

Here's the most relevant part:

[ 38%] Building CXX object examples/cpp/CMakeFiles/CGroupTest.dir/CGroupTest.cc.o
cd /home/rnsanchez/apps/bcc-copy/build/examples/cpp && /usr/bin/c++ -DHAVE_LIBDEBUGINFOD -I/usr/include/../tools/clang/include -I/home/rnsanchez/apps/bcc-copy/build/src/cc -I/home/rnsanchez/apps/bcc-copy/src/cc -I/home/rnsanchez/apps/bcc-copy/src/cc/api -I/home/rnsanchez/apps/bcc-copy/src/cc/libbpf/include/uapi -Wall  -fPIC -O3 -DNDEBUG -std=gnu++14 -MD -MT examples/cpp/CMakeFiles/CGroupTest.dir/CGroupTest.cc.o -MF CMakeFiles/CGroupTest.dir/CGroupTest.cc.o.d -o CMakeFiles/CGroupTest.dir/CGroupTest.cc.o -c /home/rnsanchez/apps/bcc-copy/examples/cpp/CGroupTest.cc
[ 38%] Linking CXX executable CGroupTest
cd /home/rnsanchez/apps/bcc-copy/build/examples/cpp && /usr/bin/cmake -E cmake_link_script CMakeFiles/CGroupTest.dir/link.txt --verbose=1
/usr/bin/c++  -Wall  -fPIC -O3 -DNDEBUG -rdynamic CMakeFiles/CGroupTest.dir/CGroupTest.cc.o -o CGroupTest  ../../src/cc/libbcc.a ../../src/cc/frontends/clang/libclang_frontend.a -Wl,--whole-archive -lclangFrontend -lclangSerialization -lclangDriver -lclangASTMatchers -lclangParse -lclangSema -lclangCodeGen -lclangAnalysis -lclangRewrite -lclangEdit -lclangAST -lclangLex -lclangBasic /usr/lib64/libLLVMBPFDisassembler.so.14 /usr/lib64/libLLVMBPFAsmParser.so.14 /usr/lib64/libLLVMCoverage.so.14 /usr/lib64/libLLVMX86CodeGen.so.14 /usr/lib64/libLLVMX86Desc.so.14 /usr/lib64/libLLVMX86Info.so.14 /usr/lib64/libLLVMLTO.so.14 /usr/lib64/libLLVMPasses.so.14 /usr/lib64/libLLVMCoroutines.so.14 /usr/lib64/libLLVMOption.so.14 /usr/lib64/libLLVMObjCARCOpts.so.14 /usr/lib64/libLLVMMCJIT.so.14 /usr/lib64/libLLVMBPFCodeGen.so.14 /usr/lib64/libLLVMFrontendOpenMP.so.14 /usr/lib64/libLLVMLinker.so.14 /usr/lib64/libLLVMIRReader.so.14 /usr/lib64/libLLVMBitWriter.so.14 /usr/lib64/libLLVMDebugInfoDWARF.so.14 -Wl,--no-whole-archive -lelf -ldebuginfod ../../src/cc/libbcc_bpf.a ../../src/cc/api/libapi-static.a ../../src/cc/usdt/libusdt-static.a ../../src/cc/libbcc-loader-static.a -lelf -lz 
/usr/bin/ld: ../../src/cc/libbcc.a(bcc_debug.cc.o): undefined reference to symbol '_ZN4llvm12MemoryBuffer16getMemBufferCopyENS_9StringRefERKNS_5TwineE'
/usr/bin/ld: /usr/lib64/gcc/x86_64-slackware-linux/12.2.0/../../../../lib64/../lib64/libLLVMSupport.so.14: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [examples/cpp/CMakeFiles/CGroupTest.dir/build.make:136: examples/cpp/CGroupTest] Error 1
make[2]: Leaving directory '/home/rnsanchez/apps/bcc-copy/build'
make[1]: *** [CMakeFiles/Makefile2:1039: examples/cpp/CMakeFiles/CGroupTest.dir/all] Error 2
make[1]: Leaving directory '/home/rnsanchez/apps/bcc-copy/build'
make: *** [Makefile:146: all] Error 2

If needed, here's the full output: https://gist.github.com/rnsanchez/e533d5c9333920467d19f2151823a584

rnsanchez avatar Sep 07 '22 12:09 rnsanchez

Looks like you are using LLVM 14 libraries ?

chenhengqi avatar Sep 07 '22 12:09 chenhengqi

Looks like you are using LLVM 14 libraries ?

Yes, 14.0.6, default with Slackware64-15.0 (and as of recent Slackware64-current).

rnsanchez avatar Sep 07 '22 13:09 rnsanchez

Cloud you please commet out line 156 and line 158 in CMakeLists.txt and see if it works?

chenhengqi avatar Sep 07 '22 13:09 chenhengqi

Cloud you please commet out line 156 and line 158 in CMakeLists.txt and see if it works?

Done. The error persists:

make[2]: Entering directory '/home/rnsanchez/apps/bcc-copy/build'
[ 38%] Building CXX object examples/cpp/CMakeFiles/CGroupTest.dir/CGroupTest.cc.o
cd /home/rnsanchez/apps/bcc-copy/build/examples/cpp && /usr/bin/c++ -DHAVE_LIBDEBUGINFOD -I/usr/include/../tools/clang/include -I/home/rnsanchez/apps/bcc-copy/build/src/cc -I/home/rnsanchez/apps/bcc-copy/src/cc -I/home/rnsanchez/apps/bcc-copy/src/cc/api -I/home/rnsanchez/apps/bcc-copy/src/cc/libbpf/include/uapi -Wall  -fPIC -O3 -DNDEBUG -std=gnu++14 -MD -MT examples/cpp/CMakeFiles/CGroupTest.dir/CGroupTest.cc.o -MF CMakeFiles/CGroupTest.dir/CGroupTest.cc.o.d -o CMakeFiles/CGroupTest.dir/CGroupTest.cc.o -c /home/rnsanchez/apps/bcc-copy/examples/cpp/CGroupTest.cc
[ 38%] Linking CXX executable CGroupTest
cd /home/rnsanchez/apps/bcc-copy/build/examples/cpp && /usr/bin/cmake -E cmake_link_script CMakeFiles/CGroupTest.dir/link.txt --verbose=1
/usr/bin/c++  -Wall  -fPIC -O3 -DNDEBUG -rdynamic CMakeFiles/CGroupTest.dir/CGroupTest.cc.o -o CGroupTest  ../../src/cc/libbcc.a ../../src/cc/frontends/clang/libclang_frontend.a -Wl,--whole-archive -lclangFrontend -lclangSerialization -lclangDriver -lclangASTMatchers -lclangParse -lclangSema -lclangCodeGen -lclangAnalysis -lclangRewrite -lclangEdit -lclangAST -lclangLex -lclang-cpp -lclangBasic /usr/lib64/libLLVMBPFDisassembler.so.14 /usr/lib64/libLLVMBPFAsmParser.so.14 /usr/lib64/libLLVMCoverage.so.14 /usr/lib64/libLLVMX86CodeGen.so.14 /usr/lib64/libLLVMX86Desc.so.14 /usr/lib64/libLLVMX86Info.so.14 /usr/lib64/libLLVMLTO.so.14 /usr/lib64/libLLVMPasses.so.14 /usr/lib64/libLLVMCoroutines.so.14 /usr/lib64/libLLVMOption.so.14 /usr/lib64/libLLVMObjCARCOpts.so.14 /usr/lib64/libLLVMMCJIT.so.14 /usr/lib64/libLLVMBPFCodeGen.so.14 /usr/lib64/libLLVMFrontendOpenMP.so.14 /usr/lib64/libLLVMLinker.so.14 /usr/lib64/libLLVMIRReader.so.14 /usr/lib64/libLLVMBitWriter.so.14 /usr/lib64/libLLVMDebugInfoDWARF.so.14 -Wl,--no-whole-archive -lelf -ldebuginfod ../../src/cc/libbcc_bpf.a ../../src/cc/api/libapi-static.a ../../src/cc/usdt/libusdt-static.a ../../src/cc/libbcc-loader-static.a -lelf -lz 
/usr/bin/ld: ../../src/cc/libbcc.a(bcc_debug.cc.o): undefined reference to symbol '_ZN4llvm12MemoryBuffer16getMemBufferCopyENS_9StringRefERKNS_5TwineE'
/usr/bin/ld: /usr/lib64/gcc/x86_64-slackware-linux/12.2.0/../../../../lib64/../lib64/libLLVMSupport.so.14: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [examples/cpp/CMakeFiles/CGroupTest.dir/build.make:137: examples/cpp/CGroupTest] Error 1
make[2]: Leaving directory '/home/rnsanchez/apps/bcc-copy/build'
make[1]: *** [CMakeFiles/Makefile2:1039: examples/cpp/CMakeFiles/CGroupTest.dir/all] Error 2
make[1]: Leaving directory '/home/rnsanchez/apps/bcc-copy/build'
make: *** [Makefile:146: all] Error 2

rnsanchez avatar Sep 07 '22 15:09 rnsanchez

Looks like you are using LLVM 14 libraries ?

Yes, 14.0.6, default with Slackware64-15.0 (and as of recent Slackware64-current).

-- Found LLVM: /usr/include 13.0.0 (Use LLVM_ROOT envronment variable for another version of LLVM)

No idea. Is there a mismatch between your LLVM toolchain and LLVM libraries ?

Typically, BCC links against a single LLVM library (llvm-config --libs --ldflags).

chenhengqi avatar Sep 07 '22 15:09 chenhengqi

-- Found LLVM: /usr/include 13.0.0 (Use LLVM_ROOT envronment variable for another version of LLVM)

No idea. Is there a mismatch between your LLVM toolchain and LLVM libraries ?

Typically, BCC links against a single LLVM library (llvm-config --libs --ldflags).

I can't say if there is a mismatch; there should not be any though (especially considering that the error happens on a fresh install as well).

If it helps knowing:

$ llvm-config --libs --ldflags
-L/usr/lib64 
-lLLVMWindowsManifest -lLLVMXRay -lLLVMLibDriver -lLLVMDlltoolDriver -lLLVMCoverage -lLLVMLineEditor -lLLVMX86TargetMCA -lLLVMX86Disassembler -lLLVMX86AsmParser -lLLVMX86CodeGen -lLLVMX86Desc -lLLVMX86Info -lLLVMBPFDisassembler -lLLVMBPFAsmParser -lLLVMBPFCodeGen -lLLVMBPFDesc -lLLVMBPFInfo -lLLVMAMDGPUTargetMCA -lLLVMAMDGPUDisassembler -lLLVMAMDGPUAsmParser -lLLVMAMDGPUCodeGen -lLLVMAMDGPUDesc -lLLVMAMDGPUUtils -lLLVMAMDGPUInfo -lLLVMOrcJIT -lLLVMMCJIT -lLLVMJITLink -lLLVMInterpreter -lLLVMExecutionEngine -lLLVMRuntimeDyld -lLLVMOrcTargetProcess -lLLVMOrcShared -lLLVMDWP -lLLVMSymbolize -lLLVMDebugInfoPDB -lLLVMDebugInfoGSYM -lLLVMOption -lLLVMObjectYAML -lLLVMMCA -lLLVMMCDisassembler -lLLVMLTO -lLLVMCFGuard -lLLVMFrontendOpenACC -lLLVMExtensions -lPolly -lPollyISL -lLLVMPasses -lLLVMObjCARCOpts -lLLVMCoroutines -lLLVMipo -lLLVMInstrumentation -lLLVMVectorize -lLLVMLinker -lLLVMFrontendOpenMP -lLLVMDWARFLinker -lLLVMGlobalISel -lLLVMMIRParser -lLLVMAsmPrinter -lLLVMDebugInfoMSF -lLLVMSelectionDAG -lLLVMCodeGen -lLLVMIRReader -lLLVMAsmParser -lLLVMInterfaceStub -lLLVMFileCheck -lLLVMFuzzMutate -lLLVMTarget -lLLVMScalarOpts -lLLVMInstCombine -lLLVMAggressiveInstCombine -lLLVMTransformUtils -lLLVMBitWriter -lLLVMAnalysis -lLLVMProfileData -lLLVMDebugInfoDWARF -lLLVMObject -lLLVMTextAPI -lLLVMMCParser -lLLVMMC -lLLVMDebugInfoCodeView -lLLVMBitReader -lLLVMCore -lLLVMRemarks -lLLVMBitstreamReader -lLLVMBinaryFormat -lLLVMTableGen -lLLVMSupport -lLLVMDemangle

$ llvm-config --components
aggressiveinstcombine all all-targets amdgpu amdgpuasmparser amdgpucodegen amdgpudesc amdgpudisassembler amdgpuinfo amdgputargetmca amdgpuutils analysis asmparser asmprinter binaryformat bitreader bitstreamreader bitwriter bpf bpfasmparser bpfcodegen bpfdesc bpfdisassembler bpfinfo cfguard codegen core coroutines coverage debuginfocodeview debuginfodwarf debuginfogsym debuginfomsf debuginfopdb demangle dlltooldriver dwarflinker dwp engine executionengine extensions filecheck frontendopenacc frontendopenmp fuzzmutate globalisel instcombine instrumentation interfacestub interpreter ipo irreader jitlink libdriver lineeditor linker lto mc mca mcdisassembler mcjit mcparser mirparser native nativecodegen objcarcopts object objectyaml option orcjit orcshared orctargetprocess passes profiledata remarks runtimedyld scalaropts selectiondag support symbolize tablegen target textapi transformutils vectorize windowsmanifest x86 x86asmparser x86codegen x86desc x86disassembler x86info x86targetmca xray

$ llvm-config --targets-built
AMDGPU BPF X86

rnsanchez avatar Sep 07 '22 19:09 rnsanchez

-- Found LLVM: /usr/include 13.0.0 (Use LLVM_ROOT envronment variable for another version of LLVM)

Your CMake reports that it found LLVM 13.

/usr/bin/ld: /usr/lib64/gcc/x86_64-slackware-linux/12.2.0/../../../../lib64/../lib64/libLLVMSupport.so.14: error adding symbols: DSO missing from command line

But actually, you are linking against LLVM 14 libraries.

chenhengqi avatar Sep 08 '22 02:09 chenhengqi

OK, based on your feedback, I decided to retrace all my steps and found out I was mistaken about the precise environment:

  • Slackware64-15.0 (fresh install) ships LLVM 13.0.0 
$ cat /usr/lib64/cmake/llvm/LLVMConfigVersion.cmake
set(PACKAGE_VERSION "13.0.0")
...
  • Slackware64-current (the one I use the most), is currently shipping LLVM 14.0.6
$ cat /usr/lib64/cmake/llvm/LLVMConfigVersion.cmake
set(PACKAGE_VERSION "14.0.6")
...

However, both of them fail building, as far as I can tell exactly the same way:

  • Slackware64-15.0, LLVM 13.0.0, GCC 11.2.0:
make  -f examples/cpp/CMakeFiles/CGroupTest.dir/build.make examples/cpp/CMakeFiles/CGroupTest.dir/build
make[2]: Entering directory '/home/rnsanchez/bcc-copy/build'
[ 38%] Linking CXX executable CGroupTest
cd /home/rnsanchez/bcc-copy/build/examples/cpp && /usr/bin/cmake -E cmake_link_script CMakeFiles/CGroupTest.dir/link.txt --verbose=1
/usr/bin/c++  -Wall  -fPIC -O3 -DNDEBUG -rdynamic CMakeFiles/CGroupTest.dir/CGroupTest.cc.o -o CGroupTest  ../../src/cc/libbcc.a ../../src/cc/frontends/clang/libclang_frontend.a -Wl,--whole-archive -lclangFrontend -lclangSerialization -lcla
ngDriver -lclangASTMatchers -lclangParse -lclangSema -lclangCodeGen -lclangAnalysis -lclangRewrite -lclangEdit -lclangAST -lclangLex -lclang-cpp -lclangBasic /usr/lib64/libLLVMBPFDisassembler.so.13 /usr/lib64/libLLVMBPFAsmParser.so.13 /usr/
lib64/libLLVMCoverage.so.13 /usr/lib64/libLLVMX86CodeGen.so.13 /usr/lib64/libLLVMX86Desc.so.13 /usr/lib64/libLLVMX86Info.so.13 /usr/lib64/libLLVMLTO.so.13 /usr/lib64/libLLVMPasses.so.13 /usr/lib64/libLLVMCoroutines.so.13 /usr/lib64/libLLVMO
ption.so.13 /usr/lib64/libLLVMObjCARCOpts.so.13 /usr/lib64/libLLVMMCJIT.so.13 /usr/lib64/libLLVMBPFCodeGen.so.13 /usr/lib64/libLLVMFrontendOpenMP.so.13 /usr/lib64/libLLVMLinker.so.13 /usr/lib64/libLLVMIRReader.so.13 /usr/lib64/libLLVMDebugI
nfoDWARF.so.13 /usr/lib64/libLLVMBitWriter.so.13 -Wl,--no-whole-archive -lelf -ldebuginfod ../../src/cc/libbcc_bpf.a ../../src/cc/api/libapi-static.a ../../src/cc/usdt/libusdt-static.a ../../src/cc/libbcc-loader-static.a -lelf -lz 
/usr/bin/ld: ../../src/cc/libbcc.a(bcc_debug.cc.o): undefined reference to symbol '_ZN4llvm12MemoryBuffer16getMemBufferCopyENS_9StringRefERKNS_5TwineE'
/usr/bin/ld: /usr/lib64/gcc/x86_64-slackware-linux/11.2.0/../../../../lib64/../lib64/libLLVMSupport.so.13: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [examples/cpp/CMakeFiles/CGroupTest.dir/build.make:137: examples/cpp/CGroupTest] Error 1
make[2]: Leaving directory '/home/rnsanchez/bcc-copy/build'
make[1]: *** [CMakeFiles/Makefile2:1039: examples/cpp/CMakeFiles/CGroupTest.dir/all] Error 2
make[1]: Leaving directory '/home/rnsanchez/bcc-copy/build'
make: *** [Makefile:146: all] Error 2
  • Slackware64-current, LLVM 14.0.6, GCC-12.2
make[2]: Leaving directory '/home/rnsanchez/apps/bcc-copy/build'
make  -f examples/cpp/CMakeFiles/CGroupTest.dir/build.make examples/cpp/CMakeFiles/CGroupTest.dir/build
make[2]: Entering directory '/home/rnsanchez/apps/bcc-copy/build'
[ 38%] Linking CXX executable CGroupTest
cd /home/rnsanchez/apps/bcc-copy/build/examples/cpp && /usr/bin/cmake -E cmake_link_script CMakeFiles/CGroupTest.dir/link.txt --verbose=1
/usr/bin/c++  -Wall  -fPIC -O3 -DNDEBUG -rdynamic CMakeFiles/CGroupTest.dir/CGroupTest.cc.o -o CGroupTest  ../../src/cc/libbcc.a ../../src/cc/frontends/clang/libclang_frontend.a -Wl,--whole-archive -lclangFrontend -lclangSerialization -lclangDriver -lclangASTMatchers -lclangParse -lclangSema -lclangCodeGen -lclangAnalysis -lclangRewrite -lclangEdit -lclangAST -lclangLex -lclang-cpp -lclangBasic /usr/lib64/libLLVMBPFDisassembler.so.14 /usr/lib64/libLLVMBPFAsmParser.so.14 /usr/lib64/libLLVMCoverage.so.14 /usr/lib64/libLLVMX86CodeGen.so.14 /usr/lib64/libLLVMX86Desc.so.14 /usr/lib64/libLLVMX86Info.so.14 /usr/lib64/libLLVMLTO.so.14 /usr/lib64/libLLVMPasses.so.14 /usr/lib64/libLLVMCoroutines.so.14 /usr/lib64/libLLVMOption.so.14 /usr/lib64/libLLVMObjCARCOpts.so.14 /usr/lib64/libLLVMMCJIT.so.14 /usr/lib64/libLLVMBPFCodeGen.so.14 /usr/lib64/libLLVMFrontendOpenMP.so.14 /usr/lib64/libLLVMLinker.so.14 /usr/lib64/libLLVMIRReader.so.14 /usr/lib64/libLLVMBitWriter.so.14 /usr/lib64/libLLVMDebugInfoDWARF.so.14 -Wl,--no-whole-archive -lelf -ldebuginfod ../../src/cc/libbcc_bpf.a ../../src/cc/api/libapi-static.a ../../src/cc/usdt/libusdt-static.a ../../src/cc/libbcc-loader-static.a -lelf -lz 
/usr/bin/ld: ../../src/cc/libbcc.a(bcc_debug.cc.o): undefined reference to symbol '_ZN4llvm12MemoryBuffer16getMemBufferCopyENS_9StringRefERKNS_5TwineE'
/usr/bin/ld: /usr/lib64/gcc/x86_64-slackware-linux/12.2.0/../../../../lib64/../lib64/libLLVMSupport.so.14: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[2]: *** [examples/cpp/CMakeFiles/CGroupTest.dir/build.make:137: examples/cpp/CGroupTest] Error 1
make[2]: Leaving directory '/home/rnsanchez/apps/bcc-copy/build'
make[1]: *** [CMakeFiles/Makefile2:1039: examples/cpp/CMakeFiles/CGroupTest.dir/all] Error 2
make[1]: Leaving directory '/home/rnsanchez/apps/bcc-copy/build'
make: *** [Makefile:146: all] Error 2

As an extra info, I had my LLVM recompiled in order to not remove any symbols, so I can check that the requested symbol is indeed present:

$ nm /usr/lib64/gcc/x86_64-slackware-linux/12.2.0/../../../../lib64/../lib64/libLLVMSupport.so.14 | grep _ZN4llvm12MemoryBuffer16getMemBufferCopyENS_9StringRefERKNS_5TwineE
000000000011d7b0 T _ZN4llvm12MemoryBuffer16getMemBufferCopyENS_9StringRefERKNS_5TwineE

The following diff did not help:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index f622aa84..a7af47a4 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -153,9 +153,9 @@ find_library(libclangRewrite NAMES clangRewrite clang-cpp HINTS ${CLANG_SEARCH})
 find_library(libclangSema NAMES clangSema clang-cpp HINTS ${CLANG_SEARCH})
 find_library(libclangSerialization NAMES clangSerialization clang-cpp HINTS ${CLANG_SEARCH})
 find_library(libclangASTMatchers NAMES clangASTMatchers clang-cpp HINTS ${CLANG_SEARCH})
-if (${LLVM_PACKAGE_VERSION} VERSION_EQUAL 15 OR ${LLVM_PACKAGE_VERSION} VERSION_GREATER 15)
+#if (${LLVM_PACKAGE_VERSION} VERSION_EQUAL 15 OR ${LLVM_PACKAGE_VERSION} VERSION_GREATER 15)
   find_library(libclangSupport NAMES clangSupport clang-cpp HINTS ${CLANG_SEARCH})
-endif()
+#endif()
 find_library(libclang-shared libclang-cpp.so HINTS ${CLANG_SEARCH})
 if(libclangBasic STREQUAL "libclangBasic-NOTFOUND")
   message(FATAL_ERROR "Unable to find clang libraries")

This one instead had the build to complete successfully:

diff --git a/src/cc/CMakeLists.txt b/src/cc/CMakeLists.txt
index ffe8feec..018afb9e 100644
--- a/src/cc/CMakeLists.txt
+++ b/src/cc/CMakeLists.txt
@@ -124,7 +124,7 @@ set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} ${llvm_lib_exclude_f
 # bcc_common_libs_for_a for archive libraries
 # bcc_common_libs_for_s for shared libraries
 set(bcc_common_libs clang_frontend
-  -Wl,--whole-archive ${clang_libs} ${llvm_libs} -Wl,--no-whole-archive
+  -Wl,--whole-archive,--copy-dt-needed-entries ${clang_libs} ${llvm_libs} -Wl,--no-whole-archive
   ${LIBELF_LIBRARIES})
 if (LIBDEBUGINFOD_FOUND)
   list(APPEND bcc_common_libs ${LIBDEBUGINFOD_LIBRARIES})
@@ -147,7 +147,7 @@ if(ENABLE_CPP_API)
   add_subdirectory(api)
   list(APPEND bcc_common_libs_for_a api-static)
   # Keep all API functions
-  list(APPEND bcc_common_libs_for_s -Wl,--whole-archive api-static -Wl,--no-whole-archive)
+  list(APPEND bcc_common_libs_for_s -Wl,--whole-archive,--copy-dt-needed-entries api-static -Wl,--no-whole-archive)
 endif()
 
 if(ENABLE_USDT)

Then I have tried, and I by tried I mean "simply adding whatever library ld was complaining" to the command line, I eventually got that specific binary to build, using this longer command line:

bcc-copy/build/examples/cpp]$ /usr/bin/c++  -Wall  -fPIC -O3 -DNDEBUG -rdynamic CMakeFiles/CGroupTest.dir/CGroupTest.cc.o -o CGroupTest  ../../src/cc/libbcc.a ../../src/cc/frontends/clang/libclang_frontend.a -Wl,--whole-archive -lclangFrontend -lclangSerialization -lclangDriver -lclangASTMatchers -lclangParse -lclangSema -lclangCodeGen -lclangAnalysis -lclangRewrite -lclangEdit -lclangAST -lclangLex -lclang-cpp -lclangBasic /usr/lib64/libLLVMSupport.so.14 /usr/lib64/libLLVMMC.so.14 /usr/lib64/libLLVMCore.so.14 /usr/lib64/libLLVMipo.so.14 /usr/lib64/libLLVMObject.so.14 /usr/lib64/libLLVMBPFDesc.so.14 /usr/lib64/libLLVMExecutionEngine.so.14 /usr/lib64/libLLVMRuntimeDyld.so.14 /usr/lib64/libLLVMBPFInfo.so.14 /usr/lib64/libLLVMBPFDisassembler.so.14 /usr/lib64/libLLVMBPFAsmParser.so.14 /usr/lib64/libLLVMCoverage.so.14 /usr/lib64/libLLVMX86CodeGen.so.14 /usr/lib64/libLLVMX86Desc.so.14 /usr/lib64/libLLVMX86Info.so.14 /usr/lib64/libLLVMLTO.so.14 /usr/lib64/libLLVMPasses.so.14 /usr/lib64/libLLVMCoroutines.so.14 /usr/lib64/libLLVMOption.so.14 /usr/lib64/libLLVMObjCARCOpts.so.14 /usr/lib64/libLLVMMCJIT.so.14 /usr/lib64/libLLVMBPFCodeGen.so.14 /usr/lib64/libLLVMFrontendOpenMP.so.14 /usr/lib64/libLLVMLinker.so.14 /usr/lib64/libLLVMIRReader.so.14 /usr/lib64/libLLVMBitWriter.so.14 /usr/lib64/libLLVMDebugInfoDWARF.so.14 -Wl,--no-whole-archive -lelf -ldebuginfod ../../src/cc/libbcc_bpf.a ../../src/cc/api/libapi-static.a ../../src/cc/usdt/libusdt-static.a ../../src/cc/libbcc-loader-static.a -lelf -lz

These were the added libs, just after -lclangBasic:

/usr/lib64/libLLVMSupport.so.14 /usr/lib64/libLLVMMC.so.14 /usr/lib64/libLLVMCore.so.14 /usr/lib64/libLLVMipo.so.14 /usr/lib64/libLLVMObject.so.14 /usr/lib64/libLLVMBPFDesc.so.14 /usr/lib64/libLLVMExecutionEngine.so.14 /usr/lib64/libLLVMRuntimeDyld.so.14

But I don't know why weren't these properly picked up during cmake.

rnsanchez avatar Sep 08 '22 12:09 rnsanchez

Looks like you are using LLVM 14 libraries ?

My ENV:

[rongtao@RT-PC ~]$ cat /etc/redhat-release 
CentOS Stream release 9
[rongtao@RT-PC ~]$ llvm-as --version
LLVM (http://llvm.org/):
  LLVM version 14.0.6
  Optimized build.
  Default target: x86_64-redhat-linux-gnu
  Host CPU: skylake

Solve this problem in https://github.com/iovisor/bcc/pull/4232

Rtoax avatar Sep 12 '22 10:09 Rtoax

Fix confirmed with https://github.com/iovisor/bcc/pull/4232/commits/b09d91d2196eebbe7d89ffcfbab4133fb5dc9abf.

rnsanchez avatar Sep 12 '22 11:09 rnsanchez