xmake Build error using shared link chain with add

Xmake Version

2.8.6

Operating System Version and Architecture

Ubuntu 22

Describe Bug

When building a dynamic link chain with a dynamic library from add_requires at the end of the chain, the build fails on Linux and succeeds on macOS. 当构建动态链接链，且链的末端是来自add_requires的动态库时，在Linux上会构建失败，但在macOS上构建成功。

The dependencies are shown in the figure: 依赖关系如图:

The links to the repo and github action results used for reproduction are as follows. 用于复现的repo和github action结果链接如下:

Results with ldd in Ubuntu: 在Ubuntu使用ldd的结果:

Run ldd ./build/linux/x86_6[4](https://github.com/traversebitree/test_xmake_shared_link_chain_with_add_package/actions/runs/7778257919/job/21207685528#step:6:5)/release/libmydylib.so
	linux-vdso.so.1 (0x00007ffd76b78000)
	libfmt.so.10 => not found
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3764c00000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f3764f1[5](https://github.com/traversebitree/test_xmake_shared_link_chain_with_add_package/actions/runs/7778257919/job/21207685528#step:6:6)000)

Results for using otool on macOS: 在macOS使用otool的结果:

Run otool -L ./build/macosx/x8[6](https://github.com/traversebitree/test_xmake_shared_link_chain_with_add_package/actions/runs/7778257919/job/21207685873#step:7:7)_64/release/main
./build/macosx/x[8](https://github.com/traversebitree/test_xmake_shared_link_chain_with_add_package/actions/runs/7778257919/job/21207685873#step:7:9)6_64/release/main:
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	@rpath/libmydylib.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1300.36.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 131[9](https://github.com/traversebitree/test_xmake_shared_link_chain_with_add_package/actions/runs/7778257919/job/21207685873#step:7:10).0.0)
./build/macosx/x86_64/release/libmydylib.dylib:
	@rpath/libmydylib.dylib (compatibility version 0.0.0, current version 0.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	@rpath/libfmt.[10](https://github.com/traversebitree/test_xmake_shared_link_chain_with_add_package/actions/runs/7778257919/job/21207685873#step:7:11).dylib (compatibility version 10.0.0, current version 10.2.1)
	/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version [13](https://github.com/traversebitree/test_xmake_shared_link_chain_with_add_package/actions/runs/7778257919/job/21207685873#step:7:14)00.36.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.0.0)

I tried to set rpath, but it still doesn't work. Because the dependency libfmt.so.10 is in /home/runner/.xmake/packages/f/fmt/10.2.1/ea2159e092e440a3866610426e31b244/lib: 我试图设置rpath，但结果依然不行。因为要依赖的 libfmt.so.10 在 /home/runner/.xmake/packages/f/fmt/10.2.1/ea2159e092e440a3866610426e31b244/lib 位置:

rule("set_rpath")
do
  on_load(function(target)
    if (target:kind() == "binary" or target:kind() == "shared") and is_plat("linux") then
      target:add("rpathdirs", "$ORIGIN/")
    end
  end)
end
rule_end()

Expected Behavior

This problem only occurs on Linux, and it's not even a problem on macOS. 这个问题只发生在Linux上，而在macOS都没这个问题。

The problem is in the linking process, which is a two-step process. 问题出在链接过程中，链接分两步:

第一步 linking.release libmydylib.so 没问题，用-L直接链接libfmt.so.10所在的目录

[ 50%]: linking.release libmydylib.so
/usr/bin/g++ -o build/linux/x86_64/release/libmydylib.so build/.objs/mydylib/linux/x86_64/release/dylibsrc/dylibsrc.cpp.o -shared -m64 -fPIC -L/home/runner/.xmake/packages/f/fmt/10.2.1/ea2159e092e440a3866610426e31b244/lib -s -lfmt
checking for the linker (ld) ... g++
checking for flags (-Wl,-rpath=@loader_path) ... ok

第二部 linking.release main 出错了，main用-L链接到libmydylib.so，但libmydylib.so需要在rpath里找到libfmt.so.10才行，找不到就报错了

[ 75%]: linking.release main
/usr/bin/g++ -o build/linux/x86_64/release/main build/.objs/main/linux/x86_64/release/src/main.cpp.o -m64 -Lbuild/linux/x86_64/release -Wl,-rpath=$ORIGIN -s -lmydylib
/usr/bin/ld: warning: libfmt.so.10, needed by build/linux/x86_64/release/libmydylib.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: build/linux/x86_64/release/libmydylib.so: undefined reference to `fmt::v10::vprint(fmt::v10::basic_string_view<char>, fmt::v10::basic_format_args<fmt::v10::basic_format_context<fmt::v10::appender, char> >)'
collect2: error: ld returned 1 exit status
error: execv(/usr/bin/g++ -o build/linux/x86_64/release/main build/.objs/main/linux/x86_64/release/src/main.cpp.o -m64 -Lbuild/linux/x86_64/release -Wl,-rpath=$ORIGIN -s -lmydylib) failed(1)
Error: Process completed with exit code 255.

So at build time, either copy libfmt.so.10 to the directory where libmydylib.so is located, or automatically set the rpath equal to the directory where the link -L is located 因此，要么在构建的时候，要么把libfmt.so.10拷贝到libmydylib.so所在目录，要么自动设置rpath等于链接-L时的目录

Project Configuration

add_rules("mode.debug", "mode.release")
set_languages("cxx20")

add_requires("fmt 10.x", {
	debug = is_mode("debug"),
	configs = {
		shared = true,
	},
})

target("main")
do
	set_kind("binary")
	add_files("src/main.cpp")
	add_deps("mydylib")
end
target_end()

target("mydylib")
do
	set_kind("shared")
	add_files("dylibsrc/dylibsrc.cpp")
	add_includedirs("dylibsrc/", { public = true })
	add_headerfiles("dylibsrc/dylibsrc.hpp", { install = true })
	add_packages("fmt")
end
target_end()

Additional Information and Error Logs

用的xmake -vD: https://github.com/traversebitree/test_xmake_shared_link_chain_with_add_package/actions/runs/7778666728/job/21208665428

Run xmake q -y
checking for platform ... linux
checking for architecture ... x86_64
updating repositories .. ok
  => download https://github.com/fmtlib/fmt/releases/download/10.2.1/fmt-10.2.1.zip .. ok
  => install fmt 10.2.1 .. ok
checking for the c++ compiler (cxx) ... gcc
checking for flags (-O3) ... ok
> gcc "-O3" "-m64"
checking for flags (-fvisibility-inlines-hidden) ... ok
> gcc "-fvisibility-inlines-hidden" "-m64"
checking for flags (-std=c++20) ... ok
> gcc "-std=c++20" "-m64"
checking for flags (-DNDEBUG) ... ok
> gcc "-DNDEBUG" "-m64"
[ 25%]: compiling.release src/main.cpp
/usr/bin/gcc -c -m64 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -Idylibsrc -DNDEBUG -o build/.objs/main/linux/x86_64/release/src/main.cpp.o src/main.cpp
checking for flags (-std=c++20) ... ok
> gcc "-std=c++20" "-m64"
[ 25%]: compiling.release dylibsrc/dylibsrc.cpp
/usr/bin/gcc -c -m64 -fPIC -O3 -std=c++20 -Idylibsrc -DFMT_LIB_EXPORT -isystem /home/runner/.xmake/packages/f/fmt/10.2.1/ea2159e092e440a3866610426e31b244/include -DNDEBUG -o build/.objs/mydylib/linux/x86_64/release/dylibsrc/dylibsrc.cpp.o dylibsrc/dylibsrc.cpp
checking for flags (-MMD -MF) ... ok
> gcc "-MMD" "-MF" "/dev/null" "-m64"
checking for flags (-MMD -MF) ... ok
> gcc "-MMD" "-MF" "/dev/null" "-m64"
checking for g++ ... /usr/bin/g++
checking for the shared library linker (sh) ... g++
checking for flags (-fPIC) ... ok
> g++ "-fPIC" "-shared" "-m64" "-m64"
[ 50%]: linking.release libmydylib.so
/usr/bin/g++ -o build/linux/x86_64/release/libmydylib.so build/.objs/mydylib/linux/x86_64/release/dylibsrc/dylibsrc.cpp.o -shared -m64 -fPIC -L/home/runner/.xmake/packages/f/fmt/10.2.1/ea2159e092e440a3866610426e31b244/lib -s -lfmt
checking for the linker (ld) ... g++
checking for flags (-Wl,-rpath=@loader_path) ... ok
> g++ "-Wl,-rpath=@loader_path" "-m64" "-m64"
[ [7](https://github.com/traversebitree/test_xmake_shared_link_chain_with_add_package/actions/runs/7778666728/job/21208665428#step:5:8)5%]: linking.release main
/usr/bin/g++ -o build/linux/x[8](https://github.com/traversebitree/test_xmake_shared_link_chain_with_add_package/actions/runs/7778666728/job/21208665428#step:5:9)6_64/release/main build/.objs/main/linux/x86_64/release/src/main.cpp.o -m64 -Lbuild/linux/x86_64/release -Wl,-rpath=$ORIGIN -s -lmydylib
/usr/bin/ld: warning: libfmt.so.10, needed by build/linux/x86_64/release/libmydylib.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: build/linux/x86_64/release/libmydylib.so: undefined reference to `fmt::v10::vprint(fmt::v10::basic_string_view<char>, fmt::v10::basic_format_args<fmt::v10::basic_format_context<fmt::v10::appender, char> >)'
collect2: error: ld returned 1 exit status
error: @programdir/core/main.lua:314: @programdir/actions/build/main.lua:148: @programdir/modules/async/runjobs.lua:320: @programdir/actions/build/kinds/binary.lua:74: @programdir/core/sandbox/modules/os.lua:378: execv(/usr/bin/g++ -o build/linux/x86_64/release/main build/.objs/main/linux/x86_64/release/src/main.cpp.o -m64 -Lbuild/linux/x86_64/release -Wl,-rpath=$ORIGIN -s -lmydylib) failed(1)
stack traceback:
    [C]: in function 'error'
    [@programdir/core/base/os.lua:[9](https://github.com/traversebitree/test_xmake_shared_link_chain_with_add_package/actions/runs/7778666728/job/21208665428#step:5:10)49]:
    [@programdir/core/sandbox/modules/os.lua:378]: in function 'execv'
    [@programdir/modules/core/tools/gcc.lua:526]:
    [C]: in function 'xpcall'
    [@programdir/core/base/utils.lua:280]:
    [@programdir/core/tool/linker.lua:221]: in function 'link'
    [@programdir/actions/build/kinds/binary.lua:74]: in function 'callback'
    [@programdir/modules/core/project/depend.lua:217]: in function 'on_changed'
    [@programdir/actions/build/kinds/binary.lua:55]: in function '_do_link_target'
    [@programdir/actions/build/kinds/binary.lua:[10](https://github.com/traversebitree/test_xmake_shared_link_chain_with_add_package/actions/runs/7778666728/job/21208665428#step:5:11)5]:
    [@programdir/actions/build/kinds/binary.lua:132]: in function '_link_target'
    [@programdir/actions/build/kinds/binary.lua:160]: in function 'jobfunc'
    [@programdir/modules/async/runjobs.lua:237]:
    [C]: in function 'xpcall'
    [@programdir/core/base/utils.lua:280]: in function 'trycall'
    [@programdir/core/sandbox/modules/try.lua:[11](https://github.com/traversebitree/test_xmake_shared_link_chain_with_add_package/actions/runs/7778666728/job/21208665428#step:5:12)7]: in function 'try'
    [@programdir/modules/async/runjobs.lua:220]: in function 'cotask'
    [@programdir/core/base/scheduler.lua:404]:

stack traceback:
	[C]: in function 'error'
	@programdir/core/base/os.lua:949: in function 'base/os.raiselevel'
	(...tail calls...)
	@programdir/core/main.lua:3[14](https://github.com/traversebitree/test_xmake_shared_link_chain_with_add_package/actions/runs/7778666728/job/21208665428#step:5:15): in upvalue 'cotask'
	@programdir/core/base/scheduler.lua:404: in function <@programdir/core/base/scheduler.lua:397>
Error: Process completed with exit code [25](https://github.com/traversebitree/test_xmake_shared_link_chain_with_add_package/actions/runs/7778666728/job/21208665428#step:5:26)5.

Feb 05 '24 02:02 traversebitree

你 add_packages 只加在 dylib target 里，那么 fmt 的 link 和 linkdirs 只对这个 target 生效，main target 是不可见的，你要让 main link 时候，也能 link 到 fmt ，得设置为 public 对 main target 导出 linkdirs/links，或者干脆 main target 里面加 add_packages

target("mydylib")
    add_packages("fmt", {public = true})

Feb 05 '24 08:02 waruqi

Bot detected the issue body's language is not English, translate it automatically.

If you add add_packages only to the dylib target, then the link and linkdirs of fmt will only take effect for this target. The main target is invisible. If you want the main link to be able to link to fmt, you must set it to public to export linkdirs for the main target. /links, or simply add add_packages to the main target.

target("mydylib")
    add_packages("fmt", {public = true})

Feb 05 '24 08:02 Issues-translate-bot

你 add_packages 只加在 dylib target 里，那么 fmt 的 link 和 linkdirs 只对这个 target 生效，main target 是不可见的，你要让 main link 时候，也能 link 到 fmt ，得设置为 public 对 main target 导出 linkdirs/links，或者干脆 main target 里面加 add_packages
target("mydylib")
    add_packages("fmt", {public = true})

@waruqi 谢谢回复，但这样令package的public=true的操作会带来链接/权限传递问题。

比如例子中，我只想让 "mydylib" 依赖 "fmt" 库，而不想让我的 "main" 也依赖 "fmt" 库。如果我设置了 add_packages("fmt", {public = true})，那么不仅linkdir会被传递，连includedir也会被传递，那么我在 "main" 里就可以 #include <fmt/core.h> 了。

这样可能小概率产生一些链接/权限传递问题，如:

链接库重名问题，比如 "main" 本身链接了一个自己写的 "fmt.dll/so"，但同时又因为add_packages("fmt", {public = true})而链接了 "fmt" 库的 "fmt.dll/so"。
暴漏实现问题，比如我只想在 "mydylib" 里可以 #include <fmt/core.h> ，但我不想暴漏这个给 "main"。

Feb 06 '24 02:02 traversebitree

Bot detected the issue body's language is not English, translate it automatically.

If your add_packages are only added to the dylib target, then the link and linkdirs of fmt will only take effect for this target. The main target is invisible. If you want the main link to be able to link to fmt, you must set it to public to export the main target. linkdirs/links, or simply add add_packages to the main target

``lua target("mydylib") add_packages("fmt", {public = true})

@waruqi Thank you for your reply, but the operation of public=true in the package will cause link/permission transfer problems.

For example, in the example, I only want "mydylib" to depend on the "fmt" library, but I don't want my "main" to also depend on the "fmt" library. If I set add_packages("fmt", {public = true}), then not only the linkdir will be passed, but also the includedir will be passed, then I can #include <fmt/core. h>.

This may cause some link/permission transfer problems with a small probability, such as:

Link library duplication problem, for example, "main" itself is linked to a self-written "fmt.dll/so", but at the same time it is linked to "fmt" because of add_packages("fmt", {public = true}) Library "fmt.dll/so".
Expose implementation issues. For example, I only want to #include <fmt/core.h> in "mydylib", but I don't want to expose this to "main".

Feb 06 '24 02:02 Issues-translate-bot

@waruqi 这是一个CMake实现的示例与结果:

https://github.com/traversebitree/test_cmake_shared_link_chain_with_vcpkg https://github.com/traversebitree/test_cmake_shared_link_chain_with_vcpkg/actions/runs/7799832121

用vcpkg作为包管理器，可执行程序"main"只依赖"mydylib"; "mydylib"只依赖"fmt"。target_link_libraries全程都是PRIVATE的，在"main.cpp"是里无法#include <fmt/core.h>的

Feb 06 '24 13:02 traversebitree

试下这个，要想原生支持，目前暂时没时间处理，你可以暂时先这个搞下

target("mydylib") do
    set_kind("shared")
    add_files("dylibsrc/dylibsrc.cpp")
    add_includedirs("dylibsrc/", { public = true })
    add_headerfiles("dylibsrc/dylibsrc.hpp", { install = true })
    add_packages("fmt")
    on_load(function (target)
        for _, pkg in ipairs(target:orderpkgs()) do
            for _, linkdir in ipairs(pkg:get("linkdirs")) do
                target:add("ldflags", "-Wl,-rpath-link=" .. linkdir, {public = true, force = true})
            end
        end
    end)
end

ruki@0d35ffa77272:/tmp/test_xmake_shared_link_chain_with_add_package$ xmake -rv
checking for gcc ... /usr/bin/gcc
checking for zig ... no
checking for zig ... no
checking for unzip ... /usr/bin/unzip
checking for git ... /usr/bin/git
checking for gzip ... /usr/bin/gzip
checking for tar ... /usr/bin/tar
/usr/bin/git rev-parse HEAD
checking for cmake ... no
checking for cmake ... /usr/bin/cmake
checking for xmake::fmt ... fmt 10.2.1
checking for gcc ... /usr/bin/gcc
checking for the c++ compiler (cxx) ... gcc
checking for /usr/bin/gcc ... ok
checking for flags (-fPIC) ... ok
checking for flags (-O3) ... ok
checking for flags (-std=c++20) ... ok
checking for flags (-DNDEBUG) ... ok
[ 25%]: cache compiling.release dylibsrc/dylibsrc.cpp
/usr/bin/gcc -c -m64 -fPIC -O3 -std=c++20 -Idylibsrc -DFMT_LIB_EXPORT -isystem /home/ruki/.xmake/packages/f/fmt/10.2.1/ea2159e092e440a3866610426e31b244/include -DNDEBUG -o build/.objs/mydylib/linux/x86_64/release/dylibsrc/dylibsrc.cpp.o dylibsrc/dylibsrc.cpp
checking for flags (-MMD -MF) ... ok
checking for flags (-fdiagnostics-color=always) ... ok
checking for the c++ compiler (cxx) ... gcc
checking for flags (-fvisibility-inlines-hidden) ... ok
[ 25%]: cache compiling.release src/main.cpp
/usr/bin/gcc -c -m64 -fvisibility=hidden -fvisibility-inlines-hidden -O3 -std=c++20 -Idylibsrc -DNDEBUG -o build/.objs/main/linux/x86_64/release/src/main.cpp.o src/main.cpp
checking for g++ ... /usr/bin/g++
checking for the shared library linker (sh) ... g++
checking for /usr/bin/g++ ... ok
checking for flags (-fPIC) ... ok
[ 50%]: linking.release libmydylib.so
/usr/bin/g++ -o build/linux/x86_64/release/libmydylib.so build/.objs/mydylib/linux/x86_64/release/dylibsrc/dylibsrc.cpp.o -shared -m64 -fPIC -L/home/ruki/.xmake/packages/f/fmt/10.2.1/ea2159e092e440a3866610426e31b244/lib -s -lfmt
checking for g++ ... /usr/bin/g++
checking for the linker (ld) ... g++
checking for flags (-fPIC) ... ok
checking for flags (-Wl,-rpath=@loader_path) ... ok
[ 75%]: linking.release main
/usr/bin/g++ -o build/linux/x86_64/release/main build/.objs/main/linux/x86_64/release/src/main.cpp.o -m64 -Lbuild/linux/x86_64/release -Wl,-rpath=$ORIGIN -s -lmydylib -Wl,-rpath-link=/home/ruki/.xmake/packages/f/fmt/10.2.1/ea2159e092e440a3866610426e31b244/lib
[100%]: build ok, spent 1.281s
ruki@0d35ffa77272:/tmp/test_xmake_shared_link_chain_with_add_package$ xmake run
hello world!
Result of 3 + 4 is 7

Feb 07 '24 13:02 waruqi

@waruqi 我测试了下，能解决问题了，但是需要加上判断，判断平台是linux。这个flag应该是macos出问题。 https://github.com/traversebitree/test_xmake_shared_link_chain_with_add_package/actions/runs/7815925997/job/21320344680

加上判断就不报错了

target("mydylib")
do
	set_kind("shared")
	add_files("dylibsrc/dylibsrc.cpp")
	add_includedirs("dylibsrc/", { public = true })
	add_headerfiles("dylibsrc/dylibsrc.hpp", { install = true })
	add_packages("fmt", { public = false })
	on_load(function(target)
		if target:is_plat("linux") then  -- 加判断
			for _, pkg in ipairs(target:orderpkgs()) do
				for _, linkdir in ipairs(pkg:get("linkdirs")) do
					target:add("ldflags", "-Wl,-rpath-link=" .. linkdir, { public = true, force = true })
				end
			end
		end
	end)
end
target_end()

https://github.com/traversebitree/test_xmake_shared_link_chain_with_add_package/actions/runs/7816057398

Feb 07 '24 14:02 traversebitree

@waruqi 但这会不会造成 -rpath-link 污染？

Feb 07 '24 14:02 traversebitree

Bot detected the issue body's language is not English, translate it automatically.

@waruqi But will this cause -rpath-link pollution?

Feb 07 '24 14:02 Issues-translate-bot

@waruqi 但这会不会造成 -rpath-link 污染？

实测了一下，不会污染 😄

Feb 07 '24 14:02 traversebitree

conan vcpkg 这个问题都处理得很好。xmake 加油 main => a.so => b.so => c.so 这样的情况很常用

Jun 24 '24 07:06 19317362

Bot detected the issue body's language is not English, translate it automatically.

conan vcpkg handles this problem very well. Come on xmake Main => a.so => b.so => c.so This situation is very common

Jun 24 '24 08:06 Issues-translate-bot

Build error using shared link chain with add_package on Linux.

Xmake Version

Operating System Version and Architecture

Describe Bug

Expected Behavior

Project Configuration

Additional Information and Error Logs