JuliaLang
macOS shared library fails during `init_julia`
Hi, I'm trying to use PackageCompiler to build a shared library which I can link into a relocatable plugin for a complex piece of C++ software, specifically, scsynth (https://supercollider.github.io/).
OS: macOS 10.14.6 (x86_64-apple-darwin18.7.0)
cpp: Apple LLVM version 10.0.1 (clang-1001.0.46.4)
Julia: 1.7.1 .app release / 1.7.1 debug built locally with git checkout v1.7.1; make debug
Currently my plugin fails during the init_julia call, which is in a class constructor not in main since this is a plugin.
int argc_ = 1;
const char* arg = "";
char** argv_ = const_cast<char**>(&arg);
init_julia(argc_, argv_);
here's the backtrace:
Executable module set to "/Applications/SuperCollider.app/Contents/Resources/scsynth".
Architecture set to: x86_64h-apple-macosx-.
(lldb) c
Process 54584 resuming
Process 54584 stopped
* thread #7, name = 'com.apple.audio.IOThread.client', stop reason = signal SIGABRT
frame #0: 0x00007fff7127e2c2 libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
-> 0x7fff7127e2c2 <+10>: jae 0x7fff7127e2cc ; <+20>
0x7fff7127e2c4 <+12>: movq %rax, %rdi
0x7fff7127e2c7 <+15>: jmp 0x7fff71278453 ; cerror_nocancel
0x7fff7127e2cc <+20>: retq
Target 0: (scsynth) stopped.
(lldb) thread backtrace
* thread #7, name = 'com.apple.audio.IOThread.client', stop reason = signal SIGABRT
* frame #0: 0x00007fff7127e2c2 libsystem_kernel.dylib`__pthread_kill + 10
frame #1: 0x00007fff71339bf1 libsystem_pthread.dylib`pthread_kill + 284
frame #2: 0x00007fff711e86a6 libsystem_c.dylib`abort + 127
frame #3: 0x00007fff6e3c1641 libc++abi.dylib`abort_message + 231
frame #4: 0x00007fff6e3c1704 libc++abi.dylib`default_terminate_handler() + 48
frame #5: 0x00007fff6e3cd19e libc++abi.dylib`std::__terminate(void (*)()) + 8
frame #6: 0x00007fff6e3cd225 libc++abi.dylib`std::terminate() + 69
frame #7: 0x00007fff6e372960 libc++.1.dylib`std::__1::thread::~thread() + 16
frame #8: 0x00007fff711e93cf libsystem_c.dylib`__cxa_finalize_ranges + 319
frame #9: 0x00007fff711e96b3 libsystem_c.dylib`exit + 55
frame #10: 0x000000010b19d64b libjulia-internal.1.7.dylib`jl_exit + 27
frame #11: 0x000000010b0d5b3d libjulia-internal.1.7.dylib`jl_vexceptionf + 285
frame #12: 0x000000010b0d5bd2 libjulia-internal.1.7.dylib`jl_errorf + 146
frame #13: 0x000000010764a44c libscjulia.0.1.0.dylib`init_julia + 300
frame #14: 0x0000000107296977 JuliaUGen_scsynth.scx`JuliaUGen::JuliaUGen::JuliaUGen(this=0x000000012734aad4) at JuliaUGen.cpp:16:9
frame #15: 0x0000000107296a85 JuliaUGen_scsynth.scx`JuliaUGen::JuliaUGen::JuliaUGen(this=0x000000012734aad4) at JuliaUGen.cpp:11:24
frame #16: 0x0000000107296d65 JuliaUGen_scsynth.scx`void detail::constructClass<JuliaUGen::JuliaUGen>(unit=0x000000012734aad4) at SC_PlugIn.hpp:242:98
frame #17: 0x0000000101565c4b scsynth`Graph_FirstCalc(inGraph=0x000000012734a580) at SC_Graph.cpp:440:9 [opt]
frame #18: 0x000000010156e5c7 scsynth`::Group_Calc(inGroup=<unavailable>) at SC_Group.cpp:69:9 [opt]
frame #19: 0x000000010156e5c7 scsynth`::Group_Calc(inGroup=<unavailable>) at SC_Group.cpp:69:9 [opt]
frame #20: 0x0000000101563efa scsynth`void SC_CoreAudioDriver::Run<true>(this=0x0000000167b47000, inInputData=0x00007fc8048be6b0, outOutputData=0x00007fc7fff05f50, oscTime=<unavailable>) at SC_CoreAudio.cpp:1370:13 [opt]
frame #21: 0x0000000101562881 scsynth`int appIOProc<true>(device=<unavailable>, inNow=<unavailable>, inInputData=0x00007fc805002700, inInputTime=<unavailable>, outOutputData=0x00007fc7fff05f50, inOutputTime=0x000000010c63d0e0, defptr=0x0000000167b47000) at SC_CoreAudio.cpp:0:10 [opt]
frame #22: 0x00007fff44c181b2 CoreAudio`HALC_ProxyIOContext::IOWorkLoop() + 4908
frame #23: 0x00007fff44c16cd4 CoreAudio`HALC_ProxyIOContext::IOThreadEntry(void*) + 122
frame #24: 0x00007fff44c16836 CoreAudio`HALB_IOThread::Entry(void*) + 72
frame #25: 0x00007fff713372eb libsystem_pthread.dylib`_pthread_body + 126
frame #26: 0x00007fff7133a249 libsystem_pthread.dylib`_pthread_start + 66
frame #27: 0x00007fff7133640d libsystem_pthread.dylib`thread_start + 13
looks like maybe somewhere in here: https://github.com/JuliaLang/PackageCompiler.jl/blob/v2.0.4/src/julia_init.c#L27-L38 ?
I'm not sure if this will turn out to be a PackageCompiler issue, scsynth issue or user error:
- the same
init_juliacode in a minimal C++ program (https://github.com/victor-shepardson/sc-julia/blob/master/plugins/JuliaUGen/test.cpp) works as expected, so the scsynth environment may be relevant. - when using a debug build of Julia 1.7.1 to get a better trace the error disappears, and it works as expected. it's even portable to another mac without Julia.
code is here: https://github.com/victor-shepardson/sc-julia
I notice that when using PackageCompiler with the 1.7.1 release build of Julia, libjulia.dylib and the entry point built by PackageCompiler (in my case libscjulia.dylib) end up in a lib directory, but everything else e.g. libjulia-internal.dylib are in lib/julia. Whereas with the debug build, everything is together in lib. However, I made sure to add both as rpaths when building my plugin:
$ otool -L build/JuliaUGen_scsynth.scx
build/JuliaUGen_scsynth.scx:
@rpath/libjulia.dylib (compatibility version 1.0.0, current version 1.7.1)
@rpath/libscjulia.0.1.0.dylib (compatibility version 0.0.0, current version 0.1.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.4)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
$ otool -l build/JuliaUGen_scsynth.scx | grep RPATH -A 2
cmd LC_RPATH
cmdsize 56
path @loader_path/../SCJulia/build/scjulia/lib/ (offset 12)
--
cmd LC_RPATH
cmdsize 64
path @loader_path/../SCJulia/build/scjulia/lib/julia/ (offset 12)
So I can't see how this difference would cause the failure.
I also notice that libscsynth uses @executable_path unlike my .scx plugin and the other Julia .dylibs:
$ otool -L SCJulia/build/scjulia/lib/libscjulia.dylib
SCJulia/build/scjulia/lib/libscjulia.dylib:
@rpath/libscjulia.0.1.0.dylib (compatibility version 0.0.0, current version 0.1.0)
@rpath/libjulia-debug.dylib (compatibility version 1.0.0, current version 1.7.1)
@rpath/libjulia-internal-debug.dylib (compatibility version 1.0.0, current version 1.7.1)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.4)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)
$ otool -l SCJulia/build/scjulia/lib/libscjulia.dylib | grep RPATH -A 2
cmd LC_RPATH
cmdsize 32
path @executable_path (offset 12)
--
cmd LC_RPATH
cmdsize 40
path @executable_path/julia (offset 12)
$ otool -l SCJulia/build/scjulia/lib/libjulia-internal-debug.dylib | grep RPATH -A 2
cmd LC_RPATH
cmdsize 32
path @loader_path/ (offset 12)
this is true for both debug and release Julia. AFAICT the dynamic linker wouldn't find libjulia-internal-debug.dylib using @executable_path, which would point to the location of scsynth which loads JuliaUGen_scsynth.scx which loads libscjulia.dylib which loads libjulia*.dylib.
If I use install_name_tool to change @executable_path to @loader_path here, nothing changes. debug still works and release still fails.
man dyld says:
The run path stack is built from the LC_RPATH load commands in the dependency chain that lead to the current dylib load.
so I think it's the use of @loader_path in JuliaUGen.scx which is actually enabling libscjulia to find further dependencies?
after more debugging, it definitely fails here: https://github.com/JuliaLang/PackageCompiler.jl/blob/ab5d6cf008b25a0868c61b21287d46ca2104ef05/src/julia_init.c#L27
where libname is "libscjulia.0.1.0.dylib", and was compiled into libscjulia itself here https://github.com/JuliaLang/PackageCompiler.jl/blob/94fdba2b99b7f0d3be9906de831c1ca6a0091488/src/PackageCompiler.jl#L583
if I start scsynth with DYLD_PRINT_RPATHS=1 DYLD_PRINT_LIBRARIES=1:
<initially loading all plugins>
...
dyld: loaded: /Users/victor/sc-julia/build/../SCJulia/build/scjulia/lib/libscjulia.dylib
RPATH successful expansion of @rpath/libscjulia.dylib to: /Users/victor/sc-julia/build/../SCJulia/build/scjulia/lib/libscjulia.dylib
...
<starting plugin, triggering init_julia call>
RPATH failed to expanding libscjulia.dylib to: /Users/victor/sc-julia/SCJulia/build/scjulia/lib/julia/libscjulia.dylib
ERROR: julia: Failed to load library at libscjulia.dylib
so the directory structure does seem to explain the debug/release difference:
when using PackageCompiler with the 1.7.1 release build of Julia, libjulia.dylib and the entry point built by PackageCompiler (in my case libscjulia.dylib) end up in a lib directory, but everything else e.g. libjulia-internal.dylib are in lib/julia. Whereas with the debug build, everything is together in lib.
I still don't understand why /Users/victor/sc-julia/SCJulia/build/scjulia/lib wouldn't be in the search path at this point, nor why it even tries to expand RPATH since there's no @rpath/ in the name passed to dlopen:
RPATH failed to expanding libscjulia.dylib to: /Users/victor/sc-julia/SCJulia/build/scjulia/lib/julia/libscjulia.dylib
^
there is no @rpath ???
like the locally built julia-debug 1.7.1, locally built 1.7.1 in release mode also works as expected and causes PackageCompiler to place everything under lib/ and not lib/julia.
this directory structure appears to be inherited from the Julia install. reflects <repo>/usr/lib in the locally built case, /Applications/Julia-1.7.app/Contents/Resources/julia/lib in the installed case.