3dem
Create statically linked binaries
We (@biochem-fan and @dkimanius) are considering distribution of statically linked and optimized binaries. Such binaries should not depend on libstdc++, glibc etc since some Linux distributions have a very old version.
For a simple C++ test program, the following worked.
$ g++ -static -o cpp_static_test cpp_static_test.cpp -static-libstdc++ -static-libgcc
$ ldd ./cpp_static_test
not a dynamic executable
#include <iostream>
#include <cmath>
#include <vector>
int main()
{
std::cout << "Hello, world." << std::endl;
std::vector<float> arr;
arr.push_back(9);
std::cout << "Sqrt(9) = " << sqrt(arr[0]) << std::endl;
}
However, doing this for RELION is not trivial; we have to make sure external dependencies (FFTW, FLTK etc) are also stand-alone.
Another point of discussion: The cpp_static_test above is 8.7 MB while a dynamically linked version is only 22 KB. Perhaps we do this only for performance critical programs (e.g. relion_refine_mpi)?
I also don't know how to do this with Intel compilers (@CharlesCongdon, @do-jason any suggestions?).
$ icpc -static -static-intel -o cpp_static_test_icc cpp_static_test.cpp
ld: cannot find -lstdc++
ld: cannot find -lm
ld: cannot find -lstdc++
ld: cannot find -lc
ld: cannot find -ldl
ld: cannot find -lc
$ icpc -static-intel -o cpp_static_test_icc cpp_static_test.cpp
icpc: warning #10237: -lcilkrts linked in dynamically, static library not available
$ ldd ./cpp_static_test_icc
linux-vdso.so.1 => (0x00007fff713c9000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f3b52d06000)
libm.so.6 => /lib64/libm.so.6 (0x00007f3b52a04000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f3b527ee000)
libc.so.6 => /lib64/libc.so.6 (0x00007f3b52420000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f3b5221c000)
/lib64/ld-linux-x86-64.so.2 (0x00007f3b5300e000)
I've tried some experiments, and it looks like the Linux version being used (and thus the gcc version) is most of the problem (just like #622). Which complicates things...
Experiment 1: RHEL 7.8 with stock gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-39)
$ g++ -static -o cpp_static_test cpp_static_test.cpp -static-libstdc++ -static-libgcc /usr/bin/ld: cannot find -lstdc++ /usr/bin/ld: cannot find -lm /usr/bin/ld: cannot find -lc collect2: error: ld returned 1 exit status $ source /opt/intel/compilers_and_libraries_2018.3.222/linux/bin/compilervars.sh intel64 $ icpc -static -o cpp_static_test cpp_static_test.cpp -static-libstdc++ -static-libgcc ld: cannot find -lstdc++ ld: cannot find -lm ld: cannot find -lstdc++ ld: cannot find -lc ld: cannot find -ldl ld: cannot find -lc $ icpc -static-intel -o cpp_static_test cpp_static_test.cpp -static-libstdc++ -static-libgcc icpc: warning #10237: -lcilkrts linked in dynamically, static library not available ld: cannot find -lstdc++
ld: cannot find -lstdc++
Experiment 2: RHEL 7.8 with gcc (GCC) 7.3.0 (installed confusingly in /opt/intel)
$ source /opt/intel/compilers_and_libraries_2018.3.222/linux/bin/compilervars.sh intel64 $ export PATH=/opt/intel/gcc7.3/bin:$PATH
$ export LD_LIBRARY_PATH=/opt/intel/gcc7.3/lib:$LD_LIBRARY_PATH
$ g++ -static -o cpp_static_test cpp_static_test.cpp -static-libstdc++ -static-libgcc
/usr/bin/ld: cannot find -lm /usr/bin/ld: cannot find -lc collect2: error: ld returned 1 exit status $ icpc -static -o cpp_static_test cpp_static_test.cpp -static-libstdc++ -static-libgcc ld: cannot find -lm ld: cannot find -lc ld: cannot find -ldl ld: cannot find -lc $ icpc -static-intel -o cpp_static_test cpp_static_test.cpp -static-libstdc++ -static-libgcc icpc: warning #10237: -lcilkrts linked in dynamically, static library not available
Experiment 3: Ubuntu 18.04.4 LTS with gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
$ g++ -static -o cpp_static_test cpp_static_test.cpp -static-libstdc++ -static-libgcc $ ldd cpp_static_test not a dynamic executable $ source /opt/intel/compilers_and_libraries_2018.3.222/linux/bin/compilervars.sh intel64 $ icpc -static -o cpp_static_test cpp_static_test.cpp -static-libstdc++ -static-libgcc $ ldd cpp_static_test not a dynamic executable $ icpc -static-intel -o cpp_static_test cpp_static_test.cpp -static-libstdc++ -static-libgcc icpc: warning #10237: -lcilkrts linked in dynamically, static library not available $ ldd cpp_static_test linux-vdso.so.1 (0x00007fff8b97f000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fe95a43d000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe95a04c000) /lib64/ld-linux-x86-64.so.2 (0x00007fe95a7db000)
Experiment 4: Ubuntu 20.04 LTS with gcc (Ubuntu 9.3.0-10ubuntu2) 9.3.0
$ g++ -static -o cpp_static_test cpp_static_test.cpp -static-libstdc++ -static-libgcc $ source /opt/intel/compilers_and_libraries_2018.3.222/linux/bin/compilervars.sh intel64 $ icpc -static -o cpp_static_test cpp_static_test.cpp -static-libstdc++ -static-libgcc In file included from cpp_static_test.cpp(1): /usr/include/c++/9/iostream(38): catastrophic error: cannot open source file "bits/c++config.h" #include <bits/c++config.h> ^ $ <playing with various -L options, I'm unable to get this to build. So let's try a more recent compiler> $ source /opt/intel/compilers_and_libraries_2020.2.254/linux/bin/compilervars.sh intel64 $ icpc -static -o cpp_static_test cpp_static_test.cpp -static-libstdc++ -static-libgcc cwcongdo@ortce-dg1g:~/RELION$ ldd cpp_static_test not a dynamic executable $ icpc -static-intel -o cpp_static_test cpp_static_test.cpp -static-libstdc++ -static-libgcc icpc: warning #10237: -lcilkrts linked in dynamically, static library not available
So, GCC version seems to matter, OS seems to matter (ubuntu likes to put stuff in /usr/lib/x86_64-linux-gnu/ - the paths should be set properly, but they may not), and which version of the Intel compiler which can handle this seems to matter. Not pretty. As we can see, some OS versions do not contain static versions of 64-bit libraries, and some compilers, such as the Intel compiler, don't contain static versions for all libraries.
This stuff is maddening...
My best known flags for mostly static linking of Intel compiler is "-mkl -static-intel -no-intel-extensions -qopenmp-link=static -static-libgcc -static-libstdc++ -Wl,--as-needed [lots of -lLIBNAME...]".
There is no TBB static library provided by Intel which is needed for CPU acceleration. It needs special build from open source to have static TBB library. All other intel compiler parts have static libraries. But there will be other issues in Linux depending on distro. Some distro does not provide static libgcc and static stdc++ library. And there may be problems in MPI linking if OFED or other interconnect library does not provide static library.
If all the required libraries have static library, then adding "-static -L/path/to/static_libraries -lLIBNAMES" will work even with icpc.
More experiments and talk with our compiler guys found the -no-cilk flag. Unfortunately, that just makes the one error go away, and on Ubuntu we still end up with a non-static binary:
$ icpc -static-intel -o cpp_static_test cpp_static_test.cpp -static-libstdc++ -static-libgcc -no-cilk $ ldd cpp_static_test linux-vdso.so.1 (0x00007ffcedf61000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa35304a000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa352c59000) /lib64/ld-linux-x86-64.so.2 (0x00007fa3533e8000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa352a55000)
Ah, some of this is an admin issue. On my RHEL 7.8 machine, if I install the glibc-static and libstdc++-static packages with yum it builds cleanly and I get a static binary (I'm not even sure installing libstdc++-static is needed). These errors went away:
ld: cannot find -lstdc++ ld: cannot find -lm ld: cannot find -lstdc++ ld: cannot find -lc ld: cannot find -ldl ld: cannot find -lc
So the key seems to be to make sure your build system has the static libraries installed, which is something optional when building a machine.
@CharlesCongdon @do-jason Thank you very much for information. @dkimanius's looking at this. If this turns out too complex, we might build within conda.
So the key seems to be to make sure your build system has the static libraries installed, which is something optional when building a machine.
Thanks for looking into this @CharlesCongdon. Was this observation on GCC, ICC or both? GCC doesn't seem to complain if a particular static library is missing. It just ignores your request.
@dkimanius Dari - where these messages are concerned, it seems to be the presence of static system libraries (e.g. GCC) that matters the most (libm, libstdc++, libld, etc.). On one machine, I forget if before or after the output I show above, I had to load the static libraries using yum/apt to make errors go away (this may be something a clever CMake coder could check). It also appears that some versions of both ICC and GCC will silently produce a non-static binary if something is missing. Most frustrating to script!