ziglang
Compiling a program on termux aarch64 produces `FileNotFound` errors
Zig Version
0.11.0-dev.1012+4b5fc5239
Steps to Reproduce and Observed Behavior
Steps to reproduce
- Install Termux (not the google play version, as that's is deprecated and has security flaws) on an aarch64 Android phone
- Download the
masteraarch64 Linux tarball from ziglang.org and extract it - Run
zig init-exe && zig build run
Observed Behavior
info: Created build.zig
info: Created src/main.zig
info: Next, try `zig build --help` or `zig build run`
warning: Encountered error: FileNotFound, falling back to default ABI and dynamic linker.
warning: Encountered error: FileNotFound, falling back to default ABI and dynamic linker.
warning: Encountered error: FileNotFound, falling back to default ABI and dynamic linker.
warning: Encountered error: FileNotFound, falling back to default ABI and dynamic linker.
warning: Encountered error: FileNotFound, falling back to default ABI and dynamic linker.
warning: Encountered error: FileNotFound, falling back to default ABI and dynamic linker.
warning: Encountered error: FileNotFound, falling back to default ABI and dynamic linker.
warning: Encountered error: FileNotFound, falling back to default ABI and dynamic linker.
All your codebase are belong to us.
Run `zig build test` to run the tests.
Interestingly, the outputted program behaves as expected, although I'm not sure if more complex programs would break.
Expected Behavior
The program should compile with no errors and run properly.
The same eight FileNotFound errors appear with this code:
// main.zig
pub fn main() void {}
Running zig build-exe rather than zig build only produces the error once.
can you show the output of the following 2 commands, where my_name_exe is what zig build-exe produces
ldd /usr/bin/env
ldd my_main_exe
/usr/ on android with termux is located at /data/data/com.termux/files/usr/, so I did ldd /data/data/com.termux/usr/etc/env with that in mind
libandroid-support.so => /data/data/com.termux/files/usr/lib/libandroid-support.so
libgmp.so => /data/data/com.termux/files/usr/lib/libgmp.so
libiconv.so => /data/data/com.termux/files/usr/lib/libiconv.so
libc.so => /system/lib64/libc.so
ld-android.so => /system/lib64/ld-android.so
libdl.so => /system/lib64/libdl.so
ldd main doesn't give any output (main is the executable)
Hmm perhaps the zig executable you are using has a dynamic linker path that is not found on your host. Maybe ldd against zig can shed some light.
ldd on zig also doesn't give any output :/
I found the native ldd on my system it was in /system/bin/ (i had downloaded gnu ldd before and it was having some weird issues). The native ldd actually has some potentially useful results! Upon running it on either the zig executable it says error: "/data/data/com.termux/files/home/coding/tools/zig/zig" has unexpected e_type: 2
On Android, ldd is a shell script that runs either linker --list or linker64 --list depending on if the executable is 32 or 64 bit. I'm gonna look at the linker64 source code to try to find out what the e_type means.
e_type: 2 is part of an ELF header and marks something as an executable. I found this in the Android source code for the linker:
if (header_.e_type != ET_DYN) {
DL_ERR("\"%s\" has unexpected e_type: %d", name_.c_str(), header_.e_type);
return false;
}
So it seems like that error might be unrelated.
That looks like it is expecting a dynamic executable and iirc zig shipped in tarballs for linux are static for maximum platform compatibility. But yeah, unrelated.
zig cc potentially gave a hint!
$ zig cc main.c
warning: Encountered error: FileNotFound, falling back to default ABI and dynamic linker.
error: unable to create compilation: LibCRuntimeNotFound
I think that since Android has a non-standard filesystem layout it doesn't know where to look for libraries.
Things seem to work fine from within proot-distro, which emulates a standard filesystem (among other things), I think this is a point in favor of Android's filesystem being the source of the problem.
Edit: you can probably disregard this; I think I was conflating two different issues.
In src/libc_installation.zig the findNativeCrtDirPosix() function looks for crt1.o with the command
cc -print-file-name=crt1.o
but this fails on Termux. There may be a useful hint here that the filename to search for in this system is crtbegin_dynamic.o instead. Maybe?
I think this is a pretty easy fix. Termux root userspace is at /data/data/com.termux/files/, and usr is located there. Therefore the location for /usr/bin/env is actually /data/data/com.termux/files/usr/bin/env. If you replace /usr/bin/env with this location in zig/system/NativeTargetInfo.zig, the error goes away.
The solution then would be to detect if we're running on Termux and alter the path accordingly. It seems that $SHELL, $PREFIX, $HOME and $LD_PRELOAD are all prefixed with /data....
Hi @DivergentClouds ,
It seems that @JerwuQu has fixed the problem in PR #15896. I hope this patch will be accepted and then someone will create a Termux package for ziglang.
Regards.
Clarifying criteria for me to consider this issue "solved":
First, adjust the logic like this:
- Try checking
/usr/bin/envdirectly. - Next, try looking for
envin each directory inPATHin reverse order. This is a heuristic to try to find it faster, since non-standard directories are probably listed first.
Can a termux user confirm that this would find /data/data/com.termux/files/usr/bin/env?
Finally, send a patch to NixOS to remove this patch.
Then this issue can be closed.
Can confirm! PATH on Termux looks like this: /data/data/com.termux/files/home/.local/bin:/data/data/com.termux/files/usr/bin
First, adjust the logic like this:
This doesn't work with Nix:
In Nix shells, /usr/bin/env is present, but it may contain the wrong dynamic linker version.
The correct approach for Nix would be:
- Step 0: If env var
NIX_BINTOOLSis set, use the content of path$NIX_BINTOOLS/nix-support/dynamic-linker. This text file contains a store path like/nix/store/46m4xx889wlhsdj72j38fnlyyvvvvbyb-glibc-2.37-8/lib/ld-linux-x86-64.so.2.
try looking for
envin each directory inPATHin reverse order
This doesn't work with Nix
Couldn't this issue be resolved by looking for env in PATH in the correct/forward order and then instead checking /usr/bin/env directly last if finding through PATH failed?
Checking forward also makes more sense since that's the order e.g. linux searches for binaries in it and (unless my understanding is incorrect) would result in the same file as which env gives and thus avoid potential future confusion.
The downside is that the Zig compiler seems to call the function that resolves env multiple times for a compilation. For example, I get the FileNotFound warning message five times when running zig build after zig init. Considering this, we should probably cache either at the call-site or within the function to avoid iterating PATH multiple times, regardless of if we iterate from the front or back.
Some paths
Termux which env: /data/data/com.termux/files/usr/bin/env
Doesn't have /usr/bin in PATH, /data/data/com.termux/files/usr/bin is last.
/usr/bin/env doesn't exist (but /bin/env does).
NixOS which env: /run/current-system/sw/bin/env
Doesn't have /usr/bin in PATH, /run/current-system/sw/bin is last.
/usr/bin/env exists but (for me) symlinks to a different executable than /run/current-system/sw/bin/env. (cureutils-9.4 vs. coreutils-full-9.4)
My Arch install which env: /usr/bin/env
/usr/bin is roughly in the middle of PATH.
Regarding NIX_BINTOOLS, that does not work outside of a nix-shell, where the variable is not present.
But anyway the point of using /usr/bin/env is that it is portable because that is the path typically hardcoded directly into shebang lines, so all unix-likes are incentivized to put something in that path that is functional. You can see that even NixOS, the distro most likely to not conform, still does it, with the exception of sandboxed build environments.
I'm still sticking with the same PATH based solution outlined above. It's better not to directly target NixOS or Termux or any particular OS, but to pick some reasonable behavior that has a chance to work everywhere, and meet in the middle with these systems.
Like I said, non-standard directories are probably listed first. We don't want a user's cute "env plus extra features" implementation from their home directory to get priority because it's less likely to have the same dynamic linker as the base system installation.
For NixOS, /usr/bin/env continues to be the best default, and in sandboxed build environments, nix should be passing an explicit target dynamic linker override to the build system. I will make sure to make this clear while working on #14281 and communicate that to NixOS maintainers.
The only alternative I am considering to this is to change the fallback logic after checking hard-coded /usr/bin/env to checking the first executable file found by searching backwards in PATH. For example, on my system that would be /run/current-system/sw/bin/[ which has the same, correct dynamic linker as /usr/bin/env.
What I like about solutions like this is that the little guys have a chance to participate. Somebody's hobby OS project could look at Zig's logic and make reasonable adjustments to their packaging and build system and make zig work correctly out of the box for their system.
Regarding NIX_BINTOOLS, that does not work outside of a nix-shell, where the variable is not present.
It does work in general: When in a Nix build (shell or sandbox), use the explicitly pinned-down version via NIX_BINTOOLS, otherwise (i.e. when NIX_BINTOOLS is unset) use the system default (/usr/bin/env).
reasonable behavior that has a chance to work everywhere, and meet in the middle with these systems.
Using PATH with reverse search order doesn't work in Nix shells, where the derivation-specific coreutils (which we want to pick up) are prepended to the default PATH that includes the system default env binary.
But using the default search order would work.
Are you sure it won't work? If it's a sandbox that means it should be excluding directories in PATH that don't belong.
Looks like it works just fine to me, poking around at a nix-shell --pure -p which.
Yeah, it works with nix-shell --pure or nix develop --ignore-environment.
But typically these are invoked as nix-shell or nix develop, for the convenience of having your personal tools in PATH.
I think it's an UX pitfall if Zig were to produce slightly different results (i.e. different linker versions) depending on this pure flag.
Outside of package management context, literally matching /usr/bin/env is correct. That is your native dynamic linker path, for all intents and purposes. In a package management context, nix should be passing an explicit target dynamic linker override to the build system, to be offered as the "native" dynamic linker path. I mentioned this over in https://github.com/ziglang/zig/issues/14281 and I will be sure to help communicate that to NixOS maintainers and ensure that packaging goes smoothly with this regard, and the correct dynamic linker value will be used.
Like I said, non-standard directories are probably listed first. We don't want a user's cute "env plus extra features" implementation from their home directory to get priority because it's less likely to have the same dynamic linker as the base system installation.
I'm not convinced of this argument for iterating in reversed order when looking at my own PATH.
flatpak, jvm and perl binary directories are all placed after /usr/bin, and I'd assume there are users that place their custom bin paths at the end of PATH as well, though perhaps not if they actually want a custom env (that would likely only work in their own scripts anyways since it's not at /usr/bin/env...)
In my view, following the "correct" order and being able to quickly solve any issues that arise on non-standard platforms by asking "what does which env give?" is a benefit.
One could even argue scanning the full PATH and giving an error or warning if more than one env is found follows zig zen the best.
Ultimately it won't matter for the vast majority of users, so reverse order would also be fine, and I could pick up my PR again if the decision is firm.
Why not just try /usr/bin/env first and do a regular forwards scan of PATH as a fallback? Sounds like that would only cause minor slowdown in rare cases and seems way more fail-safe than adding platform-specific workarounds.
I have quite a lengthy PATH on my system and have never experienced any slowdowns anywhere as a result of it.
OK, I am convinced.
- Hard-coded check against
/usr/bin/env, followed by - Finding
envin PATH directories, in regular order
Note that #18778 introduces build option --host-dynamic-linker, so the dynamic linker can now explicitly be set.
This is super helpful, thanks for adding this!
Update: --host-dynamic-linker has been removed, use -Ddynamic-linker=... instead.
I saw it pointed out a couple times that this works fine from inside a nix build, but breaks from a nix develop/nix-shell, where /usr/bin/env exists but has the wrong linker path. Did we figured out a workaround / solution to that?
I tried passing -Ddynamic-linker=$(cat $NIX_BINTOOLS/nix-support/dynamic-linker), and that fixed the linker, but then all the other dynamic linking was broken
Linker correct, but everything else broken:
❯ zig build -Ddynamic-linker=$(cat $NIX_BINTOOLS/nix-support/dynamic-linker)
❯ ldd zig-out/bin/foo
linux-vdso.so.1 (0x00007ffe04110000)
libconfig.so.11 => not found
libSDL2-2.0.so.0 => not found
libSDL2_mixer-2.0.so.0 => not found
libOpenGL.so.0 => not found
libm.so.6 => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libm.so.6 (0x00007f4cba9e4000)
libc.so.6 => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6 (0x00007f4cba7f5000)
/nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/ld-linux-x86-64.so.2 => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib64/ld-linux-x86-64.so.2 (0x00007f4cbaac9000)
Everything correct except the linker:
The relevant line is
/nix/store/p3jshbwxiwifm1py0yq544fmdyy98j8a-glibc-2.38-27/lib/ld-linux-x86-64.so.2 => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib64/ld-linux-x86-64.so.2 (0x00007f5b60259000)
❯ zig build
❯ ldd zig-out/bin/foo
linux-vdso.so.1 (0x00007ffc57bd6000)
libconfig.so.11 => /nix/store/mf65q6nb0dwzi4kv5zyflc8lvbb5xwkk-libconfig-1.7.3/lib/libconfig.so.11 (0x00007f5b60247000)
libSDL2-2.0.so.0 => /nix/store/xay51i1rxl73plhl5h7jn3z9n3ckb5fr-SDL2-2.30.2/lib/libSDL2-2.0.so.0 (0x00007f5b5fe00000)
libSDL2_mixer-2.0.so.0 => /nix/store/isqh6rr69a13zc5zmpq4vmkxh9phw99h-SDL2_mixer-2.8.0/lib/libSDL2_mixer-2.0.so.0 (0x00007f5b601f7000)
libOpenGL.so.0 => /nix/store/wmm60qwckax108ffr8x1fm2xkzmm0vmh-libglvnd-1.7.0/lib/libOpenGL.so.0 (0x00007f5b601ca000)
libm.so.6 => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libm.so.6 (0x00007f5b600e7000)
libc.so.6 => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libc.so.6 (0x00007f5b5fc13000)
/nix/store/p3jshbwxiwifm1py0yq544fmdyy98j8a-glibc-2.38-27/lib/ld-linux-x86-64.so.2 => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib64/ld-linux-x86-64.so.2 (0x00007f5b60259000)
libX11.so.6 => /nix/store/lj440816z554v8d5l6ih1p2cnzlyihww-libX11-1.8.9/lib/libX11.so.6 (0x00007f5b5face000)
libXext.so.6 => /nix/store/4a27gp9n2l14p5wa7w3dwzihyz5dlydb-libXext-1.3.6/lib/libXext.so.6 (0x00007f5b600ce000)
libXcursor.so.1 => /nix/store/3rg0k4sdlgnh58inxnahyg7c13mnx5pg-libXcursor-1.2.2/lib/libXcursor.so.1 (0x00007f5b600c1000)
libXi.so.6 => /nix/store/6ai9w3b2kl2q87gqkh8zzbdrkxfmn7gr-libXi-1.8.1/lib/libXi.so.6 (0x00007f5b600ad000)
libXfixes.so.3 => /nix/store/509d3r5zldjdcxw1c2lpkxi8r0lpnlwy-libXfixes-6.0.1/lib/libXfixes.so.3 (0x00007f5b600a5000)
libXrandr.so.2 => /nix/store/4rlwdx9876d4b0irgfim4dag5lk62vy8-libXrandr-1.5.4/lib/libXrandr.so.2 (0x00007f5b60096000)
libXss.so.1 => /nix/store/bdg27lxq1hphfv9kc6l6qmmv76xizf6w-libXScrnSaver-1.2.4/lib/libXss.so.1 (0x00007f5b60091000)
libpthread.so.0 => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libpthread.so.0 (0x00007f5b6008c000)
libfluidsynth.so.3 => /nix/store/0yawlqcanasw45qlikgpz8wdy0wnsmkp-fluidsynth-2.3.5/lib/libfluidsynth.so.3 (0x00007f5b5f9f7000)
libopusfile.so.0 => /nix/store/kwzcbrs0mj581f7jv58ngpglw2rk4qzz-opusfile-0.12/lib/libopusfile.so.0 (0x00007f5b6007d000)
libGLdispatch.so.0 => /nix/store/wmm60qwckax108ffr8x1fm2xkzmm0vmh-libglvnd-1.7.0/lib/libGLdispatch.so.0 (0x00007f5b5f93e000)
libxcb.so.1 => /nix/store/0qypm1by15j6mq7zfszcwqzwry63a45k-libxcb-1.17.0/lib/libxcb.so.1 (0x00007f5b5f913000)
libXrender.so.1 => /nix/store/prs35snp05hmk1ia26k13hggqnwvvm22-libXrender-0.9.11/lib/libXrender.so.1 (0x00007f5b5f906000)
libgomp.so.1 => /nix/store/p3ffjixpnfgkqh20nsrc13vrj3yfi0nj-gcc-13.2.0-lib/lib/libgomp.so.1 (0x00007f5b5f8b9000)
libglib-2.0.so.0 => /nix/store/3xsbahrqqc4fc3gknmjj9j9687n4hiz0-glib-2.80.0/lib/libglib-2.0.so.0 (0x00007f5b5f76e000)
libgthread-2.0.so.0 => /nix/store/3xsbahrqqc4fc3gknmjj9j9687n4hiz0-glib-2.80.0/lib/libgthread-2.0.so.0 (0x00007f5b60074000)
libsndfile.so.1 => /nix/store/bdd4409zqdxdp82rq7p8z5yxdqjqb56c-libsndfile-1.2.2/lib/libsndfile.so.1 (0x00007f5b5f6e4000)
libpulse-simple.so.0 => /nix/store/qhg3n4ld2aw0fyr5a1bb03bk8yyfnn3c-libpulseaudio-17.0/lib/libpulse-simple.so.0 (0x00007f5b5f6dd000)
libpulse.so.0 => /nix/store/qhg3n4ld2aw0fyr5a1bb03bk8yyfnn3c-libpulseaudio-17.0/lib/libpulse.so.0 (0x00007f5b5f687000)
libasound.so.2 => /nix/store/ry8s556mdvgxb06s32xff3pany9zdb9l-alsa-lib-1.2.11/lib/libasound.so.2 (0x00007f5b5f577000)
libjack.so.0 => /nix/store/iz7svswnhxzfbybcziywx7bkqm859ay5-libjack2-1.9.22/lib/libjack.so.0 (0x00007f5b5f525000)
libstdc++.so.6 => /nix/store/p3ffjixpnfgkqh20nsrc13vrj3yfi0nj-gcc-13.2.0-lib/lib/libstdc++.so.6 (0x00007f5b5f200000)
libgcc_s.so.1 => /nix/store/p3ffjixpnfgkqh20nsrc13vrj3yfi0nj-gcc-13.2.0-lib/lib/libgcc_s.so.1 (0x00007f5b5f500000)
libogg.so.0 => /nix/store/gdkfgqbq27d88hbiqqjxpv68rljjnw15-libogg-1.3.5/lib/libogg.so.0 (0x00007f5b5f4f6000)
libopus.so.0 => /nix/store/6j95x0gf0w6dr5zh7nc4rwbmr5qxf2kk-libopus-1.5.2/lib/libopus.so.0 (0x00007f5b5f48f000)
libXau.so.6 => /nix/store/qdzsf0qr6686s4m76g6v602kbj96q7bn-libXau-1.0.11/lib/libXau.so.6 (0x00007f5b5f48a000)
libXdmcp.so.6 => /nix/store/qklman82zn30jc5dikx98rncsqq5p2mb-libXdmcp-1.1.5/lib/libXdmcp.so.6 (0x00007f5b5f482000)
libdl.so.2 => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libdl.so.2 (0x00007f5b5f47d000)
libpcre2-8.so.0 => /nix/store/sik7a8dic5lxhm4cp9npibk97x9jcxwf-pcre2-10.43/lib/libpcre2-8.so.0 (0x00007f5b5f160000)
libFLAC.so.12 => /nix/store/4n40harx8nqp2lsiqmfpn3w99si3f1ny-flac-1.4.3/lib/libFLAC.so.12 (0x00007f5b5f0e0000)
libvorbis.so.0 => /nix/store/5mp95b920k7djj6ip4ha4sgm9df9xx3g-libvorbis-1.3.7/lib/libvorbis.so.0 (0x00007f5b5f0a6000)
libvorbisenc.so.2 => /nix/store/5mp95b920k7djj6ip4ha4sgm9df9xx3g-libvorbis-1.3.7/lib/libvorbisenc.so.2 (0x00007f5b5effa000)
libmpg123.so.0 => /nix/store/61cdki944hc7p8mgry2s68zk649qmz9b-libmpg123-1.32.6/lib/libmpg123.so.0 (0x00007f5b5ef9d000)
libmp3lame.so.0 => /nix/store/nd5wjm6p4il5jcyb1nd1ljk3a0rfx95k-lame-3.100-lib/lib/libmp3lame.so.0 (0x00007f5b5ef25000)
libpulsecommon-17.0.so => /nix/store/qhg3n4ld2aw0fyr5a1bb03bk8yyfnn3c-libpulseaudio-17.0/lib/pulseaudio/libpulsecommon-17.0.so (0x00007f5b5ee9f000)
libdbus-1.so.3 => /nix/store/asgpvq2q0fl49w1c65fggpzqkkiw0ch3-dbus-1.14.10-lib/lib/libdbus-1.so.3 (0x00007f5b5ee48000)
librt.so.1 => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/librt.so.1 (0x00007f5b5f472000)
libcelt0.so.2 => /nix/store/qh847bhx9qxzrlvz7iqgfs0gd3qcs23j-celt-0.11.3/lib/libcelt0.so.2 (0x00007f5b5ee2e000)
libsamplerate.so.0 => /nix/store/07xnfg3wnrrmiv9l2wizr74z9ps02vw7-libsamplerate-0.2.2/lib/libsamplerate.so.0 (0x00007f5b5ecbf000)
libmvec.so.1 => /nix/store/apab5i73dqa09wx0q27b6fbhd1r18ihl-glibc-2.39-31/lib/libmvec.so.1 (0x00007f5b5ebc6000)
libsystemd.so.0 => /nix/store/ppf4k669sbm1kyggr7pbi34ycb62j6gp-systemd-minimal-255.4/lib/libsystemd.so.0 (0x00007f5b5eab7000)
libcap.so.2 => /nix/store/01xhvvrfkpfghp6r9iywahl9vd1ypgm0-libcap-2.69-lib/lib/libcap.so.2 (0x00007f5b5f464000)
EDIT: worth noting that in my case (and I suspect most nix dev shells), the version of env in the PATH, which I can find with which env, is linked with the right version of the linker.
Here's the problem:
https://github.com/ziglang/zig/blob/e2ec54bb38eb7b157667f0a87c9eb929017c3710/src/main.zig#L3710-L3712
It must be that when I pass -Ddynamic-linker=, we no longer have resolved_target.is_native_os
And indeed if I take the produced zig build-exe command, and append -feach-lib-rpath, it works. I don't see a way to pass that in to zig build though
I edited the file/data/data/com.termux/files/usr/lib/zig/lib/std/zig/system.zig (source) to replace /usr/bin/env by /data/data/com.termux/files/usr/bin/env but I still get the same error. Not sure what the fix is now for Zig 0.12.0.
I actually had this error when trying to edit Zig source files after installing zig.vim, I added silent! before the line throwing the error to hide it until this issue is resolved.
I want to add that the message disappears if you add -target aarch64-linux-musl flag. it fixes both building zig projects and also makes drop-in c compiler work on termux as well. when you execute zig cc -v, target is aarch64-unknown-linux-musl, so maybe that's why zig doesn't know how to look for libc
