Invalid cross-device link (os error 18) when building inside pods

[huguesBouvier]

Problem

Hello!

I developing my application on kubernetes. For development, I use minikube, I mount my host repo inside a pod and I build/run from inside it. However, I have the following issue when building: Compiling libc v0.2.118 error: failed to write /home/azureuser/..../target/debug/deps/libunicode_xid-8bb8457dab823b6f.rmeta: Invalid cross-device link (os error 18)

error: could not compile unicode-xid due to previous error.

Thanks!

Steps

Install and start minikube. mount any rust repo: minikube mount /repo:/repo

Create a deployment to mount the repo inside a pod. Do cargo build inside the pod

Possible Solution(s)

No response

Notes

No response

Rustup version

rustup 1.24.3 (ce5817a94 2021-05-31)

Installed toolchains

rustc 1.57.0 (f1edd0429 2021-11-29)

[rbtcollins]

I suspect that this isn't a rustup problem, sauce you are getting this error from the compiler.

What's the full path in the error?

[g00nix]

I can confirm the issue. I am running rustup inside a podman container. Code to the full setup can be found here.

~ # ❯❯❯ rustup update
info: syncing channel updates for 'stable-x86_64-unknown-linux-gnu'
info: latest update on 2022-02-24, rust version 1.59.0 (9d1b2106e 2022-02-23)
info: downloading component 'rust-std' for 'wasm32-unknown-unknown'
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
 53.8 MiB /  53.8 MiB (100 %)  30.8 MiB/s in  1s ETA:  0s
info: downloading component 'rustfmt'
info: removing previous version of component 'rust-std' for 'wasm32-unknown-unknown'
info: removing previous version of component 'cargo'
info: rolling back changes
error: could not rename component file from '/root/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/zsh/site-functions' to '/root/.rustup/tmp/4jpq5_2ziff3ygx8_dir/bk'
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
info: latest update on 2022-03-19, rust version 1.61.0-nightly (1bfe40d11 2022-03-18)
info: downloading component 'rust-analyzer-preview'
info: downloading component 'rust-src'
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
 55.2 MiB /  55.2 MiB (100 %)  25.9 MiB/s in  2s ETA:  0s
info: downloading component 'rustfmt'
info: removing previous version of component 'rust-analyzer-preview'
info: rolling back changes
error: could not rename component file from '/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/share/doc/rust-analyzer' to '/root/.rustup/tmp/b62wowa4hn89yplv_dir/bk'
info: checking for self-updates

   stable-x86_64-unknown-linux-gnu update failed - rustc 1.58.1 (db9d1b20b 2022-01-20)
  nightly-x86_64-unknown-linux-gnu update failed - rustc 1.61.0-nightly (3b348d932 2022-02-19)

info: cleaning up downloads & tmp directories
~ # ❯❯❯ rustup update nightly
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
info: latest update on 2022-03-19, rust version 1.61.0-nightly (1bfe40d11 2022-03-18)
info: downloading component 'rust-analyzer-preview'
info: downloading component 'rust-src'
info: downloading component 'cargo'
info: downloading component 'clippy'
info: downloading component 'rust-docs'
info: downloading component 'rust-std'
info: downloading component 'rustc'
 55.2 MiB /  55.2 MiB (100 %)  23.4 MiB/s in  2s ETA:  0s
info: downloading component 'rustfmt'
info: removing previous version of component 'rust-analyzer-preview'
info: rolling back changes
error: could not rename component file from '/root/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/share/doc/rust-analyzer' to '/root/.rustup/tmp/qb9tafd8b28fe6to_dir/bk': Invalid cross-device link (os error 18)
~ # ❯❯❯                   

The solution in my case was to delete the container and the image and to run the install automation again, forcing all OS and rust-related packages to install from scratch.

[huguesBouvier]

Thanks I'll try that!

[huguesBouvier]

My solution, which is not really one was to build on host with musl and mount only the executable....

[kinnison]

I don't recommend running rustup update or similar inside containers. Sadly container runtimes don't offer full filesystem semantics which results in things going bad. Instead ensure that you have rustup, and the correct toolchain, installed in a single layer.

[sajattack]

I'm struggling with this too. Not running rustup update, but even just trying to build a crate on a minikube mount. Should I re-log it on https://github.com/rust-lang/rust ?

[rbtcollins]

the behaviour of rustup and cargo/rustc in this area is independent. As long as you're not running a custom toolchain, if rustup succeeded installing the toolchain, then the problem is very unlikely to be rustup, and you should debug cargo/rustc

[clarfonthey]

This isn't just a container issue; the issue is that rustup is assuming that the file can simply be renamed in cases where it may actually be moving between filesystems. For example, I use different ZFS datasets for ~/.rustup/downloads, ~/.rustup/tmp, and ~/.rustup/toolchains (so I can backup toolchains, but ignore downloads/tmp) and I get this error:

error: could not rename component file from '/home/ltdk/.rustup/tmp/tz7f320ij__0ru3o_file' to '/home/ltdk/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/components'
error: could not rename components file from '/home/ltdk/.rustup/tmp/d_ded9_oq2y4lbm7_file' to '/home/ltdk/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/components': Invalid cross-device link (os error 18)

I'll try and take a look in the code to see if this can be fixed.

[Diggsey]

@huguesBouvier @sajattack Your issues (which are the one described in the issue body) are not related to rustup. The error is coming from rustc although the cause is likely due to your container's configuration.

@g00nix @clarfonthey Your issues are different, and are related to rustup. As you correctly point out they are due to the rustup directories existing on different filesystems. However, this is not a supported configuration - it's expected that everything within the .rustup directory resides on a single filesystem, because rustup expects to be able to rename files between the downloads/tmp/toolchains folders. Adding support for this may be possible, but may have performance and correctness implementations that should be considered before that.

(For that matter, backing up toolchains seems like a waste of space since they are immutable... Unless these are custom toolchains, in which case it might be better to have only the custom toolchains in the ZFS dataset?)

[clarfonthey]

Toolchains aren't immutable; updating a version of stable, for example, will overwrite the data in the toolchains folder. While they aren't super important from the perspective of being able to re-download them, so is most of my root filesystem, which I also back up and snapshot for the purpose of being able to debug problems when they happen. Sure, in the particular case of Rust, I don't have to do this, but I didn't question it until now because I had just assumed that things would be copy-removed if needed, like the behaviour of mv and others.