ipetkov
Symlink cargo build artifacts instead of copying them
Motivation
Avoid copying big cargo build artifacts from the nix store, when using cargoArtifacts.
This works by just symlinking everything in target/release/build and target/release/deps. and uses rsync to copy other files. AFAIK the big files are all contained in these folders and are content-addressed by hash, so this should be safe.
What I have noticed (even before this change), is that cargo tarpaulin generally seems to rebuild the cargo artifacts, ignoring previous artifacts, even with --skip-clean, this fails as it tries to regenerate already built artifacts and tries to write into the build folders.
This should probably be investigated, so that cargo tarpaulin reuses the previously built/cached dependencies.
I have for now just removed --skip-clean in cargo tarpaulin, as it doesn't seem to have the intended effect anyway?
Without that flag, all tests run through.
Btw. I have noticed the (now removed) comment, but I have not experienced any issues other than the cargo tarpaulin above.
Checklist
- [ ] added tests to verify new behavior
- [ ] added an example template or updated an existing one
- [ ] updated
docs/API.md(or general documentation) with changes - [ ] updated
CHANGELOG.md
I gave it a spin, but realized it can't help me as I'm running with:
installCargoArtifactsMode = "use-zstd";
for reasons explained in #76.
I turned it off, but then discovered that after building dependencies Nix does bunch of shrinking ... on lots of files, and then it causes building the workspace to rebuild all the dependencies again. :/
I'm not sure if master has some error, or my setup is triggering some issues (works OK with use-zstd).
So I can't tell how much does it improve things.
It seems like the problem is not there in v0.12.1, so I rebased your commit on top of v0.12.1:
9ca63fb40f4ff761a6cbb7d5c07f5fe4c98782b9 (HEAD -> symlink-artifacts, origin/symlink-artifacts) Symlink cargo build artifacts instead of copying them
445a3d222947632b5593112bb817850e8a9cf737 (tag: v0.12.1) Update CHANGELOG
and again I'm seeing rebuilds of dependencies. So either your change is causing it too, or I messed up something somewhere.
No, I must've overlooked #76, thanks for mentioning.
I turned it off, but then discovered that after building dependencies Nix does bunch of shrinking ... on lots of files, and then it causes building the workspace to rebuild all the dependencies again. :/
Hmm weird, is maybe the host system relevant? It doesn't do that for me, it's shrinking the files, but that doesn't seem to matter and happens also on master. It only (re-)builds local crates in the repository (workspace) but reuses external dependencies. For my reference, I'm using rust-overlay with the current stable (1.69) on NixOS, maybe that's relevant?
Can you provide more info? Maybe try one of the examples, if it's working there for you (e.g. trunk)?
I'm using fenix toolchain. I reported https://github.com/ipetkov/crane/issues/323 , maybe it will shed some light.
Hi @Philipp-M thank you for the PR! When I originally explored symlinking all artifacts (instead of copying them deeply) cargo would fail with some errors about trying to write to a read-only file system (i.e. the nix store), but I hadn't considered that this might not apply to all artifacts, so kudos to spotting that!
What I have noticed (even before this change), is that cargo tarpaulin generally seems to rebuild the cargo artifacts, ignoring previous artifacts
I don't use cargo-tarpaulin much these days, but I do remember it insisting on cleaning the target directory (to recompile all deps with the right compiler flags IIRC). If it is ignoring --skip-clean it deserves some investigation, but it could be an unrelated issue
Noting @dpc 's comment on #323 (please correct me if I'm misunderstanding or misrepresenting anything)
Edit: Just checked and it seems that v0.12.1 behaves correctly. It also does the shrinking but when buildWorkspace starts it only builds local crates as expected.
Sounds like these changes (as they stand currently) are leading to rebuilds. My first guess is that cargo may be getting confused by looking at artifacts whose timestamp is older than the sources (sources in the Nix store have a timestamp in 1970) and it is resulting in rebuilding them. I wonder if the artifacts should be hardlinked (not symlinked) so that they get their own timestamp instead of inheriting the previous one...
@Philipp-M I pushed a small fix to resolve the build errors. Looks like a lot of the tests are passing but some are still failing (perhaps we shouldn't link build script output directories since scripts might need to be re-run and they expect to be able to write files? Alternatively perhaps we can take a shortcut by linking any *.rlib files or similar heuristics rather than guessing which directories are for build scripts and which ones aren't)
Thanks. Sorry haven't had time to investigate this further (since it's also a little bit hard to debug, as on my machines everything seems to be working). According to the tests only OSX has problems (though I guess this has to be tested thoroughly on different machines anyway I think). I guess to be sure, one has to deep dive into the source of how cargo etc. is deciding whether stuff has to be rebuilt on different machines. Leaving build script dirs out of the linking process, is probably not a bad idea:
When rebuilding a package, Cargo does not necessarily know if the build script needs to be run again. By default, it takes a conservative approach of always re-running the build script if any file within the package is changed ... The rerun-if-changed instruction tells Cargo to re-run the build script if the file at the given path has changed. Currently, Cargo only uses the filesystem last-modified “mtime” timestamp to determine if the file has changed. It compares against an internal cached timestamp of when the build script last ran.
If the path points to a directory, it will scan the entire directory for any modifications.
@dpc, can you provide more info, what exactly is going wrong (like hardware/OS, the exact issue)?
Just noticed #344 (thanks for further investigating btw.), I guess I can close this for now?
One thing I noticed while checking the deps folder, is that I think .so files can be safely symlinked as well.
@Philipp-M It's up to you! I decided to pick the low hanging fruit so we could make incremental updates.
Feel free to explore the .so linking optimization in this PR or a fresh one as you prefer
I tried the current master with #334 and just so you're aware: in a larger project, just using zstd is still waaay better than not using, even with current optimizations.
In our our project zstd produces: 828M for deps, and 1.5G for workspace build. no-zstd&symlinks: 4.6G and 4.1G.
@dpc fwiw this PR's symlinking only happens when inheriting existing artifacts to perform a build, but is entirely unrelated with how the resulting artifacts are organized in /nix/store