ipetkov
bug: excluding `cargoArtifacts` overrides sources
Describe the bug
Ok, this is a weird one and took me quite a few hours to sus out. The problem is that if you do not include cargoArtifacts as part of a buildPackage, it gives you only a warning that it will not reuse artifacts. "Cool, whatever right, just build the thing", is what I thought, but then I got sent down a rabbit hole as to why my lib.fileset.toSource was somehow not what I had specified. I added a ton of traces to see all the way through my logic, and found no errors. Every step along the way the file I needed in my sources was found, but then why during the build was it not found? I thought, "ok what exactly did I do differently here that I never did anywhere else" and so I tried adding back in the cargoArtifacts and bam, it worked again. Somewhere during the unpackPhase, my guess is that because cargoArtifacts isn't present it just decides that the src it was given should be disregarded.
Reproduction
Here is my situation:
I wrote a cargo plugin which uses a config file called reaper.toml or .reaper.toml. I'm writing some checks using some test data (just some cargo projects that use the cargo plugin I wrote and have a reaper.toml in their directory) that I'm re-using the craneLib from my flake that I use to build the cargo plugin. So the whole project sort of looks like this:
Cargo.toml Cargo.lock src flake.nix flake.lock test_data
And in test data there are multiple cargo projects for testing which are not part of the workspace.
Here is the code I wrote that produced the error:
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
crane.url = "github:ipetkov/crane";
fenix = {
url = "github:nix-community/fenix";
inputs.nixpkgs.follows = "nixpkgs";
inputs.rust-analyzer-src.follows = "";
};
nix-core = {
url = "github:Cloud-Scythe-Labs/nix-core";
inputs.nixpkgs.follows = "nixpkgs";
inputs.fenix.follows = "fenix";
};
flake-utils.url = "github:numtide/flake-utils";
advisory-db = {
url = "github:rustsec/advisory-db";
flake = false;
};
};
outputs =
{ self
, nixpkgs
, crane
, fenix
, nix-core
, flake-utils
, advisory-db
, ...
}:
flake-utils.lib.eachDefaultSystem (system:
let
pkgs = import nixpkgs {
inherit system;
config.allowUnfree = true;
};
inherit (pkgs) lib;
rustToolchain = nix-core.toolchains.${system}.mkRustToolchainFromTOML
./.rust-toolchain.toml
"sha256-KUm16pHj+cRedf8vxs/Hd2YWxpOrWZ7UOrwhILdSJBU=";
craneLib = (crane.mkLib pkgs).overrideToolchain rustToolchain.fenix-pkgs;
src = craneLib.cleanCargoSource ./.;
# Common arguments can be set here to avoid repeating them later
commonArgs = {
inherit src;
strictDeps = true;
buildInputs = [
rustToolchain.darwin-pkgs
];
};
# Build *just* the cargo dependencies, so we can reuse
# all of that work (e.g. via cachix) when running in CI
cargoArtifacts = craneLib.buildDepsOnly commonArgs;
# Build the actual crate itself, reusing the dependency
# artifacts from above.
cargo-reaper-drv = craneLib.buildPackage (commonArgs // {
inherit cargoArtifacts;
doCheck = false;
});
in
{
checks =
let
buildReaperExtension = { package, plugin ? package, ... }@crateArgs:
craneLib.buildPackage (crateArgs // {
pname = package;
src = (builtins.trace "found cargo-reaper config in fileset ${crateArgs.src} with file type '${(builtins.readDir crateArgs.src).${"reaper.toml"}}'" crateArgs.src);
nativeBuildInputs = (crateArgs.nativeBuildInputs or []) ++ [
# Add `cargo-reaper` as a build time dependency of this derivation.
self.packages.${system}.default
];
# Run `cargo-reaper`, passing trailing args to the cargo invocation.
# We do not symlink the plugin since the `UserPlugins` directory is in
# the `$HOME` directory which is inaccessible to the sandbox.
buildPhaseCargoCommand = ''
ls . # somehow the source here is wrong even though up to this point everything is fine?
cargo reaper build --no-symlink \
-p ${package} --lib \
--release
'';
# Include extension plugin in the build result.
installPhaseCommand = ''
mkdir -p $out/lib
mv target/release/${plugin}.* $out/lib
'';
# Bypass crane checks for target install paths.
doNotPostBuildInstallCargoBinaries = true;
});
cargoReaperConfigFilter = from:
lib.fileset.fileFilter (file:
let
found = (lib.match "\.?reaper\.toml" file.name) != null;
in
builtins.trace (if found then "cargo-reaper config successfully located: ${from}" else "unable to locate cargo-reaper config file") found)
from;
testArgs = src: {
inherit src;
version = "0.1.0";
strictDeps = true;
};
in
{
# Build the crate as part of `nix flake check` for convenience
inherit cargo-reaper-drv;
test-cargo-reaper-build =
let
root = ./tests/test_data/package_manifest;
src = lib.fileset.toSource {
inherit root;
fileset = lib.fileset.unions [
(root + "/Cargo.toml")
(root + "/Cargo.lock")
(root + "/src")
(cargoReaperConfigFilter (root + "/reaper.toml"))
];
};
cargoReaperBuildArgs = testArgs (builtins.trace "found cargo-reaper config in fileset ${src} with file type '${(builtins.readDir src).${"reaper.toml"}}'" src);
in
buildReaperExtension (cargoReaperBuildArgs // {
package = "package_extension";
plugin = "reaper_package_ext";
});
};
packages = rec {
cargo-reaper = cargo-reaper-drv;
default = cargo-reaper;
};
});
}
And adding this fixes it:
test-cargo-reaper-build =
let
root = ./tests/test_data/package_manifest;
src = lib.fileset.toSource {
inherit root;
fileset = lib.fileset.unions [
(root + "/Cargo.toml")
(root + "/Cargo.lock")
(root + "/src")
(cargoReaperConfigFilter (root + "/reaper.toml"))
];
};
cargoReaperBuildArgs = testArgs (builtins.trace "found cargo-reaper config in fileset ${src} with file type '${(builtins.readDir src).${"reaper.toml"}}'" src);
+ cargoReaperCargoArtifacts = craneLib.buildDepsOnly cargoReaperBuildArgs;
in
buildReaperExtension (cargoReaperBuildArgs // {
+ cargoArtifacts = cargoReaperCargoArtifacts;
package = "package_extension";
plugin = "reaper_package_ext";
});
Through traces I was able to confirm that the source which my other traces verify is correct is not the source that is being unpacked when cargoArtifacts is not provided.
Hi @eureka-cpu thanks for the report! Could you please provide a self-contained (working) reproduction? Either a repo or a gist with all relevant files will work.
Unfortunately I can't get the flake above working (besides having to iron things out like rust-toolchains) especially since some of the inputs are not publicly reachable I cant quite fully test whether this is a configuration bug or whether a particular fix has solved the problem
Sure, happy to 🙂 Yeah, I just wanted to write a report of how I ran into it just in case you might have an idea of what it could be (or maybe I missed something in the docs requiring cargoArtifacts as part of the build process). I'll write up a minimal repro as soon as I have time.
I didn't see anything glaringly wrong in your configuration, but at the same time I couldn't spot anything in our own sources (generally things just pass down the original source through) hence why I was hoping to poke at a repo and start debugging things more closely