Recursive registry vendoring doesn't work

[ondt]

My crate foo has a dependency on crate bar from custom registry registry1, and bar is itself dependent on crate baz from registry2. registry1 and registry2 are in fact two different URLs for the same registry, one of them is sparse+http:// and the other one is git:// (but this doesn't matter in the general case).

Even if I configure Crane with both registries, it only vendors crates from registry1. Then it tries to access registry2 during cargo check in another derivation, which of course fails due to network access issues.

How hard do you think it would be to implement proper recursive vendoring?

Is it possible to temporarily work around this issue without modifying the crates in the custom registry?

---

I tried to put the following config into .cargo/config.toml:

# registry used to download `bar`
[registries.registry1]
index = "sparse+http://example.com/api/v1/crates/"

# the registry that `bar` wants `baz` from
[source.replacement2]
registry = "git://example.com/index"
replace-with = "registry1"

It fixed the issue of trying to access registry2 (git://example.com/index) from sandbox, but it was causing errors like this:

error[E0308]: mismatched types
   --> /nix/store/8krp2f7953dmzlg6ds096dafnhyh1yga-vendor-cargo-deps/059aab26819e2c754aab8ea711140537883f6c4c885fb914bc87eaf4194b5795/bar-0.2.3/src/watermelon.rs:454:21
    |
454 |                     redacted,
    |                     ^^^^^^^^ expected `FooBarType<u8>`, found a different `FooBarType<u8>`
    |
    = note: `FooBarType<u8>` and `FooBarType<u8>` have similar names, but are actually distinct types
note: `FooBarType<u8>` is defined in crate `baz`
   --> /nix/store/8krp2f7953dmzlg6ds096dafnhyh1yga-vendor-cargo-deps/059aab26819e2c754aab8ea711140537883f6c4c885fb914bc87eaf4194b5795/baz-0.1.1/src/banana.rs:32:1
    |
32  | pub struct FooBarType<T>(Box<[T]>);
    | ^^^^^^^^^^^^^^^^^^^^^^^^
note: `FooBarType<u8>` is defined in crate `baz`
   --> /nix/store/8krp2f7953dmzlg6ds096dafnhyh1yga-vendor-cargo-deps/059aab26819e2c754aab8ea711140537883f6c4c885fb914bc87eaf4194b5795/baz-0.1.1/src/banana.rs:32:1
    |
32  | pub struct FooBarType<T>(Box<[T]>);
    | ^^^^^^^^^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `baz` are being used?

and looking at the output of cargo check we see that it was indeed working with two identical copies of baz (from both sources, most likely because: foo -> bar(registry1) -> baz(registry2), foo -> baz(registry1), foo also depends on baz):

    Checking baz v0.1.1 (registry `replacement2`)
    Checking baz v0.1.1 (registry `registry1`)

Note that it works fine when running cargo check with the modified .cargo/config.toml outside of Crane:

    Checking baz v0.1.1 (registry `registry1`)

[ipetkov]

Hi @ondt thanks for the report! I unfortunately do not have access to multiple registries that I can test with so I may need to rely on you to help me debug this

The vendoring process has two steps:

1. Download all referenced packages and put them in the Nix store, then build a symlink farm for each registry/git repo
2. Append some configuration (in the equivalent of $HOME/.cargo/config.toml to avoid touching the project's files directly) which would instruct cargo to use the vendored directories instead of trying to fetch things directly

As long as the Cargo.lock is fully up to date, 1. should complete without issues (since we crawl the entire contents of Cargo.lock); IIRC cargo should be committing any source overrides in the Cargo.lock but in case it isn't we can dig into it further

As for 2. i wonder if mixing source replacements with registries (and source vendoring) is running into a precedence issue or something else. My first advice is to try running nix build .#whatever --keep-failed and then analyzing the left-over directory; e.g. do the configs look correct, are all the packages present, and so on

[ondt]

Hello @ipetkov, sorry for the delayed response.

I unfortunately do not have access to multiple registries

You can test it with just one registry.

All that is needed is the following setup:

• Binary foo that you're building with Crane
• Libraries bar and baz located on a registry sparse+http://your.registry.com/api/v1/crates/ (we're using Kellnr but it shouldn't matter)
  • bar is dependent on baz
  • foo is dependent on bar
• Registry URL configured using registryFromSparse, and also configured in foo's .cargo/config.toml

In this setup the build of foo should work. However, if you change the URL used to access the registry when building foo (in .cargo/config.toml) to sparse+http://your.registry.com:80/api/v1/crates/ (explicitly adding port 80 or 443 or using a different domain name alias), the build fails.

Now we've got two distinct URLs for the same registry, let's call them url1 and url2:

• url1: sparse+http://your.registry.com/api/v1/crates/
• url2: sparse+http://your.registry.com:80/api/v1/crates/

After changing the URL the situation is as follows:

• Binary foo requires bar through url2
• bar still requires baz through url1; url1 was baked into bar at the time of publishing

Now, even if we configure both url1 and url2 using registryFromSparse, then Crane still fails to run subsequent cargo commands, because cargo would be trying to access url1 from the sandbox, and I believe that's because baz wasn't vendored at all, or it was vendored improperly.

:warning: Note that after changing url1 to url2 in foo's .cargo/config.toml, the foo's Cargo.lock ONLY contains url2, but cargo still accesses url1 when downloading baz. That's why I believe Crane didn't know about url1, and therefore didn't vendor baz.

I believe this issue is caused in step 1 as you described it. As for step 2 and source replacements, I suggest not to focus on it right now, it's possible that it would fix itself if we manage to fix step 1. If not, then we should open a separate issue for it.

My initial description of the issue was very condensed and kind of all over the place, sorry about that!

---

By the way, the example with port 80 is just for simplicity. We actually encountered this issue while transitioning from git to sparse. ALL the libraries in the registry must be updated to use the new sparse URL. The Crane build breaks even if just one of them uses the old git URL.

This issue also applies to scenarios with two actually separate registries, but that is not our case. (However, in this case it can't be mitigated by just by publishing all the crates again with fixed URLs, projects like these can't be built at all using Crane.)

[ipetkov]

@ondt thanks for the detailed write up!

Is it possible to condense those instructions into a flake that I could check out and repro directly? Alexandrie is an open registry I've used for testing before, but it can be any registry that works!

[szlend]

I ran into the same issue. It works fine with a cargo build outside of Nix. But inside Nix the transient dependency's registry is never referenced in the crane generated config.toml, unless it's explicitly added to the app's .cargo/config.toml. Unfortunately I can't share our private registry setup, but I'll see if I can reproduce something with the public repos.

[szlend]

@ipetkov I prepared a repro case over at https://github.com/szlend/crane-reg-test

The cargo build in the devshell works fine, but nix build fails when the cargo check tries to fetch crates from the transitive custom registry.

       > ++ command cargo check --release --locked --all-targets
       >     Updating `https://github.com/szlend/alexandrie-index` index
       > warning: spurious network error (3 tries remaining): [35] SSL connect error (OpenSSL/3.0.14: error:16000069:STORE routines::unregistered scheme)
       > warning: spurious network error (2 tries remaining): [35] SSL connect error (OpenSSL/3.0.14: error:16000069:STORE routines::unregistered scheme)
       > warning: spurious network error (1 tries remaining): [35] SSL connect error (OpenSSL/3.0.14: error:16000069:STORE routines::unregistered scheme)
       > error: failed to download from `https://crates.polomack.eu/api/v1/crates/epitech_api/0.2.0/download`

crane-reg-test-deps> will append /private/tmp/nix-build-crane-reg-test-deps-0.1.0.drv-0/source/.cargo-home/config.toml with contents of /nix/store/4w5wg6zc4xzlwcqn5dwqcg7k5mb3hkxn-vendor-cargo-deps/config.toml

[source.nix-sources-8ffc52aa4d04993f2f8485e30b54e0763e4429374c2a679592e85187efa925b5]
directory = "/nix/store/4w5wg6zc4xzlwcqn5dwqcg7k5mb3hkxn-vendor-cargo-deps/8ffc52aa4d04993f2f8485e30b54e0763e4429374c2a679592e85187efa925b5"
[source.nix-sources-c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0]
directory = "/nix/store/4w5wg6zc4xzlwcqn5dwqcg7k5mb3hkxn-vendor-cargo-deps/c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0"
[source.nix-sources-f7e93db4c478e311b0c8ac8822989922129c8770f333062d10356eb5b8bdaf04]
directory = "/nix/store/4w5wg6zc4xzlwcqn5dwqcg7k5mb3hkxn-vendor-cargo-deps/f7e93db4c478e311b0c8ac8822989922129c8770f333062d10356eb5b8bdaf04"
[source.alexandrie]
registry = "https://github.com/Hirevo/alexandrie-index"
replace-with = "nix-sources-f7e93db4c478e311b0c8ac8822989922129c8770f333062d10356eb5b8bdaf04"
[source.crates-io]
registry = "https://github.com/rust-lang/crates.io-index"
replace-with = "nix-sources-c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0"

Note that there should be two registries in the dependency chain:

• https://github.com/Hirevo/alexandrie-index
• https://github.com/szlend/alexandrie-index

Only one of them gets put into crane's generated .cargo/config.toml. A workaround is to list all transitive registries in the root project that you're building. But this isn't necessary with plain cargo (without crane).

[szlend]

Another thing to note. Registries from transitive deps are unnamed when building:

   Compiling epitech_api v0.2.0 (registry `https://github.com/szlend/alexandrie-index`)   <---- unnamed
   Compiling crane-test-dep v0.2.0 (registry `alexandrie`)

[szlend]

My attempt at fixing this is in #676