`build-std` doesn't work

[aaronmondal]

We want to use the -Zbuild-std cargo flag to rebuild the standard library sanitized. This practice is requiredy by various sanitizers to work properly, for instance ThreadSanitizer and MemorySanitizer.

Since build-std is a Cargo flag it cannot be passed via extra_rustc_flags. So I tried adding a Cargo config file like

[unstable]
build-std = ["core", "std", "alloc", "proc_macro"]

to the cargo_config in crates_repository. That doesn't seem to have any effect at all though. Bug?

AFAICS to get behavior like with build-std we'd have to use a override the default BUILD file for the standard library, but that doesn't seem like a nice solution considering that this should work with a flag in the cargo config file.

Maybe I'm overlooking something though and this is already supported somehow?

Original issue: https://github.com/allada/turbo-cache/issues/188#issuecomment-1637103453

[aaronmondal]

cc @allada

[scentini]

I wouldn't call it a bug, but rather an unsupported scenario. We don't support building the standard library from source out of the box, although it's possible to define a toolchain that does this. @daivinhtran is your proof of concept of defining a toolchain that builds the standard library from sources publicly visible?

[aran]

I'm trying to port a cargo build that uses -Z build-std=std,panic_abort for a rust_shared_library, to be used in arust_wasm_bindgen. It would be great to have an example toolchain that enables this.

[tidefield]

Inspired by https://rustc-dev-guide.rust-lang.org/building/bootstrapping.html, we can define rust_toolchain to build libstd from sources. However, such toolchain still needs the existence of prebuilt standard libraries; using prebuilt stdlibs to build libstd from sources.

To do so, we need a few components:

1. rust_stdlib_filegroup target for the prebuilt stdlibs. The prebuilt stdlibs are built by rust/x.py
2. base toolchain using prebuilt stdlibs from (1)
3. rust_library targets for sources
4. rust_stdlib_filegroup target to group all rust_librarys from (3)
5. final toolchain using stdlibs from (4). This toolchain is registered from WORKSPACE.

Looking at these components, there's a dependency cycle 5 -> 4 -> 3 -> 5 because (3) needs a rust_toolchain to build itself. To break the dependency cycle, we implement with_base_transtion rule to wrap the files from (4) and attach an incoming transition to set extra_toolchains to the base toolchain in (2). During toolchain resolution, base toolchain set by extra_toolchains command line option is preferred/selected to compile the stdlibs's sources - breaking the dependency cycle into 5 -> 4 -> 3 -> 2 -> 1.

@aran @aaronmondal let me know if you have any issue setting this up. I think building the prebuilt stdlibs (1) correctly is the main heavy-lifting to get this working.

[tidefield]

Actually, we can get (1) and (2) out of the box without additional work by using rules_rust's default toolchains. We can also set up a repo from downloaded library's source for (3). Let me see if I can create an example PR with this approach.

[tidefield]

@aaronmondal @aran

This thread has been a while. I finally got some time for this.

https://github.com/daivinhtran/rules_rust/tree/toolchain-to-rebuild-std/examples/toolchain-to-rebuild-std is an example how to bootstrap a rust toolchain and rebuild libstd. If you're using a target triple that's already in https://github.com/bazelbuild/rules_rust/blob/main/rust/platform/triple_mappings.bzl, it should be easy to follow the example.

https://github.com/daivinhtran/rules_rust/blob/toolchain-to-rebuild-std/examples/toolchain-to-rebuild-std/stdlibs.BUILD.bazel is the BUILD file for defining the standard library. You can add any custom configuration to the rust_library targets.