How to build go?

[testinfected]

Hi, I'm trying to use the example go toolchain, but it seems out of date.

Apparently it does not include some require values for GoToolChainInfo, namely compiler_flags_shared, compiler_flags_static, linker_flags_shared, linker_flags_static as well as prebuilt_stdlib and prebuilt_stdlib_shared.

I tried filling out those values, but don't know what to provide for prebuilt_stdlib. With an empty map I get the following error:

Error running analysis for root//... (prelude//platforms:default#524f8da68ea2a374)`

Caused by: Traceback (most recent call last): File , in * prelude/go/go_library.bzl:40, in go_library_impl static_pkg = compile( * prelude/go/compile.bzl:148, in compile cmd.add(cmd_args(_compile_cmd(ctx, pkg_name, pkgs, deps, compile_flags, share... * prelude/go/compile.bzl:110, in _compile_cmd stdlib_pkg_artifacts(go_toolchain, shared = shared), error: Type of parameters mismatch, expected int, actual provider_callable --> prelude/go/packages.bzl:56:19 | 56 | stdlib_pkgs = prebuilt_stdlib[ArtifactGroupInfo].artifacts | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | `

Is there an example of a working Go Toolchain somewhere?

Any help would be appreciated.

Thanks

[testinfected]

Should I understand nobody is using Buck2 to build go?

[ndmitchell]

We are using Buck2 internally to build Go. But we don't have any open source samples. That's kind of the weird middle ground - you can probably get it working, but it will require a bit of work. I'll see if anyone internally can be persuaded to shed some light (I know the build system, but not much about how Go works with it).

[podtserkovskiy]

Hey @testinfected,

I can share some context on buck2&Go integration, we are using it internally for limited purposes.

This is in active development at the moment. We will make some breaking changes time to time to make it better. So, at this moment we can't give any promise of API compatibility.

However I can answer some of your questions.

compiler_flags_shared, compiler_flags_static, linker_flags_shared, linker_flags_static

Just leave them empty for the beginning.

don't know what to provide for prebuilt_stdlib

You need to pre-build it manually using following command

export GODEBUG="installgoroot=all"
shared_flags="-gcflags -shared -asmflags -shared"
go install -pkgdir "$out_dir/prebuilt_std/static" std
go install -pkgdir "$out_dir/prebuilt_std/shared" $shared_flags std

Then wrap into into filegroups

    filegroup(
        name = "stdlib_static",
        srcs = native.glob(["prebuilt_std/static/**/*"]),
    )

    filegroup(
        name = "stdlib_shared",
        srcs = native.glob(["prebuilt_std/shared/**/*"]),
    )

And then supply it into GoToolChainInfo. I suppose you are using go_toolchain for it 😀

[sluongng]

That’s close to what we do in Bazel’s rules_go https://github.com/bazelbuild/rules_go/blob/2f29bd3956a37156f0dee39b761109a490f0a47b/go/tools/builders/stdlib.go#L115

Note that would would probably want to filter out go install generated buildid as it could make your build not reproducible.

[testinfected]

Thanks, will try that. I'm very new to Buck, so fingers crossed

[dvtkrlbs]

@podtserkovskiy hey what should be the param for gen_stdlib_importcfg should be. I am trying to make https://github.com/sluongng/buck2-go work with the current version of buck2. and currently getting an error like this

shell output

Running action: anon//:http_archive@f05825da4029d31f (http_archive) (build), local executor: env -- 'TMPDIR=/Users/dvtkrlbs/DEV/safarifruits/geomys/buck-out/v2/tmp-anon/anon/6c0c0ee7f897af38f05825da4029d31f/__http_archive__/http_archive' 'BUCK_SCRATCH_PATH=buck-out/v2/tmp-anon/anon/6c0c0ee7f897af38f05825da4029d31f/__http_archive__/http_archive' 'BUCK2_DAEMON_UUID=407d0906-13c3-4401-b87a-c2c64e02a357' 'BUCK_BUILD_ID=55f10eed-5e07-408a-934c-520e98d71512' /bin/sh buck-out/v2/gen-anon/anon/6c0c0ee7f897af38f05825da4029d31f/__http_archive__/unpack.sh
Running action: prelude//go/tools:stdlib (prelude//platforms:default#6c0c0ee7f897af38) (go_build_stdlib go_build_stdlib) (build), local executor: env -- 'TMPDIR=/Users/dvtkrlbs/DEV/safarifruits/geomys/buck-out/v2/tmp/prelude/6c0c0ee7f897af38/go/tools/__stdlib__/go_build_stdlib/go_build_stdlib' 'BUCK_SCRATCH_PATH=buck-out/v2/tmp/prelude/6c0c0ee7f897af38/go/tools/__stdlib__/go_build_stdlib/go_build_stdlib' 'BUCK2_DAEMON_UUID=407d0906-13c3-4401-b87a-c2c64e02a357' 'BUCK_BUILD_ID=55f10eed-5e07-408a-934c-520e98d71512' env 'GOEXPERIMENT=nocoverageredesign' 'GOARCH=arm64' 'GOOS=darwin' 'GOROOT=buck-out/v2/gen-anon/anon/6c0c0ee7f897af38f05825da4029d31f/__http_archive__/http_archive' 'CGO_ENABLED=1' 'GODEBUG=installgoroot=all' 'CGO_ENABLED=0' '' buck-out/v2/gen-anon/anon/6c0c0ee7f897af38f05825da4029d31f/__http_archive__/http_archive/bin/go install -pkgdir buck-out/v2/gen/prelude/6c0c0ee7f897af38/go/tools/__stdlib__/stdlib_pkgdir '-gcflags=-buildid=' std
Running action: root//:example (prelude//platforms:default#6c0c0ee7f897af38) (go_filter_srcs) (build), local executor: env -- 'TMPDIR=/Users/dvtkrlbs/DEV/safarifruits/geomys/buck-out/v2/tmp/root/6c0c0ee7f897af38/__example__/go_filter_srcs' 'BUCK_SCRATCH_PATH=buck-out/v2/tmp/root/6c0c0ee7f897af38/__example__/go_filter_srcs' 'BUCK2_DAEMON_UUID=407d0906-13c3-4401-b87a-c2c64e02a357' 'BUCK_BUILD_ID=55f10eed-5e07-408a-934c-520e98d71512' env 'GOEXPERIMENT=nocoverageredesign' 'GOARCH=arm64' 'GOOS=darwin' 'GOROOT=buck-out/v2/gen-anon/anon/6c0c0ee7f897af38f05825da4029d31f/__http_archive__/http_archive' 'CGO_ENABLED=1' /usr/bin/env 'PYTHONPATH=buck-out/v2/gen/prelude/6c0c0ee7f897af38/go/tools/__filter_srcs__/__filter_srcs__' python3 buck-out/v2/gen/prelude/6c0c0ee7f897af38/go/tools/__filter_srcs__/filter_srcs.py '--go=buck-out/v2/gen-anon/anon/6c0c0ee7f897af38f05825da4029d31f/__http_archive__/http_archive/bin/go' '--tags=' '--output=buck-out/v2/gen/root/6c0c0ee7f897af38/__example__/__filtered_srcs__.txt' buck-out/v2/gen/root/6c0c0ee7f897af38/__example__/__srcs__
Action failed: prelude//go/tools:stdlib (go_build_stdlib go_build_stdlib)
Local command returned non-zero exit code 127
Reproduce locally: `env -- 'BUCK_SCRATCH_PATH=buck-out/v2/tmp/prelude/6c0c0ee7f897af38/go/tools/__stdlib__/go_build_stdlib/go_build_stdlib' env 'GOEXPERIMENT=nocoverageredesign' 'GOARCH=arm64' 'GOOS=darwin' 'GOROOT=buck-out/v2/gen-anon/anon/6c0c0ee7f897af38f05825da4029d31f/__http_archive__/http_archive' 'CGO_ENABLED=1' 'GODEBUG=installgoroot=all' 'CGO_ENABLED=0' '' buck-out/v2/gen-anon/anon/6c0c0ee7f897af38f05825da4029d31f/__http_archive__/http_archive/bin/go install -pkgdir buck-out/v2/gen/prelude/6c0c0ee7f897af38/go/tools/__stdlib__/stdlib_pkgdir '-gcflags=-buildid=' std`
stdout:
stderr:
env: : No such file or directory
Build ID: 55f10eed-5e07-408a-934c-520e98d71512

https://github.com/dvtkrlbs/buck2-prelude here is my fork of the prelude

[dvtkrlbs]

nvm i figured it out. apparently some of the tools like go_wrapper and gen_stdlib_import_cfg were not included in the attrs declaration. defining them made the issue go away. Will open a PR about this. another question I have is if I open a PR with remote go toolchain downloading implemented would this be approved if not I will just contribute the simple fixes I did

[podtserkovskiy]

I'm sorry for the late reply. Thanks for dong this work.

I removed need to pre-compile stdlibs, so now it should be easier. I've just fixed some inconsistencies we had in GoToolchainInfo attributes and now #577 is ready for merge and will be merged shortly.

[podtserkovskiy]

if I open a PR with remote go toolchain downloading implemented would this be approved

Let's double check with @ndmitchell and @JakobDegen, but personally I'm happy to have it supported in buck

[JakobDegen]

I don't think we'd just approve it, I honestly think that more generally useful toolchain rules across languages is one of the highest value contributions buck2 could see right now. So yes, please do!

[podtserkovskiy]

BTW, I'm working on making system_go_toolchain work with go binary from PATH. Also, I'm adding some examples which will be compiled on CI and prevent breaking the system toolchain in the future.

I've just noticed we have some examples of downloading Go toolchain in no_prelude directory. You can use it together with system_go_toolchain for the inspiration 😀

[podtserkovskiy]

@dvtkrlbs @testinfected I've merged #588. So, system_go_toolchain works on Linux and MacOS, has an example of usage in examples/with_prelude/go/hello and those examples being validated on CI.

[testinfected]

Great news @podtserkovskiy ! Kudos

[dankox]

This issue is a bit old and the system_go_toolchain was published, but I'm curious how this can be used to do cross compilation of the Go binary.

I was trying to create toolchain similar to system_go_toolchain where I hardcoded env_go_os and env_go_arch, but I still need to provide stdlib which is compiled for that platform (it's using the default one which appears to be compiled always for the current platform, even though I thought it would pick it from toolchain definition) and I'm not sure how.

[podtserkovskiy]

This issue is a bit old and the system_go_toolchain was published, but I'm curious how this can be used to do cross compilation of the Go binary.

I was trying to create toolchain similar to system_go_toolchain where I hardcoded env_go_os and env_go_arch, but I still need to provide stdlib which is compiled for that platform (it's using the default one which appears to be compiled always for the current platform, even though I thought it would pick it from toolchain definition) and I'm not sure how.

@dankox setting env_go_os and env_go_arch should be enough for cross-compilation, there's no need to supply stdlib anymore, we build it on-demand. If you still see it in GoToolchainInfo please upgrade buck2/buck2-prelude.

If something doesn't work I'm here and happy to help 😀

[dankox]

@dankox setting env_go_os and env_go_arch should be enough for cross-compilation, there's no need to supply stdlib anymore, we build it on-demand. If you still see it in GoToolchainInfo please upgrade buck2/buck2-prelude.

If something doesn't work I'm here and happy to help 😀

@podtserkovskiy Hey, thanks for coming back to me for this issue:) I'm still having a problem and I have the latest prelude. I could configure the env_go_os and env_go_arch but the stdlib is causing troubles anyway.

I have pushed my setup here (it's really simple one file project where I was trying it first): https://github.com/dankox/colorx/tree/buck2-test

I'm not sure if I'm using it correctly, but it seems that the build of the main.go works fine, until it tries to import fmt package which was build for another platform. I thought it might be caused by me building first the binary for my platform and later trying to cross-compile, but even if I remove everything and go for other platform, I will get this error:

Connected to new buck2 daemon.
Action failed: root//:colorx_linux (go_compile main)
Local command returned non-zero exit code 1
Reproduce locally: `env -- 'BUCK_SCRATCH_PATH=buck-out/v2/tmp/root/ca7581441d28fa24/go_compile/main' 'GOARCH=amd64' 'GOE ...<omitted>... g -o buck-out/v2/gen/root/3a2dd161323a3ea9/__colorx_linux__/__action__4__/go_compile_out.a ./main.go (run `buck2 log what-failed` to get the full command)`
stdout:
./main.go:4:2: could not import fmt (object is [go object darwin amd64 go1.22.2 GOAMD64=v1 X:regabiwrappers,regabiargs,allocheaders,exectracer2
] expected [go object linux amd64 go1.22.2 GOAMD64=v1 X:regabiwrappers,regabiargs,allocheaders,exectracer2
])
stderr:
Build ID: 899d8947-949a-4091-a477-bb48bb370f2f
Jobs completed: 75. Time elapsed: 20.8s.
Cache hits: 0%. Commands: 6 (cached: 0, remote: 0, local: 6)
BUILD FAILED

[podtserkovskiy]

I'm sorry for missing the notification, thank you for creating the repro. I'll debug it shortly.

[podtserkovskiy]

@dankox I've done a PR https://github.com/dankox/colorx/pull/1/files I did a few changes:

• added platforms for different combinations os/arch
• added select() on toolchains//:go
• removed prelude submodule and added bundled prelude (that's the current recommended way)
• removed go_linux toolchain and colorx_linux binary

To do a cross-platform build you need to specify --target-platforms flag with a specific platform For example for darwin/amd64 your build command will look like this

buck2 build --target-platforms root//platforms:macos_x86_64 root//:colorx

Roughly target-platforms sets constraint_values , then select() on toolchains//:go maps this on go-specific target platform options.

You can't override _go_toolchain because other dependencies like go_stdlib will still use toolchains//:go

You must define your own toolchains//:go instead.

BTW, I'll add these selects to system_go_toolchain to make it cross-compile.

[dankox]

@podtserkovskiy Thanks a lot for that PR and the explanation. It really helps a lot. Now it makes much more sense to me, so thank you:)