Cache `go install`-ed binaries

[SirSova]

Description: Cache go install-ed binaries (optionally I suppose) along with go mod dependencies. Store $GOBIN folder with all installed during workflow execution binaries on post step (cache save).

Justification: In my scenario, I use tparse tool to prettify tests results. I can imagine other cases such as code generator tools. Basically pre-run/post-run scripts. For now, I turned off the cache option of this action and wrote my own using action/cache, but it adds significant complexity to keep it around multiple workflows the same way.

Are you willing to submit a PR? Sure, as soon as the feature is approved.

[HarithaVattikuti]

Hello @SirSova We appreciate your suggestion for a new feature! We'll make sure to address it when we have the opportunity

[silverwind]

I don't know what GOBIN is, but GOMODCACHE (go env GOMODCACHE aka. $GOPATH/pkg/mod) is generally cachable and caching it would speed up any go run tool@version or go install tool@version invokations, so it would be welcome to include them in the caching.

[SirSova]

GOBIN represents $(go env GOPATH)/bin. It's an env for the folder with all go-installed binaries.

So if I run go install tool@version -- it won't add a new dependency in my go.mod (meaning it won't be cached), but inside my GH workflows I do this:

go install github.com/mfridman/[email protected]
go test -json  ./... | tparse -all

It will download and build tparse tool on each run which I want to avoid. And since these installations managed by Go, I believe that it's appropriate to do using setup-go action

[silverwind]

I have been experimenting using https://github.com/actions/cache and got some good performance results by caching GOCACHE (build cache) and GOMODCACHE (modules cache) but I see it as a bit of risky activity because it relies on golang correctly invalidating its cache and I'm not fully trusting it yet.

[StefMa]

Even through I would love to have this feature build in into this action, I honestly think it's not the responsibility of this action to cache such things.

This action is designed to install go, not more. What you're doing with go is not really part of this action. Is it? 🤔

[tak11173132]

I don't know what GOBIN is, but GOMODCACHE (go env GOMODCACHE aka. $GOPATH/pkg/mod) is generally cachable and caching it would speed up any go run tool@version or go install tool@version invokations, so it would be welcome to include them in the caching.

[silverwind]

Even through I would love to have this feature build in into this action, I honestly think it's not the responsibility of this action to cache such things.

This action is designed to install go, not more. What you're doing with go is not really part of this action. Is it? 🤔

I tend to agree that caching should not be in scope of setup-* actions (do one thing), but apparently these caching features have been creeping into them and setup-go is as far as I'm aware the only setup action that enables caching by default.

I think the most important thing is that only safe things should be cached and I don't know how safe it is to cache these go directories. There could always be undiscovered cache invalidation bugs in golang.

[nferch]

I too would appreciate this feature. Even if it just cached the dependencies for something that was go install'd, that'd speed up my builds quite a bit.

I'm a bit confused/ignorant as to why this isn't happening already. I have multiple workflows that run at the same time on the same commit, is it that the first run that completes doesn't contain the cached modules in GOMODCACHE?

[zaibon]

I have been experimenting using https://github.com/actions/cache and got some good performance results by caching GOCACHE (build cache) and GOMODCACHE (modules cache) but I see it as a bit of risky activity because it relies on golang correctly invalidating its cache and I'm not fully trusting it yet.

@silverwind can you share your solution in the mean time, while this issue is being decided/worked on ?

[silverwind]

Here is what I have been experimenting with and it seemed to work. The cache key surely is too aggressive and GOVERSION and go.mod hash can likely be removed.

- uses: actions/setup-go@v5
  with:
    go-version-file: go.mod
    check-latest: true
- id: vars
  run: |
    echo "GOCACHE=$(go env GOCACHE)" >> "$GITHUB_OUTPUT"
    echo "GOMODCACHE=$(go env GOMODCACHE)" >> "$GITHUB_OUTPUT"
    echo "GOVERSION=$(go env GOVERSION)" >> "$GITHUB_OUTPUT"
- uses: actions/cache/restore@v4
  with:
    path: |
      ${{ steps.vars.outputs.GOCACHE }}
      ${{ steps.vars.outputs.GOMODCACHE }}
    key: golint-v1-${{ github.job }}-${{ runner.os }}-${{ runner.arch }}-${{ steps.vars.outputs.GOVERSION }}-${{ hashFiles('go.mod') }}
- run: make lint
- uses: actions/cache/save@v4
  with:
    path: |
      ${{ steps.vars.outputs.GOCACHE }}
      ${{ steps.vars.outputs.GOMODCACHE }}
    key: golint-v1-${{ github.job }}-${{ runner.os }}-${{ runner.arch }}-${{ steps.vars.outputs.GOVERSION }}-${{ hashFiles('go.mod') }}

[nferch]

@silverwind thanks for sharing!

That seems to work for me, although the actions/cache/restore generates warnings when it tries to overwrite files that the actions/setup-go action restored from its cache.

I was able to achieve similar results by adding my Makefile to the cache-dependency-path value, which has two effects:

• It keys the cache using the Makefile, which includes versions of the tools that are installed, so the cache is repopulated when the versions change
• It forces a separate cache key from my main build workflow, which doesn't install or run the tools. This ensures that the cache used by this workflow contains the tools.

[peterbourgon]

Running

go install github.com/mfridman/[email protected]

will create a binary named tparse in $GOPATH/bin (if $GOPATH is set) or in $HOME/go/bin (if $GOPATH is not set). That binary will be tparse at version v0.14.0, but that version is not represented in the binary filename, or in anything else that can be reasonably captured by a cache key. So you can't really cache $GOPATH/bin (or $HOME/go/bin), at least not effectively.

[zaibon]

will create a binary named tparse in $GOPATH/bin (if $GOPATH is set) or in $HOME/go/bin (if $GOPATH is not set). That binary will be tparse at version v0.14.0, but that version is not represented in the binary filename, or in anything else that can be reasonably captured by a cache key. So you can't really cache $GOPATH/bin (or $HOME/go/bin), at least not effectively.

The trick there is to allow the user to define a cache key that is generated from the script that contain go install github.com/mfridman/[email protected]. This is where the version exists, so it can be used as cache key.

[SirSova]

I suppose Go install verify the version of the binary using --version, checksum or somewhere stored in go mod cache, but right now just by caching $GOPATH/bin it won't download & build the binary again.

My working workflow with cache binaries:

 - name: Set up Go 1.22
        uses: actions/setup-go@v5
        with:
          go-version: '1.22'
          cache: false # we use our own cache for go modules, since setup-go cache doesn't save `~/go/bin`
          
      - name: Check out source code
        uses: actions/checkout@v4

      - name: Cache go modules
        uses: actions/cache@v4
        with:
          # /go/bin is for `go install`-ed tools
          path: |
            ~/.cache/go-build
            ~/go/pkg/mod
            ~/go/bin
          key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
          restore-keys: |
            ${{ runner.os }}-go-

      - name: Run tests
        run: >
        go install github.com/mfridman/[email protected]
        go test -json  ./... | tparse -all

Also good to notice. tparse itself isn't a dependency of my code, so go.mod doesn't contain any information about it. I install it manually just before the tests

P.S: I want to replace Cache go modules part with some additional config for setup-go, such as:

        uses: actions/setup-go@v5
        with:
          go-version: '1.22'
          cache: true
          cache-install: true # <-----

[Zxilly]

go install downloads the code, compiles it, and then puts the binary in a directory. But there is no standard way of describing the version of the program being installed, and each time go install is executed it is a brand new installation, so how should the cache key be designed? Should the cache key be designed to keep track of each go install call? I can almost visualize a big pile of ugly workaround code already.

[SirSova]

The first run of go install does it, but it's 100% not a brand-new installation for the next calls. Just try it out. The 2nd+ calls are almost instantaneous. It must cache at least all dependencies. The workaround described above (GH workflow) worked for me perfectly.

P.S: I see also significant difference if I use "latest" vs specific version.

[silverwind]

But there is no standard way of describing the version of the program being installed

Since go 1.16, you can use go install module@version and go run module@version to specify the version.

[Zxilly]

yes, you can specify version, but after the install no way to get that. The second install call faster because go compiler cache the intermediate object files, but the final binary still been created duplicate.

[oxisto]

Just to give some additional workaround ideas: It is possible to create a separate folder, e.g., tools with its own go.mod that looks like this (just for example)

module tools

go 1.23.4

require (
	github.com/99designs/gqlgen v0.17.61
	github.com/mfridman/tparse v0.16.0
	github.com/sqlc-dev/sqlc v1.27.0
)

Put also an additional tools.go file

//go:build tools
// +build tools

package tools

import (
	_ "github.com/99designs/gqlgen"
	_ "github.com/mfridman/tparse"
	_ "github.com/sqlc-dev/sqlc/cmd/sqlc"
)

This has the advantage that the tool dependencies will not leak into your main go.mod. Then use the additional cache-dependency-path: "**/*.sum" in your setup-go action. This will also cache your build / tools dependencies.

I wonder whether some of this workflow will be made obsolete by the new tool directive in Go 1.24 (https://tip.golang.org/doc/go1.24#tools)

[kema-dev]

I'm using this kind of steps for an action, works like a charm. However, having a flag in setup-go to add caching for global packages would be nice (or better, integrating with go tool)

runs:
  using: composite
  steps:
    - name: Set up Go
      id: setup-go
      uses: actions/setup-go@v5
      with:
        go-version-file: go.mod
        cache-dependency-path: go.sum
    - name: Get tool go.mod for cache key
      id: get-tool-go-mod-for-cache-key
      shell: bash
      run: |
        declare tool_go_mod_path="${{ runner.temp }}/tool-go.mod"
        echo "Fetching tool go.mod for cache key, storing at ${tool_go_mod_path}"
        curl --silent https://raw.githubusercontent.com/owner/repo/refs/heads/main/go.mod > "${tool_go_mod_path}"
        echo "Tool go.mod successfully fetched, stored at ${tool_go_mod_path}"
    - name: Cache Go install
      id: cache-go-install
      uses: actions/cache@v4
      with:
        path: |
          ~/go/bin
        key: go-install-${{ runner.os }}-${{ hashFiles('${{ runner.temp }}/tool-go.mod') }}
    - name: Install tool
      id: install-tool
      shell: bash
      run: |
        go install github.com/owner/repo@latest

[Kristina-Pianykh]

I'm using this kind of steps for an action, works like a charm. However, having a flag in setup-go to add caching for global packages would be nice (or better, integrating with go tool)

runs: using: composite steps: - name: Set up Go id: setup-go uses: actions/setup-go@v5 with: go-version-file: go.mod cache-dependency-path: go.sum - name: Get tool go.mod for cache key id: get-tool-go-mod-for-cache-key shell: bash run: | declare tool_go_mod_path="${{ runner.temp }}/tool-go.mod" echo "Fetching tool go.mod for cache key, storing at ${tool_go_mod_path}" curl --silent https://raw.githubusercontent.com/owner/repo/refs/heads/main/go.mod > "${tool_go_mod_path}" echo "Tool go.mod successfully fetched, stored at ${tool_go_mod_path}" - name: Cache Go install id: cache-go-install uses: actions/cache@v4 with: path: | ~/go/bin key: go-install-${{ runner.os }}-${{ hashFiles('${{ runner.temp }}/tool-go.mod') }} - name: Install tool id: install-tool shell: bash run: | go install github.com/owner/repo@latest

I'd use github.workspace instead of runner.temp to save the go.mod file to. The reason is that runner.temp is a dynamic value and changes between runs

[Zxilly]

I've done some research, and for now, I think it's possible.

In go we have a command called go version, with a provided binary name we can identify the version of the binary, like

PS C:\Users\zxilly\go\bin> go version -m air.exe
air.exe: go1.20rc3
        path    github.com/cosmtrek/air
        mod     github.com/cosmtrek/air v1.41.0 h1:6ck2LbcVvby6cyuwE8ruia41U2nppMZGWOpq+E/EhoU=
        dep     github.com/fatih/color  v1.13.0 h1:8LOYc1KYPPmyKMuN8QV2DNRWNbLo6LZ0iLs8+mlH53w=
        dep     github.com/fsnotify/fsnotify    v1.6.0  h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4HY=
        dep     github.com/imdario/mergo        v0.3.13 h1:lFzP57bqS/wsqKssCGmtLAb8A0wKjLGrve2q3PPVcBk=
        dep     github.com/mattn/go-colorable   v0.1.9  h1:sqDoxXbdeALODt0DAeJCVp38ps9ZogZEAXjus69YV3U=
        dep     github.com/mattn/go-isatty      v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y=
        dep     github.com/pelletier/go-toml    v1.9.5  h1:4yBQzkHv+7BHq2PQUZF3Mx0IYxG7LsP222s7Agd3ve8=
        dep     golang.org/x/sys        v0.0.0-20220908164124-27713097b956      h1:XeJjHH1KiLpKGb6lvMiksZ9l0fVUh+AmGcm0nOMEBOY=
        build   -buildmode=exe
        build   -compiler=gc
        build   CGO_ENABLED=1
        build   CGO_CFLAGS=
        build   CGO_CPPFLAGS=
        build   CGO_CXXFLAGS=
        build   CGO_LDFLAGS=
        build   GOARCH=amd64
        build   GOOS=windows
        build   GOAMD64=v1

and this can be used as the key of the cache.

However, that we need a mechanism to pass which hit-cached binary has been restored in order to skip go install; otherwise, if go install is executed again, go will still compile and link the corresponding binary from scratch and overwrite the one we restored from the cache.

[peterbourgon]

go version -m mybinary is not guaranteed to include important details like runtime.BuildInfo, which would be necessary to produce a unique cache key for the artifact. Also, the output of go version isn't stable.

[Zxilly]

It works mostly, cache didn't need a 100% guarantee, if we want a stable format, wr can write a wrapper in go to call buildinfo.Read

[peterbourgon]

Cache keys definitely need a 100% guarantee that they uniquely identify a specific value. If two different binaries have the same cache key then builds become non-deterministic. And my point re: runtime.BuildInfo, and I guess transitively buildinfo.Read, is that this information is not reliably available from every binary. People can build artifacts in a way that elides this metadata entirely.

[Zxilly]

I think buildinfo is always available on binaries other than wasm, especially because of the runtime.modinfo symbol, which is available even on wasm.

Can you provide an example of a buildinfo that can be stripped during go install?

[peterbourgon]

go install -trimpath -buildvcs=false -ldflags="-s -w" ...

[Zxilly]

I don't think this affects it because buildinfo lookups are based on magic, not symbol tables, so -s -w doesn't affect this. I'm not at my computer right now, can you try to see if this affects it?

[Zxilly]

https://github.com/Zxilly/go-size-analyzer/blob/master/internal/wrapper/wasm.go#L92-L121

In fact there is a more stable way, see the code above.

[peterbourgon]

None of this discussion matters. go version output is neither stable nor deterministic over identical artifacts, it can't be the basis for a cache key.