GitHub Actions Cache doesn't work unless the image is pushed into the registry

[kponichtera]

In my project I have configured Docker containers, which bundle all the application dependencies and are used locally by the developers to keep their local machines reasonably clean. I wanted to integrate these images into CI/CD to ensure maximum repeatability and avoid repeating myself when defining certain commands.

I have a custom action, which builds the "builder" image, defined as follows:

    - name: Build builder
      uses: docker/build-push-action@v3
      with:
        file: Dockerfile
        context: '.'
        tags: my-builder-image
        load: true
        cache-from: type=gha,scope=${{ github.workflow }}
        cache-to: type=gha,mode=max,scope=${{ github.workflow }}

It creates the image in the scope of Buildx builder and then loads it into Github Action runner's Docker instance.

Then in my jobs I am using the built image to execute the command in the

    - uses: addnab/docker-run-action@v3
      with:
        image: my-builder-image
        options: -w /project -v ${{ github.workspace }}:/project
        run: pylint

The build-push-action seems to call the correct docker command, reads the GHA cache and pushes it after building all the layers:

/usr/bin/docker buildx build --cache-from type=gha,scope=Verify --cache-to type=gha,mode=max,scope=Verify ...

#7 importing cache manifest from gha:12854520869560610746
#7 DONE 0.1s

#15 exporting cache
#15 preparing build cache for export 0.0s done
#15 writing layer sha256:1d578c655d03d719eb075b5759fb1d9ee02065492a5d54db2fa1c390898c3366
#15 writing layer sha256:1d578c655d03d719eb075b5759fb1d9ee02065492a5d54db2fa1c390898c3366 1.8s done
#15 writing layer sha256:927062cd74cc90148418b13fa6614da6c2f9f6c8add3b38fb39ed933d513f76e
#15 writing layer sha256:927062cd74cc90148418b13fa6614da6c2f9f6c8add3b38fb39ed933d513f76e 0.1s done
#15 writing layer sha256:9e01ad9582c1ac5f39f74b04a0a415939a41f96c342cad821a034b1482aaf2ce 0.1s done
#15 writing layer sha256:a9a04eb6545e2a381d6c68e70000bebb363068182391046868fbb320fb8cba00
#15 writing layer sha256:a9a04eb6545e2a381d6c68e70000bebb363068182391046868fbb320fb8cba00 0.2s done
#15 writing layer sha256:f12926aa5844296bacf32e0eac8c15146f9d1dcbaeefbef4de8c9996c0577c47
#15 writing layer sha256:f12926aa5844296bacf32e0eac8c15146f9d1dcbaeefbef4de8c9996c0577c47 0.1s done
#15 DONE 2.8s

However, subsequent executions of the job on the same branch do not use this cache and rebuild all the layers

When I replace load: true with push: true and put the builder image into registry, the caching works as expected. However, it also pollutes my organization's package registry with unnecessary images.

I didn't find any mention on why the caching shouldn't work when pushing is not enabled - is it by design or is this a bug?

[JCMais]

~I am having the exact same issue, I noticed the gha cache hash changes between re-runs of the same job (no git changes between each re-run)~

Okay, my issue was actually different, in my case I was adding ARGs before the steps I wanted to cache, and as one of these was changing every build, it was invalidating the cache of all future steps. Moving the ARG closer to the place where it is actually used solved it for me.

[tai-nd]

I am having the same issue with an additional error message:

#5 importing cache manifest from gha
#5 ERROR: failed to configure registry cache importer: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

[tonistiigi]

@crazy-max Any idea what is being reported in here? Obviously push doesn't have anything to do with cache export (especially with mode=max, otherwise you would need to create image but not push it), unless that pushed image itself is used as cache source instead of gha.