Multi-stage Dockerfile throws Auth error when pushed to registry

[masonhuemmer]

Actual behavior Kaniko builds the image and throws error when it attempts to push to the registry.

error pushing image: failed to push to destination : POST https://index.docker.io/v2//blobs/uploads/: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:*** Type:repository] map[Action:push Class: Name:*** Type:repository]]

Expected behavior Kaniko should be able to push the image to the registry.

To Reproduce Steps to reproduce the behavior:

1. Use Multi-stage Dockerfile Additional Information

Dockerfile

ARG VARIANT="buster"
ARG BASE_IMAGE_REFERENCE
FROM mcr.microsoft.com/vscode/devcontainers/base:0-${VARIANT} as builder

RUN TRACER_VERSION=$(curl -s https://api.github.com/repos/DataDog/dd-trace-dotnet/releases/latest | grep tag_name | cut -d '"' -f 4 | cut -c2-) \
        && curl --output /tmp/datadog-dotnet-apm_amd64.deb -sL https://github.com/DataDog/dd-trace-dotnet/releases/download/v${TRACER_VERSION}/datadog-dotnet-apm_${TRACER_VERSION}_amd64.deb

RUN echo $BASE_IMAGE_REFERENCE

FROM ${BASE_IMAGE_REFERENCE}

COPY --from=builder /tmp/datadog-dotnet-apm_amd64.deb /tmp/datadog-dotnet-apm_amd64.deb

# Download and install the Tracer
RUN mkdir -p /opt/datadog \
    && mkdir -p /var/log/datadog \
    && dpkg -i /tmp/datadog-dotnet-apm_amd64.deb \
    && rm /tmp/datadog-dotnet-apm_amd64.deb

# Enable the tracer
ENV CORECLR_ENABLE_PROFILING=1
ENV CORECLR_PROFILER={846F5F1C-F9AE-4B07-969E-05C26BC060D8}
ENV CORECLR_PROFILER_PATH=/opt/datadog/Datadog.Trace.ClrProfiler.Native.so
ENV DD_DOTNET_TRACER_HOME=/opt/datadog
ENV DD_INTEGRATIONS=/opt/datadog/integrations.json

• Kaniko Image (fully qualified with digest): gcr.io/kaniko-project/executor:debug

[masonhuemmer]

Here is the script I run to generate my config.json.

      - CREDENTIALS=$(echo -n ${DOCKER_USERNAME}:${DOCKER_PASSWORD} | base64)
      - echo "{\"auths\":{\"https://index.docker.io/v1/\":{\"auth\":\"${CREDENTIALS}\"}}}" > /kaniko/.docker/config.json

[ResidentMario]

Docker Hub API V1 is deprecated. I'm not a Kaniko dev but I expect this tool doesn't support the old API anymore.

[ResidentMario]

Actually exactly this is explicitly addressed in https://github.com/GoogleContainerTools/kaniko#known-issues.

[masonhuemmer]

After I removed the multi-stage build in the Dockerfile, I was able to push to the repository. Does Kaniko not support multi-stage builds or does it have issues with multi-stage builds?

[ryanwuer]

same issue

[thetredev]

Update 2023-05-27: Even multi-stage builds work fine for me again. @masonhuemmer at least v1.9.2-debug does.

---

Update 2023-05-26: Nevermind. My issue was due to the new GitLab project setting Allow access to this project with a CI_JOB_TOKEN being activated. Deactivating the setting made everything working like before. My issue is not related to kaniko at all and can be ignored. Leaving this comment here for anybody who might stumble upon it as well, just in case.

---

Update 2023-05-25: Maybe I'm experiencing a different issue altogether, since it also won't work with only a single layer either. Need to investigate further...

---

same issue, my Dockerfile for building an odin-lang image, using kaniko v1.9.2-debug:

ARG BUILD_BASE_IMAGE
FROM ${BUILD_BASE_IMAGE} AS builder

ARG ODIN_VERSION

USER root
RUN git clone -b ${ODIN_VERSION} --single-branch https://github.com/odin-lang/Odin.git /tmp/odin-git \
    && cd /tmp/odin-git \
    && make


FROM ${BUILD_BASE_IMAGE}
USER container

# copy odin binary
COPY --from=builder /tmp/odin-git/odin /opt/odin/odin

# copy odin libraries
COPY --from=builder /tmp/odin-git/core /opt/odin/core
COPY --from=builder /tmp/odin-git/shared /opt/odin/shared
COPY --from=builder /tmp/odin-git/vendor /opt/odin/vendor

# add odin to PATH
ENV PATH="/opt/odin:${PATH}"

The error:

INFO[0000] Resolved base name <registry-repo>:<mytag> to builder 
INFO[0000] Retrieving image manifest <registry-repo>:<mytag>
INFO[0000] Retrieving image <registry-repo>:<mytag> from registry <registry>
error building image: GET https://<registry-repo>:<mytag>: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:container/devel Type:repository]]

BUILD_BASE_IMAGE 100% exists (I can pull it via docker pull locally on my machine) and kaniko 100% has access to the private registry I'm using, because it can pull other images from there via non multi-stage Dockerfile just fine.

Edit: Just tried this Dockerfile change in combination with --target release to no avail:

- FROM ${BUILD_BASE_IMAGE}
+ FROM ${BUILD_BASE_IMAGE} AS release
USER container

[aaron-prindle]

Can anyone in the thread here confirm if they are still seeing this issue using the latest version of Kaniko as @thetredev mentioned the issue might possibly be resolved there? Additionally as @ResidentMario mentioned, the Dockerhub V1 Registry API is not supported w/ kaniko although it was unclear if that was relevant to this issue or not from the above

[thetredev]

@aaron-prindle FYI: As far as I can tell I was always using Docker Registry V2 API.

[OrkhanAlikhanov]

I consistently experienced this with the latest kaniko when using multistage build. (I pull the version tagged with :v1.18.0 but version command returns Kaniko version : v1.17.0)

docker run --rm -it gcr.io/kaniko-project/executor:v1.18.0 version
Kaniko version :  v1.17.0

We use JFrog Artifactory and when we have multistage build it fails with:

failed to push to destination docker.company.com/2320461223/my-example:3465986c361d: POST https://docker.company.com/v2/2320461223/my-example/blobs/uploads/: UNAUTHORIZED: User is unauthorized to upload to dockerv2-local/2320461223/my-example/_uploads; map[]

When we remove multistage build it succeeds.

[dongzhiwei-git]

Has this bug not been solved so far？

[KateFiroozi]

I am facing the same issue in a simple course material. I have a docker token with all permissions granted (read, write, delete). I am able to login using CLI to docker.io with same token on my local machine and then build and upload an image. But I get 404 in Postman using same token for both GET and POST on image I just uploaded.

curl --location --request GET 'https://index.docker.io/v2/repo/dsodemo/blobs/uploads/' \ --header 'Content-Type: text/plain' \ --header 'Authorization: Basic <my_valid_token>'

The repository on docker.io is public.

Dockerfile:

FROM maven:latest

WORKDIR /app

COPY .  .

RUN mvn package -DskipTests && \
    mv target/demo-0.0.1-SNAPSHOT.jar /run/demo.jar

EXPOSE 8080

CMD java  -jar /run/demo.jar

Same issue with kaniko I use in Jenkins CI:

+ /kaniko/executor --force '--context=/home/jenkins/agent/workspace/dso-demo_main' '--dockerfile=/home/jenkins/agent/workspace/dso-demo_main/Dockerfile' --insecure --skip-tls-verify '--cache=true' '--destination=docker.io/repo/dsodemo'

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "docker.io/repo/dsodemo": POST https://index.docker.io/v2/repo/dsodemo/blobs/uploads/: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:repo/dsodemo Type:repository] map[Action:push Class: Name:repo/dsodemo Type:repository]]

script returned exit code 1

the secret mountet in kaniko container has correct data and identical to one used in postman:

k create secret -n ci docker-registry regcred --docker-server=https://index.docker.io/v2 --docker-username=repo --docker-password=<token> [email protected]

kaniko image: gcr.io/kaniko-project/executor:v1.14.0-debug

I have even tried it with MFA deactivated in docker.io.

[tbernacchi]

same here, even without using multi-step build stage