Pushing images to dockerhub stopped working

[gebi]

Actual behavior

Kaniko exits with exit code 1 with the following message and and does not build the image:

error checking push permissions -- make sure you entered the correct tag name,
and that you are authenticated correctly, and try again: checking push permission for
"foo/bar:bionic-99": UNAUTHORIZED: authentication required; [map[Action:pull Class:
Name:mgit/clamav Type:repository] map[Action:push Class: Name:foo/bar Type:repository]]

This worked with the same build pipeline and no changes 3 months ago with the following image:

Using Docker executor with image gcr.io/kaniko-project/executor:debug ...
Pulling docker image gcr.io/kaniko-project/executor:debug ...
Using docker image sha256:2aa254b4837c242c7de87956438eaba70f97a2768ab0870819fd20e09df15cf6 for gcr.io/kaniko-project/executor:debug ...

Expected behavior

Kaniko to upload image to dockerhub like the version 3 months ago was able to. There where no changes, and it works if i go back to an older kaniko version.

To Reproduce Steps to reproduce the behavior:

1. ... with the following pseudo .gitlab-ci.yml

image:
  name: gcr.io/kaniko-project/executor:debug
  entrypoint: [""]

stages:
  - foo

build-foo:
  stage: foo
  script:
    - echo "{\"auths\":{\"index.docker.io\":{\"auth\":\"${CI_DOCKERHUB_AUTH}\"}}}" > /kaniko/.docker/config.json
    - >
      /kaniko/executor --context "${CI_PROJECT_DIR}/foo" --dockerfile "${CI_PROJECT_DIR}/foo/Dockerfile"
      --destination foo/bar:blub-${CI_PIPELINE_IID}"
      --destination foo/bar:blub"

2. ... build it

Additional Information

• Dockerfile Please provide either the Dockerfile you're trying to build or one that can reproduce this error.
• Build Context Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands)
• Kaniko Image (fully qualified with digest)

Using Docker executor with image gcr.io/kaniko-project/executor:debug ...
Pulling docker image gcr.io/kaniko-project/executor:debug ...
Using docker image sha256:2ec307dcf7f52dcf700ea0fbc65d448f46365cfac69567e8177bf12b80942f54 for gcr.io/kaniko-project/executor:debug ...

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [x]
Please check if this error is seen when you use --cache flag	- [ ]
Please check if your dockerfile is a multistage dockerfile	- [ ]

[gebi]

Ah, sidenote, i confirmed that the credentials still work for pushing images to dockerhub, and they work fine, both manually and as mentioned with the old kaniko version.

For now we have pinned the kaniko version to this one mgit/base:kaniko-executor-debug-stable (which also fixes the problem of kaniko unable to build images on bigger FS because of the included busybox on bigger filesystems due to 64bit inodes and the included busybox not being compiled with large file support #656 )

[giovanism]

Is this duplicate of https://github.com/GoogleContainerTools/kaniko/issues/245?

[gebi]

I don't think so, as in 245 they mention multistage builds and long build time. We have just a short build time, no multistage builds and it worked with the version of kaniko from a few months ago perfectly.

[Forst]

Encountering this issue as well. The last working version seems to be v0.16.0.

Could #957 be causing the issue perhaps?

[gebi]

it seems #1005 describes the same problem with kaniko

[gebi]

i can verify that for us too the lastest working kaniko version is v0.16.0

v0.20.0 is not able to build the image, with the following job output:

$ mkdir -p /kaniko/.docker
$ echo "{\"auths\":{\"index.docker.io\":{\"auth\":\"${DOCKERHUB_AUTH}\"}}}" > /kaniko/.docker/config.json
$ mkdir /docker-tmp
$ echo 'FROM debian:stable' >> /docker-tmp/dockerfile
$ echo 'ENTRYPOINT ["/bin/bash", "-c", "echo hello"]' >> /docker-tmp/dockerfile
$ /kaniko/executor --context /docker-tmp --dockerfile /docker-tmp/dockerfile --destination foo/bar:hello-world-latest
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "foo/bar:hello-world-latest": POST https://index.docker.io/v2/foo/bar/blobs/uploads/: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:mgit/base Type:repository] map[Action:push Class: Name:mgit/base Type:repository]]Running after_script
Uploading artifacts for failed job
ERROR: Job failed: exit code 1
FATAL: exit code 1

[diegosasw]

@gebi thanks, I confirm with the mgit/base:kaniko-executor-debug-stable image I was able to push to DockerHub but with gcr.io/kaniko-project/executor:debug I was experiencing the same error. Thanks!

This is my script

- export BASE64_CREDENTIALS="$(echo -n $DOCKER_HUB_REPOSITORY_USERNAME:$DOCKER_HUB_REPOSITORY_PASSWORD | base64)"
# @see https://github.com/GoogleContainerTools/kaniko/issues/1227 where folder may not exist
- mkdir -p /kaniko/.docker
- echo "{\"auths\":{\"https://index.docker.io/v2/\":{\"auth\":\"$BASE64_CREDENTIALS\"}}}" > /kaniko/.docker/config.json
- /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/$IMAGE_TAG/Dockerfile --destination $IMAGE_NAME:$IMAGE_TAG

Used variables to make it more meaningful for newbies (like myself 1 hour ago)

[macrotex]

This is also happening to us using the lastest version of "debug" (May 6, 2020) attempting to push to GCR. Seeing this error:

E0506 17:32:28.676141      16 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Changing to tag debug-v0.19.0 things start working again.

[tejal29]

@macrotex Can you please use kaniko v0.22.0 https://github.com/GoogleContainerTools/kaniko/releases/tag/v0.22.0 and let us know if it still exists.

[macrotex]

This is also happening to us using the lastest version of "debug" (May 6, 2020) attempting to push to GCR. Seeing this error:

E0506 17:32:28.676141      16 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

Changing to tag debug-v0.19.0 things start working again.

Version 0.22.0 fixed my issue.

[ymage]

I tried to use https://index.docker.io/v1/ instead of v2 one as docker registry url, which seemed to work for me. Is v2 endpoint really suported by kaniko ?

[GTB3NW]

Hey I still get the aws_credentials error:

Using Kubernetes executor with image gcr.io/kaniko-project/executor:debug-v0.22.0 ...

 E0515 00:08:48.365446      15 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors

[anna-codefresh]

I too am getting the same error with v0.22.0. I tested this and the latest version to work is 0.16.0 as @gebi mentioned.

[ymage]

This seems to work for docker hub :

$ export DOCKERHUB_AUTH="$(echo -n $DOCKER_HUB_REPOSITORY_USERNAME:$DOCKER_HUB_REPOSITORY_PASSWORD | base64)"
$ echo "{\"auths\":{\"https://index.docker.io/v1/\":{\"auth\":\"${DOCKERHUB_AUTH}\"}}}" > docker.json
$ docker run --rm -v $(pwd):/workspace -v $(pwd)/docker.json:/kaniko/.docker/config.json:ro gcr.io/kaniko-project/executor:v0.22.0 --context=dir:///workspace --dockerfile=Dockerfile --destination=foo/bar:latest

[anna-codefresh]

thanks @ymage. I was using the v2 docker endpoint instead of v1.

[vistaarjuneja]

+1. I am able to upload docker images with https://index.docker.io/v1 but not https://index.docker.io/v2 with the latest kaniko debug executor image. Is anyone working on this issue?

[anjeyy]

And there i was sitting the last sunday half a day thinking to be that stupid to build a simple image which i wanted to push to my private docker hub.

auths: [https://index.docker.io/v2/]

did NOT work for me.

• debug (without version)
• debug-v0.22.0
• debug-v0.19.0

did work for me.

• debug-v0.16.0

---

auths: [https://index.docker.io/v1/]

did work for me.

• debug (without version)
• debug-v0.22.0
• debug-v0.19.0
• debug-v0.16.0

---

What combination should i use, since i have no idea what the difference it's making?

• v1
• v2

[livelace]

works for me:

https://index.docker.io/v1/ + debug-v0.16.0

UP. It seems the real reason was UserAgent in config.json:

"HttpHeaders": {
  "User-Agent": "Docker-Client/19.03.12 (linux)"
}

After removing this section I haven't have any problems with pushing (even with original debug-539ddefcae3fd6b411a95982a830d987f4214251)

[tanguydelignieresaccenture]

After being struggled all the day with the issue, trying to push to dockerhub with a previous version of kaniko, debug-v0.18.0, which was fine few months ago in the same context, as @gebi, I was able to push the image using --registry-mirror index.docker.io.

@tejal29 may it be related to dockerhub hostname or default image path that has maybe changed and is no more compatible with kaniko (in old versions at least)?

I'm using auths with https://index.docker.io/v1/ in config.json.

KO:

$ docker run --rm --entrypoint "" -v /host/path/to/kaniko/config.json:/kaniko/.docker/config.json -v /host/path/to/dockerfile/directory/kaniko/20200825-001/build1:/workspace gcr.io/kaniko-project/executor:debug-v0.18.0 /kaniko/executor --context /workspace --dockerfile /workspace/Dockerfile --destination index.docker.io/tanguydelignieres/kaniko_bugs_20200825-001_build1:debug-v0.18.0
INFO[0003] Resolved base name alpine:3.9 to alpine:3.9
INFO[0003] Resolved base name alpine:3.9 to alpine:3.9
INFO[0003] Retrieving image manifest alpine:3.9
INFO[0005] Retrieving image manifest alpine:3.9
INFO[0009] Built cross stage deps: map[]
INFO[0009] Retrieving image manifest alpine:3.9
INFO[0011] Retrieving image manifest alpine:3.9
INFO[0014] Skipping unpacking as no commands require it.
INFO[0014] Taking snapshot of full filesystem...
INFO[0014] Resolving paths
INFO[0014] CMD echo "OK"
error pushing image: failed to push to destination index.docker.io/tanguydelignieres/kaniko_bugs_20200825-001_build1:debug-v0.18.0: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:tanguydelignieres/kaniko_bugs_20200825-001_build1 Type:repository] map[Action:push Class: Name:tanguydelignieres/kaniko_bugs_20200825-001_build1 Type:repository] map[Action:pull Class: Name:library/alpine Type:repository]]

$

OK with --registry-mirror index.docker.io:

$ docker run --rm --entrypoint "" -v /host/path/to/kaniko/config.json:/kaniko/.docker/config.json -v /host/path/to/dockerfile/directory/kaniko/20200825-001/build1:/workspace gcr.io/kaniko-project/executor:debug-v0.18.0 /kaniko/executor --registry-mirror index.docker.io --context /workspace --dockerfile /workspace/Dockerfile --destination index.docker.io/tanguydelignieres/kaniko_bugs_20200925-001_build1:debug-v0.18.0
INFO[0002] Resolved base name alpine:3.9 to alpine:3.9
INFO[0002] Resolved base name alpine:3.9 to alpine:3.9
INFO[0002] Retrieving image manifest alpine:3.9
INFO[0004] Retrieving image manifest alpine:3.9
INFO[0007] Built cross stage deps: map[]
INFO[0007] Retrieving image manifest alpine:3.9
INFO[0008] Retrieving image manifest alpine:3.9
INFO[0010] Skipping unpacking as no commands require it.
INFO[0010] Taking snapshot of full filesystem...
INFO[0010] Resolving paths
INFO[0010] CMD echo "OK"

$

I confirm I did not face the problem with debug-v1.0.0.

[nlamirault]

I try with Kaniko v1.0.0 on Docker Hub v2 endpoint, it fails. It works using v1.

[tejal29]

Thank you folks, I updated the docs to use v1 endpoint instead of v2

[tejal29]

@tanguydelignieresaccenture i am still trying to understand why adding --registry-mirror works for you.

[prune998]

@nlamirault how did you changed the v1/v2 endpoint for reaching dockerhub ?

I don't understand why it's so complicated to have this basic feature working... (I'm just starting with Kaniko...)

[prune998]

OK the doc is https://github.com/GoogleContainerTools/kaniko#pushing-to-docker-hub

Use a config.json file for docker as:

{
	"auths": {
		"https://index.docker.io/v1/": {
			"auth": "xxxxxxxxxxxxxxx"
		}
	}
}

[shubhag]

In the known issues section in readme, it is mentioned that Kanika does not support v1 api https://github.com/GoogleContainerTools/kaniko#known-issues. These are contradictory items in the document.

[olliekrk]

Does kaniko actually supports docker registry v2 API? I have private registry with v2 api exposed and cannot pull images from there by their tags (e.g. latest / stable / specific x.x.x version). Pull directly from docker command line works perfectly. Kaniko can only pull if I provide the exact image digest. Your docs at README.md says that v1 is not supported - so how to make this pull work with v2 API?

[anjeyy]

@olliekrk

Please see the comments above. As stated in https://github.com/GoogleContainerTools/kaniko/issues/1209#issuecomment-672159106 debug-v0.16.0 worked for me with v2.

[jgericke]

@olliekrk

Please see the comments above. As stated in #1209 (comment) debug-v0.16.0 worked for me with v2.

I can confirm I hit this with debug (as per below), switching to debug-v0.16.0 solved DockerHub auth

Pushing to: docker.io/redacted/image-builder:
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "docker.io/redacted/image-builder:": POST https://index.docker.io/v2/redacted/image-builder/blobs/uploads/: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:redacted/image-builder Type:repository] map[Action:push Class: Name:redacted/image-builder Type:repository]]

[mth-cbc]

We are using Harbor as a self hosted container registry, but I cannot get the authentification to work in the kaniko build. The error is the same as with the official Docker registry:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "hub.domain.com/repo/kaniko:f569d627": POST https://hub.domain.com/v2/repo/kaniko/blobs/uploads/: UNAUTHORIZED: unauthorized to access repository: repo/kaniko, action: push: unauthorized to access repository: repo/kaniko, action: push

I also tried the suggested workaround resulting in config.json like this:

{
        "auths": {
                "hub.domain.com/v1": {
                        "auth": "xxxxxxxxxxxx"
                }
}

But the logs/errors stayed the same, i.e. Kaniko still used the https://hub.domain.com/v2/ endpoint. Any hints how I could adapt the workaround to work with Harbor as well?

[qafro1]

I'm having the same issue