nerdctl push cannot push image which pulled from a private docker registry

[Gitkingly]

1. nerdctl pushed an image which had downloaded from docker hub, to a local Harbor repository, it succeed as expected
2. nerdctl pushed an image which had downloaded from local docker registry, to the same Harbor repository, it failed with "400 Bad Request"

nerdctl version 2.0.4

sudo nerdctl -n k8s.io images, I have below images: busybox:latest 192.168.140.16:5000/pause:3.10

The success image: sudo nerdctl -n k8s.io tag busybox:latest 192.168.140.16/library/busybox-16:latest-1 sudo nerdctl -n k8s.io push 192.168.140.16/library/busybox-16:latest-1

The failed image: sudo nerdctl -n k8s.io tag 192.168.140.16:5000/pause:3.10 192.168.140.16/library/pause-5000:3.10 sudo nerdctl -n k8s.io push 192.168.140.16/library/pause-5000:3.10

Failed logs as below: DEBU[0000] fetch response received digest="sha256:61d9e957431bdf7a34f31cbcb23fe7966ab9da5c1df35138b1e752af15b69669" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.connection=keep-alive response.header.content-length=133 response.header.content-type="application/json; charset=utf-8" response.header.date="Thu, 05 Jun 2025 12:24:02 GMT" response.header.server=nginx response.header.set-cookie="sid=b4d676795d6c9c9691404ad3834fb26f; Path=/; HttpOnly" response.header.x-request-id=e5aa4024-75f1-4832-992a-d36fce073f6b response.status="404 Not Found" size=316581 url="https://192.168.140.16/v2/library/pause-5000/blobs/sha256:61d9e957431bdf7a34f31cbcb23fe7966ab9da5c1df35138b1e752af15b69669" DEBU[0000] do request digest="sha256:61d9e957431bdf7a34f31cbcb23fe7966ab9da5c1df35138b1e752af15b69669" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.user-agent=containerd/2.0.4+unknown request.method=POST size=316581 url="https://192.168.140.16/v2/library/pause-5000/blobs/uploads/?mount=sha256:61d9e957431bdf7a34f31cbcb23fe7966ab9da5c1df35138b1e752af15b69669&from=pause" DEBU[0000] fetch response received digest="sha256:61d9e957431bdf7a34f31cbcb23fe7966ab9da5c1df35138b1e752af15b69669" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.connection=keep-alive response.header.content-length=92 response.header.content-type="application/json; charset=utf-8" response.header.date="Thu, 05 Jun 2025 12:24:02 GMT" response.header.server=nginx response.header.x-request-id=edcef50b-4867-4372-8a0a-df67ec419a9d response.status="400 Bad Request" size=316581 url="https://192.168.140.16/v2/library/pause-5000/blobs/uploads/?mount=sha256:61d9e957431bdf7a34f31cbcb23fe7966ab9da5c1df35138b1e752af15b69669&from=pause" DEBU[0000] unexpected response body="{"errors":[{"code":"BAD_REQUEST","message":"bad request: invalid repository name: pause"}]}\n" digest="sha256:61d9e957431bdf7a34f31cbcb23fe7966ab9da5c1df35138b1e752af15b69669" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip resp="&{400 Bad Request 400 HTTP/1.1 1 1 map[Connection:[keep-alive] Content-Length:[92] Content-Type:[application/json; charset=utf-8] Date:[Thu, 05 Jun 2025 12:24:02 GMT] Server:[nginx] X-Request-Id:[edcef50b-4867-4372-8a0a-df67ec419a9d]] {0xc000192bc0} 92 [] false false map[] 0xc000402f00 0xc0004de000}" size=316581 DEBU[0000] fetch response received digest="sha256:873ed75102791e5b0b8a7fcd41606c92fcec98d56d05ead4ac5131650004c136" mediatype=application/vnd.docker.container.image.v1+json response.header.connection=keep-alive response.header.content-length=92 response.header.content-type="application/json; charset=utf-8" response.header.date="Thu, 05 Jun 2025 12:24:02 GMT" response.header.server=nginx response.header.x-request-id=17401ecd-77ca-4b39-9f51-ad2c055afa50 response.status="400 Bad Request" size=881 url="https://192.168.140.16/v2/library/pause-5000/blobs/uploads/?mount=sha256:873ed75102791e5b0b8a7fcd41606c92fcec98d56d05ead4ac5131650004c136&from=pause" DEBU[0000] unexpected response body="{"errors":[{"code":"BAD_REQUEST","message":"bad request: invalid repository name: pause"}]}\n" digest="sha256:873ed75102791e5b0b8a7fcd41606c92fcec98d56d05ead4ac5131650004c136" mediatype=application/vnd.docker.container.image.v1+json resp="&{400 Bad Request 400 HTTP/1.1 1 1 map[Connection:[keep-alive] Content-Length:[92] Content-Type:[application/json; charset=utf-8] Date:[Thu, 05 Jun 2025 12:24:02 GMT] Server:[nginx] X-Request-Id:[17401ecd-77ca-4b39-9f51-ad2c055afa50]] {0xc0003302c0} 92 [] false false map[] 0xc0009bd900 0xc0009b4780}" size=881 manifest-sha256:d8c8296a995aa91d1a888d405d98f3c4a317fe2b5790b996051f7ff1184a3694: waiting |--------------------------------------| layer-sha256:61d9e957431bdf7a34f31cbcb23fe7966ab9da5c1df35138b1e752af15b69669: waiting |--------------------------------------| config-sha256:873ed75102791e5b0b8a7fcd41606c92fcec98d56d05ead4ac5131650004c136: waiting |--------------------------------------| elapsed: 0.1 s total: 0.0 B (0.0 B/s) FATA[0000] unexpected status from POST request to https://192.168.140.16/v2/library/pause-5000/blobs/uploads/?mount=sha256:61d9e957431bdf7a34f31cbcb23fe7966ab9da5c1df35138b1e752af15b69669&from=pause: 400 Bad Request

[Gitkingly]

This looks like to be a nerctl bug. Doing the same thing with Docker tag/push, it was working fine. To clarify:

1. The local harbor: 192.168.140.16:80
2. The local docker regitry: 192.168.140.16:5000 The result: Nerdctl cannot push docker images from 192.168.140.16:5000 to 192.168.140.16:80, it looks like was trying to optimize the image layers pushing, but ignored the port 5000 which was on the same machine with the harbor and failed the image pusing.

So the workaround: Put the local docker registry to another machine, e.g. 192.168.140.17:5000, then nerdctl is working as expected.

[apostasie]

nerdctl -n k8s.io push 192.168.140.16/library/busybox-16:latest-1

This will push to port 443. If you want to push to port 80, you have to state this explicitly in your image IIRC.

eg:

nerdctl -n k8s.io push 192.168.140.16:80/library/busybox-16:latest-1

Do you define this host as insecure anywhere in your config?

[Gitkingly]

@apostasie Thanks for your quick response. Actually I installed Harbor with default config, and I have configured the https connection. I did not use --insecure-registry when doing the pushing, so 443 port was using, sorry for the confusing in previous post.

When I installed docker registry on another machine, 192.168.140.17:5000, saved the same image to it, then pulled it to current machine, then I can push the image to harbor successfully.

sudo nerdctl -n k8s.io images, I have below images in current machine: busybox:latest 192.168.140.16:5000/pause:3.10 192.168.140.17:5000/pause:3.10

Pusing to harbor 192.168.140.16/library was successful: sudo nerdctl -n k8s.io tag 192.168.140.17:5000/pause:3.10 192.168.140.16/library/pause-5000:3.10-2 sudo nerdctl -n k8s.io push 192.168.140.16/library/pause-5000:3.10-2

[apostasie]

@Gitkingly

Ok.

So, are things working, or not?

Can you provide a complete / clear reproducer, including:

• the exact command you use to start your registry
• the exact commands you are then running

Thanks.

[Gitkingly]

On machine 192.168.140.17:

docker run -d -p 5000:5000 --name registry registry:2
docker pull registryk8s/pause:3.10 
docker tag registryk8s/pause:3.10 192.168.140.17:5000/pause:3.10
docker push 192.168.140.17:5000/pause:3.10

On machine 192.168.140.16 which has installed harbor with https enabled:

docker run -d -p 5000:5000 --name registry registry:2
docker pull registryk8s/pause:3.10 
docker tag registryk8s/pause:3.10 192.168.140.16:5000/pause:3.10
docker push 192.168.140.16:5000/pause:3.10

# failed
sudo nerdctl -n k8s.io pull 192.168.140.16:5000/pause:3.10 --insecure-registry
sudo nerdctl -n k8s.io tag 192.168.140.16:5000/pause:3.10 192.168.140.16/library/pause-5000:3.10
sudo nerdctl -n k8s.io push 192.168.140.16/library/pause-5000:3.10

# succeed
sudo nerdctl -n k8s.io pull 192.168.140.17:5000/pause:3.10 --insecure-registry
sudo nerdctl -n k8s.io tag 192.168.140.17:5000/pause:3.10 192.168.140.16/library/pause-5000:3.10-2
sudo nerdctl -n k8s.io push 192.168.140.16/library/pause-5000:3.10-2

[apostasie]

Thanks @Gitkingly

So, what step is not working?

Is that the pull, or the push?

Also, can you provide logs for both the step that does not work and the equivalent step that does work, matching the last example you posted above? (with nerdctl --debug-full) (format them with ticks to make it readable here).

Thanks.

[apostasie]

Note: your original logs (though they do not match your latest example), do mention: bad request: invalid repository name: pause".

This ^ looks like the error message comes from Harbor - could be a Harbor bug, or at least some incompatibility when the registry host is the same

• what version of Harbor are you using?
• what exact command did you run to run Harbor?

[Gitkingly]

That's the problem:

bad request: invalid repository name: pause"

No matter what tag I set, “nerdctl push“ just malfunctioned when the source docker registry and the target harbor are on the same machine. Note that I set the tag "192.168.140.16:5000/pause:latest" to try to fix “invalid repository name: pause“ but failed.

I don't think there was any problem with harbor, because:

1. Using "docker push" to do the same things, it was successful
2. When the source docker registry was from a different machine, "nerdctl push" was successful

$ sudo nerdctl -n k8s.io inspect 192.168.140.16/library/pause-5000:3.10
[
    {
        "Id": "sha256:873ed75102791e5b0b8a7fcd41606c92fcec98d56d05ead4ac5131650004c136",
        "RepoTags": [
            "192.168.140.16/library/pause-5000:3.10",
            "192.168.140.16:5000/pause:3.10",
            "192.168.140.16:5000/pause:latest"
        ],
        "RepoDigests": [
            "192.168.140.16/library/pause-5000@sha256:d8c8296a995aa91d1a888d405d98f3c4a317fe2b5790b996051f7ff1184a3694",
            "192.168.140.16:5000/pause@sha256:d8c8296a995aa91d1a888d405d98f3c4a317fe2b5790b996051f7ff1184a3694",
            "192.168.140.16:5000/pause@sha256:d8c8296a995aa91d1a888d405d98f3c4a317fe2b5790b996051f7ff1184a3694"
        ],
        "Parent": "",
        "Comment": "buildkit.dockerfile.v0",
        "Created": "2024-05-23T20:42:04.96991089Z",
        "DockerVersion": "",
        "Author": "",
        "Config": {
            "User": "65535:65535",
            "AttachStdin": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "WorkingDir": "/",
            "Entrypoint": [
                "/pause"
            ]
        },
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 741376,
        "VirtualSize": 741376,
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:d8bdedd33a4e22f5cb6ca641a7ada5f9e6a92ad001589480818a136184c7ce34"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

[apostasie]

@Gitkingly

Reproduced.

Harbor apparently does not allow image names at the top-level (eg: /pause). Because the image you are tagging retains information about the original repo name, Harbor will fail.

In your specific case, you can workaround this issue using a two-level repo name for the intermediary image instead of putting it at the root (eg: 192.168.140.16:5000/pause -> 192.168.140.16:5000/whatever/pause):

docker run -d -p 5000:5000 --name registry registry:2
docker pull registryk8s/pause:3.10 
docker tag registryk8s/pause:3.10 192.168.140.16:5000/whatever/pause:3.10
docker push 192.168.140.16:5000/whatever/pause:3.10

sudo nerdctl -n k8s.io pull 192.168.140.16:5000/whatever/pause:3.10 --insecure-registry
sudo nerdctl -n k8s.io tag 192.168.140.16:5000/whatever/pause:3.10 192.168.140.16/library/pause-5000:3.10
sudo nerdctl -n k8s.io push 192.168.140.16/library/pause-5000:3.10

The reason why you do not see this problem when the original image is from a different host is because then there is no layer mounting as this is a different host. Maybe a case could be made that we should not try to layer mount if the port does not match (I am a bit skeptical of this - but anyhow, that would be a containerd bug).

This is definitely a Harbor bug that maybe should be reported to them (or a limitation, that they might or might not want to address) - though with the workaround above you should be fine.

Let me know how it works for you with the suggestion.

@AkihiroSuda question + external

[apostasie]

@AkihiroSuda what are your thoughts for the containerd part?

TL;DR: When pushing to the same domain, but on a different port, containerd still tries to layer mount from the original image.

eg:

nerdctl tag somehost:1234/foo somehost:4567/bar
nerdctl push somehost:4567/bar

^ containerd will try to layer mount from /foo to /bar when pushing to the destination registry (somehost:4567), while (of course) the destination registry does not know anything about image /foo.

This is likely from: https://github.com/containerd/containerd/blob/main/core/remotes/docker/handler.go#L118 which ignores the port. Fixing it might be tricky (implied port vs. explicit) - and there is somewhat little benefit - we might even loose some functionality (eg: in some scenario, a registry may expose a read-only endpoint on port X, and a push endpoint on port Y) - but, yeah, this does not look "correct"?

[apostasie]

Reported on Harbor ^.

[AkihiroSuda]

somehost:1234 and somehost:4567 should be considered completely different hosts, and even assumed to be different tenants.

i.e., no mount should happen.