GoogleContainerTools
Pushing with kaniko is slow
Actual behavior
Pushing large images (e.g 5Gb) with kaniko to some https registries (e.g ecr, quay or gitlab contianer registry) has been identified to be quite slow when using kaniko . After some preliminary investigation (in https://gitlab.com/gitlab-org/gitlab/-/issues/241996 )it appears the slowness comes from the issue mentioned in https://github.com/golang/go/issues/47840 and is related to the http/2 implementation of net/http
Because of the slowness caused by https://github.com/golang/go/issues/47840 docker and buildx registry clients have opted to only push on http/1.1 connection. see: https://github.com/moby/buildkit/pull/1420
A few issues have been raised on (go-containerregistry) which kaniko uses that alludes to a similar http2 bottleneck:
- https://github.com/google/go-containerregistry/issues/1193
- https://github.com/google/go-containerregistry/issues/1210
Expected behavior
Pushing a 5Gb layer using kaniko to gitlab contianer registry, ecr or quay should not take > 30mins on arguably fast internet connections
To Reproduce Steps to reproduce the behavior:
- Push a random 5Gb image to gitlab container registry
- Push a random 5Gb image to gitlab container registry with http 2 disabled (by setting GODEBUG="http2client=0")
- Observe it takes almost > 40mins with http 2 enabled
Additional Information
- Setup: https://gitlab.com/suleimiahmed/registry-speed-test/-/tree/test-ff?ref_type=heads
- includes docker file
- Any kaniko that defaults to http 2 has this issue
Triage Notes for the Maintainers
| Description | Yes/No |
|---|---|
| Please check if this a new feature you are proposing |
|
| Please check if the build works in docker but not in kaniko |
|
Please check if this error is seen when you use --cache flag |
|
| Please check if your dockerfile is a multistage dockerfile |
|
From 40 minutes to 2 minutes with workaround... Thanks !
We are having the same problem. The workaround also worked for us.
We also saw improvements on builds while pushing caches and image to gcr and gar. From 13 mins to 7 mins. Thanks for workaround!
In my case it went from 38min to 2min. @zijiwork this is how you do it in gitlab:
build-ci-image:
stage: build-ci-image
variables:
GODEBUG: "http2client=0"
image:
name: gcr.io/kaniko-project/executor:v1.9.0-debug
entrypoint: [ "" ]
script:
- /kaniko/executor
--context ...
--dockerfile ...
--destination ...
Uploads to gitlab registry went from max 5 mbit/s to full line speed
build:
stage: build
image:
name: gcr.io/kaniko-project/executor:debug
entrypoint: [""]
variables:
GODEBUG: "http2client=0"
script:
- /kaniko/executor
--context "${CI_PROJECT_DIR}"
--dockerfile "${CI_PROJECT_DIR}/Dockerfile"
--destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}"
Is anyone aware of whether this bug persists in later version releases? I am running v1.23.0 and implemented this fix a few months ago. At the time, it made a very big difference, bringing our build process from around 30 minutes to less than 10. However, it seems I have started seeing longer and longer registry upload times. We have been using gitlab all along. Occasionally, it is taking 30 minutes to upload a 250mb image to our gitlab container registry.
To add details and partially answer my own question, using Kaniko v1.23.2-debug, removing the GODEBUG: "http2client=0" workaround dropped my upload time for a 500mb image from about 15 minutes to 5 minutes. So it seems that maybe part of this issue was fixed. 5 minutes for 500mb still seems excessively slow to me, but it could be a gitlab container repository issue at this point.
This problem still seems relevant. Here are two builds of the same dockerfile:
- Without workaround, timeout after 60 minutes
- With workaround, done in 5 minutes
I feel that the issue still exist and I was forced to use the workaround proposed below.
