GoogleContainerTools
Parallel build of multi-stage Docker images
Actual behavior
It looks like when I have multi-stage Docker image which have few independent stages to build everything, and then the last stage only to copy files over into the final image, those initial independent stages are not build in parallel.
Expected behavior
Independent stages initial stages could be build in parallel to speed up overall building.
To Reproduce
I used Dockerfile like:
FROM node:12-buster as node-builder
WORKDIR /app
COPY ./frontend .
RUN yarn install
RUN yarn build-staging
RUN yarn build-production
FROM golang:1.14-buster as go-builder
WORKDIR /app
COPY ./backend .
RUN make
FROM debian:buster
COPY --from=node-builder /app/dist-staging /app/staging/
COPY --from=node-builder /app/dist-production /app/production/
COPY --from=go-builder /app/backend /usr/local/bin/backend
Triage Notes for the Maintainers
| Description | Yes/No |
|---|---|
| Please check if this a new feature you are proposing |
|
| Please check if the build works in docker but not in kaniko |
|
Please check if this error is seen when you use --cache flag |
|
| Please check if your dockerfile is a multistage dockerfile |
|
(Buildkit-enabled docker build call seems to parallelize build run this.)
Hello,
We have a build with several build stage (compilation) with a final stage taking all compiled binaries. It takes severals minutes instead of some to build the image.
Looks like this issue could be easily fixed using some goroutine and a simple env/param to define max "worker" to use (like this loop)
As this issue got a lot upvote, I could work on it but is it an accepted design or is there a limitation I missed ?
Hi, I am not an author of the project but am about to fix some standing issues regarding these
which should be fixed first: https://github.com/GoogleContainerTools/kaniko/pull/2066
I think "main" still has some weird bugs with stages, most users do not use lots of stages and some problems do not arise (due to hasing etc.)
I guess @Jason Hall could clarify maybe how to go about this if its even possible, I guess it should be, but is not so trivial as stated above, since your are basically solving a DAG because every stage has a parent. So simply speaking
- building up the stage graph (graph node: stage index), and then
- defining an execution order over all nodes (coloring),
- and then building all stages in parallel with the same execution priority successively
- maybe there are shortcuts by how the Dockerfile is defined and what is possible in the sense of DAG... ?
This would speed up our builds tremendously. Honestly an implementation identical to buildkit, where the stages all start and build up to a waiting dependency (if there is one) would be a suitable implementation.
Could we first build data structure which builds this DAG, then solve an execution order (coloring) and then simply execute it in parallel. Maybe everything in different PRs. Step 3 would then finally enable this :)
Von meinem iPhone gesendet
Am 06.06.2022 um 08:12 schrieb Christopher Powroznik @.***>:
This would speed up our builds tremendously. Honestly an implementation identical to buildkit, where the stages all start and build up to a waiting dependency (if there is one) would be a suitable implementation.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.
We need this please! We have 3 stages with vue build, in docker buildkit they run in parallel, and with kaniko in serial
bumping this as it would be a nice feature that puts kaniko more in feature parity with other build tools
