GoogleContainerTools
layer created implicitly by WORKDIR is not cached
Actual behavior
When WORKDIR is called on a non-existent directory, kaniko is kind enough to create that directory for you, resulting in a layer being added. However, kaniko does not cache that layer, which means that on every invocation a completely new image is emitted from that point onwards. Inside the same stage this is non-obvious as caching mechanism still pulls, so you get a 100% cache hitrate thereafter, but the image is completely new. In multistage builds or builds that depend on the newly emitted image, this is catastrophic, as they do consider the entire image's sha when determining whether a cache is hit or not, so this will invalidate the entire cache.
Workaround is simple but silly, we just need to make sure that the directory exists before calling WORKDIR
RUN mkdir /app
WORKDIR /app
Expected behavior
When WORKDIR is called on a non-existent directory, that directory is implicitly created and the resulting layer is pushed into cache.
To Reproduce
FROM ubuntu:latest
WORKDIR /app
RUN echo "whatever"
run this build multiple times, with cache, and observe that the layers change every time
docker inspect <image>
Triage Notes for the Maintainers
| Description | Yes/No |
|---|---|
| Please check if this a new feature you are proposing |
|
| Please check if the build works in docker but not in kaniko |
|
Please check if this error is seen when you use --cache flag |
|
| Please check if your dockerfile is a multistage dockerfile |
|
this is the underlying cause of https://github.com/GoogleContainerTools/kaniko/issues/3246
problem is here https://github.com/GoogleContainerTools/kaniko/blob/main/pkg/commands/workdir.go#L74 we add to the snapshot but don't cache it.
Ran into the same issue. 3341 is not merged yet. Is there any other workaround ?
you can ensure that the workdir exists prior to running the command, not nice but not terrible either
RUN mkdir /app
WORKDIR /app
you can ensure that the workdir exists prior to running the command, not nice but not terrible either
RUN mkdir /app WORKDIR /app
since you mention it, i tried it with a multistage build. Doesn't help. removing, or moving the WORKDIR to the end of the build process (Dockerfile) is the only way it seems.
@salzig please provide more context. there are a whole host of reasons why caching fails on multi-stage builds. Notable issues are labels being applied on intermediate images https://github.com/GoogleContainerTools/kaniko/pull/3413
As said, I only changed the location of workdir from early in chain of stages to the end, and the caching problem went away.
If you're still looking for a solution to this you could give my fork a try https://github.com/mzihlmann/kaniko/releases/ It fixes this issue and a few more, mostly related to caching, if you have other issues you would like to see resolved please let me know. I know that this is not ideal and I hope we can get the changes merged here eventually but for now that's the best I can offer. If you like what you see you can support me with a star, thank you 🙇