Conflicting files while building images with Kaniko on Kubernetes

[RafaeLeal]

Actual behavior I'm using Tekton+EKS so we use kaniko to build images. A clear and concise description of what the bug is.

We started to see an error like this:

{"level":"info","msg":"Running: [/bin/sh -c mkdir /root/.ssh]","time":"2022-07-06T14:22:19Z"}
mkdir: can't create directory '/root/.ssh': File exists

I checked the base image, and it does not have this folder. Adding a RUN ls -lah /root && ls -lah /root/.ssh before this RUN mkdir /root/.ssh step, I got:

{"level":"info","msg":"Running: [/bin/sh -c ls -lah /root \u0026\u0026 ls -lah /root/.ssh ||:]","time":"2022-07-06T14:22:19Z"}
total 0      
drwx------    3 root     root          18 Jul  6 14:22 .
drwxr-xr-x    1 root     root         272 Jul  6 14:22 ..
drwx------    2 root     root          72 Jul  6 14:22 .ssh
total 12K    
drwx------    2 root     root          72 Jul  6 14:22 .
drwx------    3 root     root          18 Jul  6 14:22 ..
-rw-------    1 root     root         114 Jul  6 14:22 config
-rw-------    1 root     root        3.2K Jul  6 14:22 id_default-github-ssh-key
-rw-------    1 root     root         442 Jul  6 14:22 known_hosts

But these files are injected via ServiceAccount into kaniko's step, so it doesn't come from the base image. I tested changing to mkdir -p /root/.ssh and it passed, and it didn't add those files (thankfully)

Expected behavior I expected some sort of isolation between what's run for building the layer. So I expected to be able to run RUN mkdir <path> if that file path doesn't exist on the base image.

To Reproduce Although I'm using a Kubernetes, Tekton, etc. I believe you can safely test it locally by going to an empty folder and running this script:

#!/usr/bin/env bash
set -euo pipefail

mkdir -p ./foo

echo -e 'FROM alpine \nRUN mkdir /foo' > Dockerfile | tar -cf - Dockerfile | gzip -9 | docker run \
	--interactive -v $(pwd):/workspace -v $(pwd)/foo:/foo gcr.io/kaniko-project/executor:latest \
	--context tar://stdin \
	--log-format=json \
	--no-push \
	--tarPath=foo.tar \
	--destination=foo

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing	- [ ]
Please check if the build works in docker but not in kaniko	- [x]
Please check if this error is seen when you use --cache flag	- [ ]
Please check if your dockerfile is a multistage dockerfile	- [ ]