GoogleContainerTools
root GitLab runner builds failing on Kaniko v1.8.0-debug and newer
There seems to be a breaking change related to root GitLab runner builds in the Kaniko v1.8.0-debug release.
Actual behavior All builds were running smoothly on v1.7.0-debug. non-root builds are running smoothly on the newset Kaniko v1.9.1-debug. root builds are failing on the newset Kaniko v1.9.1-debug. With backtracing we found that the builds are failing on any Kaniko version from v.1.8.0-debug to v1.9.1-debug.
Expected behavior root builds on GitLab runner on Kaniko versions v1.8.0-debug and newer run smoothly.
To Reproduce .gitlab-ci.yml
...
build:
stage: build
image:
name: gcr.io/kaniko-project/executor:v1.8.0-debug
entrypoint: [""]
script:
- |
if [[ -z "$CI_COMMIT_TAG" ]]; then
export CI_APPLICATION_REPOSITORY=${CI_APPLICATION_REPOSITORY:-$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG}
export CI_APPLICATION_TAG=${CI_APPLICATION_TAG:-$CI_COMMIT_SHA}
else
export CI_APPLICATION_REPOSITORY=${CI_APPLICATION_REPOSITORY:-$CI_REGISTRY_IMAGE}
export CI_APPLICATION_TAG=${CI_APPLICATION_TAG:-$CI_COMMIT_TAG}
fi
image_tagged="$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG"
image_latest="$CI_APPLICATION_REPOSITORY:latest"
mkdir -p /kaniko/.docker
echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
/kaniko/executor \
--force \
--cache=true \
--context $CI_PROJECT_DIR \
--dockerfile $CI_PROJECT_DIR/Dockerfile \
$AUTO_DEVOPS_BUILD_IMAGE_EXTRA_ARGS \
--destination "$image_tagged" \
--destination "$image_latest"
rules:
- if: '$AUTO_DEVOPS_PLATFORM_TARGET == "EC2"'
when: never
- if: '$CI_COMMIT_TAG || $CI_COMMIT_BRANCH'
...
log:
Running with gitlab-runner 14.8.2 (foo)
on foo-bar-gitlab-runner-foo-bar FOO
Preparing the "kubernetes" executor 00:00
Using Kubernetes namespace: foo
Using Kubernetes executor with image gcr.io/kaniko-project/executor:v1.8.0-debug ...
Using attach strategy to execute scripts...
Preparing environment 00:12
Waiting for pod foo/runner-foo-project-17-concurrent-foo to be running, status is Pending
Waiting for pod foo/runner-foo-project-17-concurrent-foo to be running, status is Pending
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod foo/runner-foo-project-17-concurrent-foo to be running, status is Pending
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod foo/runner-foo-project-17-concurrent-foo to be running, status is Pending
ContainersNotReady: "containers with unready status: [build helper]"
ContainersNotReady: "containers with unready status: [build helper]"
Running on runner-foo-project-17-concurrent-foo via foo-bar-gitlab-runner-foo-bar...
Getting source from Git repository 00:01
Fetching changes with git depth set to 50...
Initialized empty Git repository in /foo/bar/.git/
Created fresh repository.
Checking out foo as master...
Skipping Git submodules setup
Executing "step_script" stage of the job script 00:01
$ if [[ -z "$CI_COMMIT_TAG" ]]; then # collapsed multi-line command
WARN[0000] kaniko is being run outside of a container. This can have dangerous effects on your system
INFO[0000] Retrieving image manifest registry.foo.bar.foo/bar/foo
INFO[0000] Retrieving image registry.foo.bar.foo/bar/foo from registry registry.foo.bar.foo
INFO[0000] Retrieving image manifest registry.foo.bar.foo/bar/foo
INFO[0000] Returning cached image manifest
INFO[0000] Built cross stage deps: map[]
INFO[0000] Retrieving image manifest registry.foo.bar.foo/bar/foo
INFO[0000] Returning cached image manifest
INFO[0000] Retrieving image manifest registry.foo.bar.foo/bar/foo
INFO[0000] Returning cached image manifest
INFO[0000] Executing 0 build triggers
INFO[0000] Checking for cached layer registry.foo.bar.foo/foo/bar/master/cache:foo...
INFO[0000] No cached layer found for cmd RUN apt-get update && apt-get install -y libvips && rm -rf /var/lib/apt/lists/*
INFO[0000] Unpacking rootfs as cmd RUN apt-get update && apt-get install -y libvips && rm -rf /var/lib/apt/lists/* requires it.
error building image: error building stage: failed to get filesystem from image: failed to write "security.capability" attribute to "/usr/bin/newgidmap": operation not permitted
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: command terminated with exit code 1
Additional Information
- Dockerfile
- Any docker file that uses root priviliges
- Kaniko Image (fully qualified with digest)
- gcr.io/kaniko-project/executor:v1.8.0-debug sha256:0d8408715c7bcc2dc747405936c0e72665cafb2357fb78e23eb71f90bc39624f
- gcr.io/kaniko-project/executor:v1.9.1-debug sha256:ac169723b2076f9d5804f4bc05c98397e286da6fdcdd5a09fdc179f06ccb3be1
Potentially related to these:
- https://github.com/GoogleContainerTools/kaniko/pull/1725
- https://github.com/GoogleContainerTools/kaniko/pull/1859
Related issues:
- https://github.com/GoogleContainerTools/kaniko/issues/2201
Triage Notes for the Maintainers
| Description | Yes/No |
|---|---|
| Please check if this a new feature you are proposing |
|
| Please check if the build works in docker but not in kaniko |
|
Please check if this error is seen when you use --cache flag |
|
| Please check if your dockerfile is a multistage dockerfile |
|
@kruserr can you confirm if this issue is still occurring w/ the latest version of Kaniko - v1.12.1?
@aaron-prindle is it safe to use the ignore dir to work around this issue? https://github.com/GoogleContainerTools/kaniko/issues/2201
I wonder if it is related to this. https://github.com/GoogleContainerTools/kaniko/commit/e65bce193de5615edbad0986def310372fa62719#diff-a7dc2085daa1b9679520b0f79b134a5215227e6e20424ed95ff338a02733ddf1R166