docker
Unable to docker build on Linux kernel 4.19
- [x] This is a bug report
- [ ] This is a feature request
- [x] I searched existing issues before opening this one
Expected behavior
Build without error
Actual behavior
When run docker build -t testing . this error appears on Linux kernel 4.19:
dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Invalid cross-device link
E: Sub-process /usr/bin/dpkg returned an error code (2)
The command '/bin/sh -c apt-get install -y locales && sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && locale-gen' returned a non-zero code: 100
On Linux kernel v4.18 it works. I have tested docker v18.06 and v18.09.
Steps to reproduce the behavior
On a Linux kernel 4.19 use this Dockerfile:
FROM ubuntu:16.04
# Let the container know that there is no tty
ENV DEBIAN_FRONTEND noninteractive
RUN echo "deb mirror://mirrors.ubuntu.com/mirrors.txt xenial main restricted universe multiverse" \
> /etc/apt/sources.list && \
echo "deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-updates main restricted universe multiverse" \
>> /etc/apt/sources.list && \
echo "deb mirror://mirrors.ubuntu.com/mirrors.txt xenial-security main restricted universe multiverse" \
>> /etc/apt/sources.list && \
apt-get update
# Ensure UTF-8 lang and locale
RUN apt-get install -y locales && \
sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
locale-gen
ENV LANG='en_US.UTF-8' LANGUAGE='en_US:en' LC_ALL='en_US.UTF-8'
I run docker build -t testing .
Output of docker version:
Client:
Version: 18.09.0-ce
API version: 1.39
Go version: go1.11.2
Git commit: 4d60db472b
Built: Fri Nov 9 00:05:34 2018
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 18.09.0-ce
API version: 1.39 (minimum version 1.12)
Go version: go1.11.2
Git commit: 4d60db472b
Built: Fri Nov 9 00:05:11 2018
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
Containers: 5
Running: 0
Paused: 0
Stopped: 5
Images: 32
Server Version: 18.09.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9f2e07b1fc1342d1c48fe4d7bbb94cb6d1bf278b.m
runc version: 079817cc26ec5292ac375bb9f47f373d33574949
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.19.1-1-MANJARO
Operating System: Manjaro Linux
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.41GiB
Name: javier-homepc
ID: 6ULE:QTFG:T6YJ:QIRD:URUL:GEZR:ZWUK:R2J2:NAWY:ALEY:PSSJ:EW4P
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Further information
It's a physical machine with Manjaro Linux based on Arch Linux OS (docker is from Arch Linux and Linux kernel from Manjaro)
Sounds like either an issue with overlayfs or dpkg not playing well with overlayfs.
A quick search brought me to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=836211;msg=17
@thaJeztah: Thanks. It seems the bug it's still open: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=836211
There is an even simpler example:
FROM ubuntu:18.04
RUN apt-get update
RUN apt-get install -y ffmpeg
Any fixes apart from downgrading kernel?
I posted a comment here about what was discovered and a subsequent comment shows that the default for the overlay config metacopy was switched from N to Y in kernel 4.19...
The following should do the trick to get you going ...
echo N | sudo tee /sys/module/overlay/parameters/metacopy
... remember that you'll likely have to restart your containers. The above isn't permanent, a reboot will reset the value back to Y, so add this in a script somewhere that gets called on boot.
You can also try /sbin/modprobe overlay metacopy=N though it's up to you when this is called (presumably before the module is loaded?)
You can also try adding a kernel parameter in grub (or friends) overlay.metacopy=N, though I haven't personally tried this last one to see if it works.
There is a fix for the kernel: https://www.spinics.net/lists/linux-unionfs/msg06316.html. Hope it's merged in 4.19.3.
I'm seeing the same error using buildah bud to do the build. I'm on Antergos (Arch-based) so I can just drop back to the LTS kernel.
@ffernand: "You can also try adding a kernel parameter in grub (or friends) overlay.metacopy=N, though I haven't personally tried this last one to see if it works."
I just tried it, and it does appear to be working. Antergos (Arch) Linux, in docker-compose build of two images based on Debian stretch.
what @ffernand proposed:
echo N | sudo tee /sys/module/overlay/parameters/metacopy
works for me quite well on Manjaro
what @ffernand proposed:
echo N | sudo tee /sys/module/overlay/parameters/metacopyworks for me quite well on Manjaro
Thanks!
I am still getting the same error:
...
Get:186 http://archive.ubuntu.com/ubuntu bionic/main amd64 vdpau-driver-all amd64 1.1.1-3ubuntu1 [4674 B]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 75.8 MB in 5s (15.1 MB/s)
Selecting previously unselected package multiarch-support.
(Reading database ... 4038 files and directories currently installed.)
Preparing to unpack .../multiarch-support_2.27-3ubuntu1_amd64.deb ...
Unpacking multiarch-support (2.27-3ubuntu1) ...
dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Invalid cross-device link
on Kernel Version: 4.19.4-1-MANJARO when using overlay2 Storage Driver. When using aufs everything works fine for me.
I initially found this on 4.19.1, upgrading to 4.19.4 didn't help. (4.19.4-arch1-1-ARCH)
(Running the command mentioned above did fix it)
echo N | sudo tee /sys/module/overlay/parameters/metacopy
@GH0st3rs I also upgraded my kernel to 4.19.4-arch1-1-ARCH (archlinux) but the issue was not resolved for me when metacopy was turned on...
sudo cat /sys/module/overlay/parameters/metacopy
Y
Are you sure you don't have the option overlay.metacopy=N in your command line for the kernel (via grub) or maybe you have echo N | sudo tee /sys/module/overlay/parameters/metacopy in your bash/shell init script?
@GH0st3rs It is a partial solution, it works for me right now, but of course, if i reboot the computer i will lose this fix. Waiting for a total solution.
@SalahAdDin you can make an entry into the kernel command line to make this fix permanent.
edit /etc/defaults/grub and find the line that says:
GRUB_CMDLINE_LINUX=""
change this to:
GRUB_CMDLINE_LINUX="overlay.metacopy=N"
Then, rebuild your Grub config:
update-grub
You'll need root permissions to edit/run the commands so either preface the command with sudo or switch user to the root account. e.g.
sudo vi /etc/default/grub
sudo update-grub
And, of course, the tricky part is remembering you did this so you can remove it once this issue is resolved 😜 That will be the part I screw up.
You should not change GRUB_CMDLINE_LINUX (it's used as a fallback), but add it to the end of GRUB_CMDLINE_LINUX_DEFAULT, e.g.
GRUB_CMDLINE_LINUX_DEFAULT="quiet overlay.metacopy=N"
@teohhanhui I'd say that's just a matter of preference and not an absolute. The user is better off determining which is better given their application. That is why both options exist.
From the manual:
‘GRUB_CMDLINE_LINUX’
Command-line arguments to add to menu entries for the Linux kernel.‘GRUB_CMDLINE_LINUX_DEFAULT’
Unless ‘GRUB_DISABLE_RECOVERY’ is set to ‘true’, two menu entries will be generated for each Linux kernel: one default entry and one entry for recovery mode. This option lists command-line arguments to add only to the default menu entry, after those listed in ‘GRUB_CMDLINE_LINUX’.
https://www.gnu.org/software/grub/manual/grub/html_node/Simple-configuration.html
Yes, the documentation is an excellent place to start when determining the correct usage as it applies to the problem domain that you are trying to solve.
However, if you're trying to use it to prove you are correct, then you've failed horribly. Mainly, because you are assuming you know what is best and not considering the application which the user is dealing with.
And, more importantly, you are missing the true joy of Unix and Unix/Linux like operating systems - there is more than one way to do things. It is up to the user to decide what is best given their requirements. To say there is only one correct way to do just about anything is categorically false. If it were true, we wouldn't have hundreds of varieties of GNU/Linux or the many flavors of *BSD Unix. In short, the software ecosystem that exists today wouldn't exist.
That said, this issue is about Docker not playing nicely (yet) with the new metacopy feature in Linux 4.19 and not about which is the proper method in which to append arguments to the Linux kernel at boot time. So, I'm signing off and not replying to further posts on this topic unless it relates to metacopy and Docker.
A workaround would be the same as in https://github.com/moby/moby/pull/37993: add metacopy=off on a per-mount basis, if a parameter is available.
Workaround for waiting next docker release
$ echo N | sudo tee /sys/module/overlay/parameters/metacopy
stop running container -> start container (restart does not work correctly)
ensure docker mount overlayfs not use metacopy=on
$ mount | grep "docker"
As other mentioned, this occurs on later versions of 4.19
Linux faithful 4.19.6-1-MANJARO
The mentioned echo N | sudo tee /sys/module/overlay/parameters/metacopy does resolve the issue.
