Errors while running container built on Docker Mac for armhf

[leshik]

Consider the following Dockerfile to be built for armhf as an example:

FROM debian:buster-slim

env USER root

RUN apt-get update \
 && apt-get install --no-install-recommends -y tightvncserver xfonts-base \
 && mkdir /root/.vnc \
 && echo test | vncpasswd -f >/root/.vnc/passwd \
 && chmod 600 /root/.vnc/passwd

CMD vncserver

If you then try to run the resulting image, either on Docker Mac or on Raspberry Pi, VNC won't start, complaining:

Fatal server error:
could not open default font 'fixed'

If the image were built on Raspberry Pi, there would be no error and VNC would start successfully. The image built on Mac can be fixed on Raspberry Pi by manually executing dpkg-reconfigure xfonts-base before launching vncserver (might be somehow related to fontconfig.) There is no fix that works in QEMU, however. ARM64 doesn't suffer from this issue.

Sorry for not having a simpler example, this one was taken from the real world scenario.

[tuonga]

@leshik I don't think this is a QEMU problem. I believe the issue here is the missing fonts.dir file in /usr/share/fonts/X11/misc. Suggest you run the mkfontdir command which should create fonts.dir.

/usr/share/fonts/X11/misc# mkfontdir

[leshik]

@tuonga maybe, but the problem here that it breaks only in QEMU. On real ARM hardware the same Dockerfile works correctly, be it armhf or any other.

[tuonga]

@leshik there are obviously differences between QEMU and real hardware. I guess the main question is what's keeping the fonts.dir file from being created. Since you mentioned OSX also requires manual intervention, there may be other factors at play.

[leshik]

@tuonga yes, what can I do to help troubleshooting the issue?

ps: I also have another one where the pre-built java app doesn't work if packed for ARM on Docker Mac, but it's probably different and not that easy to reproduce, so let's figure out what's wrong with this one first.

[leshik]

@tuonga looks like it's /usr/bin/mkfontscale which fails to generate the correct fonts.dir file. As it's ELF binary and strace doesn't work in QEMU I'm not sure what to do next.

[tuonga]

@leshik can you try adding mkfontdir to your startup sequence and see if that works around the issue? There is also a new version of QEMU coming out within a week or so. Let's test again and see what happens.

PS: may have to debug mkfontscale from source.

[leshik]

@tuonga adding to startup doesn't help, it's not generating anything under QEMU (but does on real hardware).

[tuonga]

@leshik That's interesting. I was able to get vncserver to run after running mkfontdir. Can you try manually from the command line? The following sequence worked for me.

$ docker run -it timtsai2018/hello bash
root@8e4dff74eac6:/# cd /usr/share/fonts/X11/misc/
root@8e4dff74eac6:/usr/share/fonts/X11/misc# vncserver
/usr/bin/xauth:  file /root/.Xauthority does not exist
Couldn't start Xtightvnc; trying default font path.
Please set correct fontPath in the vncserver script.
Couldn't start Xtightvnc process.

10/08/19 05:04:41 Xvnc version TightVNC-1.3.9
10/08/19 05:04:41 Copyright (C) 2000-2007 TightVNC Group
10/08/19 05:04:41 Copyright (C) 1999 AT&T Laboratories Cambridge
10/08/19 05:04:41 All Rights Reserved.
10/08/19 05:04:41 See http://www.tightvnc.com/ for information on TightVNC
10/08/19 05:04:41 Desktop name 'X' (8e4dff74eac6:1)
10/08/19 05:04:41 Protocol versions supported: 3.3, 3.7, 3.8, 3.7t, 3.8t
10/08/19 05:04:41 Listening for VNC connections on TCP port 5901
Font directory '/usr/share/fonts/X11/Type1/' not found - ignoring
Font directory '/usr/share/fonts/X11/75dpi/' not found - ignoring
Font directory '/usr/share/fonts/X11/100dpi/' not found - ignoring

Fatal server error:
could not open default font 'fixed'
10/08/19 05:04:43 Xvnc version TightVNC-1.3.9
10/08/19 05:04:43 Copyright (C) 2000-2007 TightVNC Group
10/08/19 05:04:43 Copyright (C) 1999 AT&T Laboratories Cambridge
10/08/19 05:04:43 All Rights Reserved.
10/08/19 05:04:43 See http://www.tightvnc.com/ for information on TightVNC
10/08/19 05:04:43 Desktop name 'X' (8e4dff74eac6:1)
10/08/19 05:04:43 Protocol versions supported: 3.3, 3.7, 3.8, 3.7t, 3.8t
10/08/19 05:04:43 Listening for VNC connections on TCP port 5901
Font directory '/usr/share/fonts/X11/Speedo/' not found - ignoring
Font directory '/usr/share/fonts/X11/Type1/' not found - ignoring
Font directory '/usr/share/fonts/X11/75dpi/' not found - ignoring
Font directory '/usr/share/fonts/X11/100dpi/' not found - ignoring

Fatal server error:
could not open default font 'fixed'
 
root@8e4dff74eac6:/usr/share/fonts/X11/misc# mkfontdir
root@8e4dff74eac6:/usr/share/fonts/X11/misc# vncserver

New 'X' desktop is 8e4dff74eac6:1

Creating default startup script /root/.vnc/xstartup
Starting applications specified in /root/.vnc/xstartup
Log file is /root/.vnc/8e4dff74eac6:1.log

root@8e4dff74eac6:/usr/share/fonts/X11/misc# uname -a
Linux 8e4dff74eac6 4.15.0-55-generic #60-Ubuntu SMP Tue Jul 2 18:22:20 UTC 2019 armv7l GNU/Linux

[leshik]

@tuonga

Oh, that's weird. Turned out it might be not QEMU issue but Docker Mac or moby instead. Something strange is happening. I was able to reproduce it with much simpler setup, here is the Dockerfile:

FROM debian:buster-slim
RUN apt-get update \
 && apt-get install --no-install-recommends -y xfonts-base

If you build an image with it, then run it and switch to /usr/share/fonts/X11/misc, you'll find there is no fonts.dir file. In this case, running mkfontdir manually fixes the issue.

BUT !!!

If you start just from bare image (docker run -it --platform armhf debian:buster-slim), then do all the steps manually (i.e. installing xfonts-base), then no matter how many times you run mkfontdir, it always generates an incorrect fonts.dir file which is 2-bytes in length and contains just 0 and the line feed.

[tuonga]

@leshik good info. I can take a look next week.

[tuonga]

@leshik I believe ultimately the issue is this:

https://bugs.launchpad.net/qemu/+bug/1805913 (also see the link in comment #5).

I was able to distill the problem into the exact same behaviour where readdir() returns NULL with the errno EOVERFLOW.

The best solution is probably going back to glibc-2.27.

[leshik]

@tuonga yeah, I can confirm it works fine on ubuntu:18.04 which ships with glibc-2.27. Any chance this could be fixed somehow on qemu part? While we can go to older glibc for now, this might not be the case in future...

[tuonga]

@leshik this problem is tracked in upstream qemu already and is best handled there. https://bugs.launchpad.net/qemu/+bug/1805913

[leshik]

@tuonga yes, I've seen that. There is also a patch for glibc at the end of this thread which might help in the meantime for OSes that ship with glibc > 2.27.

[tuonga]

@leshik good info. it seems like a hack but I will look further.