Building FCOS with qemu-user-static for different architectures

[raballew]

Feature Request

From my point of view, support for different architectures besides x86_64 is becoming increasingly important but it seems that the coreos-assembler container image does not support that yet. This means that one has to build the container image from scratch for the target architecture for any other architecture than x86_64, even though one might lack a proper build machine running on aarch64 for example.

At this stage not many options are left, one of them is using qemu-user-static but it is not supported out of the box and requires changes to coreos-assembler code base. An example implementation of what I am trying to achieve for aarch64 can be seen here, even though it does not work due to some limitations in coreos-assembler.

Desired Feature

I would love to see two things:

1. Add support for qemu-user-static that does not require patching of the coreos-assembler source code

2. Provide a multi-arch container image for coreos-assembler on quay.io that can be used by other to build their own custom FCOS on different architectures

Example Usage

This boils down to the following:

1. On any x86_64 machine build coreos-assembler from scratch:

sudo podman run --rm --pull always --privileged registry.hub.docker.com/multiarch/qemu-user-static:latest --reset -p yes
podman build --arch ${COREOS_ARCHITECTURE} -f target/src/coreos-assembler/Dockerfile -t localhost/coreos-assembler target/src/coreos-assembler/

Where:

• ${COREOS_ARCHITECTURE} - Target architecture such as aarch64 or s390x

2. Use the existing coreos-assembler container image if no rebuild is required:

sudo podman run --rm --pull always --privileged registry.hub.docker.com/multiarch/qemu-user-static:latest --reset -p yes
podman run \
    --rm -ti --security-opt label=disable --privileged \
    --arch ${COREOS_ARCHITECTURE} \
    --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 \
    -v ${PWD}/target/build/cosa:/srv/ --device /dev/kvm --device /dev/fuse \
    --tmpfs /tmp -v /var/tmp:/var/tmp --name cosa \
    quay.io/coreos-assembler/coreos-assembler:latest "${CMD}"

Where:

• ${COREOS_ARCHITECTURE} - Target architecture such as aarch64 or s390x
• ${CMD} - Some cosa command

Other Information

None.

[cgwalters]

Agreed, it may not even be really hard to add support for cross-arch builds. Since cosa already uses qemu/kvm I think the strategy that would make the most sense is for us to support running crossarch qemu for our internal bits, instead of trying to run the whole thing under podman+qemu.

[raballew]

@cgwalters Not sure if I understood everything correctly. Do you mean that you would prefer dropping podman support in favor of running everything directly on the build host?

If that is the case, I would like to argue that being able to build coreos-assembler from scratch with podman for different architectures allows users to embed the container build process in a k8s cluster with some tweaks and leverage the clusters resources to speed up everything.

[darkmuggle]

If that is the case, I would like to argue that being able to build coreos-assembler from scratch with podman for different architectures allows users to embed the container build process in a k8s cluster with some tweaks and leverage the clusters resources to speed up everything.

Sure, you can do builds of COSA via podman

Take a look at https://github.com/coreos/coreos-assembler/tree/main/docs/gangplank which allows for starting a build on one-arch but doing the work in another place (Kube/OpenShift, Podman remote). Gangplank is how FCOS is building aarch64 over SSH. Gangplank was originally targeted at doing builds via k8s and coordinates the pods for you.