bootc icon indicating copy to clipboard operation
bootc copied to clipboard
verified publisher icon containers

[composefs-backend: bootc-installing an image from OCI registry fails due to mismatched OCI Config

Open tulilirockz opened this issue 3 months ago • 12 comments

Hello, and thank you all very much for your work on the Composefs Backend for this project!

Basically, what I've been running into when debugging https://github.com/bootcrew images, is that, when a certain "bootc-supporting" OCI image is pushed to a registry (i.e. docker.io/distribution/distribution), some mismatch inside the code makes it so the CFS handling tries fetching the wrong blobs from the OCI registry instead of whatever is required for the rootfs.

Heres a reproduction:

  1. Create an empty directory with this Containerfile and build it as root:
debian-bootc Containerfile

Modified version of: https://github.com/bootcrew/debian-bootc

FROM docker.io/library/debian:unstable@sha256:35e463b0688565a6cf07b76653d94fa37e17dc396be9af10f96a67c1ca451d10

ARG DEBIAN_FRONTEND=noninteractive
# Antipattern but we are doing this since `apt`/`debootstrap` does not allow chroot installation on unprivileged podman builds
ENV DEV_DEPS="libzstd-dev libssl-dev pkg-config libostree-dev curl git build-essential meson libfuse3-dev go-md2man dracut whois"

RUN rm /etc/apt/apt.conf.d/docker-gzip-indexes /etc/apt/apt.conf.d/docker-no-languages && \
    apt update -y && \
    apt install -y $DEV_DEPS ostree

# Usually we track main but this is for the example
ENV BOOTC_COMMIT="7eaf53bec74aca9ab0ca977bdb0c725f218465eb"
ENV RUST_VERSION="1.90.0"

ENV CARGO_HOME=/tmp/rust
ENV RUSTUP_HOME=/tmp/rust
RUN --mount=type=tmpfs,dst=/tmp \
    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- --profile minimal -y --default-toolchain="${RUST_VERSION}"  && \
    git clone https://github.com/bootc-dev/bootc.git /tmp/bootc && \
    cd /tmp/bootc && git checkout "${BOOTC_COMMIT}" && \
    sh -c ". ${RUSTUP_HOME}/env ; env CARGO_FEATURES=\"composefs-backend\" make -C /tmp/bootc bin install-all install-initramfs-dracut"

ENV DRACUT_NO_XATTR=1
RUN apt install -y \
  btrfs-progs \
  dosfstools \
  e2fsprogs \
  fdisk \
  firmware-linux-free \
  linux-image-generic \
  skopeo \
  systemd \
  systemd-boot* \
  xfsprogs

RUN sh -c 'export KERNEL_VERSION="$(basename "$(find /usr/lib/modules -maxdepth 1 -type d | grep -v -E "*.img" | tail -n 1)")" && \
    dracut --force --no-hostonly --reproducible --zstd --verbose --kver "$KERNEL_VERSION"  "/usr/lib/modules/$KERNEL_VERSION/initramfs.img" && \
    cp /boot/vmlinuz-$KERNEL_VERSION "/usr/lib/modules/$KERNEL_VERSION/vmlinuz"'

# Setup a temporary root passwd (changeme) for dev purposes
# TODO: Replace this for a more robust option when in prod
RUN usermod -p "$(echo "changeme" | mkpasswd -s)" root

RUN apt remove -y $DEV_DEPS && \
    apt autoremove -y

# Update useradd default to /var/home instead of /home for User Creation
RUN sed -i 's|^HOME=.*|HOME=/var/home|' "/etc/default/useradd"

RUN rm -rf /var /boot /home /root /usr/local /srv && \
    mkdir -p /var && \
    ln -s /var/home /home && \
    ln -s /var/roothome /root && \
    ln -s /var/srv /srv && \
    ln -s sysroot/ostree ostree && \
    ln -s /var/usrlocal /usr/local && \
    mkdir -p /sysroot /boot

# Necessary for `bootc install`
RUN mkdir -p /usr/lib/ostree && \
    printf  "[composefs]\nenabled = yes\n[sysroot]\nreadonly = true\n" | \
    tee "/usr/lib/ostree/prepare-root.conf"

RUN bootc container lint
Example steps 
$ sudo -s
# mkdir /tmp/exampledir
# cd /tmp/exampledir
# (create file with vi here)
# podman build -t example-image:latest -f Containerfile .
  1. Publish image to a registry of your choosing (I'm using a distribution container)
Running/publishing to distribution locally 
# podman tag localhost/example-image:latest localhost:5005/example-image:latest
$ podman run -d --name "my-registry" -p 5005:5000 docker.io/distribution/distribution:latest
# podman push --tls-verify=false localhost:5005/example-image:latest
# podman rmi localhost:5005/example-image:latest
# podman pull --tls-verify=false localhost:5005/example-image:latest # This is the important one!
  1. Try to create a disk image with bootc install to-disk 
# fallocate -l 10G ./mydisk.img
# podman run --rm --privileged --pid=host -v /var/lib/containers:/var/lib/containers -v /dev:/dev --security-opt label=type:unconfined_t -v .:/work:Z -w /work localhost:5005/example-image:latest bootc install to-disk --wipe --via-loopback --filesystem=ext4 --composefs-backend ./mydisk.img

You should then get an error similar to this:

error: Installing to disk: Unable to pull container image containers-storage:localhost:5005/example-image:latest: Failed to pull config Descriptor { media_type: ImageConfig, digest: Digest { algorithm: Sha256, value: "sha256:d93b3d583eb626dc186d3652aa0825eccb1b73ae4ef9ff859424c700f0d6cbfb", split: 6 }, size: 5450, urls: None, annotations: None, platform:None, artifact_type: None, data: None }: failed to invoke method GetBlob: locating item named "sha256:77de736491477bb13f26f659ae54c8e1922aa9221c7630d960dbb187f5541517" for image with ID "d93b3d583eb626dc186d3652aa0825eccb1b73ae4ef9ff859424c700f0d6cbfb" (consider removing the image to resolve the issue): file does not exist
Full output with RUST_LOG=trace 
/var/home/tulip/tmp/bugrep
❯ RUST_LOG=trace podman run --rm --privileged --pid=host -v /var/lib/containers:/var/lib/containers -v /dev:/dev --security-opt label=type:unconfined_t -e RUST_LOG=trace -v .:/work:Z -w /work localhost:5005/example-image:latest bootc install to-disk --wipe --via-loopback --filesystem=ext4 --composefs-backend ./mydisk.img
TRACE starting bootc
DEBUG argv0="bootc"
 INFO Starting disk installation from none to ./mydisk.img message_id="8b7c6d5e4f3a2b1c0d9e8f7a6b5c4d3e2" bootc.source_image="none" bootc.target_device="./mydisk.img" bootc.via_loopback="true"
Automatically enabling --generic-image when installing via loopback
TRACE Preparing install
TRACE Verified uid 0 with CAP_SYS_ADMIN
TRACE OK: we're not pid 1
TRACE OK: we're in a matching user namespace with pid1
DEBUG rootless=0
TRACE Read container engine info ContainerExecutionInfo { engine: "podman-5.6.2", name: "hungry_bohr", id: "aea8510435c29d33ddf163de083cc9680be157109a491a1a53cb8b6d43fb734d", image: "localhost:5005/example-image:latest", imageid: "d93b3d583eb626dc186d3652aa0825eccb1b73ae4ef9ff859424c700f0d6cbfb", rootless: Some("0") }
DEBUG Finding digest for image ID d93b3d583eb626dc186d3652aa0825eccb1b73ae4ef9ff859424c700f0d6cbfb
TRACE exec: "/usr/bin/bootc" "exec-in-host-mount-namespace" "podman" "inspect" "d93b3d583eb626dc186d3652aa0825eccb1b73ae4ef9ff859424c700f0d6cbfb"
DEBUG Target image reference: ostree-unverified-registry:localhost:5005/example-image:latest
DEBUG Composefs required: false
TRACE base mount id 1915489293260957925, host mount id 1915489293260957925
DEBUG Already mounted from host: /dev
TRACE base mount id 64513, host mount id 64513
DEBUG Already mounted from host: /var/lib/containers
TRACE base mount id 16559041305486660447, host mount id 16559041305486660447
DEBUG Already mounted from host: /var/tmp
TRACE exec: "mount" "tmpfs" "-t" "tmpfs" "/tmp"
DEBUG Already in a mount namespace
DEBUG Setting up sys mounts
TRACE exec: "mount" "-t" "efivarfs" "efivarfs" "/sys/firmware/efi/efivars"
DEBUG SELinux state: Disabled
Installing image: docker://localhost:5005/example-image:latest
Digest: sha256:cbe2fe0e2024e1126b30f758657517336ad4c6060dcee50a2168df6695423706
TRACE Scanning directory '/usr/lib/bootc/install'
TRACE Scanning directory '/usr/local/lib/bootc/install'
TRACE Scanning directory '/etc/bootc/install'
TRACE Scanning directory '/run/bootc/install'
DEBUG No install configuration found
Bootloader: systemd
TRACE No bootupctl binary found
TRACE exec: "losetup" "--show" "--direct-io=off" "-P" "--find" "./mydisk.img"
DEBUG Allocated loopback /dev/loop2
TRACE exec: "lsblk" "-J" "-b" "-O" "/dev/loop2"
DEBUG backfilled start to 2048

DEBUG backfilled start to 4096

DEBUG backfilled start to 1052672

TRACE exec: "findmnt" "-J" "-v" "--output=SOURCE,TARGET,MAJ:MIN,FSTYPE,OPTIONS,UUID" "-N" "1"
TRACE starting bootc
DEBUG argv0="bootc"
Wiping /dev/loop2p1
Wiping device /dev/loop2p1
Wiping /dev/loop2p2
Wiping device /dev/loop2p2
/dev/loop2p2: 8 bytes were erased at offset 0x00000052 (vfat): 46 41 54 33 32 20 20 20
/dev/loop2p2: 1 byte was erased at offset 0x00000000 (vfat): eb
/dev/loop2p2: 2 bytes were erased at offset 0x000001fe (vfat): 55 aa
Wiping /dev/loop2p3
Wiping device /dev/loop2p3
/dev/loop2p3: 2 bytes were erased at offset 0x00000438 (ext4): 53 ef
Wiping /dev/loop2
Wiping device /dev/loop2
/dev/loop2: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/loop2: 8 bytes were erased at offset 0x27ffffe00 (gpt): 45 46 49 20 50 41 52 54
/dev/loop2: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/loop2: calling ioctl to re-read partition table: Success
Block setup: direct
       Size: 10737418240
     Serial: <unknown>
      Model: <unknown>
DEBUG Partitioning: label: gpt
label-id: a12e21ca-3ac7-4ca9-80d3-13d62eaea96f
size=1MiB, bootable, type=21686148-6449-6E6F-744E-656564454649, name="BIOS-BOOT"
size=512MiB, type=c12a7328-f81f-11d2-ba4b-00a0c93ec93b, name="EFI-SYSTEM"
type=0FC63DAF-8483-4772-8E79-3D69D8477DE4, name="root"

DEBUG exec: "sfdisk" "--wipe=always" "/dev/loop2"
Checking that no-one is using this disk right now ... OK

Disk /dev/loop2: 10 GiB, 10737418240 bytes, 20971520 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

>>> Script header accepted.
>>> Script header accepted.
>>> Created a new GPT disklabel (GUID: A12E21CA-3AC7-4CA9-80D3-13D62EAEA96F).
/dev/loop2p1: Created a new partition 1 of type 'BIOS boot' and of size 1 MiB.
/dev/loop2p2: Created a new partition 2 of type 'EFI System' and of size 512 MiB.
/dev/loop2p3: Created a new partition 3 of type 'Linux filesystem' and of size 9.5 GiB.
/dev/loop2p4: Done.

New situation:
Disklabel type: gpt
Disk identifier: A12E21CA-3AC7-4CA9-80D3-13D62EAEA96F

Device         Start      End  Sectors  Size Type
/dev/loop2p1    2048     4095     2048    1M BIOS boot
/dev/loop2p2    4096  1052671  1048576  512M EFI System
/dev/loop2p3 1052672 20969471 19916800  9.5G Linux filesystem

The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
TRACE ExitStatus(unix_wait_status(0))
DEBUG Created partition table
TRACE starting bootc
DEBUG argv0="bootc"
TRACE "udevadm" ["settle"]
TRACE exec: "sfdisk" "-J" "/dev/loop2"
TRACE exec: "lsblk" "-J" "-b" "-O" "/dev/loop2p3"
DEBUG backfilled start to 1052672

Creating root filesystem (ext4) on device /dev/loop2p3 (size=10.2 GB)
> mkfs.ext4 -U 4f68bce3-e8cd-4db1-96e7-fbcaf984b709 -L root -O verity /dev/loop2p3
DEBUG exec: "mkfs.ext4" "-U" "4f68bce3-e8cd-4db1-96e7-fbcaf984b709" "-L" "root" "-O" "verity" "/dev/loop2p3"
mke2fs 1.47.2 (1-Jan-2025)
TRACE ExitStatus(unix_wait_status(0))
DEBUG Labeling .
TRACE Label for . is None
DEBUG Labeling boot
TRACE Label for boot is None
DEBUG Labeling boot
TRACE Label for boot is None
Creating ESP filesystem
> mkfs.fat /dev/loop2p2 -n EFI-SYSTEM
DEBUG exec: "mkfs.fat" "/dev/loop2p2" "-n" "EFI-SYSTEM"
TRACE ExitStatus(unix_wait_status(0))
TRACE exec: "sfdisk" "-J" "/dev/loop2"
DEBUG new_with_config: Spawned skopeo pid=117585 config=ImageProxyConfig { authfile: None, auth_data: None, auth_anonymous: false, certificate_directory: None, decryption_keys: None, insecure_skip_tls_verification: None, debug: false, skopeo_cmd: None }
TRACE new_with_config:impl_request_with_fds: sending request Initialize config=ImageProxyConfig { authfile: None, auth_data: None, auth_anonymous: false, certificate_directory: None, decryption_keys: None, insecure_skip_tls_verification: None, debug: false, skopeo_cmd: None } self=ImageProxy method="Initialize"
TRACE new_with_config:impl_request_with_fds: completed request config=ImageProxyConfig { authfile: None, auth_data: None, auth_anonymous: false, certificate_directory: None, decryption_keys: None, insecure_skip_tls_verification: None, debug: false, skopeo_cmd: None } self=ImageProxy method="Initialize"
DEBUG new_with_config: Remote protocol version: 0.2.8 config=ImageProxyConfig { authfile: None, auth_data: None, auth_anonymous: false, certificate_directory: None, decryption_keys: None, insecure_skip_tls_verification: None, debug: false, skopeo_cmd: None }
DEBUG open_image: opening image self=ImageProxy imgref="containers-storage:localhost:5005/example-image:latest"
TRACE open_image:impl_request_with_fds: sending request OpenImage self=ImageProxy imgref="containers-storage:localhost:5005/example-image:latest" self=ImageProxy method="OpenImage"
TRACE open_image:impl_request_with_fds: completed request self=ImageProxy imgref="containers-storage:localhost:5005/example-image:latest" self=ImageProxy method="OpenImage"
TRACE impl_request_with_fds: sending request GetManifest self=ImageProxy method="GetManifest"
TRACE impl_request_with_fds: completed request self=ImageProxy method="GetManifest"
DEBUG finish_pipe: closing pipe self=ImageProxy pipeid=PipeId(7)
TRACE finish_pipe:impl_request_with_fds: sending request FinishPipe self=ImageProxy pipeid=PipeId(7) self=ImageProxy method="FinishPipe"
TRACE finish_pipe:impl_request_with_fds: completed request self=ImageProxy pipeid=PipeId(7) self=ImageProxy method="FinishPipe"
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageConfig, digest: Digest { algorithm: Sha256, value: "sha256:d93b3d583eb626dc186d3652aa0825eccb1b73ae4ef9ff859424c700f0d6cbfb", split: 6 }, size: 5450, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:d93b3d583eb626dc186d3652aa0825eccb1b73ae4ef9ff859424c700f0d6cbfb", split: 6 } size=5450
TRACE get_descriptor:get_blob:impl_request_with_fds: sending request GetBlob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageConfig, digest: Digest { algorithm: Sha256, value: "sha256:d93b3d583eb626dc186d3652aa0825eccb1b73ae4ef9ff859424c700f0d6cbfb", split: 6 }, size: 5450, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:d93b3d583eb626dc186d3652aa0825eccb1b73ae4ef9ff859424c700f0d6cbfb", split: 6 } size=5450 self=ImageProxy method="GetBlob"
TRACE get_descriptor:get_blob:impl_request_with_fds: completed request self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageConfig, digest: Digest { algorithm: Sha256, value: "sha256:d93b3d583eb626dc186d3652aa0825eccb1b73ae4ef9ff859424c700f0d6cbfb", split: 6 }, size: 5450, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:d93b3d583eb626dc186d3652aa0825eccb1b73ae4ef9ff859424c700f0d6cbfb", split: 6 } size=5450 self=ImageProxy method="GetBlob"
DEBUG finish_pipe: closing pipe self=ImageProxy pipeid=PipeId(6)
TRACE finish_pipe:impl_request_with_fds: sending request FinishPipe self=ImageProxy pipeid=PipeId(6) self=ImageProxy method="FinishPipe"
TRACE finish_pipe:impl_request_with_fds: completed request self=ImageProxy pipeid=PipeId(6) self=ImageProxy method="FinishPipe"
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayerGzip, digest: Digest { algorithm: Sha256, value: "sha256:77de736491477bb13f26f659ae54c8e1922aa9221c7630d960dbb187f5541517", split: 6 }, size: 309409840, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:77de736491477bb13f26f659ae54c8e1922aa9221c7630d960dbb187f5541517", split: 6 } size=309409840
TRACE get_descriptor:get_blob:impl_request_with_fds: sending request GetBlob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayerGzip, digest: Digest { algorithm: Sha256, value: "sha256:77de736491477bb13f26f659ae54c8e1922aa9221c7630d960dbb187f5541517", split: 6 }, size: 309409840, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:77de736491477bb13f26f659ae54c8e1922aa9221c7630d960dbb187f5541517", split: 6 } size=309409840 self=ImageProxy method="GetBlob"
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayerGzip, digest: Digest { algorithm: Sha256, value: "sha256:d17fc48cf4f8b207e2a0f3637707d952691cdaa9a8610f0a0e71f0e36f11d96b", split: 6 }, size: 218192133, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:d17fc48cf4f8b207e2a0f3637707d952691cdaa9a8610f0a0e71f0e36f11d96b", split: 6 } size=218192133
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayerGzip, digest: Digest { algorithm: Sha256, value: "sha256:962db9eaccebfd9b8a26c60c0bbd19a6914fa8e254b1c9085f98b4f503f659df", split: 6 }, size: 70814454, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:962db9eaccebfd9b8a26c60c0bbd19a6914fa8e254b1c9085f98b4f503f659df", split: 6 } size=70814454
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayerGzip, digest: Digest { algorithm: Sha256, value: "sha256:52835bb60d77fb4ef9a128353bc6ff471d387df19a0562d92d74fbd3a7f4eded", split: 6 }, size: 57889124, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:52835bb60d77fb4ef9a128353bc6ff471d387df19a0562d92d74fbd3a7f4eded", split: 6 } size=57889124
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayerGzip, digest: Digest { algorithm: Sha256, value: "sha256:6595998b05ddb4652e034fc9a310abf8e761ffa7f0291e9cbd36dec0935e9e33", split: 6 }, size: 50989570, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:6595998b05ddb4652e034fc9a310abf8e761ffa7f0291e9cbd36dec0935e9e33", split: 6 } size=50989570
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayerGzip, digest: Digest { algorithm: Sha256, value: "sha256:993f8805766b0d08f8a54307d781fc31ee5403f374277323725761a2d66412f8", split: 6 }, size: 41481612, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:993f8805766b0d08f8a54307d781fc31
ee5403f374277323725761a2d66412f8", split: 6 } size=41481612
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayerGzip, digest: Digest { algorithm: Sha256, value: "sha256:07e102da9a6bfbc402ba24497abb1dec4b625b1306f0c9289f3297e83890e4c0", split: 6 }, size: 831, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:07e102da9a6bfbc402ba24497abb1dec4b625b1306f0c9289f3297e83890e4c0", split: 6 } size=831
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayerGzip, digest: Digest { algorithm: Sha256, value: "sha256:6063f2fc612c746b822fdfcfdb65bf776d7c7997fb8fb8a062dae927157c197d", split: 6 }, size: 756, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:6063f2fc612c746b822fdfcfdb65bf776d7c7997fb8fb8a062dae927157c197d", split: 6 } size=756
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayerGzip, digest: Digest { algorithm: Sha256, value: "sha256:b57926e7de01d780cf6733735417573b1bdc7933c88a5184e5437ff253b4e6d6", split: 6 }, size: 574, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:b57926e7de01d780cf6733735417573b1bdc7933c88a5184e5437ff253b4e6d6", split: 6 } size=574
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayerGzip, digest: Digest { algorithm: Sha256, value: "sha256:80d494f4563bcf0429ea4f6f86374b3476d5946ea947ef7cfebf170b2a9479e9", split: 6 }, size: 233, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:80d494f4563bcf0429ea4f6f86374b3476d5946ea947ef7cfebf170b2a9479e9", split: 6 } size=233
DEBUG get_descriptor:get_blob: fetching blob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayerGzip, digest: Digest { algorithm: Sha256, value: "sha256:bd9ddc54bea929a22b334e73e026d4136e5b73f5cc29942896c72e4ece69b13d", split: 6 }, size: 34, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:bd9ddc54bea929a22b334e73e026d4136e5b73f5cc29942896c72e4ece69b13d", split: 6 } size=34
TRACE get_descriptor:get_blob:impl_request_with_fds: sending request GetBlob self=ImageProxy img=OpenedImage(1) descriptor=Descriptor { media_type: ImageLayerGzip, digest: Digest { algorithm: Sha256, value: "sha256:d17fc48cf4f8b207e2a0f3637707d952691cdaa9a8610f0a0e71f0e36f11d96b", split: 6 }, size: 218192133, urls: None, annotations: None, platform: None, artifact_type: None, data: None } self=ImageProxy img=OpenedImage(1) digest=Digest { algorithm: Sha256, value: "sha256:d17fc48cf4f8b207e2a0f3637707d952691cdaa9a8610f0a0e71f0e36f11d96b", split: 6 } size=218192133 self=ImageProxy method="GetBlob"
TRACE exec: "losetup" "-d" "/dev/loop2"
error: Installing to disk: Unable to pull container image containers-storage:localhost:5005/example-image:latest: Failed to pull config Descriptor { media_type: ImageConfig, digest: Digest { algorithm: Sha256, value: "sha256:d93b3d583eb626dc186d3652aa0825eccb1b73ae4ef9ff859424c700f0d6cbfb", split: 6 }, size: 5450, urls: None, annotations: None, platform:None, artifact_type: None, data: None }: failed to invoke method GetBlob: locating item named "sha256:77de736491477bb13f26f659ae54c8e1922aa9221c7630d960dbb187f5541517" for image with ID "d93b3d583eb626dc186d3652aa0825eccb1b73ae4ef9ff859424c700f0d6cbfb" (consider removing the image to resolve the issue): file does not exist

tulilirockz avatar Oct 25 '25 00:10 tulilirockz

Thanks for reporting this. I have faced this before and it was on my radar for bug fixes but couldn't get around to working on it. I'm not entirely sure about the core issue here , but I think it's got to do with how skopeo handles containers storage. Because if you pull the same registry image with docker://localhost:5005/example-image:latest it should work

Johan-Liebert1 avatar Oct 27 '25 10:10 Johan-Liebert1

Hm possibly related to what I wrote about in https://github.com/containers/skopeo/issues/2576

jeckersb avatar Oct 27 '25 13:10 jeckersb

Note that this also happens when running bootc update or bootc switching to anything, I get pretty much the same-ish error there. Funnily enough it works when trying to bootc switch to an image with an OSTree repository in it (i.e.: debian-bootc -> fedora-bootc (ostree))

tulilirockz avatar Oct 27 '25 15:10 tulilirockz

Are these images Docker v2s2 or OCI format? It's the former that triggers this bug now, and there's really no reason at all to use v2s2 in 2025.

skopeo inspect --raw docker://<image> will show you.

cgwalters avatar Oct 29 '25 12:10 cgwalters

Hm possibly related to what I wrote about in https://github.com/containers/skopeo/issues/2576

Yeah it's likely

cgwalters avatar Oct 29 '25 12:10 cgwalters

@cgwalters I believe this is on the OCI format, whatever podman build spits out, tbh. Even changing it to OCI on the Buildah action on gh actions seems to yield the same result

tulilirockz avatar Oct 29 '25 13:10 tulilirockz

Doing a little bit of digging, it seems the issue is likely us querying for compressed layers while, in skopeo, here

https://github.com/containers/skopeo/blob/main/vendor/go.podman.io/image/v5/storage/storage_src.go#L139-L143

they assume that an uncompressed layer will be requested. The layers array comes out to be empty, and an assumption is made that this must be a data item

https://github.com/containers/skopeo/blob/main/vendor/go.podman.io/image/v5/storage/storage_src.go#L145

I'm not quite sure how skopeo copy handles this

Johan-Liebert1 avatar Oct 30 '25 11:10 Johan-Liebert1

I think we need to use https://github.com/bootc-dev/bootc/blob/042aa21d235dd9f13f30b74d0b515e46f03f88e2/crates/ostree-ext/src/container/store.rs#L814 in composefs-rs.

cgwalters avatar Nov 03 '25 13:11 cgwalters

Turns out that building with the oci: true option on the Buildah github action fixes this. The description seems to indicate exactly what you said, Colin, it was using the docker storage format instead of the OCI one. (see this). Seems odd that podman build would spit out images on that format, though :P

tulilirockz avatar Nov 03 '25 13:11 tulilirockz

The buildah action dates from a time when OCI was still pretty new and compatibility concerns with older clients mattered I think.

cgwalters avatar Nov 03 '25 14:11 cgwalters

Funnily enough this seems to still happen, but its only when bootc installing something...

fallocate -l 20G ./bootable.img
podman run \
    --rm --privileged --pid=host \
    -it \
    -v /sys/fs/selinux:/sys/fs/selinux \
    -v /etc/containers:/etc/containers:Z \
    -v /var/lib/containers:/var/lib/containers:Z \
    -v /dev:/dev \
    -e RUST_LOG=debug \
    -v ".:/data" \
    --security-opt label=type:unconfined_t \
    "ghcr.io/bootcrew/debian-bootc:latest" bootc \
    install to-disk --composefs-backend --via-loopback /data/bootable.img --filesystem ext4 --wipe --bootloader systemd

tulilirockz avatar Nov 04 '25 23:11 tulilirockz

For me, this issue happens to the images in my local containers-storage, and only happen to oci instead of v2s2 It only fails when the local oci image contains compressed layers, but will succeed if it's un-compressed with a skopeo copy

shi2wei3 avatar Nov 21 '25 09:11 shi2wei3