Foreign layers stored in whole-image tar files are lost on load

[bpcreech]

🐞 Foreign layers stored in whole-image tar files are lost on load

Affected Rule

container_image (root cause might be write-side, in container_build but I'm treating this as a read problem in the proposed fix.)

Is this a regression?

Yes, this worked before with the Python image loader. I think this broke when we switched from the Python containerregistry.client.v2_2.docker_image to the Golang compat/reader.go.

Description

Basically, this fails:

container_import(
    name = "import_windows_base_image",
    config = "windowsservercore.1803.config.json",
    layers = [],
    manifest = "windowsservercore.1803.manifest.json",
)

container_image(
    name = "basic_windows_image_from_tar",
    base = ":import_windows_base_image.tar",
    cmd = ["echo bar"],
    operating_system = "windows",
)

The resulting manifest for basic_windows_image_from_tar is missing the URLs specified in the container_import.

compat/reader.go builds a map by diff_id. It finds and registers the foreign layer from the manifest, but then finds a stub empty layer in the tarball and then overwrites the foreign layer with not-so-helpful information from the stub.

🔬 Minimal Reproduction

See example above, build basic_windows_image_from_tar.tar, inspect the manifest.json within it, and note that the foreign layer is missing. I'll send a PR with a test case (basically, the above) which makes it more obvious.

🔥 Exception or Error

Since the foreign layer is replaced with a no-op stub with no URLs, if you push the resultant image to a container registry, docker pull yields an error unknown blob. (And, e.g., the GCR UI shows the image size as being quite small, due to missing the URLs pointing to the base image.)

🌍 Your Environment

Operating System:

$ uname -a
Linux bpcreech9048451 5.16.18-1rodete2-amd64 #1 SMP PREEMPT Debian 5.16.18-1rodete2 (2022-04-07) x86_64 GNU/Linux

Output of bazel version:

$ bazel version
WARNING: Invoking Bazel in batch mode since it is not invoked from within a workspace (below a directory having a WORKSPACE file).
Extracting Bazel installation...
Build label: 5.1.1
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Fri Apr 8 15:49:48 2022 (1649432988)
Build timestamp: 1649432988
Build timestamp as int: 1649432988

Rules_docker version:

v0.21.0

Anything else relevant?