Add support for multi-arch OCI image indexes in Bazel builder

[ar3s3ru]

I am working in a Bazel workspace that uses rules_oci to build image tarballs, and Skaffold to perform the build+deploy steps.

As I'm working on an M1 (and minikube) but deploying on a remote k8s cluster based on linux-amd64, the images I'm building are multi-arch.

When running skaffold dev, it fails with the following error:

skaffold dev -p local --port-forward
Generating tags...
 - api -> api:v0.31.1-6-g1781fcd-dirty
Checking cache...
 - api: Not found. Building
Starting build...
Found [minikube] context, using local docker daemon.
Building [api]...
Target platforms: [linux/arm64]
Loading: 
Loading: 
Loading: 0 packages loaded
Analyzing: target //backend/apps/api:image.tar (0 packages loaded, 0 targets configured)
INFO: Analyzed target //backend/apps/api:image.tar (46 packages loaded, 1470 targets configured).
INFO: Found 1 target...
[5 / 16] [Prepa] BazelWorkspaceStatusAction stable-status.txt
Target //backend/apps/api:image.tar up-to-date:
  bazel-bin/backend/apps/api/image.tar/tarball.tar
INFO: Elapsed time: 0.771s, Critical Path: 0.43s
INFO: 3 processes: 2 internal, 1 darwin-sandbox.
INFO: Build completed successfully, 3 total actions
Cleaning up...
 - No resources found
Error: uninstall: Release not loaded: postgres: release: not found
Error: uninstall: Release not loaded: redis: release: not found
Error: uninstall: Release not loaded: jaeger: release: not found
Cleaning up resources encountered an error, will continue to clean up other resources.
build [api] failed: loading manifest from tarball failed: file manifest.json not found in tar

Here is an excerpt of the BUILD.bazel file, note how format = "oci" in oci_tarball:

BUILD.bazel

go_binary(
    name = "api",
    embed = [":api_lib"],
    visibility = ["//visibility:public"],
)

linux_amd64_binary = "api_linux-amd64".format(app_name)
linux_amd64_layer = "{}_layer".format(linux_amd64_binary)
linux_amd64_image = "{}_image".format(linux_amd64_binary)

go_cross_binary(
    name = linux_amd64_binary,
    platform = "@rules_go//go/toolchain:linux_amd64",
    target = target,
)

pkg_tar(
    name = linux_amd64_layer,
    srcs = [linux_amd64_binary],
)

oci_image(
    name = linux_amd64_image,
    base = base,
    entrypoint = ["/api"],
    tars = [linux_amd64_layer],
)

linux_arm64_binary = "api_linux-arm64"
linux_arm64_layer = "{}_layer".format(linux_arm64_binary)
linux_arm64_image = "{}_image".format(linux_arm64_binary)

go_cross_binary(
    name = linux_arm64_binary,
    platform = "@rules_go//go/toolchain:linux_arm64",
    target = target,
)

pkg_tar(
    name = linux_arm64_layer,
    srcs = [linux_arm64_binary],
)

oci_image(
    name = linux_arm64_image,
    base = base,
    entrypoint = ["/api"],
    tars = [linux_arm64_layer],
)

oci_image_index(
    name = "image",
    images = [
        linux_amd64_image,
        linux_arm64_image,
    ],
)

oci_tarball(
    name = "image.tar",
    format = "oci",
    image = ":image",
    repo_tags = ["api:latest"],
)

This is happening due to the following section: https://github.com/GoogleContainerTools/skaffold/blob/1255d6c4cdff405ef07da381dda1afcbbb62f153/pkg/skaffold/build/bazel/build.go#L93-L96

tarball.LoadManifest looks for the manifest.json file in the tarball built by Bazel, but won't find it since OCI image indexes use an index.json file instead: https://github.com/google/go-containerregistry/blob/4fdaa32ee934cd178b6eb41b3096419a52ef426a/pkg/v1/tarball/image.go#L74-L79

go-containerregistry already exposes a package to load the index.json file: layout.ImageIndexFromPath

RepoTags can be deducted using the stategy explained here: https://github.com/opencontainers/image-spec/issues/796

[ar3s3ru]

I'm giving it a try in #9219.

I managed to make it load the index.json manifest, but performing the docker load operation locally is failing with the following error:

build [api] failed: loading image into docker daemon: reading from image load response: open /var/lib/docker/tmp/docker-import-3424243076/blobs/json: no such file or directory

It appears to be due to the fact that the Docker Engine API expects a very specific format for the tarball: https://docs.docker.com/engine/api/v1.43/#tag/Image/operation/ImageLoad

[aran]

This would be really useful. Looks like Docker Engine doesn't support OCI-format multi-arch tarballs out of the box, rules_oci doesn't produce Docker-format multi-arch tarballs, and skaffold doesn't currently support alternative container APIs like containerd. (https://github.com/GoogleContainerTools/skaffold/blob/main/pkg/skaffold/docker/image.go would need to be generalized). Might need a new bazel rule something like docker_tarball to make tarballs in the format docker API expects, or generalize skaffold to be able to communicate with a containerd