Guidance on using download_pkgs & install_pkgs

[robertgates55]

I'm trying to get to grips with the download_pkgs and install_pkgs rules, and struggling to find any docs or examples. Is there anything that you can point to besides https://github.com/bazelbuild/rules_docker/blob/master/docker/package_managers/README.md#download_pkgs ? Am pretty new to bazel and struggling to piece bits of it together.

Example of the (super basic!) thing I'm trying to do just to check I understand.

WORKSPACE

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "io_bazel_rules_docker",
    sha256 = "1698624e878b0607052ae6131aa216d45ebb63871ec497f26c67455b34119c80",
    strip_prefix = "rules_docker-0.15.0",
    urls = ["https://github.com/bazelbuild/rules_docker/releases/download/v0.15.0/rules_docker-v0.15.0.tar.gz"],
)

load("@io_bazel_rules_docker//repositories:repositories.bzl", container_repositories = "repositories")
container_repositories()

load("@io_bazel_rules_docker//repositories:deps.bzl", container_deps = "deps")
container_deps()

load( "@io_bazel_rules_docker//container:container.bzl","container_pull")

container_pull(
  name = "java_8_base",
  registry = "gcr.io",
  repository = "distroless/java",
  tag = "8-debug"
)

BUILD.bzl

load("@io_bazel_rules_docker//docker/package_managers:download_pkgs.bzl", "download_pkgs")

download_pkgs(
    name = "download_test",
    image_tar = "@java_8_base//image",
    packages = ["curl"]
)

➜  bazel build //:download_test --verbose_failures --sandbox_debug  

DEBUG: /private/var/tmp/_bazel_robgates/6fa9db02416501f67274dcf7fc77085a/external/rules_python/python/legacy_pip_import/pip.bzl:143:10: DEPRECATED: the pip_import rule has been replaced with pip_install, please see rules_python 0.1 release notes
INFO: Analyzed target //:download_test (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /Users/robgates/Repositories/cube/src/common/base-images/java/BUILD.bazel:3:14: Action download_test.tar failed (Exit 1): sandbox-exec failed: error executing command 
  (cd /private/var/tmp/_bazel_robgates/6fa9db02416501f67274dcf7fc77085a/sandbox/darwin-sandbox/38/execroot/__main__ && \
  exec env - \
    PATH=/usr/local/opt/[email protected]/bin:/usr/local/opt/mysql-client/bin:/Users/robgates/.rbenv/shims:/usr/local/opt/coreutils/libexec/gnubin:/usr/local/opt/gawk/libexec/gnubin:/usr/local/opt/findutils/libexec/gnubin:/usr/local/opt/gnu-sed/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/robgates/.go/bin:/usr/local/opt/go/libexec/bin \
    TMPDIR=/var/folders/87/zk040vcd3g1dhfg_mld8hzrc0000gq/T/ \
  /usr/bin/sandbox-exec -f /private/var/tmp/_bazel_robgates/6fa9db02416501f67274dcf7fc77085a/sandbox/darwin-sandbox/38/sandbox.sb /var/tmp/_bazel_robgates/install/8398ca99fbd94b1565b6a9607fa71ced/process-wrapper '--timeout=0' '--kill_delay=15' bazel-out/darwin-fastbuild/bin/download_test.sh) sandbox-exec failed: error executing command 
  (cd /private/var/tmp/_bazel_robgates/6fa9db02416501f67274dcf7fc77085a/sandbox/darwin-sandbox/38/execroot/__main__ && \
  exec env - \
    PATH=/usr/local/opt/[email protected]/bin:/usr/local/opt/mysql-client/bin:/Users/robgates/.rbenv/shims:/usr/local/opt/coreutils/libexec/gnubin:/usr/local/opt/gawk/libexec/gnubin:/usr/local/opt/findutils/libexec/gnubin:/usr/local/opt/gnu-sed/libexec/gnubin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/robgates/.go/bin:/usr/local/opt/go/libexec/bin \
    TMPDIR=/var/folders/87/zk040vcd3g1dhfg_mld8hzrc0000gq/T/ \
  /usr/bin/sandbox-exec -f /private/var/tmp/_bazel_robgates/6fa9db02416501f67274dcf7fc77085a/sandbox/darwin-sandbox/38/sandbox.sb /var/tmp/_bazel_robgates/install/8398ca99fbd94b1565b6a9607fa71ced/process-wrapper '--timeout=0' '--kill_delay=15' bazel-out/darwin-fastbuild/bin/download_test.sh)
++ guess_runfiles
++ '[' -d bazel-out/darwin-fastbuild/bin/download_test.sh.runfiles ']'
++++ dirname bazel-out/darwin-fastbuild/bin/download_test.sh
+++ cd bazel-out/darwin-fastbuild/bin
+++ pwd
++ mydir=/private/var/tmp/_bazel_robgates/6fa9db02416501f67274dcf7fc77085a/sandbox/darwin-sandbox/38/execroot/__main__/bazel-out/darwin-fastbuild/bin
++ echo /private/var/tmp/_bazel_robgates/6fa9db02416501f67274dcf7fc77085a/sandbox/darwin-sandbox/38/execroot/__main__/bazel-out/darwin-fastbuild/bin
++ sed -e 's|\(.*\.runfiles\)/.*|\1|'
+ RUNFILES=/private/var/tmp/_bazel_robgates/6fa9db02416501f67274dcf7fc77085a/sandbox/darwin-sandbox/38/execroot/__main__/bazel-out/darwin-fastbuild/bin
+ DOCKER=/usr/local/bin/docker
+ DOCKER_FLAGS=
+ [[ -z /usr/local/bin/docker ]]
++ bazel-out/host/bin/external/io_bazel_rules_docker/contrib/extract_image_id bazel-out/darwin-fastbuild/bin/external/java_8_base/image/image.tar
+ image_id=e04861f3a8efb4d6d9ca4c19615a6a9ac8e0e4caba527ebf7842f190a11d8a97
+ /usr/local/bin/docker load -i bazel-out/darwin-fastbuild/bin/external/java_8_base/image/image.tar
++ /usr/local/bin/docker run -w=/ -d --privileged e04861f3a8efb4d6d9ca4c19615a6a9ac8e0e4caba527ebf7842f190a11d8a97 sh -c '#!/usr/bin/env bash
set -ex
printf "" >> /etc/apt/sources.list.d/download_test_repos.list
# Remove /var/lib/apt/lists/* in the base image. apt-get update -y command will create them.
rm -rf /var/lib/apt/lists/*
# Fetch Index
apt-get update -y
# Make partial dir
mkdir -p /tmp/install/./partial
# Install command
apt-get install --no-install-recommends -y -q -o Dir::Cache="/tmp/install" -o Dir::Cache::archives="." curl --download-only

items=$(ls /tmp/install/*.deb)
if [ $items = ""]; then
    echo "Did not find the .deb files for debian packages curl in /tmp/install. Did apt-get actually succeed?" && false
fi
# Generate csv listing the name & versions of the debian packages.
# Example contents of a metadata CSV with debian packages gcc 8.1 & clang 9.1:
# Name,Version
# gcc,7.1
# clang,9.1
echo "Generating metadata CSV file download_test_metadata.csv"
echo Name,Version > download_test_metadata.csv
dpkg_deb_path=$(which dpkg-deb)
for item in $items; do
    echo "Adding information about $item to metadata CSV"
    pkg_name=$($dpkg_deb_path -f $item Package)
    if [ $pkg_name = ""]; then
        echo "Failed to get name of the package for $item" && false
    fi
    pkg_version=$($dpkg_deb_path -f $item Version)
    if [ $pkg_version = ""]; then
        echo "Failed to get the version of the package for $item" && false
    fi
    echo "Package $pkg_name, Version $pkg_version"
    echo -n "$pkg_name," >> download_test_metadata.csv
    echo $pkg_version >> download_test_metadata.csv
done;
# Tar command to only include all the *.deb files and ignore other directories placed in the cache dir.
tar -cpf download_test_packages.tar --mtime=1970-01-01 --directory /tmp/install/. `cd /tmp/install/. && ls *.deb`'
+ cid=7c8cdef54f41e24c09d4a9b67c648234d0e60d477d5a904d47e966e818b2aa88
+ /usr/local/bin/docker attach 7c8cdef54f41e24c09d4a9b67c648234d0e60d477d5a904d47e966e818b2aa88
You cannot attach to a stopped container, start it first
Loaded image: bazel/image:image
Target //:download_test failed to build
INFO: Elapsed time: 6.492s, Critical Path: 6.34s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully

I'm struggling to work out what from this might be wrong - any advice to see me on my way?!

[danny-skydio]

I think you might find the tests useful, they were for me.

[AndrewGuenther]

I think this is a legitimate bug. I have run into the exact same issue and it only reproduces in our CI environments. Sandbox debug logs don't offer much assistance either.

[kshcherban]

You can find example here: https://github.com/bazelbuild/rules_docker/blob/master/testing/examples/run_instruction_apt_pkgs/BUILD

[AndrewGuenther]

The offending code is these lines: https://github.com/bazelbuild/rules_docker/blob/8c3a8110a0c519929a7e79c39ac345a0f8c74d04/docker/package_managers/run_install.sh.tpl#L51-L53

If the container fails to start, the attach call will fail. The most likely cause of the run command here failing is it isn't able to connect to the docker daemon for some reason. Take a look at your configured flags and try to run the command as printed in the echoed output.

But there's an || true there!

In my testing, this doesn't actually seem to work. If the attach fails, the target will fail immediately. This script also handled this case differently than others. The purpose of the attach here is to wait for the container to exit. Other scripts handle this case much more gracefully via the following: https://github.com/bazelbuild/rules_docker/blob/8c3a8110a0c519929a7e79c39ac345a0f8c74d04/docker/util/commit.sh.tpl#L22-L28

I'd strongly suggest updating this block to use a similar mechanism to provide clearer errors when the container failed to run.

[aw185176]

I am also hitting this error on newer versions of Ubuntu, it doesn't seem related to misusing these packages as the build defs that are having problems worked until recently

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_docker!

[github-actions[bot]]

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"