List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)

[duyuyouci]

Version

21.9.0

Steps to Reproduce

./install.sh

Expected Result

E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied)

Actual Result

subscription-consumer-transactions uses an image, skipping relay uses an image, skipping nginx uses an image, skipping Building snuba-cleanup Sending build context to Docker daemon 3.584kB Step 1/5 : ARG BASE_IMAGE Step 2/5 : FROM ${BASE_IMAGE} ---> a77f8946858f Step 3/5 : RUN apt-get update && apt-get install -y --no-install-recommends cron && rm -r /var/lib/apt/lists/* ---> Running in c0c31cde30a4 Reading package lists... E: List directory /var/lib/apt/lists/partial is missing. - Acquire (13: Permission denied) Removing intermediate container c0c31cde30a4 The command '/bin/sh -c apt-get update && apt-get install -y --no-install-recommends cron && rm -r /var/lib/apt/lists/*' returned a non-zero code: 100 Service 'snuba-cleanup' failed to build : Build failed An error occurred, caught SIGERR on line 4 Cleaning up...

[aminvakil]

Is this is a clean install or are you upgrading from an older version to 21.9.0? or from 21.9.0 to a newer version?

Can you check free disk space on your system?

[duyuyouci]

from 21.9.0 to a newer version

[aminvakil]

Judging from snap stuff, this is an Ubuntu, right? Do you have apparmor enabled or some other MAC? Can you temporary disable it to make sure if it's not preventing from write?

This may seems odd, but can you check output of df -i for free inodes? :)

[aminvakil]

You can also try building the image manually, it may help debugging to be faster and easier, it's in cron folder.

Or try this simpler Dockerfile here. To be sure change 22.6.0 to whatever version you are upgrading to:

FROM getsentry/snuba:22.6.0
USER 0
RUN apt-get update && apt-get install -y --no-install-recommends cron && \
    rm -r /var/lib/apt/lists/*
CMD ["test 0"]

Just paste this into a file named Dockerfile in a clean folder and execute this command: docker build -t snubatest:1 .

[duyuyouci]

You can also try building the image manually, it may help debugging to be faster and easier, it's in cron folder.

Or try this simpler Dockerfile here. To be sure change 22.6.0 to whatever version you are upgrading to:

FROM getsentry/snuba:22.6.0
USER 0
RUN apt-get update && apt-get install -y --no-install-recommends cron && \
    rm -r /var/lib/apt/lists/*
CMD ["test 0"]

Just paste this into a file named Dockerfile in a clean folder and execute this command: docker build -t snubatest:1 .

This method works，thanks。

[aminvakil]

You can also try building the image manually, it may help debugging to be faster and easier, it's in cron folder. Or try this simpler Dockerfile here. To be sure change 22.6.0 to whatever version you are upgrading to:

FROM getsentry/snuba:22.6.0
USER 0
RUN apt-get update && apt-get install -y --no-install-recommends cron && \
    rm -r /var/lib/apt/lists/*
CMD ["test 0"]

Just paste this into a file named Dockerfile in a clean folder and execute this command: docker build -t snubatest:1 .

This method works，thanks。

So the issue only happens when you try with ./install.sh?

[duyuyouci]

You can also try building the image manually, it may help debugging to be faster and easier, it's in cron folder. Or try this simpler Dockerfile here. To be sure change 22.6.0 to whatever version you are upgrading to:

FROM getsentry/snuba:22.6.0
USER 0
RUN apt-get update && apt-get install -y --no-install-recommends cron && \
    rm -r /var/lib/apt/lists/*
CMD ["test 0"]

Just paste this into a file named Dockerfile in a clean folder and execute this command: docker build -t snubatest:1 .

This method works，thanks。

So the issue only happens when you try with ./install.sh?

yes

[aminvakil]

This is odd, please execute it with debug mode like this and paste it in a gist.

DEBUG=1 ./install.sh

[joaodlf]

I also got this when upgrading from 22.1.0 to 22.6.0.

Gist: https://gist.github.com/joaodlf/ed9c3eb911cc08c8b44599c124e441ca

[joaodlf]

You can also try building the image manually, it may help debugging to be faster and easier, it's in cron folder.

Or try this simpler Dockerfile here. To be sure change 22.6.0 to whatever version you are upgrading to:

FROM getsentry/snuba:22.6.0
USER 0
RUN apt-get update && apt-get install -y --no-install-recommends cron && \
    rm -r /var/lib/apt/lists/*
CMD ["test 0"]

Just paste this into a file named Dockerfile in a clean folder and execute this command: docker build -t snubatest:1 .

Can also confirm this has worked. But now I'm getting the following when starting the containers via docker-compose up -d:

Starting sentry_onpremise_sentry-cleanup_1 ... error ERROR: for sentry_onpremise_sentry-cleanup_1 Cannot start service sentry-cleanup: OCI runtime create failed: > container_linux.go:380: starting container process caused: exec: "/entrypoint.sh": stat /entrypoint.sh: no such file or directory: unknown

I can see an "entrypoint.sh" file at sentry/entrypoint.sh, but not at root.

docker-compose ps:

                           Name                                          Command                  State                      Ports
--------------------------------------------------------------------------------------------------------------------------------------------------
sentry_onpremise_clickhouse_1                                 /entrypoint.sh                   Up (healthy)   8123/tcp, 9000/tcp, 9009/tcp
sentry_onpremise_cron_1                                       /etc/sentry/entrypoint.sh  ...   Up             9000/tcp
sentry_onpremise_geoipupdate_1                                /usr/bin/geoipupdate -d /s ...   Exit 1
sentry_onpremise_ingest-consumer_1                            /etc/sentry/entrypoint.sh  ...   Up             9000/tcp
sentry_onpremise_kafka_1                                      /etc/confluent/docker/run        Up (healthy)   9092/tcp
sentry_onpremise_memcached_1                                  docker-entrypoint.sh memcached   Up (healthy)   11211/tcp
sentry_onpremise_nginx_1                                      /docker-entrypoint.sh ngin ...   Up             0.0.0.0:9000->80/tcp,:::9000->80/tcp
sentry_onpremise_post-process-forwarder_1                     /etc/sentry/entrypoint.sh  ...   Up             9000/tcp
sentry_onpremise_postgres_1                                   /opt/sentry/postgres-entry ...   Up (healthy)   5432/tcp
sentry_onpremise_redis_1                                      docker-entrypoint.sh redis ...   Up (healthy)   6379/tcp
sentry_onpremise_relay_1                                      /bin/bash /docker-entrypoi ...   Up             3000/tcp
sentry_onpremise_sentry-cleanup_1                             /entrypoint.sh 0 0 * * * g ...   Exit 127
sentry_onpremise_smtp_1                                       docker-entrypoint.sh exim  ...   Up             25/tcp
sentry_onpremise_snuba-api_1                                  ./docker_entrypoint.sh api       Up             1218/tcp
sentry_onpremise_snuba-cleanup_1                              ./docker_entrypoint.sh */5 ...   Restarting
sentry_onpremise_snuba-consumer_1                             ./docker_entrypoint.sh con ...   Up             1218/tcp
sentry_onpremise_snuba-outcomes-consumer_1                    ./docker_entrypoint.sh con ...   Up             1218/tcp
sentry_onpremise_snuba-replacer_1                             ./docker_entrypoint.sh rep ...   Up             1218/tcp
sentry_onpremise_snuba-sessions-consumer_1                    ./docker_entrypoint.sh con ...   Up             1218/tcp
sentry_onpremise_snuba-subscription-consumer-events_1         ./docker_entrypoint.sh sub ...   Restarting
sentry_onpremise_snuba-subscription-consumer-transactions_1   ./docker_entrypoint.sh sub ...   Restarting
sentry_onpremise_snuba-transactions-cleanup_1                 ./docker_entrypoint.sh */5 ...   Restarting
sentry_onpremise_snuba-transactions-consumer_1                ./docker_entrypoint.sh con ...   Up             1218/tcp
sentry_onpremise_subscription-consumer-events_1               /etc/sentry/entrypoint.sh  ...   Up             9000/tcp
sentry_onpremise_subscription-consumer-transactions_1         /etc/sentry/entrypoint.sh  ...   Up             9000/tcp
sentry_onpremise_symbolicator-cleanup_1                       ./docker_entrypoint.sh 55  ...   Restarting
sentry_onpremise_symbolicator_1                               /bin/bash /docker-entrypoi ...   Up             3021/tcp
sentry_onpremise_web_1                                        /etc/sentry/entrypoint.sh  ...   Up (healthy)   9000/tcp
sentry_onpremise_worker_1                                     /etc/sentry/entrypoint.sh  ...   Up             9000/tcp
sentry_onpremise_zookeeper_1                                  /etc/confluent/docker/run        Up (healthy)   2181/tcp, 2888/tcp, 3888/tcp

[chadwhitacre]

You still stuck on this @joaodlf?

[joaodlf]

You still stuck on this @joaodlf?

Well, despite some containers still showing a state of "Restarting" and sentry_onpremise_sentry-cleanup_1 "Exit 127", Sentry appears to be running smoothly.

[SyCode7]

I still had the issue, I manually commented out the line in the Dockerfile before it worked. Will there be any implications @chadwhitacre @aminvakil ?

[aminvakil]

@joaodlf that's different from cron entrypoint.sh, it should be present and executable in cron directory.

Will you please check that?

@SyCode7 What line?

[SyCode7]

@joaodlf that's different from cron entrypoint.sh, it should be present and executable in cron directory.

Will you please check that?

@SyCode7 What line?

@aminvakil - I believe lines 3 & 4. These are the commands on the lines

RUN apt-get update && apt-get install -y --no-install-recommends cron && \
    rm -r /var/lib/apt/lists/*

[aminvakil]

Ah, you're effectively removing the cron installation, rest of the container does nothing.

@duyuyouci @joaodlf @SyCode7 I'm not sure if should affect anything or not, just a wild guess, do you have a custom umask? Something different from 0022?

It can be retrieved using umask command.

[SyCode7]

@aminvakil , I just checked, umask is default - 022

[joaodlf]

@joaodlf that's different from cron entrypoint.sh, it should be present and executable in cron directory.

Will you please check that?

@SyCode7 What line?

I've got an entrypoint.sh file within my cron dir:

#!/usr/bin/env bash

if [ "$(ls -A /usr/local/share/ca-certificates/)" ]; then
  update-ca-certificates
fi

# Prior art:
# - https://git.io/fjNOg
# - https://blog.knoldus.com/running-a-cron-job-in-docker-container/

declare -p | grep -Ev 'BASHOPTS|BASH_VERSINFO|EUID|PPID|SHELLOPTS|UID' > /container.env

{ for cron_job in "$@"; do echo -e "SHELL=/bin/bash
BASH_ENV=/container.env
${cron_job} > /proc/1/fd/1 2>/proc/1/fd/2"; done } \
  | sed --regexp-extended 's/\\(.)/\1/g' \
  | crontab -
crontab -l
exec cron -f -l -L 15

umask is showing 0022.

[github-actions[bot]]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀