docbleach
DockerCompose: worker_1 error
Hello,
When I run docker-compose up using the latest images, I get the following error :
worker_1 | [2020-07-22 14:24:38,075: ERROR/MainProcess] Control command error: OperationalError("\nCannot route message for exchange 'reply.celery.pidbox': Table empty or key no longer exists.\nProbably the key ('_kombu.binding.reply.celery.pidbox') has been removed from the Redis database.\n",)
worker_1 | Traceback (most recent call last):
worker_1 | File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 439, in _reraise_as_library_errors
worker_1 | yield
worker_1 | File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 518, in _ensured
worker_1 | return fun(*args, **kwargs)
worker_1 | File "/usr/lib/python3.6/site-packages/kombu/messaging.py", line 203, in _publish
worker_1 | mandatory=mandatory, immediate=immediate,
worker_1 | File "/usr/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 605, in basic_publish
worker_1 | message, exchange, routing_key, **kwargs
worker_1 | File "/usr/lib/python3.6/site-packages/kombu/transport/virtual/exchange.py", line 70, in deliver
worker_1 | for queue in _lookup(exchange, routing_key):
worker_1 | File "/usr/lib/python3.6/site-packages/kombu/transport/redis.py", line 877, in _lookup
worker_1 | exchange, redis_key))
worker_1 | kombu.exceptions.InconsistencyError:
worker_1 | Cannot route message for exchange 'reply.celery.pidbox': Table empty or key no longer exists.
worker_1 | Probably the key ('_kombu.binding.reply.celery.pidbox') has been removed from the Redis database.
worker_1 |
worker_1 |
worker_1 | During handling of the above exception, another exception occurred:
worker_1 |
worker_1 | Traceback (most recent call last):
worker_1 | File "/usr/lib/python3.6/site-packages/celery/worker/pidbox.py", line 46, in on_message
worker_1 | self.node.handle_message(body, message)
worker_1 | File "/usr/lib/python3.6/site-packages/kombu/pidbox.py", line 145, in handle_message
worker_1 | return self.dispatch(**body)
worker_1 | File "/usr/lib/python3.6/site-packages/kombu/pidbox.py", line 115, in dispatch
worker_1 | ticket=ticket)
worker_1 | File "/usr/lib/python3.6/site-packages/kombu/pidbox.py", line 151, in reply
worker_1 | serializer=self.mailbox.serializer)
worker_1 | File "/usr/lib/python3.6/site-packages/kombu/pidbox.py", line 285, in _publish_reply
worker_1 | **opts
worker_1 | File "/usr/lib/python3.6/site-packages/kombu/messaging.py", line 181, in publish
worker_1 | exchange_name, declare,
worker_1 | File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 551, in _ensured
worker_1 | errback and errback(exc, 0)
worker_1 | File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
worker_1 | self.gen.throw(type, value, traceback)
worker_1 | File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 444, in _reraise_as_library_errors
worker_1 | sys.exc_info()[2])
worker_1 | File "/usr/lib/python3.6/site-packages/vine/five.py", line 194, in reraise
worker_1 | raise value.with_traceback(tb)
worker_1 | File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 439, in _reraise_as_library_errors
worker_1 | yield
worker_1 | File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 518, in _ensured
worker_1 | return fun(*args, **kwargs)
worker_1 | File "/usr/lib/python3.6/site-packages/kombu/messaging.py", line 203, in _publish
worker_1 | mandatory=mandatory, immediate=immediate,
worker_1 | File "/usr/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 605, in basic_publish
worker_1 | message, exchange, routing_key, **kwargs
worker_1 | File "/usr/lib/python3.6/site-packages/kombu/transport/virtual/exchange.py", line 70, in deliver
worker_1 | for queue in _lookup(exchange, routing_key):
worker_1 | File "/usr/lib/python3.6/site-packages/kombu/transport/redis.py", line 877, in _lookup
worker_1 | exchange, redis_key))
worker_1 | kombu.exceptions.OperationalError:
worker_1 | Cannot route message for exchange 'reply.celery.pidbox': Table empty or key no longer exists.
worker_1 | Probably the key ('_kombu.binding.reply.celery.pidbox') has been removed from the Redis database.
And the web interface can't be reached on port 5000. Any ideas ?
Thanks a lot :)
I ended up modifying a bit the docker-compose file, and now the web interface is accessible.
However, whenever I upload a file, I get the following error from the webApp :
web_1 | [I 200722 15:01:34 web:2064] 200 GET / (192.168.160.1) 2.49ms
plik_1 | [07/22/2020 15:01:39][INFO ][[192.168.160.4]] POST /upload
plik_1 | [07/22/2020 15:01:39][INFO ][[192.168.160.4]] POST /file/Tk3Z5RH5Vnu0uPGV/xwCStjzbXQlnRf4c/STDIN
web_1 | [E 200722 15:01:39 web:1591] Uncaught exception POST /v1/tasks (192.168.160.1)
web_1 | HTTPServerRequest(protocol='http', host='localhost:5000', method='POST', uri='/v1/tasks', version='HTTP/1.1', remote_ip='192.168.160.1', headers={'Host': 'localhost:5000', 'Connection': 'keep-alive', 'Content-Length': '172550', 'Accept': '*/*', 'X-Requested-With': 'XMLHttpRequest', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36', 'Content-Type': 'multipart/form-data; boundary=----WebKitFormBoundaryAEDdkpBolFa0ScZX', 'Origin': 'http://localhost:5000', 'Sec-Fetch-Site': 'same-origin', 'Sec-Fetch-Mode': 'cors', 'Sec-Fetch-Dest': 'empty', 'Referer': 'http://localhost:5000/', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'fr-FR,fr;q=0.9,en-GB;q=0.8,en;q=0.7,en-US;q=0.6'})
web_1 | Traceback (most recent call last):
web_1 | File "/usr/local/lib/python3.7/site-packages/kombu/utils/objects.py", line 42, in __get__
web_1 | return obj.__dict__[self.__name__]
web_1 | KeyError: 'backend'
web_1 |
web_1 | During handling of the above exception, another exception occurred:
web_1 |
web_1 | Traceback (most recent call last):
web_1 | File "/usr/local/lib/python3.7/site-packages/tornado/web.py", line 1512, in _execute
web_1 | result = yield result
web_1 | File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 1055, in run
web_1 | value = future.result()
web_1 | File "/usr/local/lib/python3.7/site-packages/tornado/concurrent.py", line 238, in result
web_1 | raise_exc_info(self._exc_info)
web_1 | File "<string>", line 4, in raise_exc_info
web_1 | File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 1063, in run
web_1 | yielded = self.gen.throw(*exc_info)
web_1 | File "/app/docbleach/controllers/upload.py", line 25, in post
web_1 | async_res = yield self.add_task(link, filename)
web_1 | File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 1055, in run
web_1 | value = future.result()
web_1 | File "/usr/local/lib/python3.7/site-packages/tornado/concurrent.py", line 238, in result
web_1 | raise_exc_info(self._exc_info)
web_1 | File "<string>", line 4, in raise_exc_info
web_1 | File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 292, in wrapper
web_1 | result = func(*args, **kwargs)
web_1 | File "/usr/local/lib/python3.7/types.py", line 277, in wrapped
web_1 | coro = func(*args, **kwargs)
web_1 | File "/app/docbleach/controllers/upload.py", line 53, in add_task
web_1 | kwargs={})
web_1 | File "/usr/local/lib/python3.7/site-packages/celery/app/base.py", line 736, in send_task
web_1 | self.backend.on_task_call(P, task_id)
web_1 | File "/usr/local/lib/python3.7/site-packages/kombu/utils/objects.py", line 44, in __get__
web_1 | value = obj.__dict__[self.__name__] = self.__get(obj)
web_1 | File "/usr/local/lib/python3.7/site-packages/celery/app/base.py", line 1182, in backend
web_1 | return self._get_backend()
web_1 | File "/usr/local/lib/python3.7/site-packages/celery/app/base.py", line 900, in _get_backend
web_1 | self.loader)
web_1 | File "/usr/local/lib/python3.7/site-packages/celery/app/backends.py", line 65, in by_url
web_1 | return by_name(backend, loader), url
web_1 | File "/usr/local/lib/python3.7/site-packages/celery/app/backends.py", line 45, in by_name
web_1 | cls = symbol_by_name(backend, aliases)
web_1 | File "/usr/local/lib/python3.7/site-packages/kombu/utils/imports.py", line 57, in symbol_by_name
web_1 | module = imp(module_name, package=package, **kwargs)
web_1 | File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
web_1 | return _bootstrap._gcd_import(name[level:], package, level)
web_1 | File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
web_1 | File "<frozen importlib._bootstrap>", line 983, in _find_and_load
web_1 | File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
web_1 | File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
web_1 | File "<frozen importlib._bootstrap_external>", line 724, in exec_module
web_1 | File "<frozen importlib._bootstrap_external>", line 860, in get_code
web_1 | File "<frozen importlib._bootstrap_external>", line 791, in source_to_code
web_1 | File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
web_1 | File "/usr/local/lib/python3.7/site-packages/celery/backends/redis.py", line 21
web_1 | from . import async
web_1 | ^
web_1 | SyntaxError: invalid syntax
web_1 | [E 200722 15:01:39 web:2064] 500 POST /v1/tasks (192.168.160.1) 170.28ms
I get below error - root@ubuntu:/docker-containers/DocBleach-Web# docker-compose up -d Building web Sending build context to Docker daemon 1.854MB [WARNING]: Empty continuation line found in: RUN apk add -t build --no-cache --update-cache ca-certificates openssl curl && update-ca-certificates && wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.23-r3/glibc-2.23-r3.apk && apk add glibc-2.23-r3.apk && apk del build wget unzip && adduser -S -u 1000 worker && chmod o+x /usr/bin/plik && echo 'URL = "https://plik.root.gg"' > /home/worker/.plikrc [WARNING]: Empty continuation lines will become errors in a future release. Step 1/13 : FROM python:3.7-alpine ---> accf77502477 Step 2/13 : EXPOSE 5000 ---> Using cache ---> 52879e55cfd8 Step 3/13 : HEALTHCHECK CMD curl --fail http://localhost:5000/ping version: '2.1'
services:
Redis
redis: image: redis
Our rest API
web: build: context: api dockerfile: Dockerfile image: docbleach/api:latest environment: - CELERY_BROKER=redis://redis/ - CELERY_RESULT_BACKEND=redis://redis/ - INTERNAL_PLIK_SERVER=${FINAL_PLIK_SERVER:-http://plik:8080} ports: - "${EXTERNAL_PORT:-9000}:5000" tmpfs: /tmp
Celery worker, the one you should be scaling using, ie to have 10 instances:
docker-compose scale worker=10
worker: build: context: worker dockerfile: Dockerfile image: docbleach/worker:latest environment: - CELERY_BROKER=redis://redis/ - CELERY_RESULT_BACKEND=redis://redis/ - FINAL_PLIK_SERVER=${FINAL_PLIK_SERVER:-https://plik.root.gg} tmpfs: /tmp
plik: image: rootgg/plik expose: - 8080 ~ ~ ~ ~ "docker-compose.yml" [noeol] 38L, 876B 38,1 All ---> Using cache ---> ea2c7f795115 Step 4/13 : ENV INTERNAL_PLIK_SERVER https://plik.root.gg ---> Using cache ---> ab399ca49a78 Step 5/13 : ADD https://plik.root.gg/clients/linux-amd64/plik /usr/bin/plik FROM openjdk:8-jre-alpine
ENTRYPOINT ["/app/entrypoint.sh"]
HEALTHCHECK CMD /usr/bin/celery inspect ping -A docbleach.tasks -d celery@$HOSTNAME
ENV FINAL_PLIK_SERVER https://plik.root.gg
Stored on Plik for now, as we don't yet have the GitHub Link
ENV DOCBLEACH_JAR https://github.com/docbleach/DocBleach/releases/download/v0.0.9/docbleach.jar
We add the Plik binary file
ADD https://plik.root.gg/clients/linux-amd64/plik /usr/bin/plik
RUN apk add --no-cache --update-cache
# Having up to date SSL certificates is always a good thing. :)
ca-certificates
openssl
python3
&&
update-ca-certificates && \
# Add glibc, wanted by plik (compiled with Go)
wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && \
wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.23-r3/glibc-2.23-r3.apk && \
apk add glibc-2.23-r3.apk && \
# Setup Python 3
python3 -m ensurepip && \
rm -r /usr/lib/python*/ensurepip && \
pip3 install --upgrade pip setuptools && \
rm -r /root/.cache && \
# Remove openssl, only needed for wget
apk del openssl wget unzip && \
# We add an user
adduser -S -u 1000 worker && \
# Add default config for Plik, required
chmod o+x /usr/bin/plik && \
echo 'URL = "https://plik.root.gg"' > /home/worker/.plikrc
"Dockerfile" 52L, 1604B 1,1 Top FROM openjdk:8-jre-alpine
ENTRYPOINT ["/app/entrypoint.sh"]
HEALTHCHECK CMD /usr/bin/celery inspect ping -A docbleach.tasks -d celery@$HOSTNAME
ENV FINAL_PLIK_SERVER https://plik.root.gg
Stored on Plik for now, as we don't yet have the GitHub Link
ENV DOCBLEACH_JAR https://github.com/docbleach/DocBleach/releases/download/v0.0.9/docbleach.jar Downloading [==================================================>] 7.643MB/7.643MB
---> Using cache ---> ec80eb0aa85a Step 6/13 : RUN apk add -t build --no-cache --update-cache ca-certificates openssl curl && update-ca-certificates && wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.23-r3/glibc-2.23-r3.apk && apk add glibc-2.23-r3.apk && apk del build wget unzip && adduser -S -u 1000 worker && chmod o+x /usr/bin/plik && echo 'URL = "https://plik.root.gg"' > /home/worker/.plikrc ---> Running in 59b6b2567c9a fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz (1/6) Installing openssl (1.1.1s-r0) (2/6) Installing brotli-libs (1.0.9-r6) (3/6) Installing nghttp2-libs (1.47.0-r0) (4/6) Installing libcurl (7.83.1-r4) (5/6) Installing curl (7.83.1-r4) (6/6) Installing build (20221118.094241) Executing busybox-1.35.0-r17.trigger OK: 16 MiB in 42 packages WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping Connecting to github.com (20.207.73.82:443) FROM python:3.7-alpine
EXPOSE 5000 HEALTHCHECK CMD curl --fail http://localhost:5000/ping
ENV INTERNAL_PLIK_SERVER https://plik.root.gg
We add the Plik binary file
ADD https://plik.root.gg/clients/linux-amd64/plik /usr/bin/plik
RUN apk add -t build --no-cache --update-cache
# Having up to date SSL certificates is always a good thing. :)
ca-certificates
openssl
# For the healthcheck
curl
&&
update-ca-certificates && \
# Add glibc, wanted by plik (compiled with Go)
wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && \
wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.23-r3/glibc-2.23-r3.apk && \
apk add glibc-2.23-r3.apk && \
# Remove openssl, only needed for wget
apk del build wget unzip && \
# We add an user
adduser -S -u 1000 worker && \
# Add default config for Plik, required
chmod o+x /usr/bin/plik && \
echo 'URL = "https://plik.root.gg"' > /home/worker/.plikrc
Install the python dependencies
ADD requirements.txt /app/ WORKDIR /app/
RUN pip3 install --no-cache-dir -r requirements.txt
COPY . /app/ "Dockerfile" 46L, 1279B 1,1 Top FROM python:3.7-alpine
EXPOSE 5000 HEALTHCHECK CMD curl --fail http://localhost:5000/ping Connecting to objects.githubusercontent.com (185.199.111.133:443) saving to 'glibc-2.23-r3.apk' glibc-2.23-r3.apk 100% |********************************| 2874k 0:00:00 ETA 'glibc-2.23-r3.apk' saved fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz (1/1) Installing glibc (2.23-r3) ERROR: glibc-2.23-r3: trying to overwrite etc/nsswitch.conf owned by alpine-baselayout-data-3.2.0-r23. 1 error; 20 MiB in 43 packages The command '/bin/sh -c apk add -t build --no-cache --update-cache ca-certificates openssl curl && update-ca-certificates && wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.23-r3/glibc-2.23-r3.apk && apk add glibc-2.23-r3.apk && apk del build wget unzip && adduser -S -u 1000 worker && chmod o+x /usr/bin/plik && echo 'URL = "https://plik.root.gg"' > /home/worker/.plikrc' returned a non-zero code: 1 ERROR: Service 'web' failed to build : Build failed
