DefectDojo
docker-compose up fails with execution permission denied
Be informative After following the docker installation instructions from github I am able to succesfully run the "docker-compose build" command, however the "docke-compose up" fails.
Bug description I am receiving a "standard_init_linux.go:228: exec user process caused: permission denied" error message while running the "docke-compose up" command.
Expected behavior No execution permission errors.
Deployment method (select with an X)
- [X] Docker Compose
- [ ] Kubernetes
- [ ] GoDojo
Environment information
- Operating System: Linux Centos8 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
- DefectDojo version (see footer) or commit message: #5869 from DefectDojo/release/2.7.1 [ (HEAD -> master, tag: 2.7.1, origin/master, origin/HEAD)]
Logs --==LOGS==-- mysql_1 | 2022-03-01T08:10:16.743633Z 0 [Note] Event Scheduler: Loaded 0 events mysql_1 | 2022-03-01T08:10:16.743893Z 0 [Note] mysqld: ready for connections. mysql_1 | Version: '5.7.37' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL) celerybeat_1 | standard_init_linux.go:228: exec user process caused: permission denied initializer_1 | standard_init_linux.go:228: exec user process caused: permission denied celeryworker_1 | standard_init_linux.go:228: exec user process caused: permission denied uwsgi_1 | standard_init_linux.go:228: exec user process caused: permission denied django-defectdojo_uwsgi_1 exited with code 1 django-defectdojo_celerybeat_1 exited with code 1 django-defectdojo_initializer_1 exited with code 1 django-defectdojo_celeryworker_1 exited with code 1 nginx_1 | Basic auth is off (HTTP_AUTH_PASSWORD not provided) nginx_1 | 2022/03/01 08:10:23 [emerg] 1#1: host not found in upstream "uwsgi:3031" in /run/defectdojo/uwsgi_server:1 nginx_1 | nginx: [emerg] host not found in upstream "uwsgi:3031" in /run/defectdojo/uwsgi_server:1 django-defectdojo_nginx_1 exited with code 1 rabbitmq_1 | 2022-03-01 08:10:25.909229+00:00 [info] <0.228.0> Feature flags: list of feature flags found: rabbitmq_1 | 2022-03-01 08:10:25.931627+00:00 [info] <0.228.0> Feature flags: [ ] implicit_default_bindings rabbitmq_1 | 2022-03-01 08:10:25.931679+00:00 [info] <0.228.0> Feature flags: [ ] maintenance_mode_status rabbitmq_1 | 2022-03-01 08:10:25.931707+00:00 [info] <0.228.0> Feature flags: [ ] quorum_queue rabbitmq_1 | 2022-03-01 08:10:25.931743+00:00 [info] <0.228.0> Feature flags: [ ] stream_queue rabbitmq_1 | 2022-03-01 08:10:25.931823+00:00 [info] <0.228.0> Feature flags: [ ] user_limits --==SNIP==--
Additional context (optional)
I noteiced that in the ouput of "mount | grep noexec" is the following: shm on /var/lib/docker/containers/42f3dff802d120a4c8b7d76bdaff72cd2ef6520cdd29a4a71f40a1344335a049/mounts/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k) shm on /var/lib/docker/containers/4c202c3d5387bf5f2ead05154bb6c87c167d1a93aedd610d60f4702ecde7ff13/mounts/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
I do not know why are these containers mounted as noexec, but I guess it is connected with the error above.
This is usually a local problem, especially with CentOS/RHEL with their security settings.
I agree with what Val said ^, here's some useful info: https://stackoverflow.com/questions/58298774/standard-init-linux-go211-exec-user-process-caused-exec-format-error
You could also see if this guide for CentOS 7 happens to work for CentOS 8: https://github.com/DefectDojo/Community-Contribs/tree/master/CentOS-7-deploy
Try to put selinux into permissive mode, look at the audit log then you can use Audit2allow to generate selinux policies.
I tried to completely disable selinux, just to see if selinux is the problem, but did not helped. Also set exec on tmpfs, /dev/shm type tmpfs (rw,nosuid,nodev) and the same for /var. No joy.
However, what is a bit suspicious for me, that mysql:5.7.37 and rabbitmq:3.9.13 are running after docker-compose up, but every other service, which originate from image: "defectdojo/defectdojo-django:${DJANGO_VERSION:-latest}", are not running and they throw the "exec user process caused: permission denied" error..
Can you try with 2.8.0? Docker compose setup has changed quite a bit in there. Also, which profile are you running? If you run in dev mode it will mount the local folder into the container which can cause permission problems.
Hi, I was having this same issue for a long time and pulled my hair out trying to fix it. In the end, I tried a few things:
- Re-mounting the partition dojo is running on with execute permission (lack of any execute permission means noexec in SE Linux iirc) -> did not work
- Making sure the docker temp directory was executable (mount as exec and also changing folder using env variable) -> did not work
- Making sure the dojo docker scripts were executable (chmod +x) -> did not work
And finally, the one that fixed it for me:
Setting the images to run as root in the docker-compose.yml file. (Specify
user: root)
Whilst I know this solution is not ideal, it was the only way I managed to fix it on RHEL 8. If any one has a better solution, I would love to hear it so that I can replicate in my own dojo instance.
I now occasionally get other permission errors similar to this, but they go away after re-building the containers.
Edit: The fixes suggested above worked on RHEL 7, but not on RHEL 8, whether that is due to server configuration changes or OS level changes, I don't know
I have in plan to take a look at this issue, running app as root is not good idea. Are you running standard docker?
Yes regular docker, I did originally try using podman but this caused other issues due to the image tags
Hi I tried to replicate your problems, but without success, on my side everything work as expected.
DD 2.8.0, images created from the source
[root@localhost ~]# cat /etc/redhat-release
CentOS Stream release 8
root@localhost ~]# docker --version
Docker version 20.10.12, build e91ed57
[root@localhost ~]# docker-compose --version
docker-compose version 1.29.2, build 5becea4c
[root@localhost ~]# sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Memory protection checking: actual (secure)
Max kernel policy version: 33
[root@localhost ~]# uname -a
Linux localhost.localdomain 4.18.0-365.el8.x86_64 #1 SMP Thu Feb 10 16:11:23 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
To install docker I was using https://docs.docker.com/engine/install/centos/
Can you drop same config param here, so we can compare it?
I threw everything away and made a new machine with a centos7 and with the latest docker, docker-compose and DD image. That config worked for me, without any problem. Probably v2.8.0 works (better) compared to v2.7.1 on Centos/RHEL.
Just tried with Defect Dojo on RHEL 8 and getting the same issue, completely fresh clone using mysql-rabbitmq profile:
Creating django-defectdojo_rabbitmq_1 ... done
Creating django-defectdojo_mysql_1 ... done
Creating django-defectdojo_uwsgi_1 ... done
Creating django-defectdojo_initializer_1 ... done
Creating django-defectdojo_celeryworker_1 ... done
Creating django-defectdojo_celerybeat_1 ... done
Creating django-defectdojo_nginx_1 ... done
Attaching to django-defectdojo_rabbitmq_1, django-defectdojo_mysql_1, django-defectdojo_initializer_1, django-defectdojo_celery beat_1, django-defectdojo_celeryworker_1, django-defectdojo_uwsgi_1, django-defectdojo_nginx_1
celerybeat_1 | standard_init_linux.go:228: exec user process caused: permission denied
mysql_1 | 2022-03-24 16:03:23+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.37-1debian10 started.
initializer_1 | standard_init_linux.go:228: exec user process caused: permission denied
mysql_1 | 2022-03-24 16:03:23+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
celeryworker_1 | standard_init_linux.go:228: exec user process caused: permission denied
mysql_1 | 2022-03-24 16:03:23+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.37-1debian10 started.
nginx_1 | Basic auth is off (HTTP_AUTH_PASSWORD not provided)
mysql_1 | 2022-03-24T16:03:23.986342Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --exp licit_defaults_for_timestamp server option (see documentation for more details).
mysql_1 | 2022-03-24T16:03:23.988074Z 0 [Note] mysqld (mysqld 5.7.37) starting as process 1 ...
django-defectdojo_celeryworker_1 exited with code 1
Configuration information:
[dojouser@redacted django-Defectdojo]$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 (Ootpa)
[dojouser@redacted django-Defectdojo]$ docker --version
Docker version 20.10.12, build e91ed57
[dojouser@redacted django-Defectdojo]$ sudo docker-compose --version
docker-compose version 1.29.2, build 5becea4c
[dojouser@redacted django-Defectdojo]$ sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Memory protection checking: actual (secure)
Max kernel policy version: 33
[dojouser@redacted django-Defectdojo]$ uname -a
Linux redacted 4.18.0-305.10.2.el8_4.x86_64 #1 SMP Mon Jul 12 04:43:18 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
If I use docker compose rather than docker-compose:
django-defectdojo-mysql-1 | Version: '5.7.37' socket: '/var/run/mysqld/mysqld.sock' port: 3306 MySQL Community Server (GPL)
django-defectdojo-initializer-1 | standard_init_linux.go:228: exec user process caused: permission denied
django-defectdojo-uwsgi-1 | standard_init_linux.go:228: exec user process caused: permission denied
django-defectdojo-celerybeat-1 | standard_init_linux.go:228: exec user process caused: permission denied
django-defectdojo-celeryworker-1 | standard_init_linux.go:228: exec user process caused: permission denied
django-defectdojo-initializer-1 exited with code 1
django-defectdojo-uwsgi-1 exited with code 1
django-defectdojo-celerybeat-1 exited with code 1
django-defectdojo-celeryworker-1 exited with code 1
I am facing the same issue with 2.9.0 version of DefectDojo in my RHEL 7.9 machine... Do we have any fix for this issue?
Or is there any alternative way of installation other than docker-compose?
Removed my temporary user: root fix in the docker compose file and re-built the images then tried running them, still getting the error on dojo 2.10.0
Now we are using docker compose v2.5.1 and docker engine 20.10.12
[dojo@server release]$ docker compose --profile mysql-rabbitmq --env-file /opt/defect-dojo/release/dojo/settings/.env.prod up --no-deps -d
[+] Running 1/5
⠿ Container release-rabbitmq-1 Started 0.6s
⠹ Container release-initializer-1 Starting 1.3s
⠹ Container release-uwsgi-1 Starting 1.3s
⠦ Container release-celeryworker-1 Starting 0.6s
⠦ Container release-celerybeat-1 Starting 0.6s
Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/wait-for-it.sh": permission denied: unknown
[dojo@server release]$ sudo docker compose --profile mysql-rabbitmq --env-file /opt/defect-dojo/release/dojo/settings/.env.prod up
[+] Running 1/0
⠿ Container release-rabbitmq-1 Running 0.0s
Attaching to release-celerybeat-1, release-celeryworker-1, release-initializer-1, release-nginx-1, release-rabbitmq-1, release-uwsgi-1
Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/wait-for-it.sh": permission denied: unknown
Okay, I seem to have found a solution!
In the docker compose file, set the value: user: ${CURRENT_UID:-defectdojo} for all images except nginx and the message broker which can work without this parameter. It has also fixed Celery for me (that container didn't like being run as root).
This has fixed the issue on my test and prod servers running RHEL 8.
Oracle Linux 8.7 (RHEL 8) up to date: user: ${CURRENT_UID:-defectdojo} - no success for me :( user: root - adding this for every service that have defined volumes type bind seems to be workaround in my case.
I believe that adding
chown ${appuser} /*.sh && \
after
chown ${appuser} /var/run/${appuser} && \
line in dockerfile.django-alpine and/or dockerfile.django-debian is the proper solution.
I believe that adding
chown ${appuser} /*.sh && \afterchown ${appuser} /var/run/${appuser} && \line in dockerfile.django-alpine and/or dockerfile.django-debian is the proper solution.
@WojTecH94 can we execute this exercise ones again, can you share your information about the environment like:
[root@localhost ~]# cat /etc/redhat-release
root@localhost ~]# docker --version
[root@localhost ~]# docker-compose --version
[root@localhost ~]# sestatus
Can you ensure that there are no indicators of how SeLinux is doing something behind? Please share your SELinux logs after the start of the app.
Red Hat Enterprise Linux release 8.7 (Ootpa) Docker version 20.10.23, build 7155243 Docker Compose version v2.15.1
The first thing I've checked was to try with selinux disabled... and it didn't help. Now I use it in permissive mode so it doesn't block anything, for me that is not the case. I'd rather say this might have something to do with rootless docker mode in which I run my containers. This is how execution files look like for me inside running container:
-rwxr-x---. 1 defectdojo root 951 Feb 13 12:18 entrypoint-celery-beat.sh -rwxr-x---. 1 defectdojo root 1.2K Feb 13 12:18 entrypoint-celery-worker.sh -rwxr-x---. 1 defectdojo root 3.7K Feb 13 12:18 entrypoint-initializer.sh -rwxr-x---. 1 defectdojo root 2.2K Feb 13 12:18 entrypoint-unit-tests-devDocker.sh -rwxr-x---. 1 defectdojo root 2.6K Feb 13 12:18 entrypoint-unit-tests.sh -rwxr-x---. 1 defectdojo root 1.2K Feb 13 12:18 entrypoint-uwsgi-dev.sh -rwxr-x---. 1 defectdojo root 1.5K Feb 13 12:18 entrypoint-uwsgi.sh -rwxr-x---. 1 defectdojo root 3.6K Feb 13 12:18 wait-for-it.sh
I've only changed owner, without it defectdojo user simply have no execution rights. If changing owner is so bad practice we can also add the "x" right for Others column and it will also do the trick (already tested it).
The first thing I've checked was to try with selinux disabled... and it didn't help. Now I use it in permissive mode so it doesn't block anything, for me that is not the case. I'd rather say this might have something to do with rootless docker mode in which I run my containers. This is how execution files look like for me inside running container:
This might fly.
-rwxrwxr-x 1 root root 951 Jan 30 14:25 entrypoint-celery-beat.sh
-rwxrwxr-x 1 root root 1228 Jan 30 14:25 entrypoint-celery-worker.sh
-rwxrwxr-x 1 root root 4469 Mar 1 17:16 entrypoint-initializer.sh
-rwxrwxr-x 1 root root 2220 Jan 30 14:25 entrypoint-unit-tests-devDocker.sh
-rwxrwxr-x 1 root root 2607 Jan 30 14:25 entrypoint-unit-tests.sh
-rwxrwxr-x 1 root root 1223 Jan 30 14:25 entrypoint-uwsgi-dev.sh
-rwxrwxr-x 1 root root 1447 Jan 30 14:25 entrypoint-uwsgi.sh
@WojTecH94 finally I have RHEL, but I can't replicate error even by running in the rootless mode.
[dsever@rhel8 ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux release 8.7 (Ootpa)
[dsever@rhel8 ~]$ docker --version
Docker version 23.0.1, build a5ee5b1
[dsever@rhel8 ~]$ docker-compose --version
Docker Compose version v2.15.1
Can you navigate me how to replicate the error?
When you clone git repo what are the permissions for the .sh files on your host server?
Really? That is your answer? So like I said earlier in running container I have permissions which cause permission denied error.
Pardon me: my containers are actually not running because of the permission denied error, which I can only fix by changing mentioned permissions.
Yes, as I said I want to know in details how you are running containers, I tried with Centos8 , RHEL 8, root and rootless and can't replicate the error. So I assume you have some specific usecase I'm missing to test.
FWIW, I have also been unable to replicate this issue. I agree with @dsever that it's hard to take a PR for an issue we can't replicate and doesn't appear to be impacting other users.