for-linux cgroups: cannot found cgroup mount destination: unknown

[x] This is a bug report
[ ] This is a feature request
[x] I searched existing issues before opening this one

Expected behavior

docker run hello-world

should run normally.

Actual behavior

docker run hello-world
docker: Error response from daemon: cgroups: cannot found cgroup mount destination: unknown.

Steps to reproduce the behavior

Install latest stable or edge builds (17.12 or 18.01), using: https://docs.docker.com/install/linux/docker-ce/ubuntu/

Output of docker version:

Client:                                    
 Version:       18.01.0-ce                 
 API version:   1.35                       
 Go version:    go1.9.2                    
 Git commit:    03596f5                    
 Built: Wed Jan 10 20:11:05 2018           
 OS/Arch:       linux/amd64                
 Experimental:  false                      
 Orchestrator:  swarm                      
                                           
Server:                                    
 Engine:                                   
  Version:      18.01.0-ce                 
  API version:  1.35 (minimum version 1.12)
  Go version:   go1.9.2                    
  Git commit:   03596f5                    
  Built:        Wed Jan 10 20:09:37 2018   
  OS/Arch:      linux/amd64                
  Experimental: false

Output of docker info:

Containers: 16                                                                    
 Running: 0                                                                       
 Paused: 0                                                                        
 Stopped: 16                                                                      
Images: 1                                                                         
Server Version: 18.01.0-ce                                                        
Storage Driver: overlay2                                                          
 Backing Filesystem: extfs                                                        
 Supports d_type: true                                                            
 Native Overlay Diff: true                                                        
Logging Driver: json-file                                                         
Cgroup Driver: cgroupfs                                                           
Plugins:                                                                          
 Volume: local                                                                    
 Network: bridge host macvlan null overlay                                        
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog    
Swarm: inactive                                                                   
Runtimes: runc                                                                    
Default Runtime: runc                                                             
Init Binary: docker-init                                                          
containerd version: 89623f28b87a6004d4b785663257362d1658a729                      
runc version: b2567b37d7b75eb4cf325b77297b140ea686ce8f                            
init version: 949e6fa                                                             
Security Options:                                                                 
 seccomp                                                                          
  Profile: default                                                                
Kernel Version: 4.4.0                                                             
Operating System: Ubuntu 16.04.3 LTS                                              
OSType: linux                                                                     
Architecture: x86_64                                                              
CPUs: 1                                                                           
Total Memory: 512MiB                                                              
Name: SRV_01                                                                      
ID: BBRL:P36V:7QOO:IRBR:X2TC:UC47:4L7N:IIIQ:OLSL:KWS2:OVAZ:TKKF                   
Docker Root Dir: /var/lib/docker                                                  
Debug Mode (client): false                                                        
Debug Mode (server): false                                                        
Registry: https://index.docker.io/v1/                                             
Labels:                                                                           
Experimental: false                                                               
Insecure Registries:                                                              
 127.0.0.0/8                                                                      
Live Restore Enabled: false                                                       
                                                                                  
WARNING: No cpuset support                                                        
WARNING: bridge-nf-call-iptables is disabled                                      
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details (AWS, VirtualBox, physical, etc.)

> uname -a

Linux DAAA_S1 4.4.0 #1 SMP Fri Jan 5 21:20:16 MSK 2018 x86_64 x86_64 x86_64 GNU/Linux

> ls /sys/fs/cgroup/
beancounter  cpu          cpuacct  freezer  memory   net_cls,net_prio  perf_event  systemd
blkio        cpu,cpuacct  devices  hugetlb  net_cls  net_prio          pids        ve

AppArmor is also deactivated. Running under root terminal/normal terminal returns same issue. Docker pull is working. Confirmed working under 17.09.1~ce-0~ubuntu

Feb 05 '18 03:02 xgenvn

Hi @xgenvn, have you fixed that issue? I'm at the same point but it occurs when docker is ran by a pipeline using gitlab ci/cd... Not sure if in my case it is a docker or a gitlab related issue

Feb 13 '18 12:02 adriantunez

@AdrianAntunez I installed specific version: 17.09.1~ce-0~ubuntu and it's working fine now.

Feb 13 '18 12:02 xgenvn

@xgenvn I will give it a try ASAP and I will let you know if it worked for me or not. Thx!!

Feb 15 '18 12:02 adriantunez

@AdrianAntunez ran into the same issue, it is working with this configuration for me:

image: docker:17.09

services: - docker:17.09-dind

Mar 17 '18 01:03 philippbussche

Same issue here, I'm running:

Version: 17.12.1-ce API version: 1.35 Go version: go1.9.4 Git commit: 7390fc6 Built: Tue Feb 27 22:17:40 2018 OS/Arch: linux/amd64

Mar 17 '18 22:03 hectornevarez

@philippbussche hey philip, so only the docker was a problem is it? Am also facing the same problem

My version

`Client: Version: 17.12.0-ce API version: 1.35 Go version: go1.9.2 Git commit: c97c6d6 Built: Wed Dec 27 20:03:51 2017 OS/Arch: darwin/amd64

Server: Engine: Version: 17.12.0-ce API version: 1.35 (minimum version 1.12) Go version: go1.9.2 Git commit: c97c6d6 Built: Wed Dec 27 20:12:29 2017 OS/Arch: linux/amd64 Experimental: true `

For me Docker Hello World is working without a issue though.

Mar 20 '18 16:03 jkbaseer

I ran into similar issue found a temp fix on another place. Temp fix: sudo mkdir /sys/fs/cgroup/systemd sudo mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd

Mar 22 '18 02:03 dinar-dalvi

Hello @dinar-dalvi thanks a lot! Do i need to do this on the route directory of the Server? or in the app folder(for example: meteor/hello-app/

Mar 22 '18 05:03 jkbaseer

@dinar-dalvi I tried this before, however the systemd is existed and mounted. Downgrading seems to fix the issue.

Mar 22 '18 06:03 xgenvn

@xgenvn: downgrading the container to which version?

Mar 22 '18 06:03 jkbaseer

@jkbaseer you might want to try

sudo apt remove docker-ce
sudo apt install docker-ce=17.09.1~ce-0~ubuntu

Mar 22 '18 07:03 xgenvn

Am i doing it rightly?

(Ps: Also i have few containers am using for other project, will that be removed as well? or is there a single command only to downgrade? )

Mar 22 '18 07:03 jkbaseer

On devuan/ascii this worked (over 18.03):

sudo apt-get install docker-ce=17.09.1~ce-0~debian

Mar 28 '18 19:03 d-a-v

@jkbaseer no, I was speaking of Ubuntu command line, I'm not sure on Mac.

Mar 29 '18 01:03 xgenvn

Hey Guys, Sorry I gave up of this trying even after installing latest Ubuntu ce. just created a new VM and used my container. And the app is back thankfully :)

It took so much to solve this Cpgroup. Appreciate all your help!

On 29 Mar 2018, at 9:23 am, Anh Tu Nguyen [email protected] wrote:

@jkbaseer no, I was speaking of Ubuntu command line, I'm not sure on Mac.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Mar 29 '18 02:03 jkbaseer

In my case the runner was host in AWS, I tried everything and nothing worked so I decided to change the AMI (it's the virtual image which includes the SO). Swaping from amazon linux v1 to amazon linux v2 apparently solved my issue. Anyway I don't know why :confused:

Mar 29 '18 07:03 adriantunez

I'm facing this too with Docker version 18.03.0-ce, build 0520e24 Solved by regressing to previous version of docker ce, as mentioned by @xgenvn

Mar 30 '18 07:03 vehsamrak

I've just tried updating to the newest version again and it seems fixed for me in Docker version 18.03.1-ce, build 9ee9f40

Jun 03 '18 16:06 lukkigi

Still facing this issue with 18.03.1~ce-0~ubuntu.

Jun 21 '18 23:06 iwan933

Ran into this as well running under root terminal/normal terminal and using 18.03.1-ce. Confirmed works when downgrade to 17.09.1~ce-0~ubuntu

> docker version

Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:17:20 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:15:30 2018
  OS/Arch:      linux/amd64
  Experimental: false

> docker info

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 1
Server Version: 18.03.1-ce
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.12.14-aufs
Operating System: Ubuntu 16.04 LTS (containerized)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 62.88GiB
Name: antlet13
ID: KJZ3:L22I:HNCT:7O2X:VF6S:TRVG:ZCTT:IPR7:BHZR:AB6I:MYUY:KIBJ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

> uname -a

Linux antlet13 4.12.14-aufs #7 SMP Tue Jan 9 00:08:41 -00 2018 x86_64 x86_64 x86_64 GNU/Linux

> ls /sys/fs/cgroup/

blkio  cpu  cpuacct  cpu,cpuacct  cpuset  devices  freezer  hugetlb  memory  net_cls  net_cls,net_prio  perf_event  pids  systemd

Jun 25 '18 00:06 notmessenger

So basically, any image that expects systemd sysfs entries fails to start with this error correct, if you are running on a non-systemd based Linux system? Would others here concur with that statement?

Jul 12 '18 22:07 CpuID

@CpuID how do I know "the image expects systemd sysfs entries" ?

Jul 16 '18 01:07 xgenvn

@CpuID how do I know "the image expects systemd sysfs entries" ?

A difficult assumption to make ahead of time. The answer might need to be Docker passing through a mocked set of sysfs entries to the container sysfs, when the host OS doesn't have them (due to a lack of systemd)? Not 100% sure if thats wise, or might lead to more confusion... We hit the issue starting the official Elasticsearch Docker images, which I don't believe actually use systemd, but maybe they call something that expects systemd to be configured at least? shrug

Jul 16 '18 01:07 CpuID

got this during a docker-compose up for concourse's dev compose config, the concourse-db (postgres image), running inside docker-machine on a mac with virtualbox

it works again once I completely destroy the docker machine and re-create it

Creating network "local-concourse_default" with the default driver
Creating local-concourse_concourse-db_1 ...
Creating local-concourse_concourse-db_1 ... error

ERROR: for local-concourse_concourse-db_1  Cannot start service concourse-db: cgroups: cannot find cgroup mount destination: unknown

ERROR: for concourse-db  Cannot start service concourse-db: cgroups: cannot find cgroup mount destination: unknown
Encountered errors while bringing up the project.

docker version

Client:
 Version:           18.06.0-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        0ffa825
 Built:             Wed Jul 18 19:04:39 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.0-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       0ffa825
  Built:            Wed Jul 18 19:13:39 2018
  OS/Arch:          linux/amd64
  Experimental:     false

docker info

Containers: 2
 Running: 1
 Paused: 0
 Stopped: 1
Images: 3
Server Version: 18.06.0-ce
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 31
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.93-boot2docker
Operating System: Boot2Docker 18.06.0-ce (TCL 8.2.1); HEAD : 1f40eb2 - Thu Jul 19 18:48:09 UTC 2018
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.858GiB
Name: snap-dev
ID: S3SW:XQYT:MUTK:OZ45:HKRO:J65H:F6WE:BDR3:5TPD:YECG:QJDU:2GKZ
Docker Root Dir: /mnt/sda1/var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
 provider=virtualbox
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

ls /sys/fs/cgroup/

blkio/      cpuacct/    devices/    hugetlb/    net_cls/    perf_event/
cpu/        cpuset/     freezer/    memory/     net_prio/   pids/

uname -a

Linux snap-dev 4.9.93-boot2docker #1 SMP Thu Jul 19 18:29:50 UTC 2018 x86_64 GNU/Linux

Jul 25 '18 18:07 eedwards-sk

I too can confirm downgrading to 17.09.1~ce-0~ubuntu solves this issue for me. Every newer Docker CE version from the stable channel version produces the error message.

I'm running Docker inside an OpenVZ container (need to deal with it sadly) so this might contribute to the problem. The cgroup filesystems seem to be mounted but I don't trust a thing that this OpenVZ container says anymore.

I'm omitting information about the machine because it's close to what others have posted already, apart from the virtualisation maybe. Let me know if you need further information, I'll add it then.

Aug 04 '18 20:08 lenn4rd

We having the same issue with Docker version 18.03.1-ce running on AWS ECS with Concourse in docker.

Aug 28 '18 07:08 kgrodzicki

@kgrodzicki me too.

In ecs. Task(is mere DockerContainer) repeat starting and ending over many times.

the cause that Task stop is that CannotStartContainerError: API error (500): cgroups: cannot find cgroup mount destination: unknown

Sep 12 '18 03:09 diggymo

I'm running Docker inside LXC containers.

Ubuntu 18.04 LXC container can run docker (Docker version 18.06.1-ce, build e68fc7a) without problems.
Alpine 3.7 LXC container can not run docker (Docker version 18.06.1-ce, build 3b19bc6ba8).

My output:

Building app
Step 1/7 : FROM node:8
 ---> 6f62c0cdc461
Step 2/7 : WORKDIR /usr/src/app
 ---> Using cache
 ---> 66cd8fd0bcfb
Step 3/7 : COPY package*.json ./
 ---> Using cache
 ---> d7dce2de9ad6
Step 4/7 : RUN npm install
 ---> Running in 067b8a8620d6
ERROR: Service 'app' failed to build: cgroups: cannot find cgroup mount destination: unknown

Both containers were launch with security options (-c security.nesting=true -c security.privileged=true)

Sep 12 '18 10:09 MarcosRZ

We have same issue with ecs ami linux v2

Sep 23 '18 21:09 dmitryawsconsulting

same issue here on Void Linux (uses runit not systemd).

$ uname -a
Linux bradbury 4.18.9_1 #1 SMP PREEMPT Thu Sep 20 05:49:31 UTC 2018 x86_64 GNU/Linux

$ docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.11
 Git commit:        
 Built:             Thu Aug 30 08:12:28 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.11
  Git commit:       v18.06.1-ce
  Built:            Thu Aug 30 08:12:28 2018
  OS/Arch:          linux/amd64
  Experimental:     false

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
d1725b59e92d: Pull complete 
Digest: sha256:0add3ace90ecb4adbf7777e9aacf18357296e799f81cabc9fde470971e499788
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: cgroups: cannot find cgroup mount destination: unknown.
ERRO[0001] error waiting for container: context canceled

Sep 25 '18 17:09 go2null