docker
Operations hang after starting system prune
I ran docker system prune to clear out old dev images. That appears to have hung with no feedback: commands like docker ps hang and there's no UI indicating what might be the problem. There were periods where Hyperkit was 140% CPU but the running average is close to idle and disk I/O has fluctuated between modest (tens of operations per second) to idle.
It's possible that everything is working correctly but it'd be nice to know whether that's the case and whether I should just leave it running (after about 10 minutes) or if restarting it will be required.
Information
- 82040D1C-9491-46F7-A12E-B917B287132F
Docker for Mac: version: 17.12.0-ce-mac49 (d1778b704353fa5b79142a2055a2c11c8b48a653)
macOS: version 10.13.3 (build: 17D47)
logs: /tmp/82040D1C-9491-46F7-A12E-B917B287132F/20180124-132812.tar.gz
failure: docker ps failed: (Failure "docker ps: timeout after 10.00s")
[OK] db.git
[OK] vmnetd
[OK] dns
[OK] driver.amd64-linux
[OK] virtualization VT-X
[OK] app
[OK] moby
[OK] system
[OK] moby-syslog
[OK] kubernetes
[OK] env
[OK] virtualization kern.hv_support
[OK] slirp
[OK] osxfs
[OK] moby-console
[OK] logs
[ERROR] docker-cli
docker ps failed
[OK] menubar
[OK] disk
Steps to reproduce the behavior
-
docker system prune -
docker ps
I tried to do a factory reset but that also hung. Killing the Docker process caused it to restart with what appears to be a clean install. Again, no UI feedback on what the last step was – the progress bar was over on the 100% side but it stayed there without feedback for at least 10 minutes.
I also experienced this behavior. Attempting to use Diagnose & Feedback while docker system prune was running hung as well. Eventually, docker system prune returned at which point the diagnose and feedback results (below) were immediately output.
(Note this was my second attempt at docker system prune -- the first one I gave up on and ended up having to restart Docker)
Docker for Mac: version: 17.12.0-ce-mac55 (18467c0ae7afb7a736e304f991ccc1a61d67a4ab)
macOS: version 10.13.3 (build: 17D102)
logs: /tmp/F4061EC1-D38E-44DE-8F3E-50BE0111AF00/20180309-103211.tar.gz
[OK] vpnkit
[OK] vmnetd
[OK] dns
[OK] driver.amd64-linux
[OK] app
[OK] virtualization VT-X
[OK] moby
[OK] system
[OK] moby-syslog
[OK] kubernetes
[OK] env
[OK] virtualization kern.hv_support
[OK] moby-console
[OK] osxfs
[OK] logs
[OK] docker-cli
[OK] disk
Fyi, mine looked like it was hanging for a while -- I googled around for answers, found this, waited some more, and it finally completed about 20-30 minutes after I initiated the command.
A lot of the material I've read indicates one must be very careful using Docker prune. In order to remove unused images "The client and daemon API must both be at least 1.25 to use this command." And, there are Docker prune options that might be a better option to Docker prune, i.e., --all , -a (for removing all unused images, not just dangling ones. This is significant because by default, Docker image prune cleans up dangling images only.
MacOS HighSierra (Docker version 18.03.1-ce, build 9ee9f40):
-
I run
docker imagesand see 37 images with repository and 52 with<none>repository (so called "dingling" or something...) - total 89 images, average image size: 500 MB -
To clean this mess up I run
docker system prune -aafter 2h of working (without any console message written by docker), I saw thatHyperkitread 9GB and write 8GB, CPU=99% allmost all time. -
After this 2 hours (all time I was unable to run
docker psordocker images) I decide to pushctrl+zbutpsandimagesstill doesn't work - so after that I restart docker (by its icon menu at the MacOs top bar). -
Then I write
docker imagesand I saw that 90% of my images was deleted (what was great, but why so long ! :/ ) last 9 images i delete bydocker rmi xxxxx
My proposition: lets make docker system prune write some feedback like % or something during execution... .
Currently I think it is good idea to break docker system prune... (and reset) after each hour and check status using docker images
Isn't it great when you're stuck waiting at the command line, wondering if this thing has become broken again? Struggling to understand how traversing a tree and deleting some files could be taking so long... Then you check your running processes and wonder to yourself...if the lost soul who wrote this thing couldn't be bothered to output some status text to the user, did he correctly implement signals to clean up and not leave itself in an inconsistent state on restart? The world will never know.
This is awful. I've wasted half a day waiting for it to complete. I eventually cancelled it so hopefully docker will clean up ok.
ran into this today too, system prune and image prune both resulted in the same behavior
If you have lots of untagged images, run the following command:
docker rmi $(docker images -f "dangling=true"-q)
There could potentially be lots of these images, and when you run:
docker system prune
It doesn't show the images as they are being untagged and deleted, so it seems like the docker command is hanging.
Just an update to @briancaffey's solution...
Sounds silly, but for me things "seemed" to work better, by copying and pasting a list of docker rmi commands into my console.
I used this to generate the list:
docker images -f "dangling=true" -q | xargs -I {} echo docker rmi -f {}
(I got visual feedback on progress, I could see which image was holding things up, etc etc)
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale comment.
Stale issues will be closed after an additional 30d of inactivity.
Prevent issues from auto-closing with an /lifecycle frozen comment.
If this issue is safe to close now please do so.
Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale
Just wanted to report that I ran into this same problem yesterday and abandoned docker system prune with CTRL + C after letting it run for about an hour with no feedback. This somehow corrupted Docker on my entire system to the point where the Docker application would not open or launch at all.
I spent the good part of a day trying to get Docker to work again; completely uninstalling Docker and reinstalling, restarting, etc. I eventually deleted all the containers and manually removed every file related to Docker and only then did a fresh install get Docker working again.
For me, this is just really, really slow (running for hours) with no user feedback of whether it's stuck or not. If I run docker system df periodically while the pruning is happening, I see available space slowly increasing. This command also takes a long time to complete while pruning is running.
It would be really helpful if it logged some more output, so that I know it's not stuck.
Same issue. It seems to me that it's pretty absurd how long this takes. I have to kill and restart often, often with very few images on my system. Sometimes I will have to do a docker reinstall because it can't recover. I tend to run prune very infrequently because I know its going to take a long time. There is no way to tell if it is stuck or just slow. As far as I can tell, it does get stuck sometimes.
@gokcan read this https://github.com/docker/for-mac/issues/2501#issuecomment-460875712
@kamil-kielczewski Is this issue under active development? The frozen label makes it a bit confusing.
@gokcan - I don't know (I not develop docker) - I only read comment #2501 and made frozen to prevent colse
Same issue on 18.09. It already mentioned having deleted containers and networks. No idea what it is doing now, if anything.
I have some additional information based on more recent observations. Based on a stack overflow answer, I have been using the following script in stead of the actual prune command:
echo "Pruning containers"
docker rm $(docker ps -f status=exited -aq)
echo "Pruning images"
docker rmi $(docker images -f "dangling=true" -q)
echo "Pruning volumes"
docker volume rm $(docker volume ls -qf dangling=true)
This script will usually complete for me. However, there seem to be a few issues with this approach:
-
At first pruning images starts of fast. After a few dozen or so it starts to slow down. It goes in little spurts from then on out, with a pretty slow delete rate overall.
-
It pegs my CPU at 800% (I have docker configured to be able to use all 8 of my cores and 10GB of my memory. I develop containers that do hefty lifting and want to maximize resources available to docker).
-
When it eventually finishes, docker stays pegged at 800%. Containers continue to function as normal (if a bit slow) but so far it has always become unresponsive after a short while. I have to do a full stop/start of docker to continue. The stop/start process can take a long time. Approximately 20-30% of the time I have to force kill the
com.docker.hyperkitprocess to get it to shut down.
My guess is that the cause of these behaviors is the same as the prune command. Running any of these script or the prune command is the only time in the normal course of my development that docker becomes unresponsive. I've been running docker on my Mac for over 2 years and it seems pretty consistent behavior across all versions I use. I tend to keep docker on the current stable release and have 18.09.2 installed right now.
Hopefully this information is helpful.
Just add --force after prune command and the action will be over before you know
docker image prune -a --force
Experienced the same issue like @Vitiell0. Tried docker system prune --all, no feedback for a long time, terminated the task. Restarted it without --all to see whether it could at least remove something successfully. Hanging again. Tried restarting docker but the whole application hung. Terminated all Docker processes, and tried starting Docker anew but this time the daemon doesn't even start (no icon shows up in top bar on Mac).
I guess it's back to nuking docker from orbit like in the good ole' days ~2 years ago (removing the image file) where this was the only thing that actually guaranteed to get things working without 2h+ debugging. My Docker has access to 3 cores and 6GB RAM.
~Edit: Looks like Docker.qcow2 doesn't exist anymore, so it seems necessary to remove and reinstall Docker nowadays instead.~ Was wrong about this one, for me the file simply wasn't indexed by locate
Edit2: Uninstalling and installing Docker again didn't help. I also needed to purge all Docker files for daemon to start again. Boils down to removing the following directories:
~/Library/Application Scripts/com.docker.helper
~/Library/Caches/com.docker.docker
~/Library/Containers/com.docker.docker
~/Library/Containers/com.docker.helper
I've run into this in the past and just ran into it now after running docker system prune.
Surprised this is not being addressed especially when the following just works:
docker rmi $(docker images -q)
macOS 10.14.6 (18G103) and Docker Desktop 2.1.0.3 (38240) (Engine: 19.03.2).
I ran docker container prune -f, Docker started eating lots of CPU, nothing happened for hours, but I thought I would give it some time. Other docker commands would just hang. After 21h 20m I did ^C. Activity Monitor still reported the com.docker.supervisor process eating ~300% CPU and the Docker process to be "Not Responding". Also the Docker menu bar icon started beachballing (when I hoovered over it).
I force-quit the Docker process, and started Docker. Now docker commands works again. The stopped containers have not been removed.
