ZoneMinder
High CPU load (100%) and multiple zmc processes
Describe Your Environment
- Version of ZoneMinder: 1.37.11
- How you installed ZoneMinder: APT
- Full name and version of OS: Ubuntu 18.04.6 LTS
- Hardware: Intel Core i5-6300U, 16GB ram, 256GB SSD, performance governor
Describe the bug
Once Zonemidner is started it quickly rises to 100% CPU load. I have 7 cemeras all UDP rtsp streams. all running 1080. 'top' reveals that there are multiple zmc processes per camera, is this normal, I'd expect one capture process per camera, and not multiple. See screenshot: https://i.imgur.com/w3wupNR.png
To Reproduce Steps to reproduce the behavior:
- Start Zoneminder
- Enable Capturing on all cameras
Expected behavior lower CPU usage than 100%
Debug Logs
I'm a retard :) I was using htop - so if I sort by process, I only get one zmc process per camera.
however - I still have massive high cpu load. Historically load with all cameras enabled is about 80% max - but now I hammers up to 100%
I have the same behavior as yours. I was on zoneminder-master respository and recently upgrade to 1.37.12 and zmc processes have been hammering the CPU.
The load average was way over 20, and the server became unresponsive. It even reaching 90+!!! .
Usually, the load average would hover around 4-7.
Downgrading to 1.36.12 (zoneminder-proposed repository) fixed it.
I cannot use 1.37.12 at all - it fails to start - and I have not had time to investigate what is going on. 1.37.11 is the latest version that is working for me - but with massive CPU load.
Downgraded to 1.36.12, and CPU load went down ~20%- thanks @anonimou0
I'm now back at 75-80% load with all cameras enabled and analysis active.
The CPU usage was going crazy with zmc using ~most~ all CPU. At least, that's what top would show.
Before the issues started, streams would capture at a steady pace of ~20 frames. Below, we can see spikes to over 80 fps (???) and going down. Also, some errors. Hope logs for zmc on one stream helps:
This affects me as well - upgrade to 1.37.13 has zmc using massive amounts of cpu, and a load average of 70. Only difference I see in the logs are a bunch of errors w.r.t. shm
Mar 29 21:01:18 zm zmeventnotification[524]: ERR [Memory map file '/dev/shm/zm.mmap.7' does not exist in zmMemAttach. zmc might not be running.] Mar 29 21:01:18 zm zmwatch[513]: ERR [Memory map file '/dev/shm/zm.mmap.5' does not exist in zmMemAttach. zmc might not be running.] Mar 29 21:01:18 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a5b0) Guest] Mar 29 21:01:18 zm zmwatch[513]: ERR [Memory map file '/dev/shm/zm.mmap.6' does not exist in zmMemAttach. zmc might not be running.] Mar 29 21:01:19 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a610) Driveway-mon] Mar 29 21:01:19 zm zmwatch[513]: ERR [Memory map file '/dev/shm/zm.mmap.7' does not exist in zmMemAttach. zmc might not be running.] Mar 29 21:01:19 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a670) Garage-mon] Mar 29 21:01:19 zm zmwatch[513]: ERR [Memory map file '/dev/shm/zm.mmap.8' does not exist in zmMemAttach. zmc might not be running.] Mar 29 21:01:20 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a6d0) Garden-mon] Mar 29 21:01:20 zm zmwatch[513]: ERR [Memory map file '/dev/shm/zm.mmap.9' does not exist in zmMemAttach. zmc might not be running.]
and some of these as well:
Mar 29 21:01:31 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf83d418) Driveway] Mar 29 21:01:32 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf83d4f0) Garage] Mar 29 21:01:32 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86ff78) Garden] Mar 29 21:01:32 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf51a8d8) Pool] Mar 29 21:01:32 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86ff90) Dogdoor] Mar 29 21:01:32 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a5b0) Guest] Mar 29 21:01:33 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a610) Driveway-mon] Mar 29 21:01:33 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a670) Garage-mon] Mar 29 21:01:33 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a6d0) Garden-mon] Mar 29 21:01:34 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a730) Pool-mon] Mar 29 21:01:34 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a790) Dogdoor-mon] Mar 29 21:01:35 zm zmwatch[513]: ERR [No mem_key for ZoneMinder::Monitor=HASH(0x55b0bf86a7f0) Guest-mon]
This affects me as well - upgrade to 1.37.13 has zmc using massive amounts of cpu, and a load average of 70. Only difference I see in the logs are a bunch of errors w.r.t. shm
So I found the issue on my side at least - the issue was caused by the change in how "DecodingEnabled" is handled. See this commit: https://github.com/ZoneMinder/zoneminder/commit/abf238a4696b6fd26c6d9b51a9a606f11cb37640
After changing the relevant channels to have "Decoding = None" everything seems to be back to normal. Apologies for the noise
Thank you @klaasvakie . That seems to be the issue.
I have 16 cameras that record without decoding in H265 in high resolution (4MP or 6MP). I use the cameras sub-streams to view these cameras on lower resolutions, which, then, has a decoding being used.
After upgrading to 1.37.11+, the Decoding option (on Source tab) comes set to Always. Re-setting it to None made the system usable again.
Currently running 1.37.13
You may be interested in the new Decoding options : Keyframes Only and On Demand. KeyFrames Only offers almost all the benefits of None, but with thumbnails and depending on keyframe interval, a watchable live stream.
Had the exact same issue upgarding to 1.36.19 lately. multiple zmc processes on all monitors (all in Mocord) and couldn't figure out the issue as none of them have the option 'Decoding Enabled'
'
Checking the database/Monitors table, all of them had 'DecodingEnabled' in True... Setting them back to 0 directly in the database did the trick. I would have had (I guess) the same result switching them to Record, changing the Decoding Enabled to unticked and then back to Mocord but anyway...
Leaving it here as it might help others
Same issue v1.36.19. Couldn't change Run States or stop and start Zoneminder from terminal. High CPU. Solution, change to Record for each Camera and untick Decoding Enabled for each camera. I then saved to a new run state. From that point I could change between my existing Run States.
Log was filled with Memory map file '/dev/shm/zm.mmap. does not exist in zmMemAttach.
@mfauvain you can't do motion detection without decoding, which is why you weren't seeing the option.
Closing since this is a simple issue of not enough cpu for the resolution.
quote: "Closing since this is a simple issue of not enough cpu for the resolution."
That is not the case - running an older (1.36.x) version with the exact same config gives me much lower CPU utilisation. So how you think this can be due to "not enough cpu" is a mystery to me. (Cpu is: Intel(R) Core(TM) i5-6300U CPU, running @ 3GHz, should be more than enough)
You'd think so, wouldn't you? The issue is less about the power of the cpu, and the design of the code.
The newer version is more correct, but less efficient (there is 1 more image copy than in the past).
The latest 1.37.33 has removed 1 image copy. You may find that it is enough for things to work.
Long term, we can't operate the way we used to. The code was designed for 640x480, not 4k res. The cameras keep getting higher res faster than we can keep up.
Ahh, I'll try out the 1.37.33+ then - see if it works.
So what needs to be changed for higher resolutions - is it a complete rewrite of the code?
Not a complete rewrite. Currently: Step 1 get h265 packet Step 2 decode to yuyv420p Step 3 convert to rgba Step 4 copy to /dev/shm
We need to eliminate Step 3, and do motion detection on the y-channel. The copy to /dev/shm is now smaller as well. We then need code to generate a jpeg from yuv420p for live view, etc.
Eventually maybe we can get rid of the copy to /dev/shm.