cloudstack icon indicating copy to clipboard operation
cloudstack copied to clipboard

CS 4.20.1 High KVM Host Memory consumption 52%

Open tatay188 opened this issue 8 months ago • 3 comments

problem

I accounted for all memory, of turned off VM's and online VM, VR, and SystemVM memory, should be 104GB RAM why is using 256GB more - It shows 384GB RAM being used of the 768GB RAM

This problem did not exist on 4.20.0

Image

HTOP is showing QEMU using 10GB per any transaction

Image

Image

~# free total used free shared buff/cache available Mem: 792484040 416579128 374594724 6220 1310188 373661560 Swap: 67108860 0 67108860

versions

Upgraded to 4.20.1 Primary storage CEPH Secondary Storage EMC NFS Guest VXLAN Public VLAN all isolated networks.

The steps to reproduce the bug

  1. Upgrade from 4.20.0 to 4.20.1
  2. reboot Hypervisor
  3. Restart SystemVMs

Utilization jumps immediately to 384GB+ ...

What to do about it?

Memory consumption should be close what is being allocated, and reserved should show the sum of all systems running on the server.

tatay188 avatar Jun 17 '25 16:06 tatay188

What's the service offering of VR and system VMs ?

weizhouapache avatar Jun 17 '25 18:06 weizhouapache

Hey @weizhouapache Thank you for checking. Let me know what else you need, logs again shows no anomalies.

For the Vrouters:

Description System Offering 12vCPU, 12GB RAM 5Gps System VM type domainrouter Provisioning type thin Storage type shared Custom disk size false CPU cap false CPU cores 12 CPU (in MHz) 2000 Memory 12288 Created 30 May 2025 23:59:12 Dynamic scaling enabled true Disk offering strictness false

for the System VMs Proxy: CPVM ID ccefd4fe-87d6-4bb4-8d3f Description System Offering For CPVM 8vCPU 8GB RAM System VM type consoleproxy Provisioning type thin Storage type shared Custom disk size false CPU cap false CPU cores 8 CPU (in MHz) 1500 Memory 8192 Created 30 May 2025 23:21:14 Dynamic scaling enabled true Disk offering strictness false

SSVM

SSVM ID 64b0ca58-fda9-4b91-b4dd- Description System Offering For SSVM 4vCPU 8GB RAM System VM type secondarystoragevm Provisioning type thin Storage type shared Custom disk size false CPU cap false CPU cores 4 CPU (in MHz) 1500 Memory 8192 Created 30 May 2025 23:27:52 Dynamic scaling enabled true Disk offering strictness false

On Jun 17, 2025, at 2:09 PM, Wei Zhou @.***> wrote:

weizhouapache left a comment (apache/cloudstack#11050) https://github.com/apache/cloudstack/issues/11050#issuecomment-2981342036 What's the service offering of VR and system VMs ?

— Reply to this email directly, view it on GitHub https://github.com/apache/cloudstack/issues/11050#issuecomment-2981342036, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACNEALJQNRMJ6CLCWMFGKRT3EBKULAVCNFSM6AAAAAB7QWFMUOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSOBRGM2DEMBTGY. You are receiving this because you authored the thread.

tatay188 avatar Jun 17 '25 18:06 tatay188

I added All requirements even those that are no in use. And is 104GB RAM including the VMs, but HTOP shows 14GB for each event not for the VM.

Image

Image

Thank you

tatay188 avatar Jun 17 '25 18:06 tatay188

@tatay188 I quickly calculate the memory usage based on the output of free command. the usage is around 53% (416579128/792484040 = 52.56%) which seems correct

~# free
total used free shared buff/cache available
Mem: 792484040 416579128 374594724 6220 1310188 373661560
Swap: 67108860 0 67108860

do you still face the issue ? have you found which threads consumed the memory ?

weizhouapache avatar Sep 08 '25 12:09 weizhouapache