VR failing health checks on certain checks 4.18

[Chih80]

ISSUE TYPE

• Bug Report

COMPONENT NAME

VR, UI

CLOUDSTACK VERSION

4.18

CONFIGURATION

advanced networking

OS / ENVIRONMENT

CS management server running on ubuntu 22.04 and hosts running on ubuntu 20.04

SUMMARY

I keep getting ROUTER.HEALTH.CHECKS indicating "has failing checks: dns_check.py, dhcp_check.py". Everything seems to be working. My VMs are online so not sure what to make of it. I'm not sure if I should just exclude those 2 checks from global config or will there be any issues excluding those 2 checks from global config.

STEPS TO REPRODUCE

EXPECTED RESULTS

ACTUAL RESULTS

[weizhouapache]

this looks same as #8177 #8569

just to confirm, did you update the name of vms ?

[Chih80]

this looks same as #8177 #8569

just to confirm, did you update the name of vms ?

I did not update the name of the VM. As a matter of fact, I didn't really do anything. I made VMs, assign it the network and that's about it.

[weizhouapache]

this looks same as #8177 #8569 just to confirm, did you update the name of vms ?

I did not update the name of the VM. As a matter of fact, I didn't really do anything. I made VMs, assign it the network and that's about it.

ok. @Chih80 do the VMs have multiple nics ?

[Chih80]

this looks same as #8177 #8569 just to confirm, did you update the name of vms ?

I did not update the name of the VM. As a matter of fact, I didn't really do anything. I made VMs, assign it the network and that's about it.

ok. @Chih80 do the VMs have multiple nics ?

Some does but I doubt this matters. This has been happening from the get-go when all the VMs had one nic.

[weizhouapache]

I checked some similar tickets #8569 #7653 #8177 #8158

• all issues happens with 4.18.0 or 4.18.1
• shared network (#8177, #7653, #8158). network type is not specified in #8569 and #8761

I suspect it has been fixed by #8741

@Chih80 are you still facing the issue ? what type of network do you use ?

[zenbiking]

I recently upgraded my environment to 4.15.2 and I'm seeing a similar issue. I spent some time digging into this a bit more on my system. Here's what I have found.

Works as expected Step 1) Create - VPC Router using the System Offering For Software Router - DHCP/DNS alerts are fine Step 2) Create VPC Guest Network with DHCP enabled - DHCP/DNS alerts are fine Step 3) Create VM in guest network - DHCP/DNS alerts are fine - VM is added to "VirtualMachines" in /root/health_checks_data.json - DNS check - entry for VM is added into /etc/hosts - DHCP check - entry for VM exists in /etc/dhcphosts.txt file

** Causes failed healthchecks ** Step 1) Create - VPC Router using the System Offering For Software Router - DHCP/DNS checks pass Step 2) Create VPC Guest Network ** without DHCP ** - DHCP/DNS checks pass Step 3) Create VM in guest network - DHCP/DNS checks fail - Entry for this VM is added to "VirtualMachines" in /root/health_checks_data.json - DNS check - Entry for this VM is not added in /etc/hosts - DHCP check - No /etc/dhcphosts.txt file created

** Causes failed healthchecks ** Step 1) Create - VPC Router using the System Offering For Software Router - DHCP/DNS alerts pass Step 2) Create VPC Guest Network with DHCP enabled - DHCP/DNS alerts are fine Step 3) Create VM in DHCP enabled guest network - DHCP/DNS alerts are fine - Entry for this VM is added to "VirtualMachines" in /root/health_checks_data.json - DNS check - Entry for this VM is added into /etc/hosts - DHCP check -Entry for this VM exists in /etc/dhcphosts.txt file Step 4) Create VPC Guest Network ** without DHCP ** - DHCP/DNS alerts pass Step 5) Create VM in non-DHCP enabled guest network - DHCP/DNS checks fail - Entry for this VM is added to "VirtualMachines" in /root/health_checks_data.json - DNS check - Entry for this VM is not added in /etc/hosts - DHCP check - Entry for this VM does not exist in the /etc/dhcphosts.txt file

Does it make sense to add entries for machines in non-DHCP enabled networks to /root/health_checks_data.json ??

[weizhouapache]

I recently upgraded my environment to 4.15.2 and I'm seeing a similar issue. I spent some time digging into this a bit more on my system. Here's what I have found.

Works as expected Step 1) Create - VPC Router using the System Offering For Software Router - DHCP/DNS alerts are fine Step 2) Create VPC Guest Network with DHCP enabled - DHCP/DNS alerts are fine Step 3) Create VM in guest network - DHCP/DNS alerts are fine - VM is added to "VirtualMachines" in /root/health_checks_data.json - DNS check - entry for VM is added into /etc/hosts - DHCP check - entry for VM exists in /etc/dhcphosts.txt file

** Causes failed healthchecks ** Step 1) Create - VPC Router using the System Offering For Software Router - DHCP/DNS checks pass Step 2) Create VPC Guest Network ** without DHCP ** - DHCP/DNS checks pass Step 3) Create VM in guest network - DHCP/DNS checks fail - Entry for this VM is added to "VirtualMachines" in /root/health_checks_data.json - DNS check - Entry for this VM is not added in /etc/hosts - DHCP check - No /etc/dhcphosts.txt file created

** Causes failed healthchecks ** Step 1) Create - VPC Router using the System Offering For Software Router - DHCP/DNS alerts pass Step 2) Create VPC Guest Network with DHCP enabled - DHCP/DNS alerts are fine Step 3) Create VM in DHCP enabled guest network - DHCP/DNS alerts are fine - Entry for this VM is added to "VirtualMachines" in /root/health_checks_data.json - DNS check - Entry for this VM is added into /etc/hosts - DHCP check -Entry for this VM exists in /etc/dhcphosts.txt file Step 4) Create VPC Guest Network ** without DHCP ** - DHCP/DNS alerts pass Step 5) Create VM in non-DHCP enabled guest network - DHCP/DNS checks fail - Entry for this VM is added to "VirtualMachines" in /root/health_checks_data.json - DNS check - Entry for this VM is not added in /etc/hosts - DHCP check - Entry for this VM does not exist in the /etc/dhcphosts.txt file

Does it make sense to add entries for machines in non-DHCP enabled networks to /root/health_checks_data.json ??

@zenbiking I tried to reproduce the issue with main/4.20 branch, however, I was not able to.

root@r-337-VM:~# cat /root/health_checks_data.json 
{
...
        "virtualMachines": "vmName=VPC-natted-001,macAddress=02:01:02:05:00:01,ip=172.21.0.90,dhcp=true,dns=true;vmName=VPC-natted-002-no-dhcp-dns,macAddress=02:01:02:c2:00:01,ip=172.21.1.214,dhcp=false,dns=false;"
    }
}

The health check data indicates that dhcp/dns are not supported (dhcp=false,dns=false) for vm on the network without Dhcp/Dns (refer to #5554).

@zenbiking can you share the file /root/health_checks_data.json in the vpc vr ? can you upgrade to 4.16+ and retry ?

[harikrishna-patnala]

I checked some similar tickets #8569 #7653 #8177 #8158

* all issues happens with 4.18.0 or 4.18.1

* shared network ([Failed VR health check #8177](https://github.com/apache/cloudstack/issues/8177), [Virtual Router Failing Health Check Due To Missing Entries in /etc/hosts #7653](https://github.com/apache/cloudstack/issues/7653), [dnsmasq-dhcp - ignored #8158](https://github.com/apache/cloudstack/issues/8158)). network type is not specified in [Router health checks failed #8569](https://github.com/apache/cloudstack/issues/8569) and [VR failing health checks on certain checks 4.18 #8761](https://github.com/apache/cloudstack/issues/8761)

I suspect it has been fixed by #8741

@Chih80 are you still facing the issue ? what type of network do you use ?

@Chih80 since the health check issues were addressed in 4.19.1, could you please upgrade your environment to that version when its feasible for you and get back if you still face the issue. You may reopen this ticket with more details to help investigating your case. Thanks.