Host HA not working even after configured oob-management

[yashi4engg]

Host HA not working even after configured oob-management
..

backend config -
Hypervisor and Management node OS - OEL9
OOB configured to use IPMI for HA .

Host HA not working even in logs can see its getting power status successfully and putting node in alert sate but nothing after that like putting to node in disconnected state/Move VMs and so .

[kiranchavala]

@yashi4engg

Could you please provide more details regarding this issue with proper formatting

or please provide the steps to reproduce the issue

[yashi4engg]

@kiranchavala I am trying to setup Host HA ... as of now i have 3 hypervisors in cluster out of these on one hypervisor i configured OOB management and enabled Host HA ..But when host is going to power off or something else cluster is able to get power status and mark as power status off in UI but its not trying to bring this up or also not marking it disconnected in UI so VMs on hypervisor are keep on showing running but not migrating .Even VMs are marked HA enabled in template.

Our Requirement -
We want as soon as any hypervisor went down or crash ..VMs running on that hypervisor should migrate to other hypervisor by default.

Our config :-
OS - OEL9
OOB band mgmt - IPMI
H/w - Dell R series servers

Steps to Reproduce - Power off hypervisor from IDrac

[kiranchavala]

@yashi4engg what is the hypervisor you are using

Were the Service offerings> compute offering HA enabled ?

[yashi4engg]

@kiranchavala - Compute offering have HA enabled . Hypervisor type - KVM with OEL9 .

If i destroy VM manually from hypervisor it came up as part of HA but if hypervisor itself powered off it wont migrate to another node and keep on showing running in UI .

[weizhouapache]

@yashi4engg
is this same as #7520 ?

[yashi4engg]

No in this issue i am looking for host Ha not VM HA ..I configured oob and able to get power status using same but cloudstack is not putting host in maint mode or trying to recover and due to that VMs running on that should visible on cloudstack as running and never migrate.

[DaanHoogland]

@yashi4engg , I am trying to reproduce, using a nested env. In my case the host HA-state cycled through Suspect-Checking-Degraded. This happens both when I shut it down out of bands and when I stop the agent. I see no attempt to fence or reboot. I also see no alert state.
Can you add more info on the sequence of states and events?

[yashi4engg]

In my case events happening -

As soon as power off host ...as per cloudstack config it keep on checking host and VM status and its able to get host power status using ipmi tool ... and mark not power status as down/Off in UI but after that it wont try to fence or recover node. Due to that all VMs were running on that host keep on showing Running in UI and host also keep on showing enabled state .

[DaanHoogland]

what I mean is the ha-statuses in the log, do you have a track of that? in the UI it shows less.

[DaanHoogland]

@yashi4engg I think I figured out what the problem is. With centos7 this is working unless you install qemu-kvm-ev. the porblem is the activity check is not giving conclusive answers so it can't know if the host is still active and won't move to the recovering or fencing states. Look for messages containing ActivityCheckFailureUnderThresholdRatio. Do you see these as well?
the short version HostHA is not supported for EL versions above 7 (or with qemu upgraded to a higher qemu-kvm-ev version)

works:

qemu-img.x86_64                          10:1.5.3-175.el7_9.6           @updates
qemu-kvm.x86_64                          10:1.5.3-175.el7_9.6           @updates
qemu-kvm-common.x86_64                   10:1.5.3-175.el7_9.6           @updates

doesn't work:

qemu-img-ev.x86_64                       10:2.12.0-44.1.el7_8.1         @centos-qemu-ev
qemu-kvm-common-ev.x86_64                10:2.12.0-44.1.el7_8.1         @centos-qemu-ev
qemu-kvm-ev.x86_64                       10:2.12.0-44.1.el7_8.1         @centos-qemu-ev