UI: VM's on crashed node show as running

[mierea]

ISSUE TYPE

• Bug Report

COMPONENT NAME

UI

CLOUDSTACK VERSION

4.18.1.0 

CONFIGURATION

N/A

OS / ENVIRONMENT

N/A

SUMMARY

When a kvm node crashes (no maintenance) the VM's running on that node will still show the status "Running" in the dashboard.
When accessing the VM itself you get the error message:
The Control Plane Status of this instance is Offline. Some actions on this instance will fail, if so please wait a while and retry.
When the kvm node restarts, after some time the VM's on it will show as "Stopped" and will have to be started manually.

STEPS TO REPRODUCE

Force reboot without maintenance a kvm node that run's compute resources

EXPECTED RESULTS

I would expect the VM's to show as Stopped/Crashed as soon as the kvm node goes down

ACTUAL RESULTS

VM looks as running:

VM details says there is an issue with control plane:

Node is offline:

[DaanHoogland]

@mierea is this
Force reboot without maintenance a kvm node that run's compute resources a ACS controlled step or out of bounds? ACS needs to be sure the host is down and not just unreachible/disconnected before marking the VMs as down.

[mierea]

@DaanHoogland out of bounds (like a hardware failure on that kvm node). The actual issue is that if a kvm node crashes the vm's on that node will still show as "Running" on ACS. (so potential customers with access to ACS will think their VM is still online while it actually isn't). Maybe in this case the status should be "Unknown"?

[DaanHoogland]

@mierea , there are some hearthbeat setting you can use to tweak this a bit, but the jist is that a host will not give a warning about it crashing.
The Disconnected state signifies that there is no longer a successful application level ping going on.
That said, there might be a need for an enhancement as you descibe :

Maybe in this case the status should be "Unknown"

[DaanHoogland]

@mierea , can you give a good definition of what you expect, given that saying anything about the VM is impossible? When would you want the status become unknown and under what conditions?

[mierea]

@DaanHoogland when the control plane status of an instance becomes Offline, the instance on that control plane should be put in a different status than Running (Offline, Unknown, Crashed, Unreacheble - any one of these is better than to see the vm as running in the dashboard)

I used the term control plane instead of node because this is the current error that can be seen in the vm's details when the node itself is offline.