Incorporate VR OOB start checks to properly HA the VR

[ggoodrich-ipp]

Description

The file VirtualNetworkApplianceManagerImpl.java is edited for a related VM HA problem. When a Host is determined to be DOWN, CloudStack attempts to VM HA any affected routers. The problem is, when the host is determined to be down, by code referenced above, the host may not actually be DOWN. On KVM for example, the host is considered DOWN if the agent is stopped on the KVM host for too long. In that case, the VMs could still be running just fine... However when we think the host is DOWN, VM HA runs on the router and as part of that it unallocates/cleans-up the router and it's 169.x.x.x control IP is unallocated. Then after it cleans it up, it tries to power on the router on another host, and as part of that it allocates a NEW 169.x.x.x control IP and writes that to the DB. However, since the router isn't actually down (we just think the host is down) the VM HA fails as the vRouter is currently still running on the problem host.

Next, in this example, when the host agent is back online again, it sends a power report to the mgmt servers, and the management servers think the router was powered-on OOB. However, the GUI will not show a control IP for the vRouter, and the DB will have the NEW control IP it tried to allocated during the failed VM HA event. Thus, leaving us unable to communicate with the vRouter.

This PR does a simple check that we can still communicate with the vRouter after any OOB power-on occurs. If we can, then we have the correct control IP in the DB and we're good - so we do nothing. If we can't communicate with the vRouter after the OOB power-on, we do a reboot of the vRouter to fix it.

Fixes: #8284

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

I ran this sql statement to simulate an OOB power on by making CloudStack believe the router is down, but the host then sending a power report stating it is running:

-- id = 157 is the row id of the virtual router in the table
update vm_instance set state='Stopped',power_state='PowerReportMissing',host_id=NULL where id=157;

I then observed that the router got marked as OOB started, and was considered healthy, and no further action was taken.

I then ran the sql statement above again, to make cloudstack believe the router is down, and then connected to the router via cloudstack-ssh and took it to run level 1 via 'init 1' to effectively make it so that it cannot be connected to.

I then observed that the router was restarted by cloudstack, and verified the logs on the management server

[ggoodrich-ipp]

Any interest in reviewing this?

[DaanHoogland]

If this is of interest to you for having it in, yes. at the moment we are in a freeze and hopefully only one or two weeks.

[ggoodrich-ipp]

Bumping this up to request inclusion of this fix.

[DaanHoogland]

@weizhouapache @rhtyd can you review?

[weizhouapache]

@DaanHoogland starting VR OOB looks not like a normal behavior

[ggoodrich-ipp]

@DaanHoogland starting VR OOB looks not like a normal behavior

@weizhouapache We aren't starting it OOB, we are just running checks when it is believed that the VR was started OOB.

[weizhouapache]

@DaanHoogland starting VR OOB looks not like a normal behavior

@weizhouapache We aren't starting it OOB, we are just running checks when it is believed that the VR was started OOB.

@ggoodrich-ipp ok. got it.

in router, /opt/cloud/bin/checkrouter.sh checks the redundant state. is it possible that the scripts works but the router does not work well ?

[DaanHoogland]

code looks good but I am missing handling of redundant state. Do we need more code to check if it is an rvr and set it to passive if it is? @ggoodrich-ipp cc @rhtyd @weizhouapache

[weizhouapache]

@ggoodrich-ipp
I do not think checkrouteranswer is reliable when VR is started OOB.
would it be better to reboot router without checking router ?

[Slair1]

code looks good but I am missing handling of redundant state. Do we need more code to check if it is an rvr and set it to passive if it is? @ggoodrich-ipp cc @rhtyd @weizhouapache

Hi @DaanHoogland
I do not think it matters if it is an RVR or not. If a VR is thought to have been powered-on OOB (occurs in scenario in description), this checks that it can communicate with the VR and if it cannot, it reboots it to get it to a clean/manageable state.

If it was a RVR or not, the mgmt servers should still be able to communicate with the VR. If it cannot, then we need to restart the VR. If it is part of a RVR setup, it should come up and join the VRRP setup.

[Slair1]

@ggoodrich-ipp
I do not think checkrouteranswer is reliable when VR is started OOB.
would it be better to reboot router without checking router ?

Hi @weizhouapache

No, rebooting a VR (if not in a RVR setup) is disruptive to the networking of any VMs using it. This only reboots it if we must - when we can't communicate with it.

[weizhouapache]

No, rebooting a VR (if not in a RVR setup) is disruptive to the networking of any VMs using it. This only reboots it if we must - when we can't communicate with it.

@Slair1 yes.
but, if check router return true, it does not mean the VR which serves as gateway, dhcp/dns server and userdata server, is working well.
rebooting VR will cause few minutes downtime (depends on vm counts and network configs) of the network, but it will avoid some issues which might happen caused by misconfigurations after VR OOB start. the network will be more stable.

[DaanHoogland]

code looks good but I am missing handling of redundant state. Do we need more code to check if it is an rvr and set it to passive if it is? @ggoodrich-ipp cc @rhtyd @weizhouapache

Hi @DaanHoogland
I do not think it matters if it is an RVR or not. If a VR is thought to have been powered-on OOB (occurs in scenario in description), this checks that it can communicate with the VR and if it cannot, it reboots it to get it to a clean/manageable state.

If it was a RVR or not, the mgmt servers should still be able to communicate with the VR. If it cannot, then we need to restart the VR. If it is part of a RVR setup, it should come up and join the VRRP setup.

Hm, so how about these,

scenario 1:

• one of a set of RVR goes down
• ACS creates a new one
• VR comes up
  it now matters wether this one is active or passive for the functionality of the over all RVR.

scenario 2:

• The complete RVR goes down
• the passive comes up
  there is no functionality

in general ACS would create a new VR and if an old VR comes up there is more to do then just check for communication, as @weizhouapache is suggesting:

No, rebooting a VR (if not in a RVR setup) is disruptive to the networking of any VMs using it. This only reboots it if we must - when we can't communicate with it.

@Slair1 yes.
but, if check router return true, it does not mean the VR which serves as gateway, dhcp/dns server and userdata server, is working well.
rebooting VR will cause few minutes downtime (depends on vm counts and network configs) of the network, but it will avoid some issues which might happen caused by misconfigurations after VR OOB start. the network will be more stable.

[yadvr]

@blueorangutan package

[nvazquez]

@blueorangutan package

[blueorangutan]

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✖️ centos7 ✖️ centos8 ✖️ debian. SL-JID 400

[nvazquez]

@blueorangutan package

[blueorangutan]

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✖️ centos7 ✖️ centos8 ✖️ debian. SL-JID 402

[blueorangutan]

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

Trillian test result (tid-1233)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34655 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3915-t1233-kvm-centos7.zip
Smoke tests completed. 88 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

[nvazquez]

@weizhouapache can you please review/test this PR?

[weizhouapache]

@ggoodrich-ipp @nvazquez
I do not think we can merge this pr for now.

1. VR has HA, cloudstack will start it on other hosts if host is determined to be DOWN. hence there are two duplicated VRs running (on old and new host). this pr cannot solve the issue.
2. if cloudstack does not start VR on other host, because the host is Up again, the control IP of VR is not changed. this pr is not needed.
3. if VR is started out-of-band (eg virsh start), CheckRouter checks if control IP is reachable. we do not know if iptables rules or services are configured correctly.

@ggoodrich-ipp did you face this issue in a real environment ? or reproduce the issue (not hack the db) in a test environment ?

[Slair1]

@ggoodrich-ipp @nvazquez
I do not think we can merge this pr for now.

1. VR has HA, cloudstack will start it on other hosts if host is determined to be DOWN. hence there are two duplicated VRs running (on old and new host). this pr cannot solve the issue.
2. if cloudstack does not start VR on other host, because the host is Up again, the control IP of VR is not changed. this pr is not needed.
3. if VR is started out-of-band (eg virsh start), CheckRouter checks if control IP is reachable. we do not know if iptables rules or services are configured correctly.

@ggoodrich-ipp did you face this issue in a real environment ? or reproduce the issue (not hack the db) in a test environment ?
@nvazquez

We did face this issue in a real environment. The scenario is when the KVM agent is stopped and CloudStack thinks the host is down, but the host is in fact up and all VMs are still up - it is just the agent that is down. In this scenario (i think the original PR description is accurate):

VM HA runs for the router and as part of that, its 169.x.x.x control IP is unallocated. Then, it tries to power on the router on another host, and as part of that process it allocates a NEW 169.x.x.x control IP and writes that to the DB. However, since the router isn't actually down (host is up, just agent is down) the VM HA then fails (as the vRouter is currently still running on the problem host). At this point, the DB is already changed - the control IP is changed.

Next, in this scenario, when the host agent is back online again, it sends a power report to the mgmt servers, and the management servers see the router as ON. However, the GUI will not show a control IP for the vRouter, and the DB will have the NEW control IP it tried to allocated during the failed VM HA event. Thus, leaving us unable to communicate with the vRouter

[weizhouapache]

@ggoodrich-ipp @nvazquez
I do not think we can merge this pr for now.

1. VR has HA, cloudstack will start it on other hosts if host is determined to be DOWN. hence there are two duplicated VRs running (on old and new host). this pr cannot solve the issue.
2. if cloudstack does not start VR on other host, because the host is Up again, the control IP of VR is not changed. this pr is not needed.
3. if VR is started out-of-band (eg virsh start), CheckRouter checks if control IP is reachable. we do not know if iptables rules or services are configured correctly.

@ggoodrich-ipp did you face this issue in a real environment ? or reproduce the issue (not hack the db) in a test environment ?
@nvazquez

We did face this issue in a real environment. The scenario is when the KVM agent is stopped and CloudStack thinks the host is down, but the host is in fact up and all VMs are still up - it is just the agent that is down. In this scenario (i think the original PR description is accurate):

VM HA runs for the router and as part of that, its 169.x.x.x control IP is unallocated. Then, it tries to power on the router on another host, and as part of that process it allocates a NEW 169.x.x.x control IP and writes that to the DB. However, since the router isn't actually down (host is up, just agent is down) the VM HA then fails (as the vRouter is currently still running on the problem host). At this point, the DB is already changed - the control IP is changed.

Next, in this scenario, when the host agent is back online again, it sends a power report to the mgmt servers, and the management servers see the router as ON. However, the GUI will not show a control IP for the vRouter, and the DB will have the NEW control IP it tried to allocated during the failed VM HA event. Thus, leaving us unable to communicate with the vRouter

@Slair1
I think the new control IP 169.254.x.x belongs to another VR which is also Running, right ?
the two VRs are duplicated (the same guest IP, same public IP, use the same image on primary storage, except different control IP). rebooting the VR with checkrouter error won't solve the problem.
ideally we should destroy one of them.

[Slair1]

@ggoodrich-ipp @nvazquez
I do not think we can merge this pr for now.

1. VR has HA, cloudstack will start it on other hosts if host is determined to be DOWN. hence there are two duplicated VRs running (on old and new host). this pr cannot solve the issue.
2. if cloudstack does not start VR on other host, because the host is Up again, the control IP of VR is not changed. this pr is not needed.
3. if VR is started out-of-band (eg virsh start), CheckRouter checks if control IP is reachable. we do not know if iptables rules or services are configured correctly.

@ggoodrich-ipp did you face this issue in a real environment ? or reproduce the issue (not hack the db) in a test environment ?
@nvazquez

We did face this issue in a real environment. The scenario is when the KVM agent is stopped and CloudStack thinks the host is down, but the host is in fact up and all VMs are still up - it is just the agent that is down. In this scenario (i think the original PR description is accurate):
VM HA runs for the router and as part of that, its 169.x.x.x control IP is unallocated. Then, it tries to power on the router on another host, and as part of that process it allocates a NEW 169.x.x.x control IP and writes that to the DB. However, since the router isn't actually down (host is up, just agent is down) the VM HA then fails (as the vRouter is currently still running on the problem host). At this point, the DB is already changed - the control IP is changed.
Next, in this scenario, when the host agent is back online again, it sends a power report to the mgmt servers, and the management servers see the router as ON. However, the GUI will not show a control IP for the vRouter, and the DB will have the NEW control IP it tried to allocated during the failed VM HA event. Thus, leaving us unable to communicate with the vRouter

@Slair1
I think the new control IP 169.254.x.x belongs to another VR which is also Running, right ?
the two VRs are duplicated (the same guest IP, same public IP, use the same image on primary storage, except different control IP). rebooting the VR with checkrouter error won't solve the problem.
ideally we should destroy one of them.

@nvazquez
Actually VM HA doesn’t try to create a new VR. It tries to boot up the same one. It fails because it is already running on another host- it fails because it can’t get a lock on the qcow2.

[nvazquez]

@Slair1 @weizhouapache what do you think on fixing the HA host down detection instead? For a host we have the Status and the ResourceState fields, and in case a host is connected but its agent is down I think its status is still reported as 'Up' in contrast to when the host is actually down in which both fields are shown disconnected

[weizhouapache]

Actually VM HA doesn’t try to create a new VR. It tries to boot up the same one. It fails because it is already running on another host- it fails because it can’t get a lock on the qcow2.

@Slair1 on recent linux releases (eg ubuntu 20.04), yes.
on some old releases (eg centos 7), the VR can still be started on another host. It is difficult to solve this problem.

now I understand what you want to do: rebootrouter to update the control IP, assuming that VR is not duplicated on two hosts.
if network/vpc have redundant VRs, it is ok.
if there is only 1 VR in the network/vpc, it will lead to downtime.

can you look for other solutions instead of rebootrouter or restart network ?

[nvazquez]

Hi @Slair1 @ggoodrich-ipp - @weizhouapache has raised a valid concern, please check the last comment

[github-actions]

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

[DaanHoogland]

@weizhouapache , do you think we should still consider this code? (also see #8284)

[weizhouapache]

@weizhouapache , do you think we should still consider this code? (also see #8284)

@DaanHoogland
this is not needed any more I think

for vmware, I suggest to set the new global setting "systemvm.release.control.ip.on.stop" to "false" so that VR Ips will be reserved. OOB start will not cause IP conflicts.

[DaanHoogland]

for vmware, I suggest to set the new global setting "systemvm.release.control.ip.on.stop" to "false" so that VR Ips will be reserved. OOB start will not cause IP conflicts.

The default is true, and scope is zone, which makes sense. But if the (ss)vm runs on vmware this is not good. so should we make this a dynamic default of some sort? I am happy to leave it as is, but want us to be aware of the consequence.

[weizhouapache]

for vmware, I suggest to set the new global setting "systemvm.release.control.ip.on.stop" to "false" so that VR Ips will be reserved. OOB start will not cause IP conflicts.

The default is true, and scope is zone, which makes sense. But if the (ss)vm runs on vmware this is not good. so should we make this a dynamic default of some sort? I am happy to leave it as is, but want us to be aware of the consequence.

I am good with default settings.
if users are not happy with it. they can change the settings. if so, VRs will consume more private Ips, so leave to users to decide.

[DaanHoogland]

closing this.
@ggoodrich-ipp please reopen or create a new issue at your will, if you think this is not good enough