KVM HA: Fix CheckOnHostAnswer success flag when there is no heartbeat by sureshanaparti · Pull Request #13373 · apache/cloudstack

sureshanaparti · 2026-06-08T11:53:05Z

Description

This PR fixes the CheckOnHostAnswer success flag when there is no heartbeat, for KVM HA.

Fixes #13371

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
Build/CI
Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

sureshanaparti · 2026-06-08T11:54:21Z

@blueorangutan package

blueorangutan · 2026-06-08T11:56:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

codecov · 2026-06-08T12:02:25Z

Codecov Report

❌ Patch coverage is 50.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.67%. Comparing base (21b2025) to head (4d298b4).
⚠️ Report is 7 commits behind head on 4.22.

Files with missing lines	Patch %	Lines
...ache/cloudstack/kvm/ha/KVMHostActivityChecker.java	0.00%	1 Missing ⚠️
...oud/hypervisor/vmware/resource/VmwareResource.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               4.22   #13373      +/-   ##
============================================
- Coverage     17.67%   17.67%   -0.01%     
+ Complexity    15792    15791       -1     
============================================
  Files          5922     5922              
  Lines        533165   533166       +1     
  Branches      65208    65208              
============================================
  Hits          94242    94242              
- Misses       428276   428277       +1     
  Partials      10647    10647

Flag	Coverage Δ
uitests	`3.69% <ø> (ø)`
unittests	`18.75% <50.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

sureshanaparti · 2026-06-08T12:03:29Z

@blueorangutan package

blueorangutan · 2026-06-08T12:04:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

sureshanaparti · 2026-06-08T12:14:40Z

@blueorangutan package

blueorangutan · 2026-06-08T12:16:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

blueorangutan · 2026-06-08T12:34:15Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 18187

weizhouapache · 2026-06-08T13:01:09Z

@sureshanaparti
there is a test failure

14:34:05 [ERROR] testCheckOnHostCommand(com.cloud.hypervisor.kvm.resource.LibvirtComputingResourceTest)  Time elapsed: 0.279 s  <<< FAILURE!
14:34:05 java.lang.AssertionError
14:34:05 	at com.cloud.hypervisor.kvm.resource.LibvirtComputingResourceTest.testCheckOnHostCommand(LibvirtComputingResourceTest.java:3133)

sureshanaparti · 2026-06-08T13:09:02Z

@blueorangutan package

blueorangutan · 2026-06-08T13:10:05Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-06-08T14:11:47Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18189

weizhouapache · 2026-06-08T14:40:19Z

@sureshanaparti
it looks like the change made no effects

cloudstack/core/src/main/java/com/cloud/agent/api/CheckOnHostAnswer.java

Lines 39 to 43 in 6bc83a3

    
           public CheckOnHostAnswer(CheckOnHostCommand cmd, String details) { 
        
               super(cmd, false, details); 
        
               determined = false; 
        
               alive = false; 
        
           }

these two lines are the same actually

                return new CheckOnHostAnswer(command, "Heart is not beating");
                return new CheckOnHostAnswer(command, false, "Heart is not beating");

sureshanaparti · 2026-06-08T15:04:56Z

@sureshanaparti it looks like the change made no effects

cloudstack/core/src/main/java/com/cloud/agent/api/CheckOnHostAnswer.java

Lines 39 to 43 in 6bc83a3

public CheckOnHostAnswer(CheckOnHostCommand cmd, String details) {

super(cmd, false, details);

determined = false;

alive = false;

}

these two lines are the same actually
                return new CheckOnHostAnswer(command, "Heart is not beating");
                return new CheckOnHostAnswer(command, false, "Heart is not beating");

@weizhouapache answer's success flag is set in the second case (result should be true which indicates the cmd is successfully processed without any errors, and then get the other details - isAlive, etc), and answer.getResult() is true here:

cloudstack/plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHostActivityChecker.java

Lines 165 to 175 in 21b2025

    
           if (answer != null) { 
        
               if (answer.getResult()) { 
        
                   hostStatusFromNeighbour = ((CheckOnHostAnswer)answer).isAlive() ? Status.Up : Status.Down; 
        
                   logger.debug("Neighboring {} returned status [{}] for the investigated {}.", neighbor.toString(), hostStatusFromNeighbour, host.toString()); 
        
                   if (hostStatusFromNeighbour == Status.Up) { 
        
                       return hostStatusFromNeighbour; 
        
                   } 
        
               } else { 
        
                   logger.debug("{} is not active according to neighbor {}, details: {}.", host.toString(), neighbor.toString(), answer.getDetails()); 
        
               } 
        
           } else {

weizhouapache · 2026-06-08T15:10:26Z

@weizhouapache answer's success flag is set in the second case (result should be true which indicates the cmd is success, and then get the other details - isAlive, etc), and answer.getResult() is true here:

cloudstack/plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHostActivityChecker.java

Lines 165 to 175 in 21b2025

if (answer != null) {

if (answer.getResult()) {

hostStatusFromNeighbour = ((CheckOnHostAnswer)answer).isAlive() ? Status.Up : Status.Down;

logger.debug("Neighboring {} returned status [{}] for the investigated {}.", neighbor.toString(), hostStatusFromNeighbour, host.toString());

if (hostStatusFromNeighbour == Status.Up) {

return hostStatusFromNeighbour;

}

} else {

logger.debug("{} is not active according to neighbor {}, details: {}.", host.toString(), neighbor.toString(), answer.getDetails());

}

} else {

ok, I will re-test

weizhouapache

verified ok

weizhouapache · 2026-06-09T06:08:56Z

@kiranchavala
could you test it too if you have time ? thanks

weizhouapache · 2026-06-18T11:34:00Z

need another approval on dev review or qa

cc @harikrishna-patnala @DaanHoogland @kiranchavala @NuxRo

DaanHoogland · 2026-06-19T07:02:01Z

    @Override
    public Answer execute(final CheckOnHostCommand command, final CitrixResourceBase citrixResourceBase) {
-        return new CheckOnHostAnswer(command, "Not Implmeneted");
+        return new CheckOnHostAnswer(command, "Not Implemented");


is the <null> needed as second parameter here as well?

[4.22.1.0-shapeblue1] Upstream PRs apache#13373 and apache#12651

kiranchavala

LGTM, VM ha kicks in after the host is determined as Down

covered both scenarios

VM HA + Host ha Disabled
VM HA + Host ha Enabled

Before fix

2026-07-06 11:09:02,217 INFO [c.c.a.m.ClusteredAgentManagerImpl] (AgentTaskPool-1:[ctx-95f49608]) (logid:e286330d) The agent from host Host {"id":1,"name":"ref-trl-11971-k-Mol8-kiran-chavala-kvm1","type":"Routing","uuid":"f381eed2-bc93-43a1-9a8a-d439d804b40a"} state determined is Disconnected

After fix

2026-07-06 11:31:19,802 INFO [c.c.a.m.ClusteredAgentManagerImpl] (AgentTaskPool-4:[ctx-e65f658b]) (logid:2e73eea3) The agent from host Host {"id":1,"name":"ref-trl-11971-k-Mol8-kiran-chavala-kvm1","type":"Routing","uuid":"f381eed2-bc93-43a1-9a8a-d439d804b40a"} state determined is Down

2026-07-06 11:31:19,840 WARN [c.c.a.AlertManagerImpl] (AgentTaskPool-4:[ctx-e65f658b]) (logid:2e73eea3) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [Host is down, name: ref-trl-11971-k-Mol8-kiran-chavala-kvm1 (id:1), availability zone: ref-trl-11971-k-Mol8-kiran-chavala, pod: Pod1] and content [Host [name: ref-trl-11971-k-Mol8-kiran-chavala-kvm1 (id:1), availability zone: ref-trl-11971-k-Mol8-kiran-chavala, pod: Pod1] is down. Starting HA on the following VMs: test-vmha].
2026-07-06 11:31:19,841 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (AgentTaskPool-4:[ctx-e65f658b]) (logid:2e73eea3) Notifying HA Mgr of to restart Instance VM instance {"id":44,"instanceName":"i-2-44-VM","state":"Running","type":"User","uuid":"d808e241-5ade-4465-bd3d-75c700fee5a4"}
2026-07-06 11:31:19,843 DEBUG [c.c.h.HighAvailabilityManagerExtImpl] (AgentTaskPool-4:[ctx-e65f658b]) (logid:2e73eea3) HA schedule restart
2026-07-06 11:31:19,847 INFO [c.c.h.HighAvailabilityManagerExtImpl] (AgentTaskPool-4:[ctx-e65f658b]) (logid:2e73eea3) Schedule vm for HA: VM instance {"id":44,"instanceName":"i-2-44-VM","state":"Running","type":"User","uuid":"d808e241-5ade-4465-bd3d-75c700fee5a4"}

weizhouapache · 2026-07-06T11:55:47Z

thanks @kiranchavala

merging

KVM HA: Fix CheckOnHostAnswer success flag when there is no heartbeat

2e142cf

boring-cyborg Bot added component:kvm component:XenServer labels Jun 8, 2026

sureshanaparti added this to the 4.22.2 milestone Jun 8, 2026

sureshanaparti linked an issue Jun 8, 2026 that may be closed by this pull request

VM HA is broken in CloudStack 4.22.1.0 #13371

Closed

add log

cf90e46

sureshanaparti requested a review from Copilot June 8, 2026 12:09

Copilot started reviewing on behalf of sureshanaparti June 8, 2026 12:10 View session

sureshanaparti changed the base branch from main to 4.22 June 8, 2026 12:12

typo

cffeaff

boring-cyborg Bot added the component:vmware label Jun 8, 2026

Copilot AI reviewed Jun 8, 2026

unit test update

4d298b4

sureshanaparti marked this pull request as ready for review June 8, 2026 13:20

sureshanaparti added the type:regression label Jun 8, 2026

weizhouapache approved these changes Jun 8, 2026

View reviewed changes

github-actions Bot mentioned this pull request Jun 8, 2026

[repo-status] Daily Status Report – June 8, 2026 #13381

Closed

winterhazel added the status:needs-testing label Jun 9, 2026

github-actions Bot mentioned this pull request Jun 9, 2026

[repo-status] Daily Status Report – June 9, 2026 #13386

Closed

harikrishna-patnala mentioned this pull request Jun 17, 2026

[4.22.1.0-shapeblue1] Upstream PRs #13373 and #12651 shapeblue/cloudstack#138

Merged

14 tasks

DaanHoogland reviewed Jun 19, 2026

View reviewed changes

kiranchavala mentioned this pull request Jun 29, 2026

VM HA not working when there's an issue with vm restart #13510

Open

harikrishna-patnala added a commit to shapeblue/cloudstack that referenced this pull request Jul 6, 2026

Merge pull request #138 from shapeblue/ha-checkonhostanswer-fix-4.22.1

f39b852

[4.22.1.0-shapeblue1] Upstream PRs apache#13373 and apache#12651

kiranchavala approved these changes Jul 6, 2026

View reviewed changes

weizhouapache merged commit 4262c02 into apache:4.22 Jul 6, 2026
41 of 43 checks passed

weizhouapache deleted the ha-checkonhostanswer-fix branch July 6, 2026 11:56

weizhouapache mentioned this pull request Jul 6, 2026

VM HA is broken in CloudStack 4.22.1.0 #13371

Closed

Uh oh!

Conversation

sureshanaparti commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

sureshanaparti commented Jun 8, 2026

Uh oh!

blueorangutan commented Jun 8, 2026

Uh oh!

codecov Bot commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sureshanaparti commented Jun 8, 2026

Uh oh!

blueorangutan commented Jun 8, 2026

Uh oh!

sureshanaparti commented Jun 8, 2026

Uh oh!

blueorangutan commented Jun 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Jun 8, 2026

Uh oh!

weizhouapache commented Jun 8, 2026

Uh oh!

sureshanaparti commented Jun 8, 2026

Uh oh!

blueorangutan commented Jun 8, 2026

Uh oh!

blueorangutan commented Jun 8, 2026

Uh oh!

weizhouapache commented Jun 8, 2026

Uh oh!

sureshanaparti commented Jun 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

weizhouapache commented Jun 8, 2026 • edited by sureshanaparti Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

weizhouapache commented Jun 9, 2026

Uh oh!

weizhouapache commented Jun 18, 2026

Uh oh!

DaanHoogland Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

kiranchavala left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

weizhouapache commented Jul 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

sureshanaparti commented Jun 8, 2026 •

edited

Loading

codecov Bot commented Jun 8, 2026 •

edited

Loading

sureshanaparti commented Jun 8, 2026 •

edited

Loading

weizhouapache commented Jun 8, 2026 •

edited by sureshanaparti

Loading

kiranchavala left a comment •

edited

Loading