Dev: bootstrap: Reject to remove qdevice when it will cause quorum lost by liangxin1300 · Pull Request #2041 · ClusterLabs/crmsh

liangxin1300 · 2026-03-03T10:15:19Z

crm cluster remove --qdevice command will be rejected if it will cause quorum lost for the cluster configured with diskless SBD, since the diskless SBD will self-fence the node without quorum

Fix issue:

cluster remove --qdevice should be rejected on the last standing node with qdevice for the 2-node diskless sbd cluster #2039

`crm cluster remove --qdevice` command will be rejected if it will cause quorum lost for the cluster configured with diskless SBD, since the diskless SBD will self-fence the node without quorum

codecov · 2026-03-03T11:02:40Z

Codecov Report

❌ Patch coverage is 77.77778% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.55%. Comparing base (386d706) to head (e558b8f).

Files with missing lines	Patch %	Lines
crmsh/bootstrap.py	50.00%	1 Missing ⚠️
crmsh/qdevice.py	85.71%	1 Missing ⚠️

Additional details and impacted files

Flag	Coverage Δ
integration	`55.52% <77.77%> (+<0.01%)`	⬆️
unit	`52.63% <77.77%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
crmsh/bootstrap.py	`88.04% <50.00%> (-0.05%)`	⬇️
crmsh/qdevice.py	`97.56% <85.71%> (-0.22%)`	⬇️

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

gao-yan · 2026-03-03T16:41:51Z

My question is, why couldn't it remove the on-disk qdevice config and restart the cluster or tell the user to restart the cluster later? Aren't there QDEVICE_RESTART/QDEVICE_RESTART_LATER for the purpose?

liangxin1300 · 2026-03-04T03:41:17Z

My question is, why couldn't it remove the on-disk qdevice config and restart the cluster or tell the user to restart the cluster later? Aren't there QDEVICE_RESTART/QDEVICE_RESTART_LATER for the purpose?

Since when the cluster is using diskless SBD, and the cluster "will" lose quorum after remvoing qdevice, after restart cluster, the existing node will be fenced.

Let me summarise all of the cases here:

reload case:

# when the cluster is not using diskless SBD, and will still has quorum after removing qdevice
# crm cluster remove --qdevice -y
INFO: Disable corosync-qdevice.service
INFO: Stopping corosync-qdevice.service
INFO: BEGIN Removing QDevice configuration from cluster
INFO: Sync file /etc/corosync/corosync.conf to sle16-2
INFO: END Removing QDevice configuration from cluster
INFO: Reloading cluster configuration after removing QDevice
INFO: Add parameter 'pcmk_delay_max=30s' for resource 'stonith-sbd'
INFO: Update SBD_DELAY_START in /etc/sysconfig/sbd: 71
INFO: Sync file /etc/sysconfig/sbd to sle16-2
WARNING: "priority" in rsc_defaults is set to 1, it was 0
WARNING: "priority-fencing-delay" in crm_config is set to 60, it was 0

restart case:

# when the cluster is using diskless SBD, and no non-stonith RA running
# crm cluster remove --qdevice -y
INFO: Disable corosync-qdevice.service
INFO: BEGIN Removing QDevice configuration from cluster
INFO: Sync file /etc/corosync/corosync.conf to sle16-2
INFO: END Removing QDevice configuration from cluster
INFO: Restarting cluster service
INFO: BEGIN Waiting for cluster
...........                                                                                                                                                                                                                            
INFO: END Waiting for cluster
WARNING: "priority" in rsc_defaults is set to 1, it was 0
WARNING: Diskless SBD requires cluster with three or more nodes. If you want to use diskless SBD for 2-node cluster, should be combined with QDevice.

restart later case:

# when the cluster is using diskless SBD, and has non-stonith RA running
# crm cluster remove --qdevice -y
WARNING: Please stop all running resources and try again
WARNING: Or use 'crm -F/--force' option to leverage maintenance mode
WARNING: Understand risks that running RA has no cluster protection while the cluster is in maintenance mode and restarting
INFO: Aborting the configuration change attempt

# leverage maintenance mode
# crm -F cluster remove --qdevice -y
INFO: Set cluster to maintenance mode
WARNING: "maintenance-mode" in crm_config is set to true, it was false
INFO: Disable corosync-qdevice.service
INFO: BEGIN Removing QDevice configuration from cluster
INFO: Sync file /etc/corosync/corosync.conf to sle16-2
INFO: END Removing QDevice configuration from cluster
INFO: Restarting cluster service
INFO: BEGIN Waiting for cluster
...........                                                                                                                                                                                                                            
INFO: END Waiting for cluster
WARNING: "priority" in rsc_defaults is set to 1, it was 0
WARNING: Diskless SBD requires cluster with three or more nodes. If you want to use diskless SBD for 2-node cluster, should be combined with QDevice.
INFO: Set cluster from maintenance mode to normal
INFO: Delete cluster property "maintenance-mode" in crm_config

reject case:

# when the cluster is using diskless SBD, and will lose quorum after removing qdevice
# corosync-quorumtool -s
...
Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2  
Flags:            Quorate Qdevice 

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
         1          1    A,V,NMW sle16-1 (local)
         0          1            Qdevice
         
# if we allow removing qdevice, after restart cluster, the existing node will be fenced by diskless SBD
# so, we reject at this time
# crm cluster remove --qdevice -y
ERROR: cluster.remove: Can't remove QDevice since the quorum will be lost for diskless SBD cluster

gao-yan · 2026-03-04T08:42:12Z

reject case:

# when the cluster is using diskless SBD, and will lose quorum after removing qdevice
# corosync-quorumtool -s
...
Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2  
Flags:            Quorate Qdevice 

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
         1          1    A,V,NMW sle16-1 (local)
         0          1            Qdevice
         
# if we allow removing qdevice, after restart cluster, the existing node will be fenced by diskless SBD
# so, we reject at this time

Well, I assume the cause is, stop of corosync-qdevice.service will cause loss of votes right away. So diskless sbd will react if the quorum is lost, given that it's still armed.

But how about a direct "systemctl stop corosync" (meaning stopping the cluster stack as a whole). Does it still trigger sbd self-reset in such a situation?

liangxin1300 · 2026-03-05T01:57:41Z

But how about a direct "systemctl stop corosync" (meaning stopping the cluster stack as a whole). Does it still trigger sbd self-reset in such a situation?

No, stop corosync will stop everything, including the sbd process
But what we discuss about this "reject case", is the user wants to remove qdevice from the cluster, which will lose the quorum and will get self-fence after removing qdevice, not discuss the user wants to stop corosync

gao-yan · 2026-03-05T07:15:18Z

Well, we are talking about how to properly remove qdevice in such a situation by gracefully shutting down the cluster first.

First of all, it's not the removal of qdevice config causing loss of votes/quorum. It's stop of "corosync-qdevice.service" alone.

Let me elaborate the situation:

With corosync-qdevie.device running:

Expected votes: 3
Total votes: 2

Stop corosync-qdevie.device will cause it to lose 1 vote while it's still expecting 3 votes:

Expected votes: 3
Total votes: 1

Hence loss of quorum and diskless SBD reacts (Don't get confused about the timing of the actual reset, since reset anyway triggers after SBD_WATCHDOG_TIMEOUT since loss of quorum has occurred).

Given that the number of expected votes remains, if to remove qdevice, the proper way is to gracefully shut down the cluster stack first of all. And the only graceful way to shut down the cluster stack in such as situation is to shut down it as a whole.

Then by removing qdevice config, the "Expected votes" will correspondingly reduce when cluster starts again, so that removal of qdevice won't be an issue.

Of course, despite the specific workflow, I'd assume the only reason that an user wants to remove qdevice is in order to change their cluster's topology. For instance for a 2-node cluster, they want to either:

Add another node to replace the qdevice

or:

Switch to another fencing mechanism that doesn't require qdevice, to replace diskless SBD

In case we want to proactively prevent users from crippling their cluster according to the requirement "Use diskless SBD only for clusters with more than two nodes, or in combination with QDevice", for example, if diskless SBD still is used, we don't want users to remove qdevice from a 2-node cluster unless they first either add another node or change the fencing mechanism. But that'd be a different topic and a different logic to achieve that, no matter whether loss of quorum could potentially occur and what this workflow should be like.

zzhou1 · 2026-03-17T13:19:19Z

#2041 (comment)

crm cluster remove --qdevice command will be rejected if it will cause quorum lost for the cluster configured with diskless SBD, since the diskless SBD will self-fence the node without quorum

Fix issue:

cluster remove --qdevice should be rejected on the last standing node with qdevice for the 2-node diskless sbd cluster #2039

This could be rephrased as crm cluster remove --qdevice command should be rejected when the number of live nodes is unable to form quorum, either disk-based sbd or diskless sbd.

Along with that, the error message could be tweaked, something like

# crm cluster remove --qdevice -y
ERROR: cluster.remove: operation is rejected since it would result in a malfunctioning cluster lacking quorum.

zzhou1 · 2026-03-17T13:40:23Z

#2041 (comment)

when the cluster is using diskless SBD, and will lose quorum after removing qdevice

corosync-quorumtool -s

...

if we allow removing qdevice, after restart cluster, the existing node will be fenced by diskless SBD

so, we reject at this time

I can understand the concern Yan raised above. Basically, the original narrative above is incorrect and misleading the actual workflow. The qdevice removing process doesn't stop qdevice when the cluster is running. In fact, crmsh bootstrap turns on "maintenance" mode first, then disable qdevice before remove it from corosync.conf, and finally restart the whole cluster all together.

The tweaked message in the earlier comment explains better.

And, one more further rephrase could be like this: crm cluster remove --qdevice executes in the running cluster, and rejection tries to avoid resulting in a malfunction.

gao-yan · 2026-03-17T17:59:57Z

It's fine to leverage maintenance mode. But despite that, as long as a procedure involves restart of the whole cluster, the proper way anyway should be stopping the whole cluster stack altogether. That's basically how a cluster or a cluster partition can gracefully shut down anyways. It just doesn't make sense to split out stop of qdevice and make a stopping cluster to unnecessarily suffer from absolute loss of votes hence possible loss of quorum.

Technically it's possible to choose the timings when to "systemctl disable" (without --now) qdevice service and when to remove the on-disk qdevice config. It's just neither sensible nor necessary to stop qdevice alone when a cluster is active and about to stop.

Be aware, without qdevice config hence without additional expected votes from qdevice, a single node or 2-node cluster still can start and form a quorate cluster. Even if corosync wait_for_all is enabled, when a partition cannot be quorate until all nodes have come online, diskless SBD won't reset a yet-to-be-quorate node/partition on startup.

As said, in case we'd want to enforce qdevice for a cluster with <= 2 nodes and diskless SBD, hence also prevent qdevice from being removed from such a setup, it should be achieved with a different logic or code path, which should be based on how many nodes are configured in corosync's nodelist.

zzhou1 · 2026-03-18T06:47:26Z

+    quorate = utils.calculate_quorate_status(expected_votes, actual_votes)
+    if not quorate and diskless_sbd and mode == QDEVICE_REMOVE:
+        # Reject to remove qdevice, as it will lose quorum
+        # then diskless SBD will self-fence the node
+        return QdevicePolicy.QDEVICE_REMOVE_REJECT


Suggested change

quorate = utils.calculate_quorate_status(expected_votes, actual_votes)

if not quorate and diskless_sbd and mode == QDEVICE_REMOVE:

# Reject to remove qdevice, as it will lose quorum

# then diskless SBD will self-fence the node

return QdevicePolicy.QDEVICE_REMOVE_REJECT

qdevice_votes = pseudo.code.to.get.corosync.conf.quorum.device.votes

quorate = utils.calculate_quorate_status(expected_votes-qdevice_votes, actual_votes-qdevice_votes)

if not quorate and diskless_sbd and mode == QDEVICE_REMOVE:

# Rejected since it would result in a malfunctioning cluster lacking quorum

# either the disk-based sbd cluster or the diskless sbd cluster

zzhou1 · 2026-03-18T06:50:36Z

    utils.check_all_nodes_reachable("removing QDevice from the cluster")
    qdevice_reload_policy = qdevice.evaluate_qdevice_quorum_effect(qdevice.QDEVICE_REMOVE)
+    if qdevice_reload_policy == qdevice.QdevicePolicy.QDEVICE_REMOVE_REJECT:
+        utils.fatal("Can't remove QDevice since the quorum will be lost for diskless SBD cluster")


Suggested change

utils.fatal("Can't remove QDevice since the quorum will be lost for diskless SBD cluster")

utils.fatal("operation is rejected since it would result in a malfunctioning cluster lacking quorum")

It wouldn't necessarily become "malfunctioning" purely based on the logic here :-) It's just that the next cluster startup will require an actual majority or "wait_for_all" to become first quorate.

What I can tell about the case in here is, an user may want to remove qdevice when not all cluster nodes are online, right? But actually it's technically feasible no matter what the quorum status is like, as long as the offline nodes are still reachable through ssh.

The actual problem is, after removal of qdevice, if they/we still want to only bring up the same subset of nodes and expect them to reach the status as before the removal of qdevice right away, it won't be in the same condition any more though. But it's actually about how quorum is first granted on cluster startup, which either requires an actual majority or "wait_for_all" (automatically enabled when "two_node" is enabled). But that's reasonable, even if qdevice is not in place.

So to me, this could be a case where we'd rather tell user "I can remove the on-disk configuration of qdevice for you, but you'll either want to restart cluster later manually by yourself, otherwise I can restart the cluster for you now but bring up all the nodes".

BTW, it's really confusing to still call it QDEVICE_RESTART_LATER when it's leveraging maintenance mode and actually restarting the cluster. The name QDEVICE_RESTART_LATER should probably be preserved for the situation where it really requires users to manually restart their cluster by themselves.

After all, as said, to determine whether a cluster would become malfunctioning, the logic in "warn_diskless_sbd()" and the warning there tell the key:

"Diskless SBD requires cluster with three or more nodes. If you want to use diskless SBD for 2-node cluster, should be combined with QDevice."

liangxin1300 added 2 commits March 3, 2026 18:04

Dev: bootstrap: Reject to remove qdevice when it will cause quorum lost

574da08

`crm cluster remove --qdevice` command will be rejected if it will cause quorum lost for the cluster configured with diskless SBD, since the diskless SBD will self-fence the node without quorum

Dev: unittests: Adjust unit test for previous commit

e558b8f

liangxin1300 requested a review from zzhou1 March 4, 2026 06:36

liangxin1300 marked this pull request as ready for review March 4, 2026 06:36

liangxin1300 requested a review from nicholasyang2022 March 4, 2026 06:36

zzhou1 requested changes Mar 18, 2026

View reviewed changes

	utils.fatal("Can't remove QDevice since the quorum will be lost for diskless SBD cluster")
	utils.fatal("operation is rejected since it would result in a malfunctioning cluster lacking quorum")

Uh oh!

Conversation

liangxin1300 commented Mar 3, 2026

Uh oh!

codecov Bot commented Mar 3, 2026

Codecov Report

Uh oh!

gao-yan commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liangxin1300 commented Mar 4, 2026

Uh oh!

gao-yan commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liangxin1300 commented Mar 5, 2026

Uh oh!

gao-yan commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zzhou1 commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zzhou1 commented Mar 17, 2026

when the cluster is using diskless SBD, and will lose quorum after removing qdevice

corosync-quorumtool -s

if we allow removing qdevice, after restart cluster, the existing node will be fenced by diskless SBD

so, we reject at this time

Uh oh!

gao-yan commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zzhou1 Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

zzhou1 Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

gao-yan Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

gao-yan commented Mar 3, 2026 •

edited

Loading

gao-yan commented Mar 4, 2026 •

edited

Loading

gao-yan commented Mar 5, 2026 •

edited

Loading

zzhou1 commented Mar 17, 2026 •

edited

Loading

gao-yan commented Mar 17, 2026 •

edited

Loading

gao-yan Mar 18, 2026 •

edited

Loading