Scale-down reconciliation gets stuck after CockroachDB node has been decommissioned

[zhengcharon]

Scale-down reconciliation gets stuck after CockroachDB node has been decommissioned

Summary

When scaling down a CrdbCluster, the operator may enter an infinite reconciliation loop after the target CockroachDB node has already been decommissioned.

In our case:

• Scaling from 5 → 4 starts normally.
• The target CockroachDB node is successfully decommissioned.
• The operator then repeatedly reports:

could not find the id of replica 4

and retries forever.

---

Environment

• CockroachDB Operator: v2.18.3
• CockroachDB: v25.4.2
• Kubernetes: v1.33.3

CrdbCluster configuration

spec:
  nodes: 4

  image:
    name: image.ccos.io/cockroachdb/cockroach:v25.4.2
    pullPolicy: IfNotPresent

  tlsEnabled: true

  grpcPort: 26258
  sqlPort: 26257
  httpPort: 8080

  maxUnavailable: 1

  resources:
    requests:
      cpu: 500m
      memory: 2Gi
    limits:
      cpu: "2"
      memory: 8Gi

  dataStore:
    supportsAutoResize: false
    pvc:
      spec:
        accessModes:
          - ReadWriteOnce
        storageClassName: ccos-hostpath-data-stor
        volumeMode: Filesystem
        resources:
          requests:
            storage: 60Gi

---

Steps to reproduce

1. Start with a healthy 5-node cluster.
2. Update CrdbCluster.spec.nodes from 5 to 4.
3. Wait for the operator to perform the scale-down.

---

Observed behavior

The operator starts decommissioning cockroachdb-4.

The Pod eventually becomes:

0/1 Running

Meanwhile:

• cockroach node status changes from

2 3 4 7 9

to

2 3 4 7

indicating that the target CockroachDB node (NodeID 9) has already disappeared.

crdb_internal.gossip_nodes still contains the node but reports:

• is_live = false
• ranges = 0
• leases = 0

The operator then repeatedly logs:

could not find the id of replica 4

from pkg/actor/decommission.go.

The reconciliation loop never converges and continues retrying indefinitely.

---

Expected behavior

The operator should complete the scale-down once the target node has been successfully decommissioned.

If the node can no longer be resolved, it should either:

• treat the decommission as already completed, or
• return a terminal, actionable error instead of retrying forever.

---

Additional context

This cluster previously experienced another scale-down failure (5 → 3) with a different error (unexpected node status).

That issue required manually deleting the Pod (and its PVC) before reconciliation could continue. After recovery, the recreated Pod joined as a new CockroachDB node, resulting in non-contiguous node IDs:

Pod	Node ID
cockroachdb-0	7
cockroachdb-1	2
cockroachdb-2	3
cockroachdb-3	4
cockroachdb-4	9

We do not know whether this earlier incident is related, but include it in case it is helpful for debugging.

---

Attachments

• Operator log covering the scale-down and repeated reconciliation failures.

operator.log

[zhengcharon]

After looking into the operator implementation and comparing it with the CLI output, I think I found the root cause of this issue.

The infinite retry on:

could not find the id of replica 4

appears to be a secondary symptom rather than the initial failure.

The first failure actually happens while checking the decommission progress.

The operator executes:

cockroach node status --decommission --format=csv

and parses the CSV using fixed column indexes:

isLive := record[8]
replicasStr := record[9]
isDecommissioning := record[10]

However, on CockroachDB v25.4.2, the CSV header is:

id,address,sql_address,build,started_at,updated_at,locality,attrs,is_available,is_live,gossiped_replicas,is_decommissioning,membership,is_draining

Therefore:

Expected by operator | Actual column -- | -- record[8] -> is_live | is_available record[9] -> replicas | is_live record[10] -> is_decommissioning | gossiped_replicas

As a result, the operator returns:

unexpected node status

on every reconcile.

While the operator keeps retrying, CockroachDB continues the decommission process, and eventually the target node disappears from the output of:

cockroach node status --format=csv

At that point, findNodeID() can no longer map the StatefulSet ordinal to a CockroachDB NodeID, resulting in:

could not find the id of replica 4

So it seems the could not find the id of replica 4 error is a consequence of the earlier CSV parsing failure rather than the original cause.

Could this be caused by a CLI output format change in newer CockroachDB versions?