feat(controller): add spec.flushOnFailover to flush a promoted master

[nmass-betpawa]

PR description

Summary

Adds an opt-in spec.flushOnFailover field. When it is set, a replica promoted during an unplanned
failover is flushed right after SLAVE OF NO ONE and before its role: master label is patched.
This is for cache workloads that must not serve data which is stale relative to the master that was
lost: the promoted pod can be behind the dead master, and today there is no way to have the new
master start empty.

The placement between those two calls is the point of doing this in the operator. The master
Service selects on RoleLabelKey: Master (internal/resources/resources.go), so while the flush
runs the promoted pod is not an endpoint of it yet and no client can read the stale dataset through
the Service.

This follows the discussion in dragonflydb/dragonfly#8051, where the conclusion was that the policy
belongs here rather than in a Dragonfly server flag.

Changes

• New optional field DragonflySpec.FlushOnFailover, default false.
• replicaOfNoOne() issues FLUSHALL after a successful SLAVE OF NO ONE, guarded by a check that
  the pod really was a replica before the call.
• Regenerated CRDs: config/crd/bases, manifests/, charts/.

Behavior

• Default false — no change for existing deployments.
• The role check matters: replicaOfNoOne() also runs during initial master election, where the pod
  is already a master and its dataset — for example one restored from a snapshot after a full
  restart — must be preserved. Only a replica -> master transition flushes.
• Rolling updates are unaffected. They promote through updatedMaster() -> replTakeover(), a
  coordinated takeover that loses no data, so flushing there would mean a cold cache and a backend
  load spike on every upgrade.
• A failing FLUSHALL returns before the role label is patched, so the pod is never advertised as
  master with a stale dataset; the reconciler retries.

Testing

Manually, on a 3-replica instance with the field enabled: delete the master pod, then confirm the
promoted pod logs flushing the promoted master and reports dbsize 0, and that the remaining
replicas come back empty after they re-sync.

[Abhra303]

Hey @nmass-betpawa, can you elaborate more on the use case? Why do you need a replica if you don't want to retain the data? If you don't want to retain the data, you can just have a single master with no replica.

[nmass-betpawa]

Hey @nmass-betpawa, can you elaborate more on the use case? Why do you need a replica if you don't want to retain the data? If you don't want to retain the data, you can just have a single master with no replica.

The replica is there for availability, not for durability.
We cache prices from a third-party provider and they move fast. A miss is cheap: we re-fetch and repopulate. A stale hit is expensive: it means quoting on a price that has since moved, and the application cannot tell a current value from one that is N seconds behind, so "possibly stale" has to be treated as "wrong". After an unplanned failover the promoted replica sits at an unknown point in the past. TTLs do not help, because the problematic key is exactly the one the master updated and the replica never received: its old value still has a valid TTL and looks fresh.

Why not a single master: that is what we run today, and a node going down is full downtime and it leaves us with an empty cache anyway. A replica plus this flag gives the same empty cache without the outage. We are not trying to keep data across a failover, we are trying not to be down during one.

Planned promotions should keep the data, which is why REPLTAKEOVER is left alone here: a takeover waits for full sync, so the dataset is valid.