Summary
Make an explicit empty desired state distinguishable from a transient config-fetch failure.
Today the controller omits node configs with zero assigned services and removes stale cp/v1/nodes/<node>.yaml objects. When an agent cannot fetch any of its configured node files, fetchAndMerge returns nil; the tick refreshes remote routes and registry state, but does not reconcile against an empty NodeConfig.
As a result, a node that loses its final assigned service can keep running stale microVMs.
Current behavior
Relevant paths:
internal/scheduler/scheduler.go: BuildNodeConfigs skips assignment entries with no services.
internal/controlplane/controller.go: publishRendered deletes legacy/current node YAML objects absent from the new placement.
internal/agent/agent.go: when fetchAndMerge returns nil, the agent returns without reconciling local VMs.
This conflates two states:
- Desired state is known and contains zero services.
- Desired state is unknown because the store is unavailable or the object is temporarily missing.
Only the first state should stop workloads.
Impact
- Removing all services from GitOps may leave previously running VMs alive.
- Moving the last service away from a node may leave a stale copy running there.
- Node drain or rescheduling can create duplicate workload instances and unsafe split-brain behavior.
- Operators cannot reliably express "this node should run nothing."
Proposed direction
- Publish an explicit rendered
NodeConfig with services: [] for every relevant registered node, including nodes that just lost their final assignment.
- Keep explicit empty state separate from fetch/list failures.
- Reconcile an explicit empty config normally so all local services are deleted.
- Define retention or tombstone semantics for nodes that are no longer registered.
- Do not turn a transient object-store failure into an implicit empty desired state.
The exact representation may be an empty node document, a revisioned assignment/tombstone, or another explicit mechanism, as long as absence is not overloaded.
Acceptance criteria
- Removing the final desired service removes its VM, health check, port forwards, TAP device, and Traefik route.
- Moving all services away from a node stops old copies before the move is considered converged.
- A transient config-store read failure preserves last-known-good running state.
- A newly registered node with no assignments has an explicit converged empty state.
- Repeated empty-state reconciliation is idempotent.
- Unit tests cover empty placement rendering and agent reconciliation.
- A validation test covers service removal from a node that becomes completely empty.
Summary
Make an explicit empty desired state distinguishable from a transient config-fetch failure.
Today the controller omits node configs with zero assigned services and removes stale
cp/v1/nodes/<node>.yamlobjects. When an agent cannot fetch any of its configured node files,fetchAndMergereturnsnil; the tick refreshes remote routes and registry state, but does not reconcile against an emptyNodeConfig.As a result, a node that loses its final assigned service can keep running stale microVMs.
Current behavior
Relevant paths:
internal/scheduler/scheduler.go:BuildNodeConfigsskips assignment entries with no services.internal/controlplane/controller.go:publishRendereddeletes legacy/current node YAML objects absent from the new placement.internal/agent/agent.go: whenfetchAndMergereturnsnil, the agent returns without reconciling local VMs.This conflates two states:
Only the first state should stop workloads.
Impact
Proposed direction
NodeConfigwithservices: []for every relevant registered node, including nodes that just lost their final assignment.The exact representation may be an empty node document, a revisioned assignment/tombstone, or another explicit mechanism, as long as absence is not overloaded.
Acceptance criteria