Skip to content

Represent empty desired state explicitly so removed services are stopped #23

Description

@artemnikitin

Summary

Make an explicit empty desired state distinguishable from a transient config-fetch failure.

Today the controller omits node configs with zero assigned services and removes stale cp/v1/nodes/<node>.yaml objects. When an agent cannot fetch any of its configured node files, fetchAndMerge returns nil; the tick refreshes remote routes and registry state, but does not reconcile against an empty NodeConfig.

As a result, a node that loses its final assigned service can keep running stale microVMs.

Current behavior

Relevant paths:

  • internal/scheduler/scheduler.go: BuildNodeConfigs skips assignment entries with no services.
  • internal/controlplane/controller.go: publishRendered deletes legacy/current node YAML objects absent from the new placement.
  • internal/agent/agent.go: when fetchAndMerge returns nil, the agent returns without reconciling local VMs.

This conflates two states:

  1. Desired state is known and contains zero services.
  2. Desired state is unknown because the store is unavailable or the object is temporarily missing.

Only the first state should stop workloads.

Impact

  • Removing all services from GitOps may leave previously running VMs alive.
  • Moving the last service away from a node may leave a stale copy running there.
  • Node drain or rescheduling can create duplicate workload instances and unsafe split-brain behavior.
  • Operators cannot reliably express "this node should run nothing."

Proposed direction

  • Publish an explicit rendered NodeConfig with services: [] for every relevant registered node, including nodes that just lost their final assignment.
  • Keep explicit empty state separate from fetch/list failures.
  • Reconcile an explicit empty config normally so all local services are deleted.
  • Define retention or tombstone semantics for nodes that are no longer registered.
  • Do not turn a transient object-store failure into an implicit empty desired state.

The exact representation may be an empty node document, a revisioned assignment/tombstone, or another explicit mechanism, as long as absence is not overloaded.

Acceptance criteria

  • Removing the final desired service removes its VM, health check, port forwards, TAP device, and Traefik route.
  • Moving all services away from a node stops old copies before the move is considered converged.
  • A transient config-store read failure preserves last-known-good running state.
  • A newly registered node with no assignments has an explicit converged empty state.
  • Repeated empty-state reconciliation is idempotent.
  • Unit tests cover empty placement rendering and agent reconciliation.
  • A validation test covers service removal from a node that becomes completely empty.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions