Skip to content

Add auto-healing and automatic workload migration #62

Description

@flo2517

Detect provider failure and redeploy the affected workloads onto healthy providers automatically, removing the human operator from the recovery path. This is where availability stops depending on someone watching a dashboard, and it is why it sits behind on-chain orchestration: an automatic redeployment is a scheduling decision, and per #50 those decisions must be made where no single party controls them.

Blocked by: ADR-016 (see ADR-012 §6) — no implementation before acceptance. Depends on #50, and on #59 for workloads with persistent volumes.

Acceptance criteria:

  • failure detection distinguishes a dead provider from a partitioned observer, and a single observer can never trigger a migration alone;
  • the decision to migrate is reproducible from finalized chain state, consistent with Move workload orchestration to on-chain smart contracts #50 — no privileged Control Plane input;
  • a workload is never running in two places at once: the old instance is provably stopped, or fenced, before the new one is reported RUNNING, honoring the authoritative-confirmation rule in AGENTS.md;
  • migration is idempotent and bounded — retries are capped, and repeated failure ends in an explicit stopped state rather than an infinite redeployment loop;
  • persistent volumes (Add replicated block volumes across providers #59) reattach in a defined order, and a migration that cannot reattach fails explicitly instead of silently starting with an empty disk;
  • lease, payment, and reputation consequences of a migration are defined: who pays for the overlap, and whether the failed provider is penalized (Add provider slashing to back availability commitments #52);
  • migration cannot be induced as an attack — a provider or observer cannot force a competitor's workloads to churn;
  • tests cover a genuine provider failure, a partitioned observer that must not trigger migration, split-brain with both instances alive, volume reattachment failure, repeated migration failure hitting the retry cap, and a flapping provider.

Original request (FR, preserved verbatim):

Détecter la panne d'un node et redéployer automatiquement le workload sur un autre provider.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions