Harden deployment status lifecycle correctness, scalability, and client parity

[artemnikitin]

Summary

Follow-up review of the deployment status implementation added by #26 found
several correctness, lifecycle, scalability, and client-parity gaps.

The most important problem is that the current status surface can look cleaner
or healthier than the real runtime: removed nodes remain forever, an old agent
incarnation can mark its replacement down, empty desired state can hide
surviving VMs, and the overview does not summarize reconciliation or health.

This is an umbrella issue. Existing focused issues remain canonical where they
overlap:

• Represent empty desired state explicitly so removed services are stopped #23 — explicit empty desired state and stopping removed services
• Report applied revisions and reconciliation conditions from agents #24 — applied revisions, reconciliation conditions, and fleet convergence
• Cleanup stale nodes/volumes #33 — cleanup of stale nodes and retained volumes

The work can be split across multiple PRs.

Node lifecycle

• Add an explicit deregistration/tombstone and retention policy for removed
  nodes. Registry records must not remain in node lists forever.
• Preserve down as a distinct lifecycle state instead of converting a
  down node to stale after the lease TTL.
• Include the agent generation/incarnation in node state updates and reject
  stale updates.
• Prove that an old process shutting down cannot mark a newer, heartbeating
  process with the same node ID as down.
• Define cleanup behavior for unexpected disappearance versus graceful
  removal, including how retained local volumes remain discoverable after
  the node record is archived or deleted.

Relevant paths:

• internal/controlplane/registry.go
• internal/controlplane/visibility.go
• internal/agent/registry_client.go

Desired state versus actual runtime

• Complete Represent empty desired state explicitly so removed services are stopped #23 so an explicit empty node config is reconciled normally,
  while a transient missing/unreadable object preserves last-known-good
  runtime state.
• Continue reporting actual-only/orphaned VMs until their deletion is
  confirmed; a desired-state-first list must not silently hide them.
• Fail multi-label config loading closed if any configured label cannot be
  fetched or parsed. Never reconcile a partial merged configuration.
• Do not report a partial configuration as successfully fetched or
  converged.

Relevant paths:

• internal/controlplane/controller.go
• internal/agent/agent.go
• internal/agent/status.go
• internal/controlplane/visibility.go

Aggregate status truthfulness

• Complete Report applied revisions and reconciliation conditions from agents #24 with an aggregate revision/system status endpoint and
  fireworkctl status.
• Distinguish lifecycle readiness from reconciliation convergence.
• Surface unhealthy services in the overview; running must not imply
  healthy.
• Report published, progressing, converged, degraded, failed, stale, down,
  and unknown states without requiring one detail request per node.
• Use distinct reasons for unsupported agent schema, missing status, stale
  heartbeat, stale observation, down node, and revision mismatch.
• Avoid relying solely on the agent wall clock for transport freshness.

Heartbeat/status input safety

• Limit registry request bodies.
• Bound condition, service, volume, name, revision, and reason-code counts
  and lengths.
• Validate enum values and reject duplicate condition types, service names,
  and volume IDs.
• Return an actionable 4xx response for invalid node status rather than a
  generic persistence 500.
• Keep rolling upgrades backward compatible and fail unsupported schemas
  closed without labeling them as merely stale.

Persistent-volume status correctness

• Preserve durable applied_size_bytes when an otherwise partial agent
  observation reports zero/omits the applied size.
• Ensure service list and service detail produce consistent disk totals.
• Acknowledge volume records only from a fresh, supported, current-revision
  observation for the service's current placement.
• Prevent stale records from a previous node placement from changing a
  shared volume's durable resize state.
• Represent unknown/not-configured storage separately from known zero
  capacity; the UI should not render both as 0 B / 0 B.

Relevant paths:

• internal/controlplane/visibility.go
• internal/controlplane/volume_records.go
• internal/statusmodel/status.go

API scalability and failure isolation

Today each visibility request lists and reads all node and volume objects. The
browser overview independently calls both node and service endpoints every
7.5 seconds, duplicating the full projection.

• Build/cache one immutable visibility snapshot for a short interval or
  expose a combined overview endpoint.
• Avoid duplicate object-store fan-out for node and service views.
• Add pagination or a documented bounded deployment size.
• A single malformed/unreadable stale node or volume record must not take
  the entire status API and UI offline; skip/quarantine it and expose a
  degraded projection reason.

CLI and UI parity

• Add local disk capacity/allocation to fireworkctl nodes and node detail.
• Add service disk summaries to fireworkctl services, with human-readable
  sizes in table output while preserving exact bytes in JSON.
• Show the service public URL in CLI detail.
• Decide and document parity for node labels, registration/update time, and
  desired/placement/observed/applied revisions.
• Add health and node filters to the service UI, matching the API/CLI.
• Cancel or sequence browser refresh requests so a slow older response
  cannot overwrite a newer view after navigation.
• Define ready_services strictly: a running service with a configured but
  not-yet-successful health check must not be counted as ready.

Test coverage

• Node cleanup, down retention, and stale-to-removed transitions.
• Overlapping old/new agent generations with out-of-order shutdown.
• Empty desired state with existing VMs and actual-only status.
• Partial multi-label fetch and parse failures.
• Aggregate convergence and unhealthy-running services.
• Oversized, malformed, duplicate, and unsupported heartbeat status.
• Durable/observed volume merge and stale shared-volume acknowledgements.
• CLI table columns and UI/CLI parity.
• Browser navigation versus periodic-refresh races.
• Object-store fan-out/caching and malformed-record failure isolation.

Acceptance criteria

• Removed infrastructure disappears from the active node list according to a
  documented retention policy without losing retained-volume recovery data.
• A live node cannot be taken down by a stale agent incarnation.
• Status never reports an empty or converged deployment while unreported
  Firecracker VMs are known to remain.
• The overview and CLI provide one authoritative current-revision status and
  clearly expose unhealthy/degraded components.
• Status ingestion and projection remain bounded and available in the presence
  of malformed, stale, mixed-version, or large inputs.
• Disk information is semantically consistent across API, CLI, and UI.