Add first-class ActorSnapshot lifecycle APIs

[EItanya]

Summary

Substrate creates a durable checkpoint whenever it suspends an Actor. Make each successful checkpoint a first-class, immutable ActorSnapshot that callers can get, list, tag, delete, and use to initialize another Actor.

The design has changed during implementation. Superseded decisions remain crossed out below so the history is visible:

• A caller explicitly promotes a suspended Actor's private checkpoint with a separate CreateActorSnapshot RPC. Ordinary suspension does not create an ActorSnapshot.
  Updated: every successful SuspendActor automatically creates an ActorSnapshot and exposes its canonical reference through Actor.latest_snapshot.
• Snapshot data is partitioned under /root/atespace/<atespace_name>/snapshots/<uuid>.
  Updated: snapshots live under <configured snapshot root>/snapshots/<opaque-id>. The physical path carries no atespace or visibility semantics, so deleting an Actor or Atespace cannot accidentally delete a tagged snapshot by prefix.
• Tagging is a boolean retention operation and also changes snapshot scope.
  Updated: a tag is a globally unique, stable alias for a snapshot. Tagging and scope changes are separate operations.
• Snapshots can only be reused within their source atespace.
  Updated: reuse is controlled by an explicit ACTOR, ATESPACE, or GLOBAL visibility scope. The default is ACTOR.

Automatic retention limits are intentionally deferred. A future n-x policy may garbage-collect untagged snapshots; any snapshot with at least one tag is pinned.

Motivation

Control planes built on Substrate need to preserve an Actor at a known state and later create an independent Actor from that state. A higher-level system may also associate a snapshot with application history or metadata that Substrate does not own.

This cannot safely be implemented above Substrate by copying storage fields or object-storage prefixes. That would bypass Substrate's compatibility checks, authorization boundary, ownership of storage layout, and reference tracking.

Current behavior

Before this proposal:

• SuspendActor created an external checkpoint stored as private resume state.
• ResumeActor restored the latest checkpoint or the ActorTemplate golden snapshot.
• PauseActor used node-local snapshots.
• External volumes survived suspend/resume independently and were not copied into the sandbox checkpoint.
• The Control API had no first-class snapshot resource or snapshot RPCs.

With this design, a successful suspend records the completed checkpoint as an immutable ActorSnapshot and sets Actor.latest_snapshot to its canonical ObjectRef. SuspendActor returns the Actor; callers inspect the snapshot with GetActorSnapshot. PauseActor remains node-local and does not create an ActorSnapshot.

Goals

• Promote a suspended Actor's current checkpoint only when explicitly requested. Create a first-class snapshot automatically after every successful suspend.
• Keep multiple snapshots of one Actor independently addressable.
• Get, list, tag, update scope, delete tags, and delete unreferenced snapshots.
• Create an independent Actor from a tagged full or data snapshot within its visibility scope.
• Keep physical storage locations private to Substrate.
• Keep a tag address stable when snapshot scope changes.
• Make tag lookup indexed rather than an O(n) snapshot scan.
• Prevent deletion of tagged or otherwise referenced snapshots.
• Preserve external-volume suspend/resume behavior while volume snapshot/clone support is designed separately.

Non-goals

• Implementing the future automatic n-x retention or garbage-collection policy.
• Adding UID/version preconditions to lifecycle RPCs; those should be designed consistently in a separate change.
• In-place rollback of an existing Actor.
• Snapshotting an Actor while it continues running.
• Cloning node-local pause snapshots.
• Snapshotting application history or state owned by external services.
• Implementing external-volume snapshot and clone support in this first version.

Proposed API

ActorSnapshot

Content scope and visibility scope are distinct:

enum SnapshotContentScope {
  SNAPSHOT_CONTENT_SCOPE_UNSPECIFIED = 0;
  SNAPSHOT_CONTENT_SCOPE_FULL = 1;
  SNAPSHOT_CONTENT_SCOPE_DATA = 2;
}

enum ActorSnapshotScope {
  ACTOR_SNAPSHOT_SCOPE_UNSPECIFIED = 0;
  ACTOR_SNAPSHOT_SCOPE_ACTOR = 1;
  ACTOR_SNAPSHOT_SCOPE_ATESPACE = 2;
  ACTOR_SNAPSHOT_SCOPE_GLOBAL = 3;
}

message ActorSnapshot {
  ResourceMetadata metadata = 1;
  ObjectRef source_actor = 2;
  string source_actor_uid = 3;
  int64 source_actor_version = 4;
  string actor_template_namespace = 5;
  string actor_template_name = 6;
  string actor_template_uid = 7;
  SnapshotContentScope content_scope = 8;
  ActorSnapshotScope scope = 9;
  repeated string tags = 10;
}

metadata.name is a server-assigned opaque snapshot ID. Snapshot contents are immutable; scope and tags are mutable control-plane metadata. The physical storage location remains private to Substrate.

ACTOR is the default visibility and permits recreating only the source Actor identity. ATESPACE permits use by Actors in the source atespace. GLOBAL permits use across atespaces, subject to compatibility and authorization checks.

Stable snapshot references and tags

Snapshots are addressed only by (atespace, name), and tagging is represented by a boolean.

Snapshots may be addressed by canonical identity or by a globally unique tag:

message ActorSnapshotRef {
  oneof reference {
    ObjectRef snapshot = 1;
    string tag = 2;
  }
}

A snapshot may have multiple tags. A tag:

• is globally unique;
• resolves directly through an index, so lookup is O(1);
• remains bound to the same canonical snapshot when its scope changes;
• pins the snapshot against garbage collection and explicit deletion; and
• may be removed independently with DeleteActorSnapshotTag.

Assigning an existing tag to the same snapshot is idempotent. Assigning it to another snapshot returns ALREADY_EXISTS.

SuspendActor automatically creates the snapshot

SuspendActor only creates a private checkpoint. A caller later invokes CreateActorSnapshot, which promotes that checkpoint idempotently.

message SuspendActorResponse {
  Actor actor = 1;
}

Every successful suspend creates one immutable ActorSnapshot from the completed durable checkpoint, sets Actor.latest_snapshot to its canonical reference, and returns the Actor containing that reference. Callers use GetActorSnapshot when they need the snapshot resource. A later suspend creates a new snapshot rather than replacing the previous resource. Existing workflow idempotency prevents retries of the same suspension from proliferating snapshots.

Snapshot data is stored at:

<ActorTemplate snapshotsConfig.location>/snapshots/<opaque-id>

The path deliberately omits Actor, Atespace, scope, and tag. Those semantics live in Substrate's database and API, which lets a tagged snapshot outlive its source Actor or Atespace and change scope without moving bytes.

Snapshot lifecycle RPCs

rpc GetActorSnapshot(GetActorSnapshotRequest) returns (ActorSnapshot) {}
rpc ListActorSnapshots(ListActorSnapshotsRequest)
    returns (ListActorSnapshotsResponse) {}
rpc TagActorSnapshot(TagActorSnapshotRequest) returns (ActorSnapshot) {}
rpc UpdateActorSnapshot(UpdateActorSnapshotRequest) returns (ActorSnapshot) {}
rpc DeleteActorSnapshotTag(DeleteActorSnapshotTagRequest)
    returns (ActorSnapshot) {}
rpc DeleteActorSnapshot(DeleteActorSnapshotRequest) returns (ActorSnapshot) {}

Tagging and scope changes are separate because a stable tag is an address, while scope is the current visibility/reuse policy. Changing scope must not change any existing tag address.

Deletion returns FAILED_PRECONDITION while the snapshot has a tag or is referenced by an Actor or ActorTemplate. After the final reference and tag are gone, deletion removes the resource and permits cleanup of its stored data.

Create an Actor from a snapshot

Extend the existing creation RPC rather than add a parallel CloneActor lifecycle:

message CreateActorRequest {
  Actor actor = 1;
  ActorSnapshotRef source_snapshot = 2;
}

The source snapshot must be tagged before explicit reuse. It may be supplied by canonical identity or tag; the created Actor stores the canonical ObjectRef in latest_snapshot.

Reuse rules:

• ACTOR: only the source Actor identity may be recreated;
• ATESPACE: the target must be in the source atespace;
• GLOBAL: the target may be in any atespace;
• FULL: the target must use the exact source ActorTemplate UID because the snapshot contains process and sandbox state;
• DATA: currently requires the exact source ActorTemplate UID; portable extraction and compatibility rules may relax this later; and
• duplicate target Actor names return ALREADY_EXISTS.

External volumes

External volumes are actor-owned live storage. They survive ordinary suspend/resume independently and are not included in the gVisor or micro-VM sandbox checkpoint.

The first version rejects snapshot creation for every Actor with external volumes.

Automatic suspension still creates the ActorSnapshot for the sandbox checkpoint, and ordinary external-volume lifecycle remains unchanged. However, this first version does not snapshot or independently clone external-volume contents when creating another Actor from a snapshot.

Future volume snapshot/clone support should use the existing volume-provider abstraction to call the CSI controller API and store opaque per-volume snapshot handles privately. Restoring would ask the provider to create new independent volumes; deleting the final snapshot reference would delete the provider snapshots.

Direct CSI integration is preferred over creating a Kubernetes VolumeSnapshot object per Actor snapshot. At Actor scale, potentially millions of CRDs in etcd and their controller reconciliation would put Actor-scale state back into the Kubernetes control plane. Drivers without snapshot/restore support can return FAILED_PRECONDITION. NFS support may use a provider-specific archive/restore implementation while the Actor remains suspended.

References, deletion, and retention

• An Actor's latest_snapshot and an ActorTemplate golden snapshot keep that snapshot alive.
• Every tag keeps its snapshot alive.
• Deleting the source Actor or Atespace does not delete a tagged snapshot.
• Scope changes do not move snapshot bytes or alter tag resolution.
• Substrate may delete stored bytes only after the resource, its tags, and all Actor or ActorTemplate references are gone.
• Ordinary suspension remains bounded to one private checkpoint per Actor. Every suspension creates a first-class snapshot; a future n-x policy will bound untagged snapshot retention.

Consistency boundary

An ActorSnapshot guarantees a consistent Substrate Actor image for the storage types it supports. It does not include caller-owned application history, state in external services, or—until provider support is added—an independently cloneable copy of external volumes.

A higher-level control plane must stop new work, wait for current work to finish, and save any history it owns before suspension. It may associate that history position with the resulting snapshot tag, but Substrate does not manage that combined checkpoint.

Compatibility

• Existing SuspendActor request and response shapes remain unchanged; the returned Actor has latest_snapshot set to the created snapshot's canonical reference.
• Existing CreateActor requests omit source_snapshot and continue from the ActorTemplate golden snapshot.
• Snapshot resources and lifecycle RPCs are additive.
• Actor.latest_snapshot_info is removed and its protobuf name and field number are reserved.
• Lifecycle UID/version preconditions remain deferred to a separate API-wide design.

Expected errors

Condition	gRPC status
Actor, snapshot, or tag does not exist	NOT_FOUND
Target Actor name or requested tag already belongs to another resource	ALREADY_EXISTS
Actor is not in a valid state for suspension	FAILED_PRECONDITION
Untagged snapshot is used as an explicit creation source	FAILED_PRECONDITION
Snapshot scope does not permit the target Actor	FAILED_PRECONDITION
Full or data snapshot uses a different ActorTemplate UID	FAILED_PRECONDITION
Tagged or referenced snapshot is deleted	FAILED_PRECONDITION
Snapshot is incomplete or incompatible with the target runtime	FAILED_PRECONDITION
Snapshot manifest or data is missing or corrupt	DATA_LOSS

Acceptance criteria

• Every successful SuspendActor creates exactly one ActorSnapshot for that suspension and exposes its canonical reference through Actor.latest_snapshot.
• A later successful suspension creates another independently addressable snapshot.
• Snapshot storage uses <configured root>/snapshots/<opaque-id> without Actor, Atespace, scope, or tag path semantics.
• Get and paginated list enforce authorization.
• Tags are globally unique, stable aliases with indexed lookup.
• A snapshot may have multiple tags, and deleting one tag does not affect the others.
• Changing scope does not change canonical identity or tag resolution.
• Tagged snapshots survive source Actor and Atespace deletion.
• Actor, Atespace, and Global scope enforce their respective reuse boundaries.
• An Actor created from a full snapshot resumes with captured process memory, rootfs delta, and durable-directory data.
• An Actor created from a data snapshot currently requires the exact source ActorTemplate UID and starts fresh with supported checkpointed data restored.
• The source and created Actor can advance independently.
• Deleting a tagged or otherwise referenced snapshot is rejected.
• Concurrent snapshot deletion and Actor creation cannot produce a dangling Actor reference.
• Existing suspend, resume, pause, create, and external-volume lifecycle workflows continue to work.
• Automatic retention/GC policy is not coupled to lifecycle RPC request fields.

Open decisions

1. What portable extraction and compatibility rules should eventually allow DATA snapshots to use a different ActorTemplate?
2. What volume-provider API and private metadata are required to snapshot and independently clone external volumes across CSI and NFS providers?
3. What n-x policy should govern garbage collection of untagged snapshots?
4. Should lifecycle RPC preconditions later use UID/version fields, a common precondition message, or an opaque etag?
5. How should telemetry identity be reconciled after restoring a FULL snapshot into another Actor?

References

• ateapi.proto
• ActorTemplate snapshot configuration
• Substrate architecture
• Storage-layout PR #568
• ActorSnapshot lifecycle PR #570

[krisztianfekete]

A note on telemetry:
A FULL snapshot includes whatever OTel SDK (telemetry) identity the workload had at startup, so an Actor created from another Actor's FULL snapshot will resume with telemetry describing it as the source/original Actor. This is similar to golden template snapshots today.

In OTel, this is what Entities will solve.

If an observer cannot reliably obtain one or more identifying attributes, it MUST NOT emit telemetry using that entity type. Instead, it SHOULD:

• delegate to the observer that can supply the full set and treat that observer as the source of truth, or
• emit a different entity type with a set of identifying attributes it can populate reliably.

A cloned actor can't know its own ate.actor.uid, so we should give restore/create-from-snapshot a defined point where the platform reconciles actor identity instead of inheriting the snapshot one. Happy to open an issue to track this on the telemetry side.

[mini-cookie]

这是来自QQ邮箱的自动回复邮件。您好，已收到您的信息，祝您工作愉快

[EItanya]

External-volume snapshots at Actor scale

Substrate should use the CSI controller API through its volume-provider abstraction. When CreateActorSnapshot is called, the provider snapshots each external volume and returns an opaque handle that Substrate stores privately with the ActorSnapshot. Creating an Actor from that snapshot asks the same provider to create new independent volumes from those handles; deleting the ActorSnapshot deletes them after references are gone.

This uses CSI's standard snapshot and restore semantics without requiring a Kubernetes resource for every snapshot. A CSI adapter can call the driver's controller service directly; provider-specific implementations remain possible. Drivers without snapshot/restore support return FAILED_PRECONDITION. For NFS, the upstream CSI driver can implement this as an archive and restore, although the Actor must remain suspended while the archive is captured.

We prefer this over Kubernetes VolumeSnapshot objects because each snapshot would otherwise become a CRD stored in etcd and processed by Kubernetes controllers. At a potential scale of millions of Actors, that would move Actor-scale state back into the Kubernetes control plane, contrary to Substrate's architecture. Kubernetes objects may still provide the small amount of cluster-level driver and credential configuration, while per-Actor snapshot state remains in Substrate's scalable store and the storage backend.

[dberkov]

External-volume snapshots at Actor scale

Michelle Au (@msau42) could you please take a look on the comment above.

[EItanya]

Updated snapshot design decisions

Following discussion with the broader Substrate team:

1. Every successful suspend will automatically create a snapshot. We expect to add an n-x retention/garbage-collection policy later, but retention policy is out of scope for this change.

2. Snapshot data in GCS will be partitioned by atespace using the layout:

   /root/atespace/<atespace_name>/snapshots/<uuid>
   

3. Snapshots will have an ownership/visibility scope with these values:

   • ACTOR (default)
   • ATESPACE
   • GLOBAL

   This is separate from the existing FULL/DATA content scope, so the API should use distinct names for the two concepts.

4. Tagging a snapshot protects it from garbage collection and makes it explicitly referenceable. Untagged snapshots remain eligible for the future retention policy.

[HavenXia]

I might missed the broader discussion - but should we make different GCS bucket based on snapshot visibility scope (e.g. a global bucket and per-atespace buckets).

Also I proposed in the atespace design doc that tagged snapshots and defualt ACTOR scoped snapshots are both partitioned by atespace but under different subdirectory paths, e.g.

• /atespaces/<atespace_name>/actors/<actor_uuid>/snapshots/<snapshot_id>(an actor may have more than one snapshots?)
• /atespaces/<atespace_name>/tags/<tag_name>/

[EItanya]

I might missed the broader discussion - but should we make different GCS bucket based on snapshot visibility scope (e.g. a global bucket and per-atespace buckets).

Also I proposed in the atespace design doc that tagged snapshots and defualt ACTOR scoped snapshots are both partitioned by atespace but under different subdirectory paths, e.g.

• /atespaces/<atespace_name>/actors/<actor_uuid>/snapshots/<snapshot_id>(an actor may have more than one snapshots?)
• /atespaces/<atespace_name>/tags/<tag_name>/

Personally I don’t feel that snapshots need to be stored by any particular scope. The scope is given by substrate itself and the DB entry. We can also keep metadata here if we also want it collocated. Doing it that way ensures we don’t have to worry about delete semantics or actor + atespace for tagged snapshots. Also, the snapshots are an impl detail of substrate, that’s the security boundary for them, not the bucket.

[HavenXia]

That makes sense. With snapshots as first-class resources in our system, the path doesn't need to carry any semantics — permissions and lifecycle all go through the API anyway.

Snapshot data in GCS will be partitioned by atespace using the layout:
/root/atespace/<atespace_name>/snapshots/<uuid>

One followup on the GC side then: a GLOBAL snapshot would still physically sit under some atespace's prefix? So atespace deletion can't be a blind prefix delete — cleanup has to consult the database.

[EItanya]

One followup on the GC side then: a GLOBAL snapshot would still physically sit under some atespace's prefix? So atespace deletion can't be a blind prefix delete — cleanup has to consult the database.

My plan is just to remove the prefix altogether for that exact reason, the atespace doesn't really mean anything anymore, ownership is strictly semantic.

[HavenXia]

One followup on the GC side then: a GLOBAL snapshot would still physically sit under some atespace's prefix? So atespace deletion can't be a blind prefix delete — cleanup has to consult the database.

My plan is just to remove the prefix altogether for that exact reason, the atespace doesn't really mean anything anymore, ownership is strictly semantic.

LGTM, you might want to update the issue posts to reflect that to avoid future reader confusion.

[EItanya]

LGTM, you might want to update the issue posts to reflect that to avoid future reader confusion.

Updated