ateapi: add ActorSnapshot lifecycle APIs

[Eitan Yarmush (EItanya)]

Implements #529.

This branch is built on top of the storage-layout commit from #568. The PR targets main independently, so it currently includes that first commit; once #568 merges, this branch can be rebased and the PR will contain only the ActorSnapshot lifecycle commit.

• creates a first-class ActorSnapshot on every successful suspend
• exposes the new snapshot through Actor.latest_snapshot; callers fetch details with GetActorSnapshot
• adds get, list, delete, tag, tag deletion, and scope update APIs
• supports full/data content scope and actor/atespace/global visibility
• adds globally unique, stable snapshot tags with O(1) indexed lookup
• stores tags on the snapshot itself, avoiding a redundant reverse-index set and resume read
• keeps tag addresses stable when snapshot scope changes
• permits tagged snapshots to initialize actors within their visibility scope
• currently requires the exact source ActorTemplate UID for both full and data snapshots
• pins snapshots while they have tags and keeps them when an atespace is deleted

Validation:

• go test ./cmd/ateapi/internal/controlapi ./cmd/ateapi/internal/store/ateredis ./cmd/atecontroller/internal/controllers
• make test
• make verify
• Kind certificate-auth E2E on gVisor and micro-VM runtimes
• external-volume lifecycle under gVisor

[Dmitry Berkovich (dberkov)]

An additional hop to persistence will be required for every resume operation. It introduces latency on the hot path. For 100M+ agents it will increase memory for redis or memory for indexing data in postgress.

Will it be simpler to save ActorSnapshotScope for tagged versions only? It will be cheaper solution and will not be used very often.

[Eitan Yarmush (EItanya)]

Good point. We can add more data to the Actor so we don’t need to query on the hot path. However this feels like a hack around redis issues vs a real requirement, this could be a single query in Postgres

[Julian Gutierrez Oschmann (juli4n)]

Can you document these values and their semantic?

[Dmitry Berkovich (dberkov)]

Template ID for data supposed to match too.

I still dont know how exactly Zoe Zhao (@zoez7) will implement the template snapshoting but most likely what needs to be validated is a common layer of snapshot. We need to use data part of the snapshot, If template is full and template version does not match.

For gVisor it still not possible to extract durDir from the snapshot, however it is possible for microVM.

I would validate right now just template ID and create a TODO to fix it, once Zoe added support for multi versioning and Fabricio Voznika (@fvoznika) will add support for gVisor to extract data part from the memory.

[Dmitry Berkovich (dberkov)]

The actor already has field "latest_snapshot", what is a reason to return full snapshot as a separate field?

[Dmitry Berkovich (dberkov)]

out of curiosity, why lock is required? The code supposed to add a new instance of actorSnapshot ?

[Dmitry Berkovich (dberkov)]

I have completed an initial review pass and wanted to share high-level feedback before doing a deep dive into the PR.

My primary concern is generalizing "tagged" snapshots alongside golden and untagged ones. The current design introduces an extra persistence read on the hot path, which offers no advantage in 99% of cases.

Typically, an actor's lifecycle follows a simple sequence: run -> suspend -> run -> suspend. Since tagging is rarely needed, snapshot details can remain internal to the actor rather than being exposed to an external entity.

There are two distinct types of tagging:
Per-actor: Used to roll back to a well-defined state. These snapshots should not be garbage-collected by the retention policy, but they must be deleted when the actor itself is deleted.
Per-atespace: The snapshot lifecycle is tied directly to the atespace and cleaned up when the atespace is deleted. This type is only used during actor creation, and I am not aware of any use case for reverting an actor to this kind of snapshot.

Regarding golden snapshots: the current layer requires a full redesign to support management by CPU type. While the new ActorSnapshot entry could potentially facilitate this, I suggest excluding it for now. We can extend the ActorSnapshot layer to handle golden snapshots once we initiate that broader redesign.

By scoping down the tagging use cases, the entire PR can be significantly simplified.

What do you think?

[Eitan Yarmush (EItanya)]

My primary concern is generalizing "tagged" snapshots alongside golden and untagged ones. The current design introduces an extra persistence read on the hot path, which offers no advantage in 99% of cases.

If the main design is the extra read then I can solve that as per my comment here: #570 (comment)

Typically, an actor's lifecycle follows a simple sequence: run -> suspend -> run -> suspend. Since tagging is rarely needed, snapshot details can remain internal to the actor rather than being exposed to an external entity.

I'm not sure we know that tagging is rarely needed, we're making that assumption

There are two distinct types of tagging:
Per-actor: Used to roll back to a well-defined state. These snapshots should not be garbage-collected by the retention policy, but they must be deleted when the actor itself is deleted.
Per-atespace: The snapshot lifecycle is tied directly to the atespace and cleaned up when the atespace is deleted. This type is only used during actor creation, and I am not aware of any use case for reverting an actor to this kind of snapshot.

It's not clear to me that these are the only ones. We specifically discussed having globally scoped snapshots which can be shared outside of the atespace.

By scoping down the tagging use cases, the entire PR can be significantly simplified.

I think there are tradeoffs to both approaches. The ways that I can imagine scoping this down would also build us into corners. I think the other viable option would be tracking n-1 snapshots directly on the actor, and then the tagging operation would actually create the ActorSnapshot object. What do you think?