Skip to content

example: Add agent-sandbox suspend/resume example with pod snapshots - #5213

Open
vicentefb wants to merge 8 commits into
ray-project:masterfrom
vicentefb:raySuspendResumeArtifacts
Open

example: Add agent-sandbox suspend/resume example with pod snapshots #5213
vicentefb wants to merge 8 commits into
ray-project:masterfrom
vicentefb:raySuspendResumeArtifacts

Conversation

@vicentefb

@vicentefb vicentefb commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Why are these changes needed?

Follow-up to the existing Agent Sandbox sample (ray-operator/config/samples/agent-sandbox/, docs: Sandboxed Code Execution with Ray and Agent Sandbox (https://docs.ray.io/en/latest/cluster/kubernetes/examples/rayjob-agent-sandbox.html)).

This sample shows how to suspend sandboxes at the turn boundary and resuming them before the next turn, using GKE Pod Snapshots (https://docs.cloud.google.com/kubernetes-engine/docs/concepts/pod-snapshots) through the Agent Sandbox Python SDK's snapshot extension (k8s_agent_sandbox.gke_extensions.snapshots).

Related issue number

Labels

  • If this PR has user-facing changes that require documentation updates at release time, I have added the doc-updates-required label.
  • If this PR contains breaking changes, I have added the breaking-change label.

Checks

  • I've made sure the tests are passing.
  • Testing Strategy
    • Unit tests
    • Manual tests
    • This PR is not tested :(

Manual test instructions

  • sandboxed_snapshot_execution.py — Ray actors with remote suspend()/resume() methods; the driver suspends every sandbox at the turn boundary ("model is thinking") and resumes before the next turn. Restore is proven two ways: turn 0 writes state to /tmp (wiped on a plain pod restart) and turn 1 reads it back after the cycle, and every resume asserts the SDK's restored_from_snapshot flag. Any failed turn fails the job.
  • sandbox-snapshot.yaml — ServiceAccount, PodSnapshotStorageConfig, PodSnapshotPolicy (manual triggers, snapshots grouped per sandbox), SandboxTemplate (networkPolicyManagement: Unmanaged — the demo connects to the sandbox pod IP directly), SandboxWarmPool.
  • rbac.yaml — Role/RoleBinding for the Ray workers (sandbox patch, pod reads, podsnapshot.gke.io resources).
  • ray-job.yaml — RayJob wiring it together (rayproject/ray:2.55.0-py311; the SDK requires Python ≥3.11).

Tested end-to-end on a GKE Standard cluster (1.36.3, --enable-pod-snapshots, gVisor node pool), with agent-sandbox v1.0.0 and k8s-agent-sandbox==1.0.0. Full setup steps are in the companion docs page ray-project/ray#65779

Measured: warm-pool claim 0.2–0.6s, suspend (snapshot + upload + pod termination) ~5–6s, resume (pod recreate + memory restore) ~4.3s; /tmp state written before suspension was read back successfully after every resume, with restored_from_snapshot=True.

Comment thread ray-operator/config/samples/agent-sandbox/snapshots/ray-job.yaml

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit d1c0729. Configure here.

@andrewsykim andrewsykim changed the title example: Add agent-sandbox suspend/resume with GKE Pod Snapshots sample example: Add agent-sandbox suspend/resume example with pod snapshots Sep 2, 2026
Comment thread ray-operator/config/samples/agent-sandbox/snapshots/ray-job.yaml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants