[Proposal] Add first-class rollout groups for isolated, concurrent, verifiable agent execution

[acsoto]

Summary

Introduce rollout groups as a first-class AgentMesh control-plane capability for running multiple isolated, stateful, verifiable executions of the same work item.

This is inspired by volcano-sh/agentcube#267, which positions AgentCube as a stateful, isolated, concurrent rollout execution layer for agentic RL and verifiable agent tasks.

The idea fits AgentMesh well, but the responsibility boundary should be explicit:

• AgentMesh owns rollout intent, fan-out, scheduling policy, budgets, lifecycle, evaluation, selection, business state, and audit.
• A sandbox execution provider owns the isolated workspace/process/container lifecycle.
• AgentCube could be one Kubernetes-backed provider, but AgentMesh should not require it. Local containers and other sandbox systems should remain possible.

This would give AgentMesh an early concrete use case for reviewed and coordinated execution without making AgentMesh an RL trainer.

Why this fits the existing architecture

AgentMesh already has most of the required semantics in its formal design:

• A Run is one complete execution trajectory with a stable LangGraph thread_id.
• An Attempt is an infrastructure execution lease inside a Run. Lease expiry or worker replacement creates another Attempt, not another logical trajectory.
• The task domain already reserves an exception to the "one active Run per Work Item" invariant for speculative execution, with explicit winner/loser convergence.
• The orchestrator design already defines concurrency limits, budget admission, checkpoints, fork semantics, leases, fencing, and reconciliation.
• The evaluation design already separates business-affecting acceptance criteria from ordinary quality scores.
• The local runtime design already calls for short-lived, resource-limited, network-denied-by-default sandboxes.

The missing abstraction is not another name for Run. It is a durable entity that groups sibling Runs created from the same immutable input snapshot.

Proposed terminology and mapping

Rollout concept	AgentMesh mapping
Verifiable task definition	Task/Work Item snapshot + Acceptance Criteria + input ArtifactRefs
One logical rollout	One Run
Worker retry after crash/lease expiry	A new Attempt in the same Run
Stateful trajectory	Run thread_id + checkpoints + workspace/artifact lineage
N-way fan-out	A new RolloutGroup containing sibling Runs
Evaluator/reward	Versioned evaluator result recorded as CriterionResult and/or Score
Best-of-N winner	Explicit selection result; never inferred from whichever Run finishes first
Resume	Continue the same Run and Thread after guards
Fork	Create a new child Run from an allowed checkpoint; never rewrite the original history

Keeping Run = one rollout avoids a parallel lifecycle model and preserves the current Task/Run/Attempt distinction.

Proposed domain model

RolloutGroup

A durable RolloutGroup (alternative names: CandidateSet or EvaluationRunGroup) could contain:

• id, tenant/project, target Work Item ID and immutable snapshot/version digest
• mode: best_of_n, evaluation, trajectory_collection, or shadow
• requested and admitted rollout count
• candidate matrix or homogeneous candidate specification:
  • Agent Definition Version
  • model/prompt/runtime policy version
  • deterministic seed when supported
  • graph template/version
• hierarchical budget:
  • group total
  • per-Run maximum
  • max concurrency
  • deadline
• isolation profile and sandbox provider policy
• evaluator definitions/versions and thresholds
• selection/convergence policy
• status, aggregate result, winner Run ID, and cancellation reason
• idempotency key, actor, timestamps, and policy snapshot

Each child Run should additionally retain:

• rollout_group_id and stable rollout_index
• exact candidate/input snapshot digest
• optional parent_run_id and fork_checkpoint_ref
• sandbox execution binding/workspace reference
• termination reason

Standardized result

A per-Run rollout result should be machine-readable and reference immutable evidence:

{
  "run_id": "...",
  "status": "SUCCEEDED",
  "output_artifacts": ["artifact-version-ref"],
  "evaluator_results": [
    {
      "evaluator": "repo-tests",
      "version": "sha256:...",
      "score": 1,
      "passed": true,
      "evidence_artifacts": ["artifact-version-ref"]
    }
  ],
  "usage": {
    "wall_time_ms": 1234,
    "tokens": 5678,
    "cost": {"currency": "USD", "micros": 12345}
  },
  "termination_reason": "completed"
}

Large traces, patches, logs, test reports, and workspace snapshots remain Artifacts. Business tables keep references and safe summaries.

Lifecycle

1. Create a RolloutGroup from a versioned Work Item snapshot.
2. Validate policy, evaluator, isolation profile, total budget, and maximum fan-out.
3. Admit all or a policy-defined subset of the requested Runs.
4. Create sibling Runs with distinct Run IDs, Thread IDs, and sandbox/workspace scopes.
5. Schedule them using existing lease/fencing semantics.
6. Collect candidate Artifacts, traces, usage, and termination reasons.
7. Run versioned evaluators in isolated evaluator executions.
8. Apply the selection policy and record the winner/aggregate metrics.
9. Cancel or allow remaining Runs to finish according to the convergence policy.
10. Publish only the selected result through the normal Task command/acceptance path.

Group restart and reconciliation must be driven from PostgreSQL business state, not Redis pending entries or telemetry.

Isolation and side-effect rules

Concurrent rollouts must never share a writable workspace by default.

Each Run needs:

• a separate Thread/checkpoint namespace
• a separate writable workspace/sandbox namespace
• scoped credentials, Artifact access, and network policy
• explicit CPU, memory, process, disk, time, and egress limits
• audited sandbox lifecycle and resource usage

A common read-only base snapshot may be content-addressed and shared, but writable overlays must be independent.

Speculative/best-of-N execution should default to no irreversible external side effects. A rollout may prepare an ActionIntent, patch, or candidate Artifact, but publishing an external write should happen only after winner selection and normal policy/approval checks. This avoids N duplicate side effects.

Resume, checkpoint, and fork semantics

• Resume continues the same Run/Thread and workspace only after business snapshot, policy, credential, and sandbox-binding guards pass.
• Attempt replacement must not increment the logical rollout count.
• Fork creates a new Run with explicit parent/checkpoint provenance and a fresh writable workspace overlay.
• Checkpoint compatibility and workspace snapshot compatibility are separate checks.
• An unavailable or expired workspace snapshot must produce a visible recovery state; AgentMesh must not silently resume with an empty workspace.
• Original Run history, evaluator evidence, and lineage are immutable.

Sandbox provider boundary

Add a provider-neutral execution port instead of embedding Kubernetes/AgentCube concepts in the Task domain. A provider contract could support operations such as:

• create an isolated execution from an image/base Artifact and isolation profile
• attach/recover using an opaque execution handle
• execute or connect the Agent runtime
• checkpoint/snapshot when supported
• collect logs, output files, resource usage, and termination reason
• cancel/terminate and garbage-collect
• report capabilities such as snapshot support, runtime class, startup latency, and locality

An AgentCube adapter could translate an AgentMesh Assignment/Run into AgentCube sandbox lifecycle operations. AgentMesh would keep authoritative Task/Run/RolloutGroup state; AgentCube would keep infrastructure execution state. Reconciliation would compare both through opaque provider references.

Because AgentCube is currently in proposal/early-design phase, the initial AgentMesh contract should be validated with a deterministic fake or local container provider before committing to a specific AgentCube API.

Evaluation semantics

Evaluators must be versioned and reproducible:

• deterministic test/script evaluators should be preferred for verifiable tasks
• evaluator input uses immutable candidate ArtifactRefs and a controlled environment
• evaluator output includes score/pass, evidence, runtime, and failure classification
• evaluator failure is distinct from candidate failure
• only an evaluator bound to a required AcceptanceCriterion may affect Task completion
• ordinary quality/reward Scores remain analysis data
• winner selection records policy/version and tie-breaking evidence

This supports best-of-N, regression evaluation, trajectory collection, preference data, and downstream RL reward generation without making AgentMesh responsible for PPO/GRPO policy optimization.

Suggested MVP

Deliver this behind a feature gate and avoid requiring the full general-purpose multi-agent DAG scheduler.

Increment 1: architecture contract

• add an ADR clarifying Run = one rollout, Attempt != rollout, and the control-plane/runtime boundary
• update Task Domain, Orchestrator, Local Runtime, Artifact, and Evaluation L2 documents
• define RolloutGroup, standardized result, evaluator, sandbox provider, and reconciliation contracts

Increment 2: deterministic best-of-N vertical slice

• homogeneous N sibling Runs from one immutable Task snapshot
• strict global/per-group concurrency and budget caps
• separate Thread/workspace scope for every Run
• deterministic evaluator (for example, a test command against candidate artifacts)
• explicit winner selection and aggregate query
• cancellation and restart-safe reconciliation
• deterministic fake/local sandbox provider

Increment 3: pluggable isolated runtime

• provider capability discovery and scheduling filters
• optional workspace snapshot/resume/fork
• Kubernetes/AgentCube adapter exploration
• broader candidate matrices, shadow execution, and trajectory export

Acceptance criteria for the first runnable slice

• One request can create N logical Runs from the exact same immutable input snapshot.
• Every Run has a distinct Run ID, Thread ID, Attempt chain, trace correlation, and writable workspace.
• Replacing a failed/expired Attempt does not create an extra logical rollout.
• No writable file or checkpoint state leaks between sibling Runs.
• Fan-out cannot exceed admitted concurrency, cost, token, wall-time, or artifact limits.
• Process restart does not duplicate Runs, evaluators, winner selection, or business effects.
• Evaluator identity/version and evidence are retained with every score.
• The selected result is reproducible from stored candidate/evaluator references and selection policy.
• Canceling the group converges all active Runs to terminal or explicit outcome-unknown states.
• Ordinary single-Run tasks remain unchanged when the feature gate is disabled.
• The end-to-end test demonstrates a small verifiable task and computes pass@N/best-of-N metrics.

Non-goals

• Implementing an RL trainer or policy optimizer in AgentMesh.
• Making AgentCube a required dependency.
• Treating telemetry or Langfuse as the business source of truth.
• Running arbitrary evaluator code inside the AgentMesh control-plane process.
• Reusing Attempt as the rollout identity.
• Allowing speculative Runs to publish duplicate irreversible side effects.

Open questions

1. Is RolloutGroup the clearest public name, or should the business API use CandidateSet while observability uses "rollout"?
2. Should the first slice allow only homogeneous N-way execution, or also a candidate matrix across Agent/Model/Prompt versions?
3. Should evaluators be modeled as specialized Runs from the beginning, or as a narrower worker contract that can later become Runs?
4. Which workspace snapshot guarantees are required before supporting resume across sandbox replacement?
5. Should a RolloutGroup belong directly to a Work Item, or should a reusable Dataset/Evaluation Campaign own multiple groups?
6. Where should this land in the roadmap: as the first concrete Phase 2/3 bridge, or as a separate verifiable-execution track?

References

• AgentCube rollout proposal
• AgentCube overview
• AgentMesh task and execution domain
• AgentMesh orchestrator and scheduler
• AgentMesh local agent runtime
• AgentMesh observability and evaluation