Worked example: a scene-building agent

Text description → a populated, physically valid, explorable 3D space — a USD stage you open, orbit, walk through, relight and simulate — with the objects in it generated by Articraft and RTX preview rendering on NVIDIA hardware.

The artifact is the stage, not a picture of it. That distinction drives the verifier, and it is the first thing to get right.

This document exists because the library tells you to follow a build order and never shows that order applied end to end. Here it is applied to a domain neither distilled agent covers.

Nothing here has been built. It is a design and a build sequence, written against the references, with the decisions already made so you can argue with them rather than start from a blank page.

Why this domain is worth building

It exercises the one configuration the two case studies do not have.

	Articraft	Claude Code	Scene agent
What decides "done"	a compiler	a human	physics + multi-view space checks
Verifier tier	gating	advisory	gating
Action space	narrow SDK	open-world shell	narrow SDK
Second budget axis	—	—	GPU-seconds
Delegates to	—	subagents	another whole agent

Physics is a genuine oracle. Interpenetration, objects falling through the floor, exploding solver velocities, a joint that cannot reach its declared angle, an object no sampled viewpoint can see — all mechanically decidable. So you get to build the strongest verifier the library describes (reference 03, tier 1), and the loop can exit on a check rather than on a person.

It also composes two agents, which is the first time this library has had to describe that.

Two sentences

npx skills add alfredzhang98/agent-creator-skill --skill agent-creator

Build it — one sentence to your coding agent:

Using the agent-creator skill, build me an agent that turns a sentence into an explorable 3D space I can open and walk around in Isaac Sim, generating the objects in it with Articraft.

Use it — one sentence to the agent you just built:

A small study: a standing desk against the left wall, an adjustable lamp and a mug on the desk, a rug, daylight from a window on the left.

That is the whole interface. Everything below is what the skill decides on your behalf, written out so you can disagree with it.

You are asked exactly one thing during the build, because it cannot be inferred: is a human watching, what is the per-run budget, and is Articraft really the asset source you want? The rest — action space, verifier tier, sandbox, cost caps, staging, typed loop — is chosen from the request and wired in whether or not you knew to ask.

What you should see back, before any code

The skill declares its routing first. For this request it should look like this — and if a line is wrong for your setup, that is the moment to say so:

Building: a sentence -> an explorable USD stage, objects generated by Articraft

  Action space   a scene DSL; raw USD forbidden   <- output has a constrainable format
  Verifier       gating                           <- physics is a decisive oracle
  Delegation     yes, one subagent per asset      <- generation reads far more than it writes
  Skills         2 found, both nvidia/skills       <- your OK needed before install
  Sandbox        container, no network, GPU passthrough   <- always
  Budget         tokens + GPU-seconds, estimated pre-call <- always
  Persistence    staging -> promote, full traces          <- always
  Not yet        progressive disclosure, memory   <- context does not outgrow the window here

Say if any of that is wrong. Otherwise I'll start.

The skills it will offer you

Searching happens without being asked; installing does not. Expect something close to this, because it is what the registry actually returns today:

Found, for the rendering half:
  nvidia/skills/omniverse-usd-performance-tuning   2029 installs
  nvidia/skills/omniverse-realtime-viewer          1968 installs
  npx skills add nvidia/skills --skill omniverse-usd-performance-tuning --agent claude-code
  npx skills add nvidia/skills --skill omniverse-realtime-viewer --agent claude-code

Nothing fits the Articraft half — that stays a delegation, not a skill.

Two things to notice. The publisher is the vendor whose renderer you are targeting, which is the strongest provenance signal available; and the second half of the request found nothing, which is the ordinary outcome and is reported rather than papered over.

Say yes and those two skills supply USD layer and performance conventions your action space would otherwise have to encode by hand. Say no and everything below still works — you just write more of Step 1 yourself.

Do not accept them silently on the agent's say-so. A skill's body becomes your agent's instructions; reference 16 audits it for exactly that, and the audit is worth reading before you approve.

What the skill decides here, and why

Decision	What it picks for this request	Because
Action space	a scene DSL, raw USD forbidden	the output has a constrainable format
Verifier tier	gating	physics is a decisive oracle
Delegation	one subagent per asset	asset generation reads far more than it writes
Sandbox	container, GPU passthrough, no network	the model authors code that then executes
Budget	tokens and GPU-seconds, estimated pre-call	one render can outcost the conversation
Disclosure / memory	not yet	context does not outgrow the window at this size

Do not reorder the build. It is a dependency chain, not a table of contents: the action space determines whether a verifier is possible at all, and the verifier determines how the loop can exit. Built backwards, both get rewritten.

Step 1 — Action space (reference 10)

The one decision that cannot be repaired later.

Do not let the model write USD Python

Raw USD is the target format. Emitting it directly means no eager validation, no entity-named error messages, no derived collision geometry, and no mechanical QC — which is to say, no verifier. Everything else in this document depends on not doing this.

Define a scene DSL

scene = Scene(units="m", up="z")
scene.floor(material="oak", size=(6, 4))

desk = scene.add("desk", asset="articraft: a standing desk, 1.4m wide, steel legs")
lamp = scene.add("lamp", asset="articraft: an articulated desk lamp, 3 joints")
mug  = scene.add("mug",  asset="library: ceramic_mug_01")

scene.place(lamp, on=desk, at=(0.30, 0.20), facing=desk.front)
scene.place(mug,  on=desk, at=(-0.15, 0.05))

scene.light("window", direction=(-1, 0, -0.3), lux=8000)

# Viewpoints are for PREVIEW and for checking, not for framing a photo.
# An orbit and a standing eye-height walk are enough to catch what a single
# hero camera hides.
scene.preview(orbit=True, eye_height=1.6)

# The model's own contract. The compiler runs its baseline battery regardless.
scene.expect_reachable(desk, from_="door")        # the space is navigable
scene.expect_seen_from(lamp, views=">=3")         # not just from one angle
scene.expect_lit(min_lux=150, coverage=0.9)       # no dead-dark regions
scene.expect_stable(after_seconds=2.0)
scene.expect_articulation(lamp, joint="elbow", reaches_deg=110)

Three constraints, taken from Articraft

Coordinates are semantic. place(on=desk, at=(x, y)) is surface-local. Models are poor at composing transform matrices in their heads, and every world-space placement is a chance to be silently 30cm off.

The compiler derives collision geometry. Agents may not author it. This removes the entire visual/physics desync error class rather than validating against it — the same move Articraft makes, and the reason it can trust its own QC.

Semantic helpers replace tuned numbers. facing=desk.front, not a quaternion. If the DSL cannot express a quaternion, the agent cannot get one wrong.

The test for whether your action space is right

What the DSL cannot express, the agent cannot break.

If it cannot express "place the mug 0.5m below the desk surface", that failure never happens and your verifier never needs a check for it. Every check you can design away is worth more than a check you implement.

Step 2 — Verifier (reference 03), gating, three layers

Layer 1  COMPILE
  Does the scene graph resolve? Do all assets exist? Are units consistent?
  Are there placement cycles (a on b, b on a)?

Layer 2  PHYSICS — settle for N seconds, then measure
  Interpenetration between any two colliders
  Anything below floor level
  Solver velocity above a sanity threshold (numerical explosion)
  Declared articulations actually reach their declared angles

Layer 3  SPACE — sample viewpoints, do not trust one
  Every expect_seen_from object visible from at least N sampled views
  Lighting coverage: what fraction of the floor is above min_lux
  Navigability: a capsule of human size can reach every expect_reachable
    target from the entry point without clipping
  No viewpoint inside geometry; no holes in the floor or walls
  Exposure sane across the sampled set (a technically-correct all-black
  frame is the archetypal silent success in this domain)

Layer 2 is your compiler. It is the reason this agent can have a gating verifier at all.

Feedback is typed signals, never strings

Signal(
    severity="failure",
    code="interpenetration",
    entities=["lamp.base", "desk.top"],
    measured=0.012, threshold=0.0, units="m",
    fix_hint="raise 'lamp' by ~0.012 along +z, or use place(on=desk) and let "
             "the compiler seat it",
)

Validate the space, never one frame

A single render is a screenshot of one opinion about the scene. Grade the agent on it and the agent optimises for it: the hero shot looks right and everything behind the camera is garbage — furniture floating, the far wall missing, a lamp with no bulb. It is the reward-hacking failure from reference 04, arriving through the viewport.

Sample instead. An orbit plus a walk at eye height costs a handful of cheap low-resolution frames and asks a much harder question: is this true from anywhere you might stand? That is also what the user actually wanted, because they asked for a space, not a photograph.

Three rules from the reference, all load-bearing:

One primary issue at a time. One interpenetration cascades into five downstream warnings. Give the model five co-equal failures and it patches the easiest, not the causal one. Rank: physics → visibility → aesthetics.
Attribution-scoped. Report only what this revision introduced. A scene with pre-existing warnings that reports all of them teaches the model that this channel is noise.
Justified allowances. A hinge pin genuinely overlaps its barrel. allow_overlap(a, b, reason=…) requires a non-empty reason, scopes to named entities, still emits a warning, and echoes into the report. Clamp any agent-supplied tolerance so it cannot neutralise the check.

Step 3 — Tools (reference 02)

The smallest surface that lets the model work.

Tool	Read-only	Notes
`read_scene` / `write_scene` / `patch_scene`	r / w / w	`patch_scene` uses the multi-hunk tool from `agentkit/tools/patch.py`
`find_asset`	✓	Search what already exists. Search before generate is the largest single cost saving in this design
`generate_asset`		Delegates to Articraft — see Step 5
`compile_scene`		DSL → USD, derives colliders
`simulate`		Runs physics, returns typed signals
`preview`		Renders a sampled set of viewpoints (orbit + walk) at low resolution, with segmentation masks
`inspect_space`	✓	Reads back the measurements — visibility counts per object, lighting coverage, reachability. The model reasons over numbers, not over an image it cannot see
`export_stage`		Writes the USD the user opens. The deliverable

Do not add a general shell. You have a DSL. A shell turns a zero-line permission layer into a multi-thousand-line one that must re-derive command semantics to decide whether a string is read-only (reference 13 has the measured cost).

Step 4 — Sandbox (reference 04)

The model authors two kinds of executable content. Both need an OS boundary:

Articraft's CadQuery scripts (asset generation)
The USD/Python the scene DSL compiles to

container:  --network=none
            --gpus device=N
            --pids-limit  --memory  --read-only asset library
            tmpfs workspace, non-root user
parent:     wall-clock kill, then reap. Simulations hang.

Isolation buys autonomy. Compile and simulate run inside the sandbox and therefore need no permission prompt. That is the reason to build the boundary — not to restrain the agent, but to let it iterate without asking.

Step 5 — Delegation to Articraft (reference 09)

asset="articraft: …" spawns a subagent.

AgentDefinition(
    agent_type="AssetSmith",
    when_to_use="Generate one articulated asset from a text description",
    tools=("Read", "Write", "Patch", "compile_model", "probe_model"),
    max_turns=40,
    isolation="worktree",            # parallel asset generation must not collide
    omit_project_instructions=True,  # it does not need scene-layer rules
)

Allowlists replace, never extend. The scene agent's simulate and preview grants must not reach the asset agent.
One subagent per asset, run in parallel. Asset generation is the phase that parallelises cleanly: independent outputs, no shared writes.
Cache by description hash. A chair generated once is never generated again. At $0.5–$2 per asset this is the difference between a viable loop and an expensive one.

Step 6 — Budget (reference 07) — you will need to extend the library here

agentkit/cost_meter.py knows tokens. You have a second axis:

turn cost = token cost + GPU-seconds × rate
hard cap  = both, together

The GPU axis behaves differently and needs its own handling:

Estimate before the call, not after. A full-resolution sweep of a dozen viewpoints can cost more than the entire conversation; post-hoc accounting notices too late.
Tier the sampling. Four 512px viewpoints while iterating; the full sampled set at full resolution only on the final confirmation. Most turns do not need to look pretty, they need to answer one question.
Separate call caps for simulate and preview, not just a shared total.

This is a gap in the skill, stated plainly. A multi-modal cost axis is the kind of thing only a third distilled agent surfaces. When you have run this, it belongs in reference 07.

Step 7 — Loop and state (references 01, 08)

budget gate → LLM → tool gauntlet → compile → simulate → preview
                                                  ↓
                                          typed signals
                                                  ↓
                        fresh verify of latest revision? → promote
                                    ↓ no
                              inject signals, next turn

Name every exit: completed, max_turns, cost_limit, physics_unstable, asset_generation_failed, space_check_failed. In a batch of two hundred scenes, one generic failure code is unsortable.
Freshness gate. Every mutation bumps a revision counter; a finish attempt compares it against the revision at the last successful simulate. The model cannot declare success over a scene it has since changed.
Staging → promote. Only verified scenes enter the library. Failed runs keep their staging directory — a failure you cannot inspect is a failure twice.
Keep every trajectory. For this domain the traces are the asset.

Suggested layout

scene-agent/
├── dsl/            scene.py, placement.py, expectations.py   ← Step 1
├── compile/        to_usd.py, colliders.py                   ← derived geometry
├── verify/         physics.py, space.py, signals.py          ← Step 2
├── tools/          scene_tools.py, asset_tools.py            ← Step 3
├── sandbox/        container.py                              ← Step 4
├── agents/         scene_agent.py, asset_smith.py            ← Step 5
├── assets/         cache/  library/                          ← Step 5
└── runs/           <run-id>/{staging,transcript.jsonl,metadata.json}

Everything under tools/, the loop, permissions, state and delegation comes from templates/agentkit/. What you write is dsl/, compile/ and verify/ — the two domain-specific boxes reference 00 predicts.

Build sequence

Each stage has one acceptance question, and it is always the same question.

Stage 1 — the loop turns. Library assets only, no Articraft, one desk and one mug, and a stage you can actually open. Then deliberately break it twice: place the mug 0.1m below the desk surface, and put a wall between the lamp and every viewpoint. Does the verifier catch both, and can the model fix them from the error text alone?

Stage 2 — one generated asset. Adds the delegation path and the asset cache. Check that the second run with the same description does not regenerate.

Stage 3 — three assets in parallel. Exercises worktree isolation and the GPU budget. Check that a failure in one asset does not abort the other two.

Stage 4 — articulations. A lamp, a drawer, a door. expect_articulation becomes load-bearing.

Stage 5 — a ten-object room. Context pressure. This is where the compaction ladder in agentkit/provider.py starts to matter.

The acceptance question, at every stage

When it fails, can the model fix it from the error message alone, with no human input?

If not, the problem is your error text, not the model. Reference 10's error-message rules are the thing to reread: name the exact entity, embed measured-versus-threshold, compute a fix hint, and admit heuristic conservatism honestly so the agent does not chase false positives.

What will go wrong

Each of these is in a pitfalls section already.

Failure	Reference
Writing raw USD, then discovering nothing can be validated before it runs	10
Reporting all warnings instead of newly-introduced ones → the model stops reading them	03
Reporting N co-equal failures → the model patches the easiest, not the cause	03
Treating a child process as a security boundary → model-authored simulation code reads `~/.ssh`	04
Prompting for everything → the user clicks through the one that mattered	13
Not caching assets → the same chair, regenerated, forever	09
Grading on one hero frame → a scene that is correct only from that angle	your domain; sample viewpoints
A preview that is technically valid and entirely black, reported as success	your domain; must be layer 3
A hung simulation with no wall-clock kill	04
Placing the volatile asset list in the tool description → cache thrash	06, 11

If you are typing rather than reading

Build: