You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add a way to evaluate interview / multi-turn skills — skills whose whole value is a back-and-forth with the user (e.g. grill-skill, grilling, domain-modeling). Today Caliper is a single-shot harness and structurally cannot test them past the first turn.
Surfaced while using Caliper to ablate/shorten our own grill-skill and evaluate-skill (see PRD: triggering eval, PRD: skill-comparison). We could only harden those evals to test first-turn discipline, not the full interview.
The limitation (grounded in the code)
Every backend implements HarnessBackend.run(task_id, attempt, prompt, ...) — one prompt in, one transcript out (caliper/harness/base.py). The transcript can hold many ConversationTurns (assistant + tool calls), but nothing ever answers the agent's questions back. There is no user turn after the first. See CONTEXT.md → "Single-shot harness".
Consequence for interview skills:
We cannot test "asks Q1 → waits → user answers → asks Q2 …".
The only honest single-shot test is first-turn discipline: given an open prompt, did the agent ask exactly one question and stop (not dump all questions, not fabricate the user's answers, not run ahead)? That catches a lot, but leaves the elicitation of later turns unmeasured.
What to build (shape, not final design)
A multi-turn execution path where a user-simulator (a second LLM, driven by a rubric/persona in the spec) plays the human across N turns, so the skill-under-test experiences a real conversation. The transcript then spans the whole dialogue and expect:/assert: can judge the interview, not just turn one.
Hard constraint: keep the CLI dead-simple
The CLI must stay trivial for humans and agents. A multi-turn spec must not force existing single-shot specs to change or grow ceremony. Strong bias toward: single-shot stays the zero-config default; multi-turn is opt-in via one obvious spec block.
Open questions (to think slowly about — do NOT lock yet)
Spec surface. What's the smallest addition? A per-task user_sim: block (persona + goals + a stop condition)? A scripted turns: list for deterministic replays vs. a free-form simulated user? Can both share one field?
Who plays the user — reuse the judge backend, a dedicated user.backend, or the skill backend with a flipped role? Reproducibility cost of each.
Termination — max-turns cap, a goal-satisfied signal from the simulator, or the skill declaring done. How to avoid runaway/cost blowups.
Determinism & pass@k meaning — a simulated user adds a second stochastic actor; pass@k now conflates skill + simulator. Do we pin/seed the simulator? How is this reported so the number stays honest (cf. the orchestrator footnote in Add openclaw harness backend (orchestrator-style, install + pinned worker) #12)?
Transcript shape — do simulated user turns get role="user" in ConversationTurn, and do judges need to distinguish real-prompt vs simulated turns?
Scope — is scripted replay (deterministic, cheap) a good v1 that de-risks the free-form simulator v2?
Spike first
By hand, wire a two-turn exchange for grill-skill (open prompt → agent asks one question → a canned/simulated answer → agent continues) on claude-code, and confirm the transcript is judgeable end-to-end before designing the spec surface.
Summary
Add a way to evaluate interview / multi-turn skills — skills whose whole value is a back-and-forth with the user (e.g.
grill-skill,grilling,domain-modeling). Today Caliper is a single-shot harness and structurally cannot test them past the first turn.The limitation (grounded in the code)
Every backend implements
HarnessBackend.run(task_id, attempt, prompt, ...)— onepromptin, one transcript out (caliper/harness/base.py). The transcript can hold manyConversationTurns (assistant + tool calls), but nothing ever answers the agent's questions back. There is no user turn after the first. SeeCONTEXT.md→ "Single-shot harness".Consequence for interview skills:
What to build (shape, not final design)
A multi-turn execution path where a user-simulator (a second LLM, driven by a rubric/persona in the spec) plays the human across N turns, so the skill-under-test experiences a real conversation. The transcript then spans the whole dialogue and
expect:/assert:can judge the interview, not just turn one.Hard constraint: keep the CLI dead-simple
The CLI must stay trivial for humans and agents. A multi-turn spec must not force existing single-shot specs to change or grow ceremony. Strong bias toward: single-shot stays the zero-config default; multi-turn is opt-in via one obvious spec block.
Open questions (to think slowly about — do NOT lock yet)
user_sim:block (persona + goals + a stop condition)? A scriptedturns:list for deterministic replays vs. a free-form simulated user? Can both share one field?user.backend, or the skill backend with a flipped role? Reproducibility cost of each.role="user"inConversationTurn, and do judges need to distinguish real-prompt vs simulated turns?Spike first
By hand, wire a two-turn exchange for
grill-skill(open prompt → agent asks one question → a canned/simulated answer → agent continues) onclaude-code, and confirm the transcript is judgeable end-to-end before designing the spec surface.Out of scope (for now)
References
caliper/harness/base.py(HarnessBackend.run,ConversationTurn)CONTEXT.md→ "Single-shot harness"