Skip to content

Commit 53a7bc5

Browse files
GiniGini
authored andcommitted
test: add controlled Claude SDK live harness
1 parent 7bbb9f5 commit 53a7bc5

4 files changed

Lines changed: 105 additions & 3 deletions

File tree

‎docs/CLAUDE-SDK-LIVE-E2E.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Claude Agent SDK live E2E evidence
2+
3+
## Scope
4+
5+
This is the controlled, opt-in proof for ONEVibe's `claude_sdk` provider. It creates one small document task against a real server-side Anthropic credential and may incur provider usage. It is never part of unit tests or the normal `npm run check` workflow.
6+
7+
## Preconditions
8+
9+
Configure the ONEVibe API process with a server-only `ANTHROPIC_API_KEY`. The browser receives only Claude-provider readiness; it never receives the credential or task runtime state directory.
10+
11+
Verify readiness before running:
12+
13+
```sh
14+
curl -fsS http://127.0.0.1:4311/api/runtime
15+
```
16+
17+
The `claude_sdk` provider must report `available: true`. If it does not, the harness exits before creating a task.
18+
19+
## Controlled proof
20+
21+
```sh
22+
ONEVIBE_E2E_URL=https://onevibe.example \
23+
npm run e2e:claude-sdk
24+
```
25+
26+
The harness creates a Document-mode task that asks Claude to write one local `README.md`. It then verifies:
27+
28+
1. a completed task state;
29+
2. the governed host-workspace execution boundary;
30+
3. a recorded Claude SDK session ID and run lifecycle evidence;
31+
4. a non-empty local `README.md`; and
32+
5. a valid ONEVibe evidence chain.
33+
34+
This is proof of the configured SDK path, not a claim of microVM isolation, gateway enforcement, browser validation, OpenVTC approval, or production credential federation. Those require their own gated evidence.

‎docs/MANUS-PARITY.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ This is the implementation gate, not a marketing checklist. **I** means behavior
1111
3. **P** Automatic multi-step planning — five persisted, outcome-aware stages exist. Native Claude SDK tasks may refine their human-readable titles through a validated, evidence-recorded server MCP tool; ONEComputer runs project a bounded in-sandbox plan control file through the same server validation path. Arbitrary stage creation/reordering and richer agent-generated decomposition remain pending.
1212
4. **P** Live plan progress — live statuses exist and each continuation resets the durable plan lifecycle with evidence linking the prior run. Richer provider-native progress mapping remains pending.
1313
5. **P** Per-step elapsed time — plan transitions now persist start/completion timestamps, render elapsed duration, and append timing evidence. Native-agent-derived granular step mapping remains pending.
14-
6. **I** Narrated execution — typed transcript deltas from demo, SDK, or remote runtime.
14+
6. **I** Narrated execution — typed transcript deltas from demo, SDK, or remote runtime. A controlled opt-in live harness now verifies the Claude SDK lifecycle, workspace artifact, and evidence chain only when server-side credential readiness is explicitly configured.
1515
7. **P** Interruptible follow-up chat — resumable same-session follow-ups work after a turn; explicitly retained ONEComputer sandboxes can reuse their active boundary for a continuation, and active non-interruptible provider turns accept bounded guidance queued for the next turn. Users can review and retract queued guidance before it reaches a provider; cancellation records metadata-only control evidence, not the guidance text. Users can stop a running, pending, input-waiting, or approval-waiting task while preserving workspace/evidence. True provider-native live interruption/injection remains pending.
1616
8. **I** Expandable task messages — long user and grouped assistant turns collapse and expand in place.
1717
9. **I** Structured waiting state — runtime parks durably, UI renders the focused request, and an answer resumes the same execution.

‎package.json‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,12 +10,13 @@
1010
"dev:web": "vite",
1111
"build": "tsc -b && tsc -p tsconfig.server.json && vite build",
1212
"check": "npm run lint && npm run test && npm run build && npm run check:e2e-harness",
13-
"check:e2e-harness": "tsc --ignoreConfig --noEmit --target ES2022 --module NodeNext --moduleResolution NodeNext --types node scripts/onecomputer-live-e2e.ts",
13+
"check:e2e-harness": "tsc --ignoreConfig --noEmit --target ES2022 --module NodeNext --moduleResolution NodeNext --types node scripts/onecomputer-live-e2e.ts scripts/claude-sdk-live-e2e.ts",
1414
"lint": "oxlint src server",
1515
"test": "vitest run",
1616
"preview": "vite preview",
1717
"wallet": "tsx server/wallet-cli.ts",
18-
"e2e:onecomputer": "tsx scripts/onecomputer-live-e2e.ts"
18+
"e2e:onecomputer": "tsx scripts/onecomputer-live-e2e.ts",
19+
"e2e:claude-sdk": "tsx scripts/claude-sdk-live-e2e.ts"
1920
},
2021
"dependencies": {
2122
"@anthropic-ai/claude-agent-sdk": "^0.3.210",

‎scripts/claude-sdk-live-e2e.ts‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
/**
2+
* Controlled live proof for a configured ONEVibe + Claude Agent SDK server.
3+
* This is opt-in: it sends one small task to the real provider and may incur
4+
* provider usage. It refuses to create a task when server-side readiness is
5+
* absent, so local/demo environments never fall back silently.
6+
*/
7+
const baseUrl = (process.env.ONEVIBE_E2E_URL ?? 'http://127.0.0.1:4311').replace(/\/$/, '')
8+
const timeoutMs = Math.max(60_000, Number(process.env.ONEVIBE_E2E_TIMEOUT_MS ?? 5 * 60_000))
9+
10+
type Snapshot = {
11+
id: string
12+
status: string
13+
securityContext?: { executionBoundary?: string; runtimeSessionId?: string }
14+
events: Array<{ type: string; label?: string; payload: Record<string, unknown> }>
15+
}
16+
17+
const request = async <T>(pathname: string, init?: RequestInit) => {
18+
let response: Response
19+
try {
20+
response = await fetch(`${baseUrl}${pathname}`, { ...init, headers: { 'Content-Type': 'application/json', ...init?.headers } })
21+
} catch (error) {
22+
throw new Error(`Cannot reach ONEVibe at ${baseUrl}${pathname}: ${error instanceof Error ? error.message : 'network failure'}`)
23+
}
24+
const body = await response.json().catch(() => ({})) as T & { error?: string }
25+
if (!response.ok) throw new Error(`${pathname} returned ${response.status}${body.error ? `: ${body.error}` : ''}`)
26+
return body
27+
}
28+
29+
const waitForTerminalSnapshot = async (taskId: string) => {
30+
const deadline = Date.now() + timeoutMs
31+
let latest: Snapshot | undefined
32+
while (Date.now() < deadline) {
33+
latest = await request<Snapshot>(`/api/tasks/${encodeURIComponent(taskId)}`)
34+
if (['completed', 'failed', 'cancelled'].includes(latest.status)) return latest
35+
await new Promise((resolve) => setTimeout(resolve, 2_000))
36+
}
37+
throw new Error(`Task ${taskId} did not reach a terminal state within ${timeoutMs}ms (last state: ${latest?.status ?? 'unreadable'})`)
38+
}
39+
40+
const main = async () => {
41+
const readiness = await request<{ providers: Array<{ id: string; available: boolean; detail: string }> }>('/api/runtime')
42+
const claude = readiness.providers.find((provider) => provider.id === 'claude_sdk')
43+
if (!claude?.available) throw new Error(`Claude SDK is not available at ${baseUrl}: ${claude?.detail ?? 'runtime status unavailable'}`)
44+
const created = await request<{ id: string }>('/api/tasks', {
45+
method: 'POST',
46+
body: JSON.stringify({
47+
prompt: 'Create exactly one file named README.md with a title and one concise sentence stating that this is a governed Claude Agent SDK validation artifact. Do not access the network, credentials, or any path outside the current workspace.',
48+
provider: 'claude_sdk', mode: 'document', projectId: 'project_onevibe', references: [], attachments: [], skills: ['document', 'security_review'],
49+
}),
50+
})
51+
const task = await waitForTerminalSnapshot(created.id)
52+
if (task.status !== 'completed') throw new Error(`Claude SDK task ${task.id} ended ${task.status}`)
53+
if (task.securityContext?.executionBoundary !== 'host_process') throw new Error(`Expected governed host workspace boundary, found ${task.securityContext?.executionBoundary ?? 'unknown'}`)
54+
if (!task.securityContext?.runtimeSessionId) throw new Error('Claude SDK session identity was not recorded')
55+
if (!task.events.some((event) => event.type === 'run_started' && event.label === 'Claude Agent SDK started')) throw new Error('Claude SDK run-start event missing')
56+
if (!task.events.some((event) => event.type === 'run_completed' && event.label === 'Claude Agent SDK completed')) throw new Error('Claude SDK completion evidence missing')
57+
const readme = await request<{ content: string }>(`/api/tasks/${encodeURIComponent(task.id)}/file?path=README.md`)
58+
if (!readme.content.trim()) throw new Error('Expected a non-empty README.md from the Claude SDK task')
59+
const evidence = await request<{ valid: boolean }>(`/api/tasks/${encodeURIComponent(task.id)}/evidence`)
60+
if (!evidence.valid) throw new Error('Evidence chain verification failed')
61+
console.log(JSON.stringify({ taskId: task.id, status: task.status, executionBoundary: task.securityContext.executionBoundary, sessionRecorded: Boolean(task.securityContext.runtimeSessionId), evidenceValid: evidence.valid }, null, 2))
62+
}
63+
64+
main().catch((error: unknown) => {
65+
console.error(error instanceof Error ? error.message : error)
66+
process.exitCode = 1
67+
})

0 commit comments

Comments
 (0)