Mission: use agentchat to make a real, tested change in a real repo

Outcome

A developer points agentchat at a real repository (chakra itself first), asks it for a small real change, and the agent finds the code, edits it, runs the tests, reads a failure, fixes it, and leaves a green tree. If it goes wrong, /undo puts the files back. No manual copy-paste between the agent and a shell.

Exercise

make dogfood          # agentchat TUI, --workspace ., exec allowlist (go test / go build / go vet), checkpoint on

Then ask: "make pass its tests" (pick a live P3 papercut or a deliberately broken test).

What to look at: did it discover the right files without being told paths; did it run the tests itself; when a test failed, did it see the failing assertion (not a truncated middle); did the final tree pass make test independently; does /undo restore the pre-edit state.

Done when

The exercise passes end to end on chakra with a local or hosted model, and its output is right by eye. Each failure it shows gets filed as a ticket on this mission.

Not in this mission

Linux sandbox backend (agent/ext/exec: a Linux sandbox backend, after establishing what a CI runner permits #32), P3: darwin covers dogfooding.
Exec allowlist growth and scale (agent/ext/exec: a mutable allowlist, and who is allowed to widen it #34, agent/ext/exec: tool-set scale — one tool per command does not survive a large allowlist #33), waiting until the exercise needs them.
Provider, memory and composition backlog (Phase 5/6 epics Agent SDK Phase 5: provider control & decoding fidelity #16, Agent SDK Phase 6: test-time compute & routing reliability #17), P3.
Skills SEP-2640 compliance (host/skills: unimplemented SEP-2640 host MUSTs — no-prefetch, content-bound approval, frontmatter re-verification #50), P2, and serving chakra's roster (Epic: serve agents — declare chakra's own roster over the agents extension #43), epic. Neither is needed to run this exercise.

docs(CLAUDE.md): issues and missions convention, active mission #57

#56, #60 and #62 merged. #56: chakra tracks mcpkit main. #60: --workspace-config (exec, LSP, repo map) and make dogfood, closing #58. #62: the chakra/<branch> checkout layout. Progress: 1/8. Next ready: #59.

Exercise @ ef8bcb5: could not run with a model. ANTHROPIC_API_KEY wasn't set in the session that built it. What did run: agentchat built with the dogfood config lists all 11 tools (5 file tools, run_test/run_test_all/run_vet/run_build, goto_definition/find_references), and run_test and run_vet passed inside sandbox-exec on darwin with an empty test cache.
Friction: the repo is several Go modules, so run_test needed scripts/test-pkg.sh to reach packages outside the root module (fixed in #60). The banner shows model . (#61).

Exercise @ 5a3d1d7 (local, DeepSeek deepseek-chat): failed at step 1 on the TUI. After two parallel search_files calls it hung with no visible approval prompt, then dropped keystrokes (#66, parked). A --ui plain smoke test earlier did work: run_test ./ext/exec ran sandboxed and passed, ~7k tokens over 3 steps.
The exercise moves to the web surface. Chakra is the SDK; the TUI is one surface. New P1s: #64 (make it runnable from the browser), #65 (file tools lack readOnlyHint, so every read asks). Next: #64.

#123 merged. An edit with two or more hunks is now asked about one change at a time, and only the ticked hunks run. A note on the tool result tells the model what was left out (closes #39). Progress: [21/28 tickets closed]. Next ready: #7 and #109, then #124 (filed from #123).

Exercise @ 0d44482: a short run aimed at #39. DeepSeek over agentweb, fresh DOGFOOD_DIR, default read-only-auto. A small script played the browser and answered each elicit-request through RespondToAsk, unticking change_2 on the edit form. One turn ("create tmp_hunks.txt with alpha/beta/gamma, uppercase all three in ONE edit_file call, read it back, delete it"): 6 steps, 19k input tokens.

Three asks: "Create tmp_hunks.txt?", "Apply 3 changes to tmp_hunks.txt? Untick any you don't want.", and "Delete tmp_hunks.txt?".
edit_file reported "2 edit(s) applied", and the re-read showed ALPHA / beta / GAMMA.
The model got it right without being led. It said not every edit was applied, named beta → BETA as the declined one, and offered to apply it rather than retrying on its own.
delete_file used the new hash and the tree stayed clean.

Friction: the browser UI itself still can't show the per-hunk form (#124). Ctrl-C took 1.18s after the turn (#117).

#125 merged. Docs only: docs/HOOKS.md maps where behavior hooks into a turn, and CLAUDE.md asks every PR to name its hook. Progress: [21/28 tickets closed]. Next ready: #124, #109 and #7, all in review as #126, #127 and #128.

Exercise: not run for this merge, since nothing that runs changed. The next run follows #126, in a browser.

#126, #127 and #128 merged.

agentweb: answer asks from their schema, so Approve approves #126 (closes agentweb: the web Accept button sends no confirm field, and multi-field asks show as accept/decline only #124): the browser answers an ask from its schema, so Approve approves, and an edit gets one checkbox per change.
offload: name stored results by content; read_file: read a line range #127 (closes dogfood: exercise turns hit the step limit and re-read large files #109): offloaded results get a ref named by content, and read_file reads line ranges.
agent: a step chain before every model call; compaction and the step budget run on it #128 (closes agent: mid-turn compaction (a single turn that grows past the budget) #7): a step chain runs before every model call, with mid-turn compaction, a context-budget floor, and a step-budget note.

Progress: [24/28 tickets closed]. Next ready: #105 and #117 (both P3), then #22.

Exercise @ 27fad58: three turns, DeepSeek over agentweb, driven through the real UI. Headless Chrome typed into the composer, and every ask was answered by clicking Approve on the page, with no script calling RespondToAsk. Default read-only-auto, and compaction on from the dogfood config.

Turn 1, investigate only ("how does RunTurn treat ErrMaxSteps / ErrContextBudget"): 8 steps, 46k input tokens.
- Every read used a line range: host/app.go 772-900, then 855-877, plus 45-line slices of runner.go and persistence.go.
- Nothing was read whole. The agent/host: user-input transform seam (@file references and friends) #27 investigate turn took 20 steps and 680k tokens, and the agent: a tool call with invalid JSON arguments drops out of the session store and the Watch stream #104 one hit the 40-step cap.
- The answer traced the keep branch correctly.
Turn 2, the hunk exercise: 6 steps, 77k input.
- Three asks, all approved in the page. The "Apply 3 changes" form showed three checkboxes, and change 2 was unticked before Approve.
- The file came out ALPHA / beta / GAMMA. The model named beta → BETA as declined without being asked.
- delete_file cleaned up, and the tree is clean.
Turn 3, repeated read: read_file on ext/lsp/client.go twice. Both landed on res:76bb151ac273, and the second stub said the content was identical. The model reported the file unchanged instead of paging through it again.
Not triggered on this run:
- Compaction: the largest request was well under the 60k budget.
- The step-budget note: no turn got within 10 of the 40-step cap.
Both need a long turn to show up.

Friction:

Turn 1 tried goto_definition with a symbol from another file, twice, and got the "no symbol named X in this file" refusal. It recovered with search_files.
Ctrl-C took 0.14s.

Exercise @ 27fad58: the real one, end to end on #105. No merge this time, so progress is unchanged at [24/28 tickets closed]. The run produced #130, which closes #105, and found #129.

DeepSeek over agentweb, driven through the page in headless Chrome. The composer was typed into and every ask was approved by clicking, with all hunks kept. Dogfood defaults: read-only-auto, compaction on. One turn asked for the fix in a user's words, without naming files: "make run_test handle a /... path above several modules, add a test, run the tests, leave the tree green".

How it went against the exercise's checklist:

Found the files without paths: yes. search_files for run_test found scripts/test-pkg.sh, surfaces/dogfood_exec_test.go and examples/dogfood/workspace.json. It reproduced the bug first (run_test ./ext/... → matched no packages).
Ran the tests itself: yes, 15 test or vet runs across the turn.
Saw a failure and fixed it: yes. Its new test failed at steps 20 and 22. It read the failure, worked out that a failing command is a result and not IsError, and was green at step 26.
Final tree passes make test independently: yes. I ran make test after the turn and every module passed. The agent's test fails against main's script, so it really tests the change.
/undo restores the pre-edit state: half. /undo from the browser composer restored all three files' content byte for byte, but left them 0600 and dropped scripts/test-pkg.sh's executable bit, so git status showed a mode change. Filed as ext/checkpoint: /undo restores files as 0600 and drops the executable bit #129 (P2, on this mission).

Cost: 39 steps of the 40 cap, 1.06M input and 8k output tokens. Compaction never fired. Every request was under the 60k budget, so the cost is the resend of a long turn rather than any one oversized request. It finished on step 39 with a summary rather than hitting the cap.

Friction:

Approval for a write that then failed: the first write_file on scripts/test-pkg.sh asked "Create scripts/test-pkg.sh (3507 bytes)?" for a file that already exists. The tool then refused it for having no expect_hash. The approval prompt says Create whenever expect_hash is absent, so the user approved a write that couldn't run.
Formatting: the agent didn't run a gofmt check, and its test file wasn't gofmt-clean. make check-fmt isn't on the exec allowlist, so CI would have caught it, not the agent. I fixed it in test-pkg.sh: run every module below a /... path that spans several #130.
Ctrl-C: took 0.29s.

#130 merged (closes #105). run_test runs each module below a /... path that spans several. The dogfood agent wrote the change in the previous exercise run. Progress: [25/29 tickets closed]. Next ready: #129 (P2, /undo drops file modes), then #117, with #22 and the parked #34 also open.

Exercise @ 0320bbc: short run aimed at #105. DeepSeek over agentweb, one turn: "run the tests for everything under ext/ with run_test, as a single path". 2 steps, 6k input tokens. One run_test ./ext/... call ran ext/checkpoint, ext/exec, ext/files and ext/lsp and exited 0. The 7da5ad2 run spent three steps hand-naming the modules. The model noticed every result was (cached) and said so rather than reporting a fresh pass.

Friction: none new. Tree clean.

#132 and #133 merged.

checkpoint: restore each file's mode along with its content #132 (closes ext/checkpoint: /undo restores files as 0600 and drops the executable bit #129): /undo restores each file's mode along with its content.
lsp: the repo map reads symbols without opening files #133 (closes ext/lsp: quitting while gopls is still loading waits out a 2s shutdown timeout #117): the repo map reads symbols without opening files, so gopls stays idle, the map is ready in 2s instead of 20s, and Close is quick.

Progress: [27/30 tickets closed]. Left: #131 (P3, two papercuts), and #34 (waiting). #22 moved off the mission to waiting.

Exercise @ eb4a80f: the full run, and every check passed. DeepSeek over agentweb, driven through the page in headless Chrome, with every ask approved by clicking. The task was #131's first item, asked in a user's words without file names: "when write_file targets an existing file without expect_hash, the approval says Create; make it say the write will be refused. Find where the prompt is built, change it, add a test, run the tests, leave the tree green." One turn, 26 steps, 506k input tokens.

Found the code without paths: yes. search_files for the prompt text found renderEditApproval in ext/files/extension.go.
Ran the tests itself: yes: run_test_one, run_test ./ext/files, run_vet and run_test_all.
Saw a failure and fixed it: yes. Its first test run (step 21) failed. It fixed the test (step 22), and ./ext/files passed at step 23.
Final tree passes make test independently: yes, every module. The change is gofmt-clean.
/undo restores the pre-edit state: yes. "turn-1-3cfb3a: 2 file(s) restored", git status empty, and modes unchanged (-rw-r--r--). The run at 27fad58 left 0600 files and a dropped executable bit (ext/checkpoint: /undo restores files as 0600 and drops the executable bit #129).

Shutdown (#117), in a fresh session with the repo map on: Ctrl-C took 0.08s both 3s and 10s after ready, against 2.0 to 4.1s before #133.

Friction: the agent tried main/NOTES.md and main/ext/files/... as paths once (step 2), prefixing the checkout's directory name, and got path errors before using the right relative paths. Minor, and it recovered in one step.

As I read "done when", this run meets it: the exercise passes end to end on chakra with a hosted model, and the output is right by eye.

#134 merged (closes #131).

The approval for a write over an existing file without expect_hash now says the write will be refused, instead of "Create". The dogfood agent wrote that part in the last run.
The agent has run_fmt_check, and check-fmt now covers new, untracked files.

Progress: [28/30 tickets closed]. Left: only #34, which is waiting.

Exercise @ 23e850f: short run aimed at #134. DeepSeek over agentweb, one turn: create a badly formatted untracked tmp_fmt.go, run the format check, delete it. 4 steps, 23k input tokens.

run_fmt_check ran make check-fmt inside the exec sandbox. It exited 2 and named tmp_fmt.go, a file git doesn't track yet, which main's check would have missed.
delete_file cleaned up, and the tree is clean.

Friction: none new.

With the full run at eb4a80f passing every check, and only the parked #34 open, I think this mission is done.

Closing: the mission is done by its own definition.

The full exercise passed end to end at eb4a80f with a hosted model (DeepSeek over agentweb, driven through the browser). The agent fixed a real issue (#131's first item) from a request that didn't name any files. It found the code, edited it, ran the tests itself, read a failure and fixed it, and left a tree that passed make test independently. /undo then restored the pre-edit state exactly, modes included (#57 (comment)). Every failure the runs showed was filed here and closed.

Final count: 28 of 29 tickets closed. The one left, #34 (a mutable exec allowlist), is waiting and moved off this mission, since no run needed it.

What the mission produced, roughly in order: make dogfood and make dogfood-web, the files, exec, LSP and checkpoint extensions, per-hunk approval, the step chain with mid-turn compaction, content-addressed offloading and ranged reads, the browser approving asks properly, /undo keeping file modes, and a fast shutdown.

#89 (host completion) is the active mission now.

Merge pull request #140 from panyam/docs/active-mission-89

Navigation Menu

Repository navigation

Mission: use agentchat to make a real, tested change in a real repo #57

Outcome

Exercise

Done when

Not in this mission

36 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Mission: use agentchat to make a real, tested change in a real repo #57

Description

Outcome

Exercise

Done when

Not in this mission

Activity

panyam commented on Oct 14, 2026

panyam commented on Oct 15, 2026

36 remaining items

panyam commented on Oct 26, 2026

panyam commented on Oct 26, 2026

panyam commented on Oct 26, 2026

panyam commented on Oct 26, 2026

panyam commented on Oct 26, 2026

panyam commented on Oct 26, 2026

panyam commented on Oct 26, 2026

panyam commented on Oct 26, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions