[P2] [feature-request] Split file authoring from code execution — give agents write-file / edit-file / run-file (plus notebook cell edit/delete/rerun) so they refactor instead of re-paste

[lucastononro]

Summary

Today there is exactly one verb in the sandbox skill surface: execute-code. To author a module the agent runs execute-code with body Path("src/foo.py").write_text(...). To edit it, run execute-code and re-write the whole file. To run an existing module, run execute-code with import foo; foo.go(). Notebooks are append-only — append-notebook-cell and run-notebook-cell can grow a notebook but nothing in the skill set can edit a cell, delete a cell, or re-run a cell.

The agent has the volume and the preamble, but not the verbs. Every authoring intent gets funneled through execute-code, which auto-saves to /sessions/{sid}/scripts/step_NN_slug.py (backend/skills/execute-code/handler.py:89-99) — i.e. the audit log fills up with file writes that are not steps. And every prompt that says "factor reusable code into src/" is fighting tool-shape gravity: when the cheapest path is "append a new throwaway script", the agent appends.

This proposes splitting the surface along the verbs the prompts already use, and updating the prompts to teach DRY actively rather than describe it statically. Plus: extend the same treatment to notebooks (edit/delete/rerun cells) and reaffirm that notebook cells can import from src/ (already true after #51, but nothing in the prompts says so).

Verb	Today	Proposed
Write a new file at a path	execute-code + .write_text	write-file
Edit part of a file in place	execute-code + read-modify-write	edit-file
Execute an existing file on disk	execute-code + runpy.run_path / re-paste	run-file
Headless one-shot Python	execute-code	execute-code (unchanged)
Append a notebook cell	append-notebook-cell	(unchanged)
Append + execute a cell	run-notebook-cell	(unchanged)
Edit an existing cell's source	—	edit-notebook-cell
Delete a cell	—	delete-notebook-cell
Re-run an existing cell against the live kernel	—	rerun-notebook-cell

The proposal is purely additive at the skill level — no signature changes to execute-code or the existing notebook skills. The audit-log behavior of scripts/ becomes meaningful again because file authoring no longer bleeds into it.

Current state — evidence

Aspect	Reality	Gap
Authoring a module	execute-code(code="Path('src/x.py').write_text(...)") — auto-saves the write call as scripts/step_NN_*.py (backend/skills/execute-code/handler.py:89-99)	Audit log conflated with authoring. No file_updated event distinct from file_created
Editing a module	Full overwrite via another execute-code call. No read-modify-write pattern anywhere in backend/skills/ (notebook_store is the only exception, and only because nbformat gives it a structured object)	No edit-file skill; no anchor / patch primitive
Running an existing file	execute-code(code="import x; x.go()") — every reuse re-pastes the call site	No run-file skill; reuse looks identical to authoring in the audit trail
Edit a notebook cell	notebook_store.apply_source_update exists (backend/services/notebook_store.py:296-316) but is wired only to the frontend HTTP PUT path — no skill exposes it to the agent	Service layer ready; skill missing
Delete a notebook cell	No function in notebook_store, no skill	Both missing
Re-run an existing notebook cell	run-notebook-cell only appends a new cell. The persistent kernel is there (backend/services/kernel_manager.py:292-328) but no skill targets an existing cell_id	Skill missing
src/ is on sys.path for execute-code	✓ — backend/services/sandbox.py:75-95 (shipped in #51)	none
src/ is on sys.path for notebook cells	✓ — kernel preamble at backend/services/kernel_manager.py:110 runs the same SDK preamble; workdir=/data/sessions/{sid} at :324	none
Prompts teach DRY actively	✗ — every agent YAML has a "Workspace is a real Python repo" block (chat.yaml:173-182, eda.yaml:60-69, data_prep.yaml:68-84, feature_eng.yaml:59-69, trainer.yaml:69-88, orchestrator.yaml:217-221) but the language is descriptive ("Any .py file you write under src/ can be imported") rather than prescriptive ("Before pasting code for the third time, edit the existing module")	blocker (prompt side)
Prompts mention notebook → src/ import parity	Partial — eda.yaml:66-69 says "promote a helper out of a notebook to src/loaders.py so data_prep can import it" but doesn't say the same notebook can re-import it	minor

Concrete failure today, illustrating the conflation:

# Agent decides to build a reusable loader. Three execute-code calls:
#  1) Path("src/loaders.py").write_text("def load(): ...")   → scripts/step_03_path_src_loaders_py.py
#  2) Path("src/loaders.py").write_text("def load(): ...\ndef load_val(): ...")   → scripts/step_04_path_src_loaders_py.py
#  3) from loaders import load_val; df = load_val(); print(df.shape)   → scripts/step_05_from_loaders.py
#
# audit log: 3 entries, 2 of which are file writes that are not steps.
# Replaying scripts/ as a notebook would re-author the same module twice.

Proposed design

1. New file skills

backend/skills/write-file/

• handler.py wraps services.volume.write_to_volume(content, path) (it already exists at backend/services/volume.py:200-235 and handles both str and bytes)
• Inputs: path (string, must start with /sessions/{sid}/ or be relative to it — handler resolves), content (string), overwrite (bool, default true)
• Emits file_created (new) or file_updated (overwrite) — UI distinguishes
• Returns {path, bytes_written}
• Does NOT auto-save into scripts/. Authoring is not an execution step.

backend/skills/edit-file/

• handler.py reads the file via services.volume.read_volume_file_async (backend/services/volume.py:84-95), applies the patch, writes back via write_to_volume
• v1 supports two modes:
  • mode="replace" — old/new string pair (must match exactly once, else error — same shape as the Edit tool the agent already knows)
  • mode="overwrite" — content replaces the whole file (escape hatch when the file is too dirty to anchor)
• Emits file_updated, never file_created
• Returns {path, bytes_written, replacements}

backend/skills/run-file/

• handler.py calls services.sandbox.run_code(code=f"import runpy; runpy.run_path({path!r}, run_name='__main__')", session_id, ...) — so the file runs in a fresh sandbox under the existing preamble (src/ on sys.path, workdir at the session)
• Inputs: path, heavy (bool — same as execute-code's heavy flag, picks training profile)
• Audit-log behavior: writes a one-liner runpy.run_path(...) to scripts/step_NN_run_<filename>.py. The audit log records "we ran X", not "X's body" (which is already on disk as src/X.py).
• Returns standard execute-code result shape (stdout, stderr, returncode)

2. New notebook skills

backend/skills/edit-notebook-cell/

• Wraps a new notebook_store.edit_cell(session_id, name, cell_id, source, cell_type=None) helper that mutates nb.cells under the existing _lock(key) and calls save(...) (backend/services/notebook_store.py:119-145). Preserves outputs by default (mirrors what apply_source_update does at :312-314).
• Inputs: notebook_name, cell_id, source, cell_type (optional)
• Emits the same UI event the frontend's PUT path already emits, so the cell updates live
• Returns {cell_id, source_len}

backend/skills/delete-notebook-cell/

• Wraps a new notebook_store.delete_cell(session_id, name, cell_id) helper
• Returns {cell_id, remaining_cells}

backend/skills/rerun-notebook-cell/

• Look up the cell by cell_id, clear its outputs, re-execute against the long-lived kernel (kernel_manager.execute or whatever the current entry point is — same kernel that run-notebook-cell uses, so state persists)
• Same outputs streaming path as run-notebook-cell (on_cell_event at backend/services/notebook_store.py:242-293 already handles cell-id-keyed output events)
• Returns {cell_id, exec_count, duration_ms, had_error, outputs} — matches run-notebook-cell's shape

3. execute-code is not deprecated

It's still the right tool for genuine one-shot Python — "give me the dtype of column X", "what's the shape of this parquet", "print the head". The contract becomes:

execute-code is for headless, throwaway Python. Auto-saved to scripts/ as the audit log. For anything you'd want to call again, write a file with write-file and run-file it.

Same handler, same schema, same auto-save. We just stop using it as the universal hammer.

4. Prompt updates — teach DRY actively, not descriptively

Today's shared "Workspace is a real Python repo" block (verbatim in chat.yaml:173-182, data_prep.yaml:68-84, feature_eng.yaml:59-69, trainer.yaml:69-88, plus near-clones in eda and orchestrator) describes the convention. It needs to prescribe a workflow with the new verbs. Suggested replacement block (shared across every agent that today has execute-code):

## Authoring vs running — use the right verb

Your session is a real Python repo. `/sessions/{sid}/src/` is on sys.path,
`/sessions/{sid}/` is your cwd. The skills split along the verbs you need:

- write-file(path, content)         create or replace a file at `path`
- edit-file(path, mode, ...)        change part of an existing file
- run-file(path)                    execute a file in the sandbox
- execute-code(code)                one-shot throwaway Python; auto-saved to scripts/
- append-notebook-cell / run-notebook-cell / edit-notebook-cell /
  delete-notebook-cell / rerun-notebook-cell  for interactive work

**Rule of three:** before pasting code into a third execute-code call on
the same theme, stop and refactor. write-file(`src/{name}.py`, ...) the
shared body, then run-file it (or `from {name} import ...` in your next
cell). This holds for notebook cells too — `import` from `src/` works
inside a kernel cell.

**Edit, don't repaste.** If you'd be about to write-file a near-duplicate
of an existing module, edit-file the existing one. `scripts/` is the
audit log of what ran; `src/` is the library that survives. Keep `src/`
small and edited; let `scripts/` accumulate.

Per-agent tail keeps the existing examples (src/cleaning.py for data_prep, src/features.py for feature_eng, src/training.py for trainer). The shared block above replaces the descriptive paragraph; the example bullets stay.

5. Notebook ↔ src/ import parity is real — say so

backend/services/kernel_manager.py:110 runs the SDK preamble in the kernel at boot, so sys.path.insert(0, '/data/sessions/{sid}/src') is already there for notebook cells. Today no prompt makes this explicit. The shared block above ("import from src/ works inside a kernel cell") fixes that. Add to eda specifically: "When you find yourself pasting the same helper into a third notebook cell, write-file it to src/{name}.py and import it in the next cell — the kernel will find it without restart."

6. Audit-log hygiene

The current behavior conflates "the agent ran some code" with "the agent authored a file". After the split:

• execute-code → auto-saves to scripts/step_NN_*.py (today's behavior, unchanged)
• run-file → auto-saves a one-line runpy.run_path("src/X.py") to scripts/step_NN_run_X.py (so the audit log still records what was run, but the body lives next door)
• write-file, edit-file, edit-notebook-cell, delete-notebook-cell → emit file_created / file_updated events for the UI, but do not write into scripts/. Authoring is not a step.

Optional: rename the auto-save filename for run-file to make it visually distinct (scripts/step_NN_run_X.py vs scripts/step_NN_X.py) so a quick ls scripts/ shows the difference between one-shots and reuses.

7. Frontend — src/ deserves to be visible

Today /sessions/{sid}/src/ is invisible in the tree until someone writes there, and there's no file_updated event distinct from file_created (frontend/src/app/page.tsx:580-601). Two small additions:

• Handle file_updated events — update the node in place, flash it briefly so the user sees what changed.
• Pin src/ open by default in the explorer; collapse scripts/ by default (it's the audit log, not the working area). This makes the DRY pattern visible to humans too.

Philosophy

A few principles this issue is anchored in:

1. Audit logs record what ran, not what was authored. A scripts/ directory full of Path("src/foo.py").write_text(...) is not an audit log of an experiment — it's a recording of typing. The split makes scripts/ reliably mean "things that ran" so it stays useful: you can re-execute it, you can show it to a human reviewer, you can replay it as a notebook.

2. DRY is a tool-shape problem before it's a prompt problem. When the cheapest verb is "throw a fresh script at the wall", agents throw fresh scripts. Adding edit-file and run-file doesn't force DRY, but it removes the gravitational pull away from it. Prompts can then articulate a workflow ("rule of three: refactor on the third paste") that maps onto verbs the agent actually has.

3. Skills should be the verbs in the prompts. If the prompt says "factor reusable code into src/ and import it", the agent should not have to simulate that with a string-literal .write_text() call inside execute-code. The prompt and the tool surface should rhyme: one verb per intent. The notebook surface today already does this — append-notebook-cell is a cleaner verb than "execute-code that happens to write a .ipynb". Extend the same treatment to files.

4. Don't deprecate the hammer; add the screwdriver. execute-code is genuinely the right tool for one-shot exploratory Python and stays. The split is additive. Migration is: new agents get the new skills, existing flows keep working, prompts steer the agent toward the new verbs over time.

5. Notebooks aren't a parallel universe. The same DRY discipline applies inside a notebook — pasting the same loader into three cells is no better than pasting it into three scripts. The fix is the same: write-file it to src/, then import it from the next cell. Today's prompts treat notebooks and scripts as separate worlds; the new shared block treats them as two front-ends to the same library.

Files to change

Backend — new skills:

• backend/skills/write-file/{SKILL.md, schema.yaml, handler.py} — wraps services.volume.write_to_volume (backend/services/volume.py:200-235); emits file_created / file_updated
• backend/skills/edit-file/{SKILL.md, schema.yaml, handler.py} — read via read_volume_file_async (backend/services/volume.py:84-95), apply patch, write back; emits file_updated
• backend/skills/run-file/{SKILL.md, schema.yaml, handler.py} — wraps services.sandbox.run_code with runpy.run_path(path) body; writes a one-line audit script
• backend/skills/edit-notebook-cell/{SKILL.md, schema.yaml, handler.py} — calls new notebook_store.edit_cell helper
• backend/skills/delete-notebook-cell/{SKILL.md, schema.yaml, handler.py} — calls new notebook_store.delete_cell helper
• backend/skills/rerun-notebook-cell/{SKILL.md, schema.yaml, handler.py} — looks up cell by id, clears outputs, re-executes against the persistent kernel

Backend — service helpers:

• backend/services/notebook_store.py — add edit_cell(session_id, name, cell_id, source, cell_type=None) next to append_cell (L193); add delete_cell(session_id, name, cell_id). Both use the existing _lock(key) pattern and save(...) (L119-145). apply_source_update (L296-316) stays as-is for the frontend HTTP PUT path; the new helpers are the per-cell agent surface.
• backend/services/volume.py — no change; the primitives are already there (write_to_volume, read_volume_file_async)
• backend/services/sandbox.py — no change; the src/ on sys.path work shipped in [P1] [feature-request] Treat the sandbox as a real Python repo — let the agent import its own modules across execute_code calls #51

Backend — no-op-but-documented:

• backend/skills/execute-code/handler.py:89-99 — no functional change. Update SKILL.md to say "for one-shot throwaway Python — for anything reusable, use write-file + run-file."

Agent prompts — replace the "Workspace is a real Python repo" block with the prescriptive one from §4 (keep per-agent example bullets):

• backend/agents/chat.yaml:173-182
• backend/agents/eda.yaml:60-69
• backend/agents/data_prep.yaml:68-84
• backend/agents/feature_eng.yaml:59-69
• backend/agents/trainer.yaml:69-88
• backend/agents/orchestrator.yaml:217-221

Add the six new skills to each agent's skills: list wherever execute-code is already listed.

Frontend:

• frontend/src/app/page.tsx:580-601 — handle file_updated events alongside file_created; flash the updated node briefly.
• Component(s) that render the file tree — pin src/ open by default, collapse scripts/ by default.

Tests:

• backend/tests/skills/test_write_file.py, test_edit_file.py, test_run_file.py — happy path + failure path (write to invalid path, edit with non-matching anchor, run nonexistent file)
• backend/tests/skills/test_edit_notebook_cell.py, test_delete_notebook_cell.py, test_rerun_notebook_cell.py — happy path + failure path (cell_id not found)
• backend/tests/services/test_notebook_store.py — edit_cell preserves outputs; delete_cell updates the on-volume .ipynb

Acceptance criteria

• write-file(path="src/loaders.py", content="def load(): ...") emits file_created, file lands on volume, no entry appears in /sessions/{sid}/scripts/.
• edit-file(path="src/loaders.py", mode="replace", old="def load(): ...", new="def load(n=1): ...") emits file_updated, file mutated correctly, no scripts/ entry.
• run-file(path="src/loaders.py") executes the module in a sandbox; scripts/ gets a one-line runpy.run_path("src/loaders.py") audit entry, not the file body.
• edit-notebook-cell(notebook_name="x", cell_id="abc", source="new") updates the cell source, preserves its outputs (same behavior as apply_source_update's output preservation at notebook_store.py:312-314).
• delete-notebook-cell(notebook_name="x", cell_id="abc") removes the cell; UI updates live.
• rerun-notebook-cell(notebook_name="x", cell_id="abc") clears the cell's outputs, re-runs against the persistent kernel, streams new outputs back. Variables set in earlier cells remain in scope.
• A notebook cell containing from loaders import load works after write-file(path="src/loaders.py", ...) — no kernel restart needed (relies on existing [P1] [feature-request] Treat the sandbox as a real Python repo — let the agent import its own modules across execute_code calls #51 plumbing).
• An end-to-end trainer run on a small dataset produces: at least one src/*.py module, at least one edit-file call against it, and at least one run-file call against it. The agent demonstrably reuses instead of repastes.
• No regression to execute-code — same schema, same auto-save, all existing tests pass.
• All six new skills appear in chat, eda, data_prep, feature_eng, trainer, orchestrator agent YAMLs alongside execute-code.
• All six agent YAMLs use the new shared "Authoring vs running" block (§4) verbatim.

Out of scope

• File deletion (delete-file) — useful but orthogonal; add later if needed.
• Cross-session promotion of src/ modules (per-session by design — that's a separate "templates" feature).
• Git-style three-way merge or fuzzy patching for edit-file — the Edit tool's exact-anchor model is enough for v1.
• Notebook cell reordering / drag-and-drop (UI concern).
• A unified diff format for edit-file (anchor mode is simpler and matches what agents are already good at).
• Re-running multiple cells or "run from here" (rerun-notebook-cell is per-cell; multi-cell can come later).

[lucastononro]

Implemented in #175, merged into staging-v0.0.5 (ledger: STAGING-v0.0.5.md). Will land on main via the staging merge — closing.