feat: day 3 agent loop - llm chat+fallback, /agent/run API, AI-free tests

Author: aadityakumarsah

docs/day3-agent-loop.md

@@ -0,0 +1,85 @@
+# Day 3 — Agent loop: line-level report
+
+Standing-rule report: what, why, what changed, and exactly where (file:lines).
+
+## What
+
+Built the agent's brain (the loop), upgraded the LLM "phone", exposed the loop
+as an HTTP API, pinned it with AI-free tests, and proved it live. The system
+now goes: task → model thinks → real tool call in a Docker sandbox → answer —
+all visible step by step.
+
+## Why
+
+- **The loop is the product:** everything after this (sessions, tools, memory,
+  the frontend) hangs off `run_agent`. Day 3 made it real and testable.
+- **Resilience planted early:** fallback provider (Gemini/etc.) if OpenRouter
+  dies; `achat` keeps the event loop responsive; max-10-step guard is the
+  robot's seatbelt.
+- **Traceability = future observability:** every step is returned in the API
+  response — Day 45's replays, forked from Day 3's JSON.
+
+## What changed (file → lines)
+
+### `orchestrator/agent.py` (new, 111 lines) — the loop
+| Lines | Content |
+|---|---|
+| 8 | `MAX_STEPS = 10` guard |
+| 10-18 | system prompt: sandbox rules, tool, `<answer>`/`<bash>` contract |
+| 20-33 | `run_bash` tool schema |
+| 36-45 | `_summarize`: command output → short context for the model |
+| 48-111 | `run_agent`: message loop, tool execution, `<bash>` fallback, `<answer>` extraction, max-steps return |
+
+### `orchestrator/llm.py` (edited) — the phone
+| Lines | Content |
+|---|---|
+| 1 | `asyncio` import |
+| 21-23 | fallback provider settings (`LLM_FALLBACK_*`) |
+| 26-43 | `_completion` (shared provider call) |
+| 59-88 | `chat`: full message with `tool_calls` + automatic fallback |
+| 91-96 | `achat`: async twin, runs LLM call off the event loop |
+| 46-56 | `ask_llm` unchanged (ping) |
+
+### `orchestrator/main.py` (edited) — the door
+| Lines | Content |
+|---|---|
+| 5 | `from agent import run_agent` |
+| 18-19 | `AgentRunRequest {task}` |
+| 47-51 | `POST /agent/run`: 422 on empty task, else run and return `{ok, answer, steps}` |
+
+### Tests (new) — no AI, no network, no Docker
+| File | Lines | Covers |
+|---|---|---|
+| `test_agent.py` | 22-34 | immediate answer (0 tools) |
+| `test_agent.py` | 37-66 | real tool call → step recorded |
+| `test_agent.py` | 69-101 | `<bash>` tag fallback |
+| `test_agent.py` | 103-121 | max-steps guard fires |
+| `test_llm.py` | 1-81 | fallback routing + achat (4 tests) |
+| `test_agent_api.py` | 1-29 | `/agent/run` wiring + 422 |
+
+### Docs (new)
+| File | Topic |
+|---|---|
+| `docs/day3-step1-agent-loop.md` | step 1 report |
+| `docs/day3-step2-llm-upgrade.md` | step 2 report |
+| `docs/day3-step3-agent-run.md` | step 3 report |
+| `docs/day3-step4-agent-tests.md` | step 4 report |
+| `docs/day3-step5-live-demo.md` | live demo + reproduce |
+| `docs/day3-agent-loop.md` | this report |
+
+## Verified
+
+- `ruff check` / `ruff format --check` — clean
+- `mypy` — clean
+- `pytest` — **13 passed** (no network, no Docker)
+- Live (Step 5): 2-step tool run (`ls -la /` → `python3 --version` → answer)
+  and 0-tool run (`2+2` → `4`); 4 free-tier calls
+- CI (GitHub Actions): orchestrator ruff+mypy+pytest, gateway clippy+test,
+  frontend build
+
+## How it works in one sentence
+
+`POST /agent/run` takes a task; `run_agent` loops: model (via `llm.achat`)
+thinks → if it asks, `sandbox.sandbox_exec` runs the command in the Docker
+sandbox and the result is fed back → until an `<answer>` appears or the
+10-step guard trips; every tool call is kept in `steps[]` and returned as-is.

docs/day3-step1-agent-loop.md

@@ -0,0 +1,102 @@
+# Day 3, Step 1 - The agent loop engine (`orchestrator/agent.py`)
+
+The brain learns to press its own button. This file turns the orchestrator from a
+remote-controlled robot into one that thinks: decide a command -> run it in the sandbox ->
+read the output -> repeat -> answer.
+
+## The topic (ELI5)
+
+An **agent loop** is: THINK -> ACT -> LOOK -> THINK -> ... -> DONE.
+
+- THINK: the LLM decides the next shell command (or says "I'm done")
+- ACT: we run that command in the sandbox (the `/exec` button from Step 2)
+- LOOK: we show the output back to the LLM
+- Repeat until the LLM answers, or we hit the safety limit (10 steps)
+
+The LLM can press the button two ways:
+1. **Tool calling (primary):** the LLM returns a structured
+   `{"name": "run_bash", "arguments": {"cmd": "ls -la /"}}` message (the industry-standard
+   way - what Devin / Claude Code do).
+2. **Tag language (fallback):** some free models can't do tool calling, so we also teach:
+   `<bash>command</bash>` = press the button, `<answer>text</answer>` = I'm done.
+   Either way, the loop works. This resilience is the interview-gold part.
+
+## What I did, file by file (with line ranges)
+
+### NEW `orchestrator/agent.py` (111 lines) - the loop engine
+- Lines 1-6: imports. Note lines 5-6: `import llm` and `import sandbox` (module imports,
+  NOT `from x import y`). Why: tests monkeypatch `llm.chat` and `sandbox.sandbox_exec`;
+  module-style imports let the patch take effect at call time. (Learned the hard way -
+  function imports silently ignored the patch and hit the real network.)
+- Line 8: `MAX_STEPS = 10` - the "unplug the robot" limit.
+- Lines 10-18: `SYSTEM_PROMPT` - the rules of the game: one tool, network limited to
+  GitHub/PyPI/npm, prefer short commands, `<answer>` protocol, `<bash>` fallback.
+- Lines 20-33: `BASH_TOOL` - the JSON "button spec" in OpenAI tool-calling format
+  (`type: function`, name `run_bash`, one required arg `cmd`).
+- Lines 36-45: `_summarize()` - turns an ExecResult into a short readable string for the LLM
+  (handles timeouts, stderr-only, normal stdout, no output).
+- Lines 48-53: `run_agent(task, max_steps=10)` - builds the message history
+  `[system, user task]` and an empty `steps` trace log.
+- Lines 55-59: the loop head - one LLM call per iteration via `llm.chat(messages, tools=...)`.
+  `# noqa: BLE001` is intentional: LLM providers throw a zoo of exception types (verified
+  live - no common base class exists), and the loop must survive any of them.
+- Lines 62-89: **tool-calling path** - if the model returned `tool_calls`, parse the JSON
+  args (bad JSON -> empty cmd -> a helpful error result), run it with `sandbox.sandbox_exec`,
+  record the step, append the result as a `role: "tool"` message, continue the loop.
+- Lines 91-99: **tag fallback path** - no tool_calls but `<bash>cmd</bash>` present: run the
+  last one, append the output as a plain `user` message, continue.
+- Lines 101-105: **done path** - extract `<answer>...</answer>`, or treat any remaining text
+  as the final answer; return `{"ok": True, "answer": ..., "steps": [...]}`.
+- Lines 107-111: **safety exit** - 10 steps without an answer -> `{"ok": False, "error": "max steps ..."}`.
+
+### `orchestrator/llm.py` (was 30 lines, now 51)
+- Line 2: added `from typing import Any`.
+- Lines 34-51: NEW `chat(messages, tools=None, timeout_s=120)` - like `ask_llm` but takes the
+  full message history and optional tools, and returns the whole assistant message as a dict
+  (via `model_dump()`), including `tool_calls` when the model wants a tool.
+  `ask_llm` (lines 21-31) is unchanged and still powers `/llm/ping`.
+- Effect: the phone now speaks the tool-calling dialect; the agent loop uses it.
+
+### NEW `orchestrator/test_agent.py` (118 lines) - CI-safe fake-brain tests
+- Lines 6-18: `_tool_msg()` helper - builds a fake tool-calling message.
+- Lines 21-31: `test_answers_without_tools` - model answers immediately -> ok, no steps.
+- Lines 33-57: `test_uses_bash_tool` - model requests `ls -la /`, gets a fake result,
+  then answers -> 1 tool step recorded.
+- Lines 59-87: `test_tag_fallback_when_no_tool_calling` - `<bash>echo hi</bash>` path.
+- Lines 89-101: `test_max_steps_guard` - model loops forever -> ok False, "max steps", 10 steps.
+- Lines 103-106: `run_agent_sync()` - small helper wrapping `asyncio.run`.
+- Key detail: `fake_chat` fakes are **sync** (llm.chat is sync) and `fake_exec` fakes are
+  **async** (sandbox_exec is awaited) - mismatching these makes tests explode with
+  coroutine errors (learned the hard way, twice).
+
+## Why (the decisions)
+- **Why a trace (`steps` list)?** Every action is recorded end-to-end. That's Day 45's
+  observability planted early - you can see exactly what the robot did.
+- **Why module imports?** Testability (see above).
+- **Why MAX_STEPS = 10?** Free-tier models are rate-limited (~50 calls/day); 10 steps is a
+  generous ceiling that also prevents infinite loops and cost blowups.
+- **Why both tool calling AND tags?** Free OpenRouter models rotate; the loop must keep
+  working no matter which model the auto-router picks.
+- **Why catch-all with noqa?** Verified: litellm exception hierarchy has no common base.
+  A blind catch with an explanation is the correct engineering choice here.
+
+## What changed / what it brings
+- The orchestrator can now run an autonomous multi-step agent task with one function call:
+  `run_agent("list files and tell me the python version")`.
+- No new dependencies (uses `llm.py` + `sandbox.py` + stdlib `json`/`re`).
+- Test count: orchestrator now 7 tests (3 API + 4 agent), all CI-safe (no network, no Docker).
+
+## Proof (test run, 9 Aug 2026)
+```
+$ uv run pytest -q
+7 passed, 1 warning in 2.46s
+
+$ uv run ruff check .       -> All checks passed!
+$ uv run ruff format --check . -> 7 files already formatted
+$ uv run mypy agent.py ...  -> Success: no issues found
+```
+
+## What's next (Day 3, Steps 2-5)
+- Step 2: expose the loop as `POST /agent/run` in `orchestrator/main.py`
+- Step 5: LIVE demo - the robot solves "list files and tell me the Python version" for real,
+  using the OpenRouter key and the Docker sandbox.

docs/day3-step2-llm-upgrade.md

@@ -0,0 +1,71 @@
+# Day 3 — Step 2: Upgrade the phone (`orchestrator/llm.py`)
+
+## Topic
+
+`orchestrator/llm.py` is the "phone" of the agent: the only module allowed to
+talk to an LLM provider (OpenRouter today, via LiteLLM). This step upgraded it
+from a single sync call into a resilient, async-capable layer.
+
+Half of this step already landed during Day 3 Step 1 (the `chat(messages,
+tools=None)` function that returns the full message with `tool_calls`, used by
+the agent loop). This step finished the job.
+
+## What I did
+
+1. **Added fallback provider settings** to `LLMSettings`
+   (`llm.py:21-23`): `LLM_FALLBACK_PROVIDER`, `LLM_FALLBACK_API_KEY`,
+   `LLM_FALLBACK_MODEL`. These were already documented in
+   `infra/.env.example` but were never implemented.
+2. **Extracted `_completion()`** (`llm.py:26-43`): the actual LiteLLM call,
+   parameterized by provider/key/model, so both `chat` and the fallback path
+   share one code path.
+3. **Automatic fallback in `chat()`** (`llm.py:59-88`): if the primary call
+   throws (provider down, rate limit, bad key), it retries once with the
+   fallback provider — but only if `LLM_FALLBACK_API_KEY` is configured.
+   Otherwise it re-raises the original error.
+4. **Added `achat()`** (`llm.py:91-96`): async twin of `chat` that runs the
+   blocking LiteLLM call in a worker thread via `asyncio.to_thread`, so the
+   server event loop is never blocked.
+5. **Agent loop now uses the async phone** (`agent.py:57`):
+   `await llm.achat(messages, tools=[BASH_TOOL])`.
+6. **New tests** (`test_llm.py`, 4 offline tests): primary path passes tools +
+   key through; fallback fires when primary fails and dials the backup; no
+   fallback configured means the error propagates; `achat` returns the same
+   result off the event loop.
+
+## Why
+
+- **Resilience:** `run_agent`/`/agent/run` depend on one provider. If OpenRouter
+  is down or rate-limited, the whole agent dies. The fallback (e.g. Gemini via
+  a second key) keeps the loop alive for a fraction of the code cost — it was
+  already promised by `.env.example`, now it actually works.
+- **Event-loop hygiene:** FastAPI's endpoints are async; `run_agent` awaited
+  a sync LiteLLM call, which would freeze the entire server for up to 120s on
+  one slow request. `achat` moves that off-thread.
+- **Tests stay offline:** the fallback logic is pure routing around a
+  monkeypatched `litellm.completion` — no network, no keys needed.
+
+## What change it brings
+
+- If you put `LLM_FALLBACK_PROVIDER=gemini`, `LLM_FALLBACK_API_KEY=...`,
+  `LLM_FALLBACK_MODEL=gemini-2.5-flash` in `infra/.env`, agent requests
+  automatically use Gemini when OpenRouter fails.
+- The agent loop no longer blocks the FastAPI event loop.
+- Behavior with a single key is unchanged (verified with a real call: reply
+  `ok`, no tool_calls).
+
+## Where (file → lines)
+
+| File | Lines | Change |
+|---|---|---|
+| `orchestrator/llm.py` | 1-9 | imports + env path (asyncio added) |
+| `orchestrator/llm.py` | 12-23 | `LLMSettings` fallback fields |
+| `orchestrator/llm.py` | 26-43 | new `_completion()` (extracted) |
+| `orchestrator/llm.py` | 46-56 | `ask_llm` unchanged |
+| `orchestrator/llm.py` | 59-88 | `chat` with fallback logic |
+| `orchestrator/llm.py` | 91-96 | new `achat()` |
+| `orchestrator/agent.py` | 57 | loop uses `await llm.achat(...)` |
+| `orchestrator/test_llm.py` | 1-81 | new file, 4 offline tests |
+
+Verified: `ruff check` clean, `mypy` clean, `pytest` 11 passed
+(3 sandbox + 4 agent + 4 llm), plus 1 live call to the real API.

docs/day3-step3-agent-run.md

@@ -0,0 +1,69 @@
+# Day 3 — Step 3: Expose the loop to the world (`orchestrator/main.py`)
+
+## Topic
+
+The agent loop (`agent.py`) was a library function. This step gives it a door:
+`POST /agent/run` on the orchestrator API. Anyone can now send a task and get
+back the full trace of the agent's thinking + the final answer.
+
+## What I did
+
+1. **Imported the loop** (`main.py:5`): `from agent import run_agent`.
+2. **Request model** (`main.py:18-19`): `AgentRunRequest` with a single `task`
+   field, validated by Pydantic.
+3. **The endpoint** (`main.py:47-51`):
+   - Empty/whitespace task → HTTP `422` (fail fast, no LLM call wasted).
+   - Otherwise `await run_agent(req.task)` → returns the loop's dict directly:
+     `{ok, answer, steps}` (or `{ok: false, error, steps}` on failure).
+4. **API tests** (`test_agent_api.py`, new file):
+   - `test_agent_run_wiring` — monkeypatches `main.run_agent` (the name bound
+     in main's namespace, not the source module) so no network is touched;
+     asserts 200 + `ok`/`answer`/`steps` shape.
+   - `test_agent_run_rejects_empty_task` — asserts 422 + message.
+5. **Fixed the environment** for the live demo: the Rust sandbox-gateway
+   (port 8080) was built but not running — the first demo attempt failed with
+   `httpx.ConnectError` at the gateway. Restarted the prebuilt binary
+   (`sandbox-gateway/target/debug/sandbox-gateway`).
+
+## Why
+
+- **Traceability by design:** the response keeps every `steps[]` entry —
+  each tool call, its command, and the raw sandbox output. That's the seed of
+  Day 45's observability (replay "what did the agent do?"): today it's just
+  JSON in an HTTP response, later it becomes stored sessions + live streams.
+- **The demo proved the whole stack:** HTTP → FastAPI → agent loop → LiteLLM
+  → real model → real tool call → Docker sandbox → extracted `<answer>`.
+
+## What change it brings
+
+- `curl -X POST localhost:8000/agent/run -H 'Content-Type: application/json' -d '{"task": "..."}'`
+  now drives the full agent. Response example (real run):
+
+```json
+{
+  "ok": true,
+  "answer": "The directories in the root directory (`/`) are: bin, boot, dev, ...",
+  "steps": [
+    {
+      "type": "tool",
+      "cmd": "ls -la /",
+      "result": { "exit_code": 0, "stdout": "total 56\ndrwxr-xr-x ...", "duration_ms": 3746 }
+    }
+  ]
+}
+```
+
+- Bad input is rejected with 422 before any API calls are spent.
+
+## Where (file → lines)
+
+| File | Lines | Change |
+|---|---|---|
+| `orchestrator/main.py` | 5 | import `run_agent` |
+| `orchestrator/main.py` | 18-19 | `AgentRunRequest` model |
+| `orchestrator/main.py` | 47-51 | `POST /agent/run` endpoint (422 guard + call) |
+| `orchestrator/test_agent_api.py` | 1-29 | new file, 2 offline API tests |
+
+Verified: `ruff` clean, `mypy` clean, `pytest` **13 passed**, plus a live
+end-to-end run over real HTTP (server on :8010, real LLM, real Docker sandbox —
+1 tool step, answer with directory list).

docs/day3-step4-agent-tests.md

@@ -0,0 +1,65 @@
+# Day 3 — Step 4: Tests that need no AI (`orchestrator/test_agent.py`)
+
+## Topic
+
+The agent loop must be testable **without any AI** — no API key, no network,
+no Docker. Step 4 is the test suite that proves the loop's behavior by
+swapping the real brain (LLM) and hands (sandbox) for scripted fakes.
+
+## What I did
+
+A monkeypatched **fake brain** (`llm.chat`) plus a **fake hand**
+(`sandbox.sandbox_exec`) drive `run_agent` in 4 offline tests:
+
+1. **Answers immediately** (`test_agent.py:22-34`) — fake brain replies
+   `<answer>42</answer>` on the first turn. Asserts `ok`, `answer == "42"`,
+   `steps == []` (no tool was ever needed).
+2. **Uses the tool** (`test_agent.py:37-66`) — fake brain emits a real
+   `tool_calls` entry on turn 1 (via the `_tool_msg` helper, `:8-19`), then
+   `<answer>files listed</answer>` on turn 2. Fake hand executes `ls -la /`.
+   Asserts the tool result landed in `steps[0]` with the right `cmd`.
+3. **`<bash>` tag fallback** (`test_agent.py:69-101`) — fake brain ignores tool
+   calling and answers in plain text with `<bash>echo hi</bash>`; the loop must
+   parse the tag and still execute the command. Asserts `steps[0]["cmd"] == "echo hi"`.
+4. **Never finishes → max-steps guard** (`test_agent.py:103-121`) — fake brain
+   loops forever issuing tool calls. Asserts `ok is False`, error mentions
+   "max steps", and exactly `MAX_STEPS` steps were recorded (no infinite loop).
+
+`run_agent_sync` (`:124-125`) wraps the async loop in `asyncio.run` so tests
+stay plain sync functions.
+
+## Why
+
+- **CI-safe by construction:** the loop's only two I/O boundaries are
+  monkeypatched in every test — the imports are just `asyncio`, `json`,
+  `typing` and `agent`. No `httpx` socket ever opens, no Docker container ever
+  starts, no key needed. Tests run anywhere, in seconds, for free.
+- **Deterministic:** the fake brain always behaves the same, so the loop's
+  logic (tool-call routing, tag fallback, guard rails) is tested in
+  isolation — exactly what you can't get from a real, flaky model.
+- **Guard rail proven:** the max-steps test pins the loop's one real safety
+  property — a runaway model cannot run forever.
+
+## What change it brings
+
+- `pytest` now validates the core loop offline: 4 tests, ~0 API cost, 0
+  network calls, 0 Docker calls.
+- Any future edit to the loop (new tool, different prompt, guard changes) is
+  caught in seconds without touching the wallet.
+- This is the seed of the project's test discipline: fakes at the boundaries,
+  real integrations proven separately with 1-2 live calls.
+
+## Where (file → lines)
+
+| File | Lines | Change |
+|---|---|---|
+| `orchestrator/test_agent.py` | 1-5 | imports (loop constants + stdlib only) |
+| `orchestrator/test_agent.py` | 8-19 | `_tool_msg` fake assistant tool-call message |
+| `orchestrator/test_agent.py` | 22-34 | test: immediate answer, no tools |
+| `orchestrator/test_agent.py` | 37-66 | test: real tool call → result in steps |
+| `orchestrator/test_agent.py` | 69-101 | test: `<bash>` tag fallback path |
+| `orchestrator/test_agent.py` | 103-121 | test: max-steps guard fires |
+| `orchestrator/test_agent.py` | 124-125 | `run_agent_sync` async wrapper |
+
+Verified: `pytest` — 13 passed total, including these 4, with no network or
+Docker involved (all boundaries monkeypatched).