feat: recovery_candidates wired into process_plan_tick + Pipeline 3 telemetry + experiments 008/009

[Fede654]

TL;DR

Adds Codex Path D — deterministic recovery-candidate synthesis — at the right architectural layer (the autonomous loop's per-step retry handler in `agent_loop.py:process_plan_tick`). Plus 4 commits of additive telemetry on the embodied service and 2 new primitives_lab experiments that empirically motivate the work.

The body model `gemma-andy:e4b-v2-2-3-q8_0` is a selector, not a searcher: given an explicit substitute it executes 100% (lesson 7 + experiment 009: 20/20), given only a negation it re-emits the failing target 60-100% (lesson 8). The right division of labour: the autonomous loop has world state in scope (via `fetch_bot_*` REST calls) → it synthesises candidates deterministically → Andy executes via selection.

What lands

Architectural — recovery_candidates at the autonomous-loop retry layer

• `agents/recovery_candidates.py` (new, 195 lines, pure-Python, no I/O) — `maybe_synthesize_substitute(step, embodied_result, *, bot_position, nearby_blocks, inventory) -> Optional[str]`.
  • Subtype detection via regex on `execution_results[0].details`:
    • `target_occupied` → `_find_air_with_neighbor` returns nearest face-adjacent air voxel with a solid neighbor
    • `no_solid_neighbor` → 2-step "support-then-target" using existing column solids
    • `bot_self_position` → horizontal neighbor of the bot with a solid block below
    • `missing_inventory` → swap to first available block from `_placement_substitutes_in_inventory`
  • Returns `None` for unmodeled subtypes (preserves existing retry-with-backoff fallback).
• `tests/test_recovery_candidates.py` (new, 162 lines, 6 unit tests, all pass).
• `agents/agent_loop.py:process_plan_tick` (+34 lines) — between `verify_step` failure and `step.retries++`, when retries remaining ≥ 2, calls the synthesiser using fresh `fetch_bot_status/nearby/inventory` and re-runs `call_embodied` with the substitute intent in the SAME tick. Substitute intent is NOT persisted to `plan.json` (Plan stays under Steve's authority). Logs new events: `substitute_synthesised`, `substitute_verify`, `substitute_synthesis_failed` (replayable per the loop's structured-JSON convention).

Pipeline 3 — Embodied-service telemetry

Field session 2026-05-10 (during PR #11 work) spent debug time correlating bot HTTP errors back to dispatcher state. These four commits surface the data the operator needs without changing service behaviour:

• `tool_dispatch` logEvent now carries `details` (verbatim bot error message). One line, hooks the existing `r.details` field that `foldBotResponse` already produces.
• `ollama_call_done` logEvent now carries `think_excerpt` (first 500 chars of `` block). Enables grep-friendly auditing without dumping nested `plan` JSON.
• `parse_failed` logEvent now carries `raw_length_chars` + `truncated` boolean (heuristic: non-empty AND no `}` in last 50 chars OR no `}` at all). The 2026-05-10 num_predict=512 truncation incident motivated this — distinguishes "model output truncated" from "model output unparseable".
• Heuristic refinement (post-review) — drop the `]` check and short-circuit on `last_brace === -1` to fix two correctness bugs flagged by code review (length-dependent verdict on missing `}`; AND-combinator missing one-sided truncation when the inner `]` lands near end but outer `}` doesn't).

primitives_lab experiments

• `008_recovery_generalization.yaml` — 4 variants, each narrating a distinct `bot_action_failed` mode WITHOUT naming a substitute. Result: 17/20 re-emit the failing position (negative result). This is what motivated the architectural reframe.
• `009_explicit_substitute_recovery.yaml` — same 4 modes but recovery intent NAMES the substitute. Result: 20/20 substitute-match, 0/20 re-emit (e2e validated — sample 1 of each variant placed in-world). This is the empirical basis for `recovery_candidates.py`.

Empirical context (vault links external to this repo, summary inline)

The body model's behaviour now characterised by 3 ladder runs:

• Lesson 7 (experiment 007, n=10): 9-10/10 when intent NAMES substitute ("Use oak_planks instead of oak_log")
• Lesson 8 (experiment 008, n=5): 0-2/5 when intent only NARRATES failure ("do not re-emit (7, 65, 35)")
• Lesson 9 (experiment 009, n=5): 5/5 across ALL FOUR 008 modes when intent NAMES substitute

The pattern: selection works, search doesn't. The candidate synthesiser does the search; the model does the select.

The architectural reframe was suggested by Codex (`gpt-5.4`) in a single-pass audit (job `opinion-1778391655600-e69694b4`, log at `~/.bridge-ai/opinions/20260510-024055-hermes-agent-codex-second-opinion.md` on Fede's machine — quoted findings: "the architectural bug is outsourcing feasible local recovery search to the body model" / "missing boundary is which layer is responsible for generating feasible alternatives"). Confidence: medium-high.

Per-commit list (bottom → top)

f540419 test(recovery_candidates): TDD step 1 — failing tests for substitute synthesiser
c0eb53e feat(daemoncraft): recovery_candidates — deterministic substitute synthesis
ef894f6 feat(agent_loop): wire recovery_candidates into process_plan_tick
456a4b7 feat(embodied-service): tool_dispatch logEvent now carries details field
40654d8 feat(embodied-service): ollama_call_done log carries think_excerpt
1a90bcb feat(embodied-service): parse_failed log carries length + truncation flag
4f555aa fix(embodied-service): tighten parse_failed truncation heuristic
fb44fff feat(primitives_lab): experiment 008 — recovery generalization
789ff2d feat(primitives_lab): experiment 009 — explicit substitute recovery

7 files changed, +583/-1.

Test plan

• `pytest tests/test_recovery_candidates.py` → 6/6 pass (committed at `c0eb53e` + cherry-picked here).
• `python -c "import ast; ast.parse(open('agents/agent_loop.py').read())"` → SYNTAX OK.
• `python -c "import sys; sys.path.insert(0, 'agents'); from agent_loop import process_plan_tick"` → IMPORT OK.
• Telemetry verification curl probes: each new field (`details`, `think_excerpt`, `raw_length_chars`, `truncated`) confirmed present in subsequent log events post-restart of the embodied service.
• Experiment 009 e2e: sample 1 of each variant placed substitute block in-world; samples 2-5 returned `ok=false` because their substitute coord was now occupied (correctly reflecting world-state change).
• Live run with a Plan-driven Sparky exercising `process_plan_tick` substitute path — deferred to first session after merge.

Notes for the reviewer

• The substitute is computed in-memory and never written to `plan.json`. Plan stays under Steve's (cloud LLM) authority. The synthesiser is bounded to one extra `call_embodied` per tick.
• The `step.retries < step.max_retries - 1` guard preserves the last retry budget for the un-substituted fallback case — if synthesis somehow makes things worse, the loop still has at least one un-substituted retry before `step.exhausted`.
• `recovery_candidates.py` covers 4 subtypes derived from `bot/server.js` `place()` throw sites. Other failure modes (`reach_too_far`, `face_obstructed`, `server_blockUpdate_timeout`, mining failures) fall through to existing retry. Subtype-by-subtype additions are cheap once observed.

🤖 Generated with Claude Code