Skip to content

feat(console): unread nudge on the read path (pull/list) + final-check convention#253

Open
ivanmkc wants to merge 11 commits into
masterfrom
feat/inbox-followups
Open

feat(console): unread nudge on the read path (pull/list) + final-check convention#253
ivanmkc wants to merge 11 commits into
masterfrom
feat/inbox-followups

Conversation

@ivanmkc

@ivanmkc ivanmkc commented Jul 8, 2026

Copy link
Copy Markdown
Owner

What

Closes the biggest gap left after #252: the nudge only fired on writes. If the human typed after the agent's last push, the message was never surfaced — the eval passed only because its seed existed before the push.

  • Server: scoped GET /state (pull) now sets x-termchart-inbox-unread (same once-per-new-message semantics as the write path).
  • CLI pull: prints the content nudge from that header.
  • CLI list: workspace-wide unread summary via GET /inbox?all=1 — the one read agents still do when done writing, and the only surface where cross-scope messages can appear. Re-announces until read (an explicit catalog read should show what's unread); stderr-only, --json stdout stays clean.
  • Skill: final-check convention — run termchart list as your last termchart action. Plugin 0.16.4; stale marketplace entry (0.15.0) synced.
  • Eval: new TC-INBOX-MIDTASK case — seed_inbox.py gains @await scopes posted by a background watcher only once the agent's board appears, modelling a human typing mid-task. The write-path nudge can't fire; the list summary must catch it.

Verification

Live (local viewer): push → human message after → pull nudges with content; list re-announces until read + shows cross-scope unread; quiet after a real termchart inbox read; --json stdout still valid JSON. Touched unit tests 33/33.

Agent eval: K=3 × {Claude, Gemini} × 3 cases (PROMPTED / UNPROMPTED / MIDTASK) running now — results will be posted as a comment before undrafting.

Known race (documented in the case): if the agent's list lands before the watcher posts the seed (~0.5s poll), MIDTASK can miss — acceptable for an eval of a genuinely asynchronous channel.

…der, list summary)

The nudge fired only on writes (push/patch/scoped status) — a human message
arriving after the agent's LAST write was never surfaced. Close the gap on
both halves of the read loop:

- server: scoped GET /state (pull) now sets the x-termchart-inbox-unread
  header, same once-per-new-message semantics as the write path
- cli pull: prints the content nudge off that header
- cli list: prints a workspace-wide unread summary via GET /inbox?all=1 —
  the one read an agent still does when it's done writing, and the only
  place cross-scope messages can surface; re-announces until read (an
  explicit catalog read should show what's unread); stderr-only so --json
  stdout stays clean
- skill: final-check convention — run `termchart list` as the last
  termchart action; plugin 0.16.4, marketplace entry synced (was stale 0.15.0)
- eval: TC-INBOX-MIDTASK — seed_inbox.py gains @await scopes posted by a
  background watcher only after the agent's board appears (a human typing
  mid-task); run.sh wires the watcher up and kills it after the runner

Verified live: pull nudges after a post-push message; list re-announces
until read and shows cross-scope unread; quiet after a real read; --json
stdout unpolluted. Touched tests 33/33.
…ecklist tick

- TC-INBOX-DECIDE: agent posts two chips, blocks on inbox --wait; console_human.py
  (the simulated human, launched by run.sh) clicks 'green' via GET /suggest ->
  POST /inbox action once the chips are up. The choice reaches the agent only
  through the console. Deterministic gate: button color prop is the clicked id.
- TC-INBOX-TICK: human ticks an editable Checklist out-of-band (POST /interact —
  produces NO inbox event by design); the agent must observe it by polling pull
  and reply naming the item. Gates: tick landed in the spec + reply names it.
- pull-grep objectives now grep raw `pull` output (content in --json is an
  escaped string, so quote-anchored patterns silently miss — caught in dry-run).

Dry-run verified: watcher clicks + ticks; inbox --wait returns the action;
all objective commands pass against simulated-agent state.
…rop name

Run 1 near-miss: the flow worked end-to-end (chips, --wait click, green button
per judge, reply) but the agent expressed the color without a literal "color"
prop. Gate on content containing the clicked id and NOT the un-clicked one
(blue is also Mantine's default primary, so a click-blind agent still fails).
The inbox cases share one viewer/workspace; an agent following the list
unread-summary in one case can consume another case's seed (observed: DECIDE
read UNPROMPTED's seed before that case ran -> false FAIL). Singleton case-sets
+ a per-set clear/seed/watch/run cycle make that state-sharing unrepresentable.
…naming requirement

Run-1 miss was real signal: the agent parsed pull --json (content is a
double-encoded string), extracted an empty item id, and replied 'You checked: '.
Keep the strict objective; stop steering the agent into the harder surface.
The scenario said the item 'shows "checked": true' — spaced — but interact
re-stringifies the spec compact ("checked":true), so agents that grepped the
prompt's literal form polled for 2 minutes against a pattern that cannot match.
Say compact explicitly; the parse robustness burden stays on the agent.
@ivanmkc

ivanmkc commented Jul 8, 2026

Copy link
Copy Markdown
Owner Author

K=3 × 2 agents × 5 cases — repeat-run grid (isolated mode)

Run Claude Gemini
1 4/5 (TICK miss) 5/5
2 4/5 (TICK miss) 5/5
3 5/5 5/5

Core console flows are stable: 24/24 case-runs green across PROMPTED, UNPROMPTED, MIDTASK, and DECIDE — every repeat, both agents. In particular:

  • MIDTASK 6/6 — the new list unread-summary reliably catches a message that arrives after the agent's last push (this PR's headline change).
  • DECIDE 6/6 (after gate fix) — chips → human click → inbox --wait returns the action → agent applies the clicked choice. The agent→human decision loop works end-to-end on both agents.

TICK (checklist observation) exposed two real papercuts, both diagnosed from transcripts and fixed in-flight (each run re-reads the cases, and run 3 passed both agents on the final config):

  1. Double-encoded content: Claude's poll parsed pull --json, where content is a JSON-string-encoded spec — its extraction returned empty and it replied "You checked: " with no item name. (Prompt no longer steers to --json; the deeper UX note stands: machine-parsing boards requires a double decode.)
  2. Compact stringify: the prompt's own "checked": true (spaced) can never match the interact-rewritten spec ("checked":true, compact) — Claude polled 2 min against an unmatchable grep. Prompt now says compact explicitly.

Harness hardening shipped along the way: CASE_ISOLATED=1 (one clear+seed+watch cycle per case) after observing genuine cross-case interference — DECIDE's agent followed the list summary and consumed UNPROMPTED's seed before that case ran. Singleton case-sets make the shared-state false-FAIL class unrepresentable.

CI green. Ready for review.

@ivanmkc ivanmkc marked this pull request as ready for review July 8, 2026 03:44
…low coverage

Mechanical battery (9/9): delivery/order, ack-advance, SIGINT<3s, --json lines,
two concurrent followers, backoff growth on server death, RESTART recovery,
auto-provisioned-wsid semantics. It caught a real bug: after a viewer restart
(or clear) the server seq regresses and a live follower's stale cursor filtered
the first post-restart message out FOREVER. follow() now detects the regression
(cursor < ours, empty events), logs 'server ring reset', rewinds to 0, and
re-polls — unit-tested via the maxPolls hook (sinces 0,7,0).

Two agent cases: TC-INBOX-FOLLOW (background follow catches a mid-work @await
seed while multi-step drawing) and TC-INBOX-FOLLOWDECIDE (DECIDE forced through
--follow, no --wait — the agent must bound the unbounded stream itself). The
pair measures whether --follow can stand in for --wait under real agents.
FOLLOWDECIDE r1: Claude guessed --id/--label/--type chips, got bare 'Unknown
flag' errors, and burned its whole follow window on the guess-retry loop.
@ivanmkc

ivanmkc commented Jul 9, 2026

Copy link
Copy Markdown
Owner Author

Heavy --follow coverage + the --wait removal question

Mechanical stress battery (follow_battery.sh, committed — 9/9 against a live viewer): in-order delivery without dups, ack advancement (nudge cleared), SIGINT exit <3s even mid-poll, --json line integrity, two concurrent followers both receiving, backoff growth (1s→2s→4s→8s) with the follower surviving server death, server-restart recovery, and auto-provisioned-wsid semantics (a typo'd wsid waits silently by design — it can't 404).

It caught a real bug: after a viewer restart or clear, the server's seq regresses and a live follower's stale cursor filtered the first post-restart message out forever (one lost message per restart — and Cloud Run restarts routinely). follow() now detects the regression (server cursor < ours with empty events), logs server ring reset … resyncing, rewinds to 0, and re-polls. Unit-tested via the maxPolls hook (since sequence 0,7,0), plus live-verified in the battery. Detection latency is up to one long-poll hold (~25s) — acceptable.

Agent evals, K=3 × 2 agents:

Case Claude Gemini
FOLLOW (background listener catches a mid-work message) 3/3 2/3 †
FOLLOWDECIDE (--follow standing in for --wait) 1/3 3/3
(reference: DECIDE with --wait, from the earlier grid) 3/3 3/3

† the one Gemini FOLLOW miss passed both deterministic gates (board carried the console content + reply posted — the follow mechanics worked); only the LLM-rubric leg failed (partial banner application or judge blip).

Answer to "can we remove --wait in lieu of --follow": no — keep both. The data says they're different execution models, not redundant flags:

  • --wait is a bounded single call returning data in-band — it fits a turn-based tool loop natively, and it makes unrelated errors cheap: in DECIDE, agents fumbled suggest's flags too, recovered in a retry loop, and went 6/6.
  • --follow never exits, so every agent had to self-orchestrate (background + file + timeout). Set up once, first it works great (FOLLOW 5/6, 6/6 on deterministic gates). But composed with any other friction it fails compoundingly: Claude's FOLLOWDECIDE losses were suggest flag-guessing burning entire follow timeout windows, with the click landing in a file it wasn't watching. 1/3 vs --wait's 3/3 on the same task.
  • Removing --wait saves almost nothing anyway — it's a query param + timeout branch on the same endpoint follow uses.

Drive-by product fix: suggest's unknown-flag error now names the real chip forms (--items '<json>' / repeated --item id:label) — Claude guessed --id/--label/--type and a bare Unknown flag sent it into a guess-retry loop. Also refreshed inbox-watch's mechanism-1 example to the current content-inline nudge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants