Make agent actions visible in the web terminal - #575
Merged
Conversation
Activity frames were broadcast once over SSE and then lost. A client that reloaded or connected late had no way to see what the agent had done, and agent-driven panel changes produced no activity frame at all. Keep the last 50 accepted frames in an in-process ring and serve them newest-first from GET /api/agent-activity/recent, on the same loopback auth surface as the existing POST. Panel routes mirror agent-sourced focus, visibility and arrange broadcasts into the ring as synthetic tool frames, one frame per action. The kind vocabulary gains config and ui for the non-panel emitters.
An operator watching the terminal saw only a fraction of what the agent did. Nine tool families mutated state silently: logbook entries and publishes, plan writes and validations, queue stops, Phoebus drives, script runs with control-system writes, lattice edits, config patches, and window moves. Artifact saves were equally invisible. Emit one activity frame per successful mutation, placed after every refusal and failure return so a blocked action never reports itself. Withdrawn queue stops are distinguished from executed ones. Config frames carry key paths only, never values, since .mcp.json values can include credentials and the ring is persistent and GET-served. Artifact saves go through a store listener that hands frames to a worker thread, filtering notebook auto-saves so one execution cannot flood the ring. Unit runs used to leak real HTTP posts from these emit sites toward any live web terminal on the port. A directory-wide conftest guard now stubs both posters, with a marker escape hatch for the two suites that assert on the wire itself.
A user could not tell what the agent was doing to their workspace. Agent-driven panel switches looked identical to their own clicks, hides and shows left no trace, and the activity strip spoke raw tool names for a handful of tools and stayed silent for the rest. Agent focus and arrange now glow the affected tile body; users with reduced motion get a held static ring instead of the pulse. The strip words each action with a verb and the panel's display label, hides and shows included, and a click opens a history popover fed from the server ring so the record survives reload. Per-panel badges are acknowledged by server timestamp in localStorage, so seen activity stays cleared across reloads while unseen activity comes back. The rail scrolls an off-screen badge into view and tooltips the touch time. The session page connects its strip to the event stream it was booted for, and chat activity lines phrase tool names in plain language. Formatting and the history popover move to their own modules (activity-format.js, activity-history.js); the strip keeps the live line and delegates.
Sending a gallery artifact to the logbook notified panel focus with agent attribution. Every connected client had its workspace switched to ARIEL for one person's click, and the action rendered as agent activity when a human did it. The gallery now posts a same-origin navigate message to its host page, which activates ARIEL locally with no server broadcast. The host accepts only path URLs, rejecting javascript: and protocol-relative forms. A standalone gallery with no parent keeps the old behavior. The success card also builds its DOM directly instead of interpolating server values into innerHTML.
The activity surfaces had unit coverage but no end-to-end proof in a real browser, and several acceptance checks selected no tests at all. Eight Playwright tests pin the behaviors that only a live page can show: tile glow fires for agent focus and stays quiet for a human click on the same panel, hides land on the strip with a label and no glow, the history popover renders a burst as distinct rows and the same rows after reload, badge acknowledgments hold across reload while unseen activity restores, send-to-logbook navigates the sender without touching a second client, rejected navigation URLs are delivered and refused, and reduced motion holds a static ring. Negative assertions read document-start observer logs with a bounded settle after deferred paths, so a wrongly fired glow cannot slip between a barrier and its assertion.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A user sharing a workspace with the OSPREY agent could not tell what the agent was doing. Panel switches driven by the agent looked identical to the user's own clicks, most mutating tools (logbook writes, queue stops, Phoebus drives, lattice edits, config patches, script runs with control-system writes) produced no visible trace at all, and the activity record died with the SSE frame — a reload or late-joining client saw nothing.
What this does
GET /api/agent-activity/recent, and mirrors agent-sourced panel focus/visibility/arrange broadcasts into the ring as synthetic tool frames.javascript:and protocol-relative URLs rejected by the host.Notes for reviewers
python_executedeployment kill switch string-matchesexecution_mode == "readwrite"(case variants bypass it) andexecute_filehas no kill switch — the new emits make such runs visible, but the gate deserves its own PR; in container-split deployments an in-thread gallery can emit agent-attributed frames for human deletes (documented instartup.py, needs origin plumbing).