You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Follow-up to #49 (engagement detection). #49's fix opens the eligibility gate -- sessions whose .evo/ was created mid-run now correctly flip has_evo_engaged: true and pass the fanout filter. But the delivery half of the directive system is broken: queued directives never reach the agent's context as [EVO DIRECTIVE id=...] banners.
Repro
In a real claude --print PostTrainBench run on alpha.7 (the run that validated #49 end-to-end):
grep -oE "\[EVO DIRECTIVE id=[A-F0-9]+\]" /home/ubuntu/ptb-run.log
-> no matches
Agent never invoked evo ack <event_id>. Agent's prose continued the original bypass plan it was on before the directive fired -- proving the directive content never entered its context.
Diagnosis
Two halves of the directive system:
Half
Status
Queue: evo direct enqueues into events/workspace.jsonl + filters eligible sessions
Delivery: hook-drain reads events on each hook fire, splices [EVO DIRECTIVE id=...] banner into hook response, moves event to delivered/
broken -- delivery never fires
PostToolUse hooks fire constantly during the run (every Bash, Read, Write tool call). Each fire SHOULD trigger drain.py to:
Scan inject/events/workspace.jsonl for unread directives addressed to this session
Splice each as an [EVO DIRECTIVE id=<event_id>] ... [END EVO DIRECTIVE] banner into the hook response's additionalContext (or equivalent host-specific mechanism)
Write the event id to inject/delivered/<event_id>-<sid>.json (or similar marker) so it's not re-delivered
The delivered/ dir stays empty for the entire run -- the splice step never happens.
Suspected source area
plugins/evo/src/evo/drain.py (the Python drain hand-off from the Rust hook-drain binary). The Rust binary at bin/evo-hook-drain-rs/src/main.rs correctly hands off via handoff_to_drain(...) when should_handoff returns true. But the Python side either:
Isn't being invoked (the Rust handoff fails silently)
IS invoked but doesn't scan workspace-broadcast events (only per-session/per-exp targeted directives)
Scans events but the banner-splicing code path is broken for the additionalContext envelope
The session record has has_evo_engaged: true, so the eligibility check passes if it's even being evaluated. The fact that NOTHING ever lands in delivered/ suggests the scan isn't running at all -- not just a filter mismatch.
Why this matters
The entire evo direct mid-run injection mechanism is non-functional. Use cases broken:
Mid-run course correction (user observes the agent making a wrong call, fires a directive to redirect)
Operator interventions during long autonomous runs
Cross-agent / dashboard-driven nudges
The verifier subagent's [EVO DIRECTIVE ...] pre-experiment annotations (if those route through the same channel)
In our PostTrainBench run, the agent was reasoning itself out of using /evo:optimize ("instead of going through the full evo:optimize ceremony, I'll directly drive experiments"). The directive content was crafted specifically to course-correct this. Queue confirmed delivery (fanout=1). Agent never saw it. Run continued the bypass.
Bonus minor bug surfaced during diagnosis
evo direct-status <id> fails with git rev-parse --show-toplevel returned non-zero exit status 128 when invoked outside the workspace directory. Other evo subcommands (evo direct, evo status) work fine from arbitrary cwd. direct-status should be workspace-agnostic the same way -- or at least error with a clear "must be in a workspace" message instead of leaking the git internal failure.
Out of scope
Re-design of the additionalContext envelope shape
Per-host delivery (claude-code / codex / opencode / etc. -- focus on claude-code first since the repro is there)
Follow-up to #49 (engagement detection). #49's fix opens the eligibility gate -- sessions whose
.evo/was created mid-run now correctly fliphas_evo_engaged: trueand pass the fanout filter. But the delivery half of the directive system is broken: queued directives never reach the agent's context as[EVO DIRECTIVE id=...]banners.Repro
In a real
claude --printPostTrainBench run on alpha.7 (the run that validated #49 end-to-end):Agent runs
evo init,evo new,evo runbaseline normally. Session record at.evo/run_0000/inject/sessions/<sid>.jsonlandshas_evo_engaged: true(the hook-drain: session not registered when.evo/is created mid-session underclaude --print#49 fix works -- confirmed via on-disk check).From a separate shell on the same host, fire a directive:
Output:
Fanout=1 confirms the session was eligible -- engagement check passed.
Wait 3 minutes. Check
.evo/run_0000/inject/state:Grep the agent's stream-json log for the banner:
Agent never invoked
evo ack <event_id>. Agent's prose continued the original bypass plan it was on before the directive fired -- proving the directive content never entered its context.Diagnosis
Two halves of the directive system:
evo directenqueues intoevents/workspace.jsonl+ filters eligible sessions[EVO DIRECTIVE id=...]banner into hook response, moves event todelivered/PostToolUse hooks fire constantly during the run (every Bash, Read, Write tool call). Each fire SHOULD trigger drain.py to:
inject/events/workspace.jsonlfor unread directives addressed to this session[EVO DIRECTIVE id=<event_id>] ... [END EVO DIRECTIVE]banner into the hook response'sadditionalContext(or equivalent host-specific mechanism)inject/delivered/<event_id>-<sid>.json(or similar marker) so it's not re-deliveredThe
delivered/dir stays empty for the entire run -- the splice step never happens.Suspected source area
plugins/evo/src/evo/drain.py(the Python drain hand-off from the Rust hook-drain binary). The Rust binary atbin/evo-hook-drain-rs/src/main.rscorrectly hands off viahandoff_to_drain(...)whenshould_handoffreturns true. But the Python side either:The session record has
has_evo_engaged: true, so the eligibility check passes if it's even being evaluated. The fact that NOTHING ever lands indelivered/suggests the scan isn't running at all -- not just a filter mismatch.Why this matters
The entire
evo directmid-run injection mechanism is non-functional. Use cases broken:[EVO DIRECTIVE ...]pre-experiment annotations (if those route through the same channel)In our PostTrainBench run, the agent was reasoning itself out of using
/evo:optimize("instead of going through the full evo:optimize ceremony, I'll directly drive experiments"). The directive content was crafted specifically to course-correct this. Queue confirmed delivery (fanout=1). Agent never saw it. Run continued the bypass.Bonus minor bug surfaced during diagnosis
evo direct-status <id>fails withgit rev-parse --show-toplevel returned non-zero exit status 128when invoked outside the workspace directory. Otherevosubcommands (evo direct,evo status) work fine from arbitrary cwd.direct-statusshould be workspace-agnostic the same way -- or at least error with a clear "must be in a workspace" message instead of leaking the git internal failure.Out of scope