Skip to content

fix(orchestrator): allow an out-of-process deadline on the target agent - #58

Open
shehral wants to merge 1 commit into
hexo-ai:mainfrom
shehral:fix/target-agent-timeout
Open

fix(orchestrator): allow an out-of-process deadline on the target agent#58
shehral wants to merge 1 commit into
hexo-ai:mainfrom
shehral:fix/target-agent-timeout

Conversation

@shehral

@shehral shehral commented Aug 26, 2026

Copy link
Copy Markdown

The problem

SIA bounds every subprocess except the one that matters most:

SHELL_TIMEOUT:  int = 30
EVAL_TIMEOUT:   int = 600
DOCKER_TIMEOUT: int = 3600

But _stream_to_log, the single place the target agent is launched, ends with:

for line in process.stdout:
    ...
return process.wait()          # no timeout

A bare wait() blocks indefinitely. timeout= appears exactly once in orchestrator.py, on the evaluator. So a target agent that hangs on a network call stalls the whole run, not just its own generation.

Why the agent can't bound itself

This came up in practice. One generation wrote an agent that set three internal timeouts and still deadlocked, because all three lived inside the event loop that had stopped. A deadline enforced by the thing being timed is not a deadline. It has to be held outside the process.

The change

An optional SIA_TARGET_TIMEOUT (seconds).

  • Unset (default): behaviour is byte-for-byte unchanged. 129 passed, 1 skipped.
  • Set: a daemon timer kills the process at the deadline and writes the reason into the run log, so the generation is attributable afterwards rather than just missing.
  • Malformed value: ignored, no raise.

Roughly 25 lines, entirely inside _stream_to_log. The Popen call stays in this module's namespace, so it remains patchable in tests as the docstring promises.

Verification

scenario result
unset + fast process rc=0 in 0.0s, not killed
=3 + process sleeping 60s killed at 3.0s, rc=-9
=3 + fast process rc=0, not killed
="not-a-number" ignored, no crash
upstream suite, unset 129 passed, 1 skipped

The reason is written to the log, so a killed generation is distinguishable from a crashed one.

SIA bounds its evaluator (EVAL_TIMEOUT=600), its shell (SHELL_TIMEOUT=30) and
its containers (DOCKER_TIMEOUT=3600), but the one subprocess that makes network
calls is launched with a bare process.wait(), which blocks indefinitely. A
single hung generation therefore stalls the entire run rather than costing one
generation.

An agent cannot reliably bound itself here: one we watched set three internal
timeouts and still deadlocked, because all three lived inside the loop that had
stopped. The deadline has to be held outside the process.

Adds an optional SIA_TARGET_TIMEOUT (seconds). When unset -- the default --
behaviour is byte-for-byte unchanged and the existing suite passes (129 passed,
1 skipped). When set, a daemon timer kills the process at the deadline and
writes the reason to the run log so the generation is attributable afterwards.
A non-numeric value is ignored rather than raising.

Verified: inert when unset; kills a hung process at the deadline; does not
touch a process that finishes first; tolerates a malformed value.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant