fix(orchestrator): allow an out-of-process deadline on the target agent - #58
Open
shehral wants to merge 1 commit into
Open
fix(orchestrator): allow an out-of-process deadline on the target agent#58shehral wants to merge 1 commit into
shehral wants to merge 1 commit into
Conversation
SIA bounds its evaluator (EVAL_TIMEOUT=600), its shell (SHELL_TIMEOUT=30) and its containers (DOCKER_TIMEOUT=3600), but the one subprocess that makes network calls is launched with a bare process.wait(), which blocks indefinitely. A single hung generation therefore stalls the entire run rather than costing one generation. An agent cannot reliably bound itself here: one we watched set three internal timeouts and still deadlocked, because all three lived inside the loop that had stopped. The deadline has to be held outside the process. Adds an optional SIA_TARGET_TIMEOUT (seconds). When unset -- the default -- behaviour is byte-for-byte unchanged and the existing suite passes (129 passed, 1 skipped). When set, a daemon timer kills the process at the deadline and writes the reason to the run log so the generation is attributable afterwards. A non-numeric value is ignored rather than raising. Verified: inert when unset; kills a hung process at the deadline; does not touch a process that finishes first; tolerates a malformed value.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
SIA bounds every subprocess except the one that matters most:
But
_stream_to_log, the single place the target agent is launched, ends with:A bare
wait()blocks indefinitely.timeout=appears exactly once inorchestrator.py, on the evaluator. So a target agent that hangs on a network call stalls the whole run, not just its own generation.Why the agent can't bound itself
This came up in practice. One generation wrote an agent that set three internal timeouts and still deadlocked, because all three lived inside the event loop that had stopped. A deadline enforced by the thing being timed is not a deadline. It has to be held outside the process.
The change
An optional
SIA_TARGET_TIMEOUT(seconds).129 passed, 1 skipped.Roughly 25 lines, entirely inside
_stream_to_log. ThePopencall stays in this module's namespace, so it remains patchable in tests as the docstring promises.Verification
rc=0in 0.0s, not killed=3+ process sleeping 60src=-9=3+ fast processrc=0, not killed="not-a-number"The reason is written to the log, so a killed generation is distinguishable from a crashed one.