Skip to content

fix(worker): one-shot run exits 1 on success — invert to exit 0 - #169

Open
RealDiligent wants to merge 1 commit into
mini-router:mainfrom
RealDiligent:fix/critical-issue-worker-once-exit-code
Open

fix(worker): one-shot run exits 1 on success — invert to exit 0#169
RealDiligent wants to merge 1 commit into
mini-router:mainfrom
RealDiligent:fix/critical-issue-worker-once-exit-code

Conversation

@RealDiligent

Copy link
Copy Markdown
Contributor

Problem

validator/src/eval_backend/worker.py main() one-shot branch:

if not args.loop:
    raise SystemExit(process_once(session_factory, settings))

process_once returns 1 when it processed a job and 0 when the queue was empty — a "did work" signal for the --loop sleep decision, not a process exit status (the --loop branch uses it as if processed == 0: sleep(...)). Passing it to SystemExit inverts exit semantics: a run that successfully processes a job exits 1 (which supervisors/cron/systemd oneshot/CI read as failure), and an empty queue exits 0.

Impact

Any one-shot invocation (python -m eval_backend.worker per timer tick, instead of --loop) reports every successful evaluation as a failed process — spurious alerts/retries or a unit stuck failed.

Fix

-    if not args.loop:
-        raise SystemExit(process_once(session_factory, settings))
+    if not args.loop:
+        process_once(session_factory, settings)
+        raise SystemExit(0)

A completed one-shot pass exits 0 whether or not the queue had work; genuine failures already raise (the except Exception: ... raise inside process_once) and still exit nonzero.

Tests

New validator/tests/test_worker_exit.py stubs the DB/engine bootstrap and asserts: one-shot exits 0 for both process_once returns (1 = did work, 0 = idle), and a raising process_once still propagates (nonzero). Verified the process_once==1 case exits 1 on the old code and 0 with the fix.

Risk / tradeoffs

Minimal. --loop behavior is unchanged. Only the one-shot exit code changes, from an inverted value to correct success/failure semantics.

Closes #168

@github-actions github-actions Bot added eval Evaluation changes validator Validator backend changes labels Jul 13, 2026
@RealDiligent

Copy link
Copy Markdown
Contributor Author

Included the same stale-test fix as noted in #160 (the pre-existing red on main's test_wrong_theta_length_fails) so this PR's test-router passes; the substantive change is the validator worker exit code. Happy to drop that commit once #160/#161 lands.

@RealDiligent
RealDiligent force-pushed the fix/critical-issue-worker-once-exit-code branch from 70c5843 to fe64f2c Compare July 20, 2026 12:05
@RealDiligent

Copy link
Copy Markdown
Contributor Author

Rebased onto current main; dropped the obsolete validate-submission test commit (already on main). Substantive change is still the one-shot worker exit code. Watching CI.

@RealDiligent

Copy link
Copy Markdown
Contributor Author

Status check: CI is green, PR is MERGEABLE, no auto-eval/close on this infra PR. Ready for maintainer merge.

@RealDiligent

Copy link
Copy Markdown
Contributor Author

Rebased onto current main. Watching CI.

@RealDiligent

Copy link
Copy Markdown
Contributor Author

CI green + MERGEABLE after rebase. Ready for maintainer merge.

The non --loop branch did `raise SystemExit(process_once(...))`. process_once
returns 1 when it handled a job and 0 when the queue was empty — a "did work"
signal for the --loop sleep decision, not an exit status. So a one-shot run
that successfully processed a job exited 1, which supervisors/cron/systemd read
as failure (and an empty queue looked like success).

Run the pass and exit 0 on normal completion; genuine failures still raise and
exit nonzero. Added tests covering both process_once returns and the failure
path.

Closes mini-router#168
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

eval Evaluation changes validator Validator backend changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

worker: one-shot (non --loop) run exits 1 on success, so supervisors see every processed job as a failure

1 participant