Skip to content

fix: a paused task says what stopped it and how to fix it; an unavailable connector hands the call to the next one - #57

Merged
pegner-ag merged 1 commit into
mainfrom
fix/paused-notice-failover
Oct 19, 2026
Merged

pegner-ag merged 1 commit into
mainfrom
fix/paused-notice-failover

Conversation

@pegner-ag

Copy link
Copy Markdown
Owner

Found in production on 2026-10-19: the Antigravity connector refused every call because its hooks were not approved. The reception fell back to tasks, and those tasks paused and retried. Nothing in the chat said why.

  • Chat bridge: a provider failure that pauses a task (a failed call, a paused acceptance check, a Gate stopped before deciding) is posted once per pause. The post says what stopped the task, gives the reason as the server reports it, says where to fix an unavailable connector (the Connectors page or ooat connectors), and says that OOAT tries again by itself.
  • Reception: when the reception cannot decide, its reply names the provider failure.
  • Task summary: a new paused_by field, filled only within the reader's cap and shown in the inbox's stuck list.
  • Gateway: a connector that cannot start the call (UNAVAILABLE, QUOTA_EXHAUSTED) hands it to the next candidate in price order. A pin is never bypassed. A failure that may have reached the provider (API_ERROR, TIMEOUT) is never tried elsewhere. When no candidate can start the call, the error names each one.

Tests: 1477 passed, 7 skipped; the web type check is clean and the node tests pass 35/35. The quota test now expects that the same call moves on (until now, only the next call did).

🤖 Generated with Claude Code

…, an unavailable connector hands the call to the next one

- Chat bridge: a provider failure that pauses a task (a failed call, a paused acceptance check, a Gate stopped
  before deciding) is posted once per pause: what stopped it, the reason, where to fix an unavailable connector,
  and that OOAT tries again by itself.
- Reception: when it cannot decide, its reply names the provider failure.
- Task summary: `paused_by` (within the reader's cap), shown in the inbox's stuck list.
- Gateway: a connector that cannot start the call (UNAVAILABLE, QUOTA_EXHAUSTED) hands it to the next candidate;
  a pin is never bypassed, and a failure that may have reached the provider is never tried elsewhere.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
# When no decision connector can answer, the cheapest text tier answers the same typed questions (spec §6).
FALLBACK_TIER = "economy"
FALLBACK_TIMEOUT_S = 120
NEVER_RAN = ("UNAVAILABLE", "QUOTA_EXHAUSTED") # the call did not reach the provider: the next connector may serve it

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assumption doesn't hold for the agentic CLI adapters. Several adapters raise QUOTA_EXHAUSTED/UNAVAILABLE after the run, based on the CLI's final status or text:

  • adapters/antigravity-cli _failure() / parse_stream() map a non-SUCCESS result whose text matches rate limit|quota|… or log in|401 to these codes;
  • adapters/codex _failure() and adapters/claude-code parse_result() do the same.

An agent that edits files and then hits its usage limit partway through therefore reports QUOTA_EXHAUSTED. With this change the next connector runs on a project folder that is already partly changed. It also runs on tokens that were spent but recorded as ran=False (cost 0 in _invoke). This contradicts the docstring's own rule: "any other failure may have reached the provider, or changed files, so it is raised".

Suggested fix: skip failover when request.workspace is not None. Or better, let a connector mark a failure as "not started", e.g. a ConnectorError(..., started=False) raised only from pre-flight checks such as a missing binary, unapproved hooks or a missing secret, and fail over only on that. Either way, please add a test with a workspace request.

it is raised. When none could start, the cheapest one's code is raised, naming each failure."""
failed = []
for candidate in candidates:
self._check_budget(request, candidate.estimate)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_check_budget now runs for each candidate in turn. Suppose the cheapest connector is UNAVAILABLE and the next one costs more than the remaining budget. The call then raises BUDGET, so the runtime asks a budget question (runtime.py:385) instead of pausing on a provider outage. The UNAVAILABLE reason collected in failed is lost, and the person is asked to raise the budget when the actual problem is a connector to fix. Consider stopping at a candidate over budget and raising the provider failures collected so far, or adding them to the BUDGET message. A test for this case would help.

["⏸ The task is stopped: the provider returned an error.", "Reason: HTTP 529"]]


def test_a_pause_above_the_cap_is_told_without_its_reason():

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the test name says the pause "is told without its reason", but the test asserts that nothing is posted (chat.sent == []). Something like test_a_redacted_event_is_not_told_as_a_pause would match the behaviour.

@claude

claude Bot commented Oct 19, 2026

Copy link
Copy Markdown

Verdict: changes suggested

The chat notice, the reception's reason and paused_by look good. The gateway failover needs one fix before merge.

  1. Failover can run a second agent on a partly changed project folder. NEVER_RAN assumes that UNAVAILABLE/QUOTA_EXHAUSTED mean the call never started. The antigravity-cli, codex and claude-code adapters also raise these codes after a run, from its final status text, e.g. a usage limit hit partway through an edit. The next connector then works on already-changed files, and the spent tokens are recorded at cost 0. Suggested fix: no failover for workspace requests, or an explicit "not started" flag set only by pre-flight checks. Details are in the inline comment.
  2. The budget is checked per candidate. If the cheapest connector is unavailable and the next one is over budget, the result is a BUDGET question instead of a provider pause, and the outage reason is lost.
  3. Minor: one bridge test's name doesn't match what it asserts.

Checked and fine:

  • Pins are never bypassed.
  • Every candidate has already passed _exclusion, so data class, acknowledgement and hook rules still apply in the gateway.
  • paused_by is gated by the reader's cap (visible), and redacted events are not posted in the chat.
  • resume_after and last_failure share provider_failure(), so they stay consistent.
  • The docs (docs/chat-bridge.md, docs/description.md, core/description.md) are updated together with the code.
  • The ledger is untouched, and no R3 path, secret or personal data is involved.

I did not run the tests in this review. The PR reports 1477 passed and 7 skipped.

@pegner-ag
pegner-ag merged commit cc9e932 into main Oct 19, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant