fix(locomo): bounded retry on invalid reader completions + protocol-pinned reasoning effort - #178
Merged
Merged
Conversation
fazpu
enabled auto-merge (squash)
July 29, 2026 17:07
Coverage reportClick to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two answer-stage robustness fixes, both protocol-identity-relevant, driven by measured v5-strong smoke evidence (2026-07-29):
Bounded reader retry. The reader step intermittently receives non-JSON completions from
openai/gpt-5.6-luna(3/8 questions in one pass, 1/8 in another), permanently zeroing those questions. Now: up to 2 retries (3 attempts total), only for invalid-completion errors (ProviderInvalidResponseError) after at least one tool call. Every attempt counts against the per-question call budget, the global call budget, and the cost ceiling. Provider outages (transport/HTTP errors) are not retried — the exception hierarchy separates them. Per-itemreader_attemptsand summarytotal_reader_retriesrecord the mechanism's activity.Reasoning effort pinned in the protocol. Pinning luna to effort
nonevia ambientREMEMBERSTACK_OPENROUTER_REASONING_EFFORT_MAPraised the smoke 1/8 → 3/8 — but ambient env is invisible to the run pin, violating reproducibility. Nowanswer_agent_reasoning_effortlives on the protocol dataclass (nonefor full-v5-strong,None/absent for full-v5) and is applied per-request via a newModelRequest.reasoning_effortfield with explicit-set-wins semantics. Engine seats never set the field, so worker behavior under the env map is byte-identical.Both fields join the protocol fingerprint (prepare + validate + registry pin check). Fingerprints roll for both entries; all prior runs were smoke-tier diagnostics, nothing published is invalidated. Names and CLI keys unchanged;
--protocolremains prepare-only.Evidence
Smoke v5s-glm52 (pre-pin): 1/8,
answer_readerfailures 3. Smoke v5s-glm52-b (env-pinned): 3/8, failures 1. This PR makes the winning configuration reproducible from the run pin alone and stops the residual ~12% reader-failure bleed before the full benchmark.Reviews
Grok-4.5 adversarial review: APPROVE WITH NOTES — verified retry boundedness, single-count budget accounting, exception MRO (outages not retried),
model_fields_setsentinel semantics, engine call sites unchanged, fingerprint coverage on both paths. Notes are test-hardening suggestions (fingerprint identity baseline assert, explicit outage-non-retry test, cost-ceiling mid-retry test), non-blocking.Claude (final review): verified budget interactions, MRO, fingerprint derivation in prepare and validate, ran the affected suites (93 + 7 = 100 passed).
I have read and agree to the RememberStack Contributor License Agreement v1.0.