fix(locomo): v8 — twenty-word answers + first-step invalid-completion retry - #195
Merged
Conversation
Conv-47 scored 91/150 under v7: the six-word cap caused 7 invalid answers and constrained 19 longer gold answers, while 23/150 questions failed on first-step non-JSON completions. Raise the guarded answer limit to twenty words and extend the shared two-retry allowance to pre-tool invalid completions.
fazpu
enabled auto-merge (squash)
July 31, 2026 16:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two answer-stage fixes measured on the conv-47 v7 re-score (91/150), shipped together as the v8 protocol identity:
invalid_completion_attemptscounter now gates the same bounded budget for first-step and reader-position failures alike;reader_attemptssemantics unchanged (validator-compatible); new optionalfirst_step_retries/total_first_step_retriesfields (defaults keep old states parsing); provider outages remain non-retried.Protocol:
full-v8/full-v8-strong, adapter version bumped, tool catalog hash unchanged, locked fingerprints updated, v7 error strings migrated. Docs: design changelog, decisions.md D78 record, benchmarks corpus next-steps note.Expected effect on conv-47: mechanical losses (30 items ≈ 20%) recovered → ceiling ≈ 115–120/150 vs 91 under v7. Rerun follows merge.
Reviews
Grok-4.5: APPROVE WITH NOTES — verified budget non-escape (first-step retries count against per-question, global, and cost budgets), reader-position regression intact, old-state parse compatibility, fingerprint/identity coherence incl. error strings; notes are stale secondary docs (runbook, overall_design index), non-blocking.
Claude (final review): verified unified retry gate, validator compatibility, 20-word guard both sides, ran benchmarks suite (77 passed) + ruff format --check.
I have read and agree to the RememberStack Contributor License Agreement v1.0.