Skip to content

fix(locomo): v8 — twenty-word answers + first-step invalid-completion retry - #195

Merged
fazpu merged 1 commit into
mainfrom
fix/locomo-v8-answer-stage
Jul 31, 2026
Merged

fix(locomo): v8 — twenty-word answers + first-step invalid-completion retry#195
fazpu merged 1 commit into
mainfrom
fix/locomo-v8-answer-stage

Conversation

@fazpu

@fazpu fazpu commented Jul 31, 2026

Copy link
Copy Markdown
Member

What

Two answer-stage fixes measured on the conv-47 v7 re-score (91/150), shipped together as the v8 protocol identity:

  1. Answer-length rule: six → twenty words. The six-word cap caused 7 outright failures and made 19 misses structurally unwinnable (gold answers are enumerations longer than six words; the agent's compressed half-answers get judged wrong). New rule: shortest phrase that fully names the requested entities/values, at most twenty words, no explanations. Guard + message updated in the runner; tests pin 20 passes / 21 fails on both sides.
  2. First-step invalid-completion retry. Under v7's larger prompts (mean 22.8K tokens_in/question), luna emits non-JSON on the first agent step for ~15% of questions (23/150), which the fix(locomo): bounded retry on invalid reader completions + protocol-pinned reasoning effort #178 retry deliberately did not cover — those questions died with zero retries. A unified invalid_completion_attempts counter now gates the same bounded budget for first-step and reader-position failures alike; reader_attempts semantics unchanged (validator-compatible); new optional first_step_retries / total_first_step_retries fields (defaults keep old states parsing); provider outages remain non-retried.

Protocol: full-v8/full-v8-strong, adapter version bumped, tool catalog hash unchanged, locked fingerprints updated, v7 error strings migrated. Docs: design changelog, decisions.md D78 record, benchmarks corpus next-steps note.

Expected effect on conv-47: mechanical losses (30 items ≈ 20%) recovered → ceiling ≈ 115–120/150 vs 91 under v7. Rerun follows merge.

Reviews

  • Grok-4.5: APPROVE WITH NOTES — verified budget non-escape (first-step retries count against per-question, global, and cost budgets), reader-position regression intact, old-state parse compatibility, fingerprint/identity coherence incl. error strings; notes are stale secondary docs (runbook, overall_design index), non-blocking.

  • Claude (final review): verified unified retry gate, validator compatibility, 20-word guard both sides, ran benchmarks suite (77 passed) + ruff format --check.

  • I have read and agree to the RememberStack Contributor License Agreement v1.0.

Conv-47 scored 91/150 under v7: the six-word cap caused 7 invalid answers and constrained 19 longer gold answers, while 23/150 questions failed on first-step non-JSON completions. Raise the guarded answer limit to twenty words and extend the shared two-retry allowance to pre-tool invalid completions.
@fazpu
fazpu enabled auto-merge (squash) July 31, 2026 16:28
@github-actions

Copy link
Copy Markdown

Coverage report

This PR does not seem to contain any modification to coverable code.

@fazpu
fazpu merged commit 0ef5454 into main Jul 31, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant