Recipe ergonomics v4: hydrated hybrid RRF, tool guidance, agent loop guards - #152
Merged
Conversation
…guards Issue #149: claims_hybrid_rrf now ends with hydrate_claims so the envelope carries claim text plus RRF ranking scores (not bare UUIDs). Descriptor when-to-use sentences steer tool choice; recipe versions bumped for ON CONFLICT DO NOTHING rollout. Answer-agent prompt forbids identical tool+arguments retries and requires a claims search before Unknown. Protocol RS-LoCoMo-Full-v3 → v4 with recomputed tool catalog hash.
…x findings) Fable's final pass over the Grok implementation, resolving all six Codex findings. Major 1: the recipe's answer now carries the WHOLE chain's hydration drops. The executor returned only the final step's envelope, so stale nominations dropped by the search passes feeding fuse -> hydrate vanished from the caller-visible denominator -- Codex reproduced two upstream drops reported as zero. Accumulation lives in a pure _chain_answer helper with a direct test. Major 2: the new tool-choice descriptions advertised channels that do not exist -- "phrase-anchored" for a purely semantic search, "semantic+lexical" for a chain that runs the same semantic search twice. Both descriptions now state exactly what runs. The catalog pin is recomputed accordingly (f0492ced...). A real lexical channel remains future work. Minor 3: hydrate_claims now lints at exactly one input (new max_inputs cap in the linter) and the executor refuses ambiguous inputs instead of silently consuming the first -- a lint-clean chain runs exactly as written. Minor 4: the loop guard no longer forbids legitimate query refinement -- "change the arguments meaningfully or switch tools" replaces the hard switch instruction; the identical tool+arguments prohibition already prevents loops. Minor 5: the two current v3 references in locomo_benchmark_analysis.md now say v4. Minor 6: the hydration tests asserted set-equality on pre-ordered zero-drop mocks; the new tests exercise drop accumulation across a chain and the two-input lint rejection. Grok's guidance test now asserts the honest wording. Validation: pyright 0 errors, 614 passed, 311 skipped, ruff clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GKENhTLJg1HqhbdwCmmkbc
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #149. Protocol
RS-LoCoMo-Full-v3→RS-LoCoMo-Full-v4.Why
Live benchmark traces (2/8 run, 2026-07-27): the commodity agent called
identity_as_offive times with identical arguments and died; calledclaims_verbatimeight times and exhausted its budget; never once choseclaims_hybrid_rrf— whose envelope carried only claim UUIDs with no text; answered a WHEN-question without tryingentity_timeline. Five of six remaining failures were agent-policy against an unhelpful tool surface.What
claims_hybrid_rrfis now usable: newhydrate_claimsprimitive (query engine + executor dispatch + linter registration) appended to its chain — the envelope carries hydrated evidence text plus the RRF ranking. Same_confirm_claimscurrency path asclaims_verbatim.dropped_by_hydration, not just the final step's (upstream drops were silently discarded with the intermediate envelopes).ORDER BY version DESCrollout verified). Catalog pin recomputed twice — after Grok's changes and again after the honesty edits — finalf0492ced…, derived, never guessed.Process
Implemented by Grok from a written brief → Codex review (2 major, 4 minor — all reproduced with evidence, including recomputing the catalog hash independently) → Fable final review resolving all six findings with tests (drop accumulation across a chain; two-input hydrate lint rejection; honest-wording guidance assertions).
Validation
pyright 0 errors; 614 passed, 311 skipped; ruff clean. Break-then-restore proof on the hydration behaviour test.
Contributor agreement