fix(agents): honest tool-choice claims + recall-safe search combo in agent skill/template#272
Open
rusel95 wants to merge 1 commit into
Open
fix(agents): honest tool-choice claims + recall-safe search combo in agent skill/template#272rusel95 wants to merge 1 commit into
rusel95 wants to merge 1 commit into
Conversation
The grepai skill and agent-setup template claimed grepai 'Replaces ALL built-in search tools' / 'NEVER use the built-in Grep tool'. Measured on real repos, grepai's top-10 is a ranking, not an exhaustive result set: it can miss ground-truth files that keyword grep finds trivially (e.g. the @main entry point). Absolute override claims teach agents to trust top-10 as complete coverage - silently losing recall. Replace the override framing with a recall-safe combo that keeps the token savings without losing anything: grepai for ~10 ranked chunks + 'git grep -ilE <keywords>' for the exhaustive candidate file list (names only, ~zero tokens). Also route anchor queries (@main, 'func main(') straight to grep, and document --path/.grepaiignore for docs-polluted results. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
.claude/skills/grepai/SKILL.mdskill tells agents:and the
agent-setuptemplate ends with "Only use Grep for exact string searches if needed".Measured on real repos, that claim doesn't hold — and it teaches agents a failure mode.
grepai searchreturns the top ~10 ranked chunks: a ranking layer, not an exhaustive result set. On one measured repo it missed the ground-truth file on 2 of 3 intent queries (e.g. the@mainapp entry point absent from top-10, whilegit grep '@main'finds it in one hit — reproduced on two independent runs, with and without--pathscoping). An agent told "NEVER use Grep" treats top-10 as complete coverage and silently loses recall — a missed relevant file is worse than a noisy one, because the agent believes it checked everything.Fix
Replace the override framing with guidance that keeps grepai's token savings without losing recall:
grepai search --json --compactfor ~10 ranked chunks plusgit grep -ilE '<keywords>'for the exhaustive candidate file list. File names only ≈ grep's full recall at ~1% of the tokens of a content dump. Read ranked hits first, then relevant-looking checklist files grepai didn't rank.@main,func main() — grep wins outright.--pathor add a.grepaiignore(kept from existing docs).grepai traceremains the only tool for call graphs.Changes
.claude/skills/grepai/SKILL.md— rewritten claims + tool-choice table + recall-safe combo (command reference kept)cli/agent_setup.go— adds a "Completeness Check (recall-safe)" section and checklist step to the generated CLAUDE.md instructions (template text only, no logic)go build ./...andgo test ./cli/pass; existingagent_setuptests only assert markers/keywords and are unaffected.Honest claims sell better: agents that never get burned by a silent miss keep using grepai.
🤖 Generated with Claude Code