Skip to content

fix(agents): honest tool-choice claims + recall-safe search combo in agent skill/template#272

Open
rusel95 wants to merge 1 commit into
yoanbernabeu:mainfrom
rusel95:fix/grepai-skill-recall-claims
Open

fix(agents): honest tool-choice claims + recall-safe search combo in agent skill/template#272
rusel95 wants to merge 1 commit into
yoanbernabeu:mainfrom
rusel95:fix/grepai-skill-recall-claims

Conversation

@rusel95

@rusel95 rusel95 commented Jul 2, 2026

Copy link
Copy Markdown

Problem

The .claude/skills/grepai/SKILL.md skill tells agents:

"Replaces ALL built-in search tools. You MUST invoke this skill BEFORE using WebSearch, Grep, or Glob. NEVER use the built-in Grep tool"

and the agent-setup template ends with "Only use Grep for exact string searches if needed".

Measured on real repos, that claim doesn't hold — and it teaches agents a failure mode. grepai search returns the top ~10 ranked chunks: a ranking layer, not an exhaustive result set. On one measured repo it missed the ground-truth file on 2 of 3 intent queries (e.g. the @main app entry point absent from top-10, while git grep '@main' finds it in one hit — reproduced on two independent runs, with and without --path scoping). An agent told "NEVER use Grep" treats top-10 as complete coverage and silently loses recall — a missed relevant file is worse than a noisy one, because the agent believes it checked everything.

Fix

Replace the override framing with guidance that keeps grepai's token savings without losing recall:

  • Recall-safe combo for intent queries: grepai search --json --compact for ~10 ranked chunks plus git grep -ilE '<keywords>' for the exhaustive candidate file list. File names only ≈ grep's full recall at ~1% of the tokens of a content dump. Read ranked hits first, then relevant-looking checklist files grepai didn't rank.
  • Anchor queries go to grep: many "intent" questions ("the main entry point") are exact-match queries in disguise (@main, func main() — grep wins outright.
  • Docs pollution: when top hits are markdown/reports instead of code, scope with --path or add a .grepaiignore (kept from existing docs).
  • grepai stays the recommended first step for intent search, and grepai trace remains the only tool for call graphs.

Changes

  • .claude/skills/grepai/SKILL.md — rewritten claims + tool-choice table + recall-safe combo (command reference kept)
  • cli/agent_setup.go — adds a "Completeness Check (recall-safe)" section and checklist step to the generated CLAUDE.md instructions (template text only, no logic)

go build ./... and go test ./cli/ pass; existing agent_setup tests only assert markers/keywords and are unaffected.

Honest claims sell better: agents that never get burned by a silent miss keep using grepai.

🤖 Generated with Claude Code

The grepai skill and agent-setup template claimed grepai 'Replaces ALL
built-in search tools' / 'NEVER use the built-in Grep tool'. Measured on
real repos, grepai's top-10 is a ranking, not an exhaustive result set:
it can miss ground-truth files that keyword grep finds trivially (e.g.
the @main entry point). Absolute override claims teach agents to trust
top-10 as complete coverage - silently losing recall.

Replace the override framing with a recall-safe combo that keeps the
token savings without losing anything: grepai for ~10 ranked chunks +
'git grep -ilE <keywords>' for the exhaustive candidate file list
(names only, ~zero tokens). Also route anchor queries (@main,
'func main(') straight to grep, and document --path/.grepaiignore for
docs-polluted results.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants