Skip to content

Add anti-hallucination skill#10

Open
victorfelisbino wants to merge 1 commit into
groupzer0:mainfrom
victorfelisbino:skills/anti-hallucination
Open

Add anti-hallucination skill#10
victorfelisbino wants to merge 1 commit into
groupzer0:mainfrom
victorfelisbino:skills/anti-hallucination

Conversation

@victorfelisbino

Copy link
Copy Markdown

Adds a new skill at vs-code-agents/skills/anti-hallucination/SKILL.md following the existing single-file SKILL.md + YAML-frontmatter convention used by the other skills in this repo.

What the skill does

It biases the agent toward evidence and explicit uncertainty by refusing the four most common hallucination shapes in coding agents:

  1. Invented surface area — APIs, file paths, package names, config keys, env vars, CLI flags the agent has not actually seen
  2. "Should work" without proof — verdicts on ship / pass / safe without naming the check that produced them
  3. Admin-only proof — passing as an admin / owner / service account and claiming end-user behavior is also correct
  4. "No issues found" without running a check — clean-bill reviews on code or PRs the agent never opened

It also defines a four-rung verification ladder (source → repro → diff → outcome) and an opinionated response-hygiene section.

Why I think it fits this repo

The existing agents in this repo (Planner, Architect, Implementer, Critic, Code-Reviewer, Security, QA, UAT, DevOps) all have moments where wrong-but-confident is more expensive than slow-but-right. The skill includes a per-agent usage table at the bottom mapping each agent to the specific shape it should refuse — so it's not a generic "be careful" doc, it has a concrete hook for each role you already ship.

It also slots cleanly next to code-review-checklist and engineering-standards: those skills are about what to look for, this one is about what kind of answer the agent is allowed to send back.

Evidence / test harness

A 5-prompt before/after test harness lives in the upstream repo where the skill originated: https://github.com/victorfelisbino/my-agent-memory/blob/main/skills/general/anti-hallucination/test-prompts.md. Each prompt baits one of the four shapes and ships with a binary pass/fail rule. Pass bar: ≥3/5 prompts pass with the skill loaded and fail without it. A results template and a redacted example results file are in the same folder so reviewers can see the expected before/after shape without running it themselves.

I have not yet run the harness in this repo's exact agent + model setup — that's a known gap I'm open about. If maintainers want a recorded run before merge, I can do one against code-reviewer.agent.md or security.agent.md and post results in this PR.

Notes for reviewers

  • No scripts/, content-only — same shape as engineering-standards and analysis-methodology.
  • License: MIT, attribution to @victorfelisbino, with an upstream: link in the frontmatter pointing back to the source so updates can be tracked.
  • No behavioral changes to any existing agent .agent.md file — agents have to explicitly load the skill, same as the rest of the skills directory.
  • Related to my earlier issue AI memory / validation architecture — thought you might find this interesting #9 — this is the concrete probe rather than another conversation.

Happy to revise the framing, rename, restructure, or drop sections you find too prescriptive. If the right move is to start with an issue / discussion first instead of a PR, just say and I'll close this and convert.

Adds vs-code-agents/skills/anti-hallucination/SKILL.md following the existing skill convention (YAML frontmatter, MIT, single SKILL.md per folder).

The skill prevents the four most common hallucination shapes in coding agents: invented surface area, 'should work' without proof, admin-only proofs, and 'no issues found' without running a check. It also includes a per-agent-role usage table so Planner / Architect / Implementer / Critic / Code-Reviewer / Security / QA / UAT / DevOps each have a clear hook.

A 5-prompt before/after test harness with a results template and a redacted example is maintained in the upstream repo (my-agent-memory) and linked from the skill. The skill is content-only; no scripts/.
victorfelisbino added a commit to victorfelisbino/my-agent-memory that referenced this pull request May 26, 2026
- docs/roadmap.md: Wave 2 marked IN PROGRESS with done item linking to groupzer0/vs-code-agents#10
- docs/status.md: public probe moved into Real-today (PR open, awaiting review) and out of Planned
- skills/general/anti-hallucination/results-examples/example-redacted.md: redacted before/after example so reviewers can see the expected harness shape without running it themselves

The PR itself adds vs-code-agents/skills/anti-hallucination/SKILL.md upstream, matching their YAML + single-SKILL.md convention, with a per-agent usage table mapping Planner / Architect / Implementer / Critic / Code-Reviewer / Security / QA / UAT / DevOps each to the specific hallucination shape they should refuse.

Co-authored-by: Victor Felisbino <victor.felisbino@provisionsgroup.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant