Skip to content

feat(cas): complete GrandmaIntent coverage - #2024

Open
3nesdeniz wants to merge 1 commit into
NVIDIA:mainfrom
3nesdeniz:agent/grandma-intent-coverage
Open

feat(cas): complete GrandmaIntent coverage#2024
3nesdeniz wants to merge 1 commit into
NVIDIA:mainfrom
3nesdeniz:agent/grandma-intent-coverage

Conversation

@3nesdeniz

Copy link
Copy Markdown

Summary

  • activate GrandmaIntent at the OF_CONCERN tier
  • extract the existing Grandma probe subjects into S005hate and S006items intent stubs while retaining the existing S003productkeys source
  • add the missing S006items detector mapping and correct the adjacent S003 stub metadata and substance spelling
  • add focused regression coverage for legacy subject preservation, intent metadata, prompt conversion, and detector parity

Fixes #1887.

Why this is not a duplicate

As of 2026-08-06, the required duplicate checks found no open PR referencing #1887 and no open PR matching GrandmaIntent or S006items. This implements the scope Patricia clarified in the issue rather than duplicating another branch or PR.

Verification

  • python -m pytest -q tests/probes/test_probes_grandma.py tests/cas/ — 252 passed, 1 skipped
  • python -m pytest -q tests/probes/test_probes.py -k grandma — 35 passed, 1,321 deselected
  • black --check garak/probes/grandma.py garak/intents/S003.py tests/probes/test_probes_grandma.py
  • python -m json.tool garak/data/cas/intent_detectors.json
  • git diff --check
  • direct integration check with only S003productkeys,S005hate,S006items: active at OF_CONCERN, 90 aligned prompts, 30 per intent, and all three detector mappings resolved

A full local suite attempt reached 4,453 passed and 110 skipped. Its two observed failures were unrelated to this change: the optional soundfile/librosa packages were absent for AudioAchillesHeel. The run was then interrupted after an unrelated Hugging Face Hub download stalled for more than 20 minutes.

Scope notes

This does not modify IntentProbe base behaviour, remove the legacy Grandma probe classes, or hand-edit plugin_cache.json; the repository's cache-maintenance workflow handles that file after merge.

AI assistance

OpenAI Codex was used for repository navigation, implementation support, and test drafting. The commit includes an AI attribution trailer. This PR is intentionally opened as a draft so the human submitter can review every changed line and the verification evidence before marking it ready.

Extract the legacy Grandma probe subjects into intent stubs, connect S006items to its detector, and enable GrandmaIntent at the OF_CONCERN tier. Add focused regression coverage for stub intent metadata, subject preservation, and detector parity.

Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Enes Deniz <142517728+3nesdeniz@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings August 6, 2026 09:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR completes CAS coverage for the “Grandma” probe family by promoting GrandmaIntent to an active OF_CONCERN intent-driven probe, extracting legacy Grandma subjects into CAS intent stubs, ensuring detector mappings are present, and adding regression tests to keep intent metadata, stub content, and detector parity aligned.

Changes:

  • Activate GrandmaIntent at Tier.OF_CONCERN and set it active = True; fix a substance spelling in legacy probe params.
  • Add CAS intent stub text files for S005hate and S006items, and correct the S003 product keys stub intent code.
  • Add a missing S006items detector mapping and introduce focused tests for stub preservation, prompt conversion, and detector parity.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file
File Description
tests/probes/test_probes_grandma.py Adds regression tests ensuring legacy subjects are preserved via stubs, detectors resolve as expected, and prompt conversion retains stub subject text.
garak/probes/grandma.py Corrects substance spelling and activates GrandmaIntent at OF_CONCERN.
garak/intents/S003.py Fixes S003productkeys stub intent code to match the intended leaf name.
garak/data/cas/intent_stubs/S006items.txt Introduces intent stubs representing legacy Substances coverage.
garak/data/cas/intent_stubs/S005hate.txt Introduces intent stubs representing legacy Slurs coverage.
garak/data/cas/intent_detectors.json Adds detector mapping for S006items to ensure detector resolution parity with legacy probes.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@patriciapampanelli

Copy link
Copy Markdown
Collaborator

@3nesdeniz Thanks for working on this. Holding off on review since this is still a draft. Mark it ready when you're done and we'll take a look.

@3nesdeniz
3nesdeniz marked this pull request as ready for review August 7, 2026 20:12
@3nesdeniz

Copy link
Copy Markdown
Author

Thanks, Patricia. The implementation and focused verification are complete, and the branch is current with main. I've marked the PR ready for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve GrandmaIntent coverage to match the existing probes in use.

3 participants