fix(probes): return no attempts when the prompt set is empty - #2027
Open
feiiiiii5 wants to merge 1 commit into
Open
fix(probes): return no attempts when the prompt set is empty#2027feiiiiii5 wants to merge 1 commit into
feiiiiii5 wants to merge 1 commit into
Conversation
probe() indexed prompts[0] before checking the list was non-empty, so a probe whose prompt set was empty (e.g. emptied by translation) raised IndexError instead of producing no attempts. This mirrors the empty-set guard already present in IntentProbe.probe. Closes NVIDIA#2026 Signed-off-by: fei <204683769+feiiiiii5@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root Cause
Probe.probe()indexesprompts[0]without first checking that the prompt set is non-empty (garak/probes/base.py). A probe whose prompt set is empty — for example one emptied by the langprovider translation path (reproduced intests/probes/test_probes.py::test_probe_prompt_translation[probes.sysprompt_extraction.SystemPromptExtraction]in #2022) — raisesIndexError: list index out of rangeinstead of producing no attempts. The progress bar two lines earlier already tolerateslen(prompts) == 0; only the subscript does not.Fix
Return
[](with a warning naming the probe) when the copied prompt set is empty, before the progress bar and theprompts[0]check. This mirrors the empty-set guard already present inIntentProbe.probe().Test
test_probe_with_empty_prompts_returns_no_attempts: aProbewithprompts = []returns[]instead of raising.pytest tests/probes/test_probes.py::test_mint_attempt tests/probes/test_probes.py::test_probe_with_empty_prompts_returns_no_attempts→ 5 passed.Diff scope
2 files, +24/-1:
garak/probes/base.py,tests/probes/test_probes.py.AI Disclosure
AI-assisted implementation and regression test; human review of the empty-set semantics (no behavior change for non-empty probes; consistent with
IntentProbe) and final diff before submission.Closes #2026