Skip to content

fix: stop FileDetector format mismatch from aborting the run - #2029

Open
patriciapampanelli wants to merge 1 commit into
NVIDIA:mainfrom
patriciapampanelli:fix/detector-format-mismatch-degrades
Open

fix: stop FileDetector format mismatch from aborting the run#2029
patriciapampanelli wants to merge 1 commit into
NVIDIA:mainfrom
patriciapampanelli:fix/detector-format-mismatch-degrades

Conversation

@patriciapampanelli

Copy link
Copy Markdown
Collaborator

Summary

Found while reviewing #1952, which adds an S008execfileformats.FileIsExecutable intent mapping. With it applied, --spec probes.grandma.GrandmaIntent,intent:S008exec yields 256 attempts and zero evaluations.

FileDetector.detect() raises ValueError when attempt.notes["format"] does not match valid_format, and the harness calls detectors with no try/except, so the run aborts: queued probes never execute, no eval or completion records are written, and cli.main exits 0. The mapping only exposes this.

The raise becomes a warning and one None per output, not 0.0, since the outputs are not filenames at all. The evaluator already counts None as nones, and StringDetector and HFDetector.graceful_fail use the same idiom.

Reproduction

python -m garak --target_type test.Repeat --spec probes.test.Blank,probes.test.Test \
  -d fileformats.FileIsPickled --generations 1 --report_prefix filedetector-guard-before

Before: probes.test.Test never runs; the report holds 1 attempt, 0 eval, 0 completion.
After: both probes run, 9 attempts, 2 eval, 1 completion.

Both exit 0 — the exit code never signalled the abort.

…borting the run

Signed-off-by: Patricia Pampanelli <ppampanelli@nvidia.com>
@patriciapampanelli patriciapampanelli changed the title degrade FileDetector format mismatch to unscored attempt instead of a… fix: stop FileDetector format mismatch from aborting the run Aug 7, 2026
@patriciapampanelli patriciapampanelli added the bug Something isn't working label Aug 7, 2026

@jmartin-tech jmartin-tech left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This make sense to me as a behavior change since the introduction of support for None responses from detectors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants