Fix: unreachable Conversation branch in Probe.probe() pre-translation notes

[adityasingh2400]

Probe.probe() records notes["pre_translation_prompt"] so that Attempt.prompt_for() can hand back the original-language prompt after forward translation. The third isinstance check repeated the Message condition instead of testing for Conversation, so a probe with Conversation prompts left notes as None and prompt_for() returned the translated prompt even when asked for the probe language. That is what detectors.misleading and the other prompt_for(self.lang_spec) consumers read, so with a langprovider configured they were scoring against the wrong text. The branch body also used turn.context, an attribute Turn has never had, so it would have raised AttributeError if it had ever been reachable.

TreeSearchProbe._create_attempt further down the same file already has the intended form of this branch, so this change brings Probe.probe() back in line with it. The duplicate condition arrived in e2d3667. Thanks to @feiiiiii5 for the diagnosis in the issue.

Closes #2020

Why this is not a duplicate

• gh pr list --repo nvidia/garak --state open --search "2020 in:body" returns nothing related.
• A PR search for pre_translation_prompt returns only closed PRs (improve prompt translation support for Conversation related types #1441, fix: encoding probes storing translated text in pre_translation_prompt #1483, probe: Base class for iterative probes and FITD as an example #1414, Configurable system prompt #1337, Feature: multilingual machine translation #943), none of which touch this branch.
• A search of open PRs for Conversation plus translation returns only feat: Add resume scan functionality to continue interrupted scans #1531 (resume scan), which does not touch probes/base.py.

Tests run

Python 3.12, macOS, CPU only.

$ python -m pytest tests/langservice/probes/test_probes_base.py -k "pre_translation_conversation" -q
1 passed, 188 deselected in 18.23s

$ python -m pytest tests/langservice/probes/test_probes_base.py -k "base_" -q
6 passed, 183 deselected in 3.47s

$ python -m pytest tests/langservice/probes/test_probes_base.py -q
2 failed, 149 passed, 38 skipped in 974.89s

$ python -m pytest tests/test_attempt.py -q
23 passed in 8.24s

$ python -m pytest tests/probes/test_probes.py -q
7 failed, 1349 passed in 298.69s

$ black --check tests/langservice/probes/test_probes_base.py
All done!

Restoring garak/probes/base.py to main and rerunning the new test makes it fail on assert isinstance(notes_prompt, Conversation), so it is a real regression test rather than a restatement of current behaviour.

Every failure above is pre-existing in my environment rather than caused by this change, and I checked each one by restoring garak/probes/base.py to main and confirming the same failure.

• tests/probes/test_probes.py: the 7 failures are all test_probe_metadata on plugins whose optional dependencies I do not have locally, namely probes.audio.AudioAchillesHeel (wants soundfile and librosa), probes.suffix.BEAST, probes.suffix.GCG, and the three probes.topic.Wordnet* entries. None of them reach Probe.probe().
• test_atkgen_probe_translation[probes.atkgen.Tox]: fails locally only because accelerate is not installed.
• test_probe_prompt_translation[probes.sysprompt_extraction.SystemPromptExtraction]: fails with IndexError: list index out of range at garak/probes/base.py:402, which is the isinstance(prompts[0], str) check at the top of probe(). Its prompt set comes back empty locally. This reproduces identically with garak/probes/base.py restored to main, so it is not related to this change, but flagging it in case it is news.

black --check garak/probes/base.py also flags one pre-existing line further down the file, the logging.debug call in the intent-filtering probe() override. That flag is present on an unmodified checkout of main too, so I left it alone rather than widen this PR.

Reverting only the garak/probes/base.py hunk makes the new test fail with assert None is not None on notes["pre_translation_prompt"], so it is a real regression test rather than a restatement of current behaviour.

AI assistance

AI assistance was used for this change. I used Claude to trace the call path from Probe.probe() through Attempt.prompt_for() to the detector consumers, and to draft the fix and the regression test. I reviewed every changed line, confirmed against TreeSearchProbe._create_attempt and against the Turn dataclass that content is the correct attribute, and ran the tests above locally.

[jmartin-tech]

LGTM thanks @adityasingh2400, credit also goes to @feiiiiii5 for reporting the issue with details pointing to this solution.

[adityasingh2400]

Thanks for the review. Agreed on the credit, @feiiiiii5's report already had the unreachable branch pinned down, so the fix mostly followed from that.

[feiiiiii5]

Thanks for picking this up so quickly, @adityasingh2400 — the fix matches my diagnosis exactly (dead Message branch → Conversation, and turn.content not turn.context), and aligning with TreeSearchProbe._create_attempt is the right call. Appreciate the thorough pre-existing-failure triage too; the base.py:402 prompts[0] IndexError you flagged when a translated prompt set comes back empty is worth a follow-up issue.

[adityasingh2400]

Thanks @feiiiiii5, and the credit is yours. Your issue report had the diagnosis in it already, both the dead Message branch and the turn.content typo, so this was mostly a matter of writing down what you had found.

Agreed on the follow-up. I re-checked it against current main just now so the report is accurate rather than from memory:

garak/probes/base.py:402 is still if isinstance(prompts[0], str):, and nothing between the prompts copy and that line tests whether the list is empty. len(prompts) is used for the progress bar two lines earlier, so a zero-length set reaches the subscript and raises IndexError rather than returning no attempts.

I have not opened the issue yet. Happy to write it up with the reproduction path if that is useful, or leave it to you since it is your find originally. Either way it is independent of this PR, which only touches the pre-translation notes branch.

[feiiiiii5]

Thanks — I went ahead and opened the follow-up: #2026 with the empty-prompt-set repro and a suggested fix direction (return [] before the prompts[0] subscript). Happy to turn it into a PR with a regression test, or review yours if you’d rather take it.

[adityasingh2400]

Please take it, it is your find and your issue write-up already has the fix direction in it. I am not opening PRs against garak right now, so it would only sit waiting on me.

Re-checked #2026 against current main so the report holds: if isinstance(prompts[0], str): is still there, immediately after the total=len(prompts) progress bar, with nothing in between testing for an empty list. So the early return [] you suggested is the right shape.

Happy to review when you open it.

[feiiiiii5]

Opened it: #2027 — early return [] with a warning when the prompt set is empty, plus a regression test. Would appreciate your review whenever convenient!