fix: default config_root to _config in InjectAtbash

[anugram]

What

garak.probes.encoding.InjectAtbash declared __init__(self, config_root=None).
Every other plugin in the project uses config_root=_config — 188 of 189
__init__ definitions taking config_root, with this as the sole exception.

Tracing Configurable._load_config(None):

• hasattr(None, "plugins") is False, so local_root = None
• isinstance(None, dict) is False
• hasattr(None, "probes") is False
• plugins_config stays {}, the namespace lookup misses, and
  _apply_config() is never called

The result is that configuration targeting this probe is silently discarded and
DEFAULT_PARAMS win instead.

Scope of impact

Direct instantiation only. _plugins.load_plugin() calls
klass(config_root=config_root) explicitly, so probes loaded through the normal
plugin machinery — which is every CLI run — were unaffected. The bug surfaces
when InjectAtbash() is constructed directly, e.g. from library code, a
notebook, or a test.

Reproduction

Setting a narrowed payload list for two sibling encoding probes on
_config.plugins.probes["encoding"]:

_config.plugins.probes["encoding"] = {
    "InjectROT13": {"payloads": ["default"]},
    "InjectAtbash": {"payloads": ["default"]},
}

Before the fix:

InjectROT13  payloads: ['default']
InjectAtbash payloads: ['default', 'xss', 'slur_terms']

After the fix:

InjectROT13  payloads: ['default']
InjectAtbash payloads: ['default']

Changes

• garak/probes/encoding.py — InjectAtbash.__init__ now defaults
  config_root to _config, matching every other plugin.
• tests/plugins/test_plugins.py — new test_plugin_config_root_default,
  parametrized over all plugins, asserting __init__ declares config_root
  with _config as its default. This turns the convention into an enforced
  invariant rather than something maintained by hand.

The test was written first and observed failing on exactly one case before the
fix was applied.

Tests run

Before the fix, with the new test in place:

$ python -m pytest tests/plugins/test_plugins.py -k config_root -v
FAILED tests/plugins/test_plugins.py::test_plugin_config_root_default[probes.encoding.InjectAtbash]
  - AssertionError: config_root must default to garak._config so direct
    instantiation honours global config
    assert None is _config
     +  where None = <Parameter "config_root=None">.default

1 failed, 359 passed, 360 deselected in 3.41s

After the fix:

$ python -m pytest tests/plugins/test_plugins.py -v
720 passed

Wider suite:

$ python -m pytest tests/probes/ tests/plugins/ -q
2839 passed, 1 skipped in 601.61s (0:10:01)

Formatted with black --config pyproject.toml.
Environment: Python 3.12.3, Linux.

Not a duplicate

$ gh pr list --repo nvidia/garak --state open --search "InjectAtbash"
no pull requests match your search in nvidia/garak

$ gh pr list --repo nvidia/garak --state open --search "config_root"
Showing 6 of 6 pull requests in nvidia/garak that match your search
#1673  fix: modernize Cohere generator (#1384)
#1806  Implement OpenAIResponsesGenerator
#1261  Branch jailbreakv
#1291  Generator that allows for multiple requests
#1306  Generator: Add support for Google Gemini models
#1289  Feature: translation cache

All six are generator changes that reference config_root incidentally, as any
new generator __init__ does. Confirmed none touch the files changed here:

$ for n in 1673 1806 1261 1291 1306 1289; do
    printf "PR #%s: " "$n"
    gh pr diff "$n" --repo nvidia/garak --name-only 2>/dev/null \
      | grep -E "probes/encoding\.py|plugins/test_plugins\.py" || echo "no overlap"
  done
PR #1673: no overlap
PR #1806: no overlap
PR #1261: no overlap
PR #1291: no overlap
PR #1306: no overlap
PR #1289: no overlap

AI assistance

AI assistance was used in investigating this issue, drafting the test, and
preparing this description. I have reviewed every changed line, run the tests
above myself, and understand the config-loading path the change affects.

[jmartin-tech]

Nice catch, thanks for the validation test.

[anugram]

Follow-up question @jmartin-tech, if you have a moment.

After this merged I noticed #2014 fixed the same underlying problem in a different shape: RileyIsnt's signature was correct, but the body passed the module-level _config to super().__init__ instead of threading config_root through. The test I added here can't catch that — it only inspects the signature default.

I've prototyped a behavioral version: pick a boolean from a plugin's DEFAULT_PARAMS, pass it via a GarakSubConfig rather than the _config module, instantiate directly, and assert the value landed. Using a distinct config object is the key part — where the config root is _config, a plugin that wrongly hardcodes it is indistinguishable from one that threads it through, which is why the existing test_instantiate_* tests can't see this class of bug either.

It's verified in both directions: reverting #2014's one-liner makes it fail with the right message, and on current main it finds nothing.

The costs are real, though. It adds roughly two minutes to CI, skips about 209 of ~360 plugins (anything needing an API key, an executable or a model can't be constructed in a test environment), and needs a fairly broad construction-exception skip clause.

So before I invest more: is that trade worth it to you? I could scope the parametrisation to only plugins that yield a usable boolean key, which would cut most of the skips and most of the runtime. Or if a preventive test that currently finds nothing isn't worth the CI time, that's a perfectly good answer and I'll drop it.

[jmartin-tech]

@anugram, I appreciate the extra look. At this time I think the extra CI test is probably not needed.

I suspect a more static analysis based approach be taken via pylint standards for classes that implement Configurable may be a more efficient way to ensure things like the pattern in #2014 do not recur. If/when the lint standards are tuned and applied to all PRs such a rule will likely be one of the first adds.

[anugram]

Makes sense, and a static check is the better fit — it catches the pattern at analysis time instead of paying for it on every CI run. I'll drop the runtime test.

One thing I noticed while looking: load-plugins= is currently empty in pylintrc, so a custom checker looks like the natural vehicle. I'd be glad to prototype one for Configurable implementers covering both shapes — a config_root default that isn't _config, and super().__init__ being handed _config where the passed config_root was available.

Is that useful input now, or would you rather it waited until the baseline lint standards are settled? Happy either way, I just don't want to build ahead of the plan.

[jmartin-tech]

Maybe review #1793 and offer feedback there.