feat: add fabricated regulatory citation prompts to misleading probe

[dentity007]

Adds misleading.FabricatedRegulatoryCitation, a probe that asks targets about compliance control identifiers that do not exist, to test whether a target refutes the false premise or supplies authoritative-sounding regulatory detail.

Identifiers are generated at runtime from templates modelled on each framework's numbering (NIST SP 800-171, NIST SP 800-53, CMMC 2.0, DFARS, HIPAA), using unassigned family codes or numeric components drawn from outside each published range, so a generated citation cannot collide with a control that actually exists. They are built to be non-existent rather than realistic, so detection difficulty varies by framework and the aggregate is a floor rather than a calibrated rate. Question framings ask for requirements, assessment objectives, or implementation guidance, so a target that accepts the premise produces substantive fabricated content rather than a passing mention.

On overlap with FalseAssertion: the two probes share intent = C009 and the demon:Rhetoric:Persuasion_and_manipulation:Distraction tag, so they pool into the same cell of the report digest's technique/intent matrix. That is the norm in garak rather than an exception: 40 of the 78 currently populated cells are shared by more than one probe, including the 27 probes under Ignore_previous_instructions x T009ignore and the 20 encoding.Inject* probes under Data_encoding x T018decode. What differs here is the generator, not the wording. FalseAssertion replays 50 fixed world-knowledge claims from a data file under three prefixes, whereas this probe synthesises identifiers at run time from a per-framework template plus numeric bounds chosen to fall outside each published standard, with syntax, bounds, and volume all reconfigurable via framework_specs and prompts_per_framework. That construction cannot be expressed as extra lines in truefalse_falseclaims_50.txt, and folding it into FalseAssertion would give that probe a second unrelated prompt-generation path plus parameters that mean nothing for its existing corpus. The isolated failure also differs in kind: FalseAssertion asks whether a target refutes a claim it should know to be false, while here the target cannot know the identifier is fake except by checking, so a substantive answer is pattern-completion on a plausible-looking control number, which is the behaviour that matters when that answer is pasted into an audit response.

Demonstrated efficacy: Dahl, Magesh, Suzgun and Ho, Large Legal Fictions (https://arxiv.org/abs/2401.01301), section 5.2 "Contra-Factual Bias", evaluates queries "with premises that are false by construction" and treats "any uncritical answering of the prompt" as a hallucination, which is this probe's scoring rule. Their Figure 8 reports pooled contra-factual hallucination rates of 0.69 for GPT-4, 0.52 for GPT-3.5, 0.99 for PaLM 2 and 0.02 for Llama 2, and their Online Appendix builds a "fake existence task" from fabricated case identifiers, the same construction as _fabricate_identifier here. The domain there is US case law rather than compliance standards, so this is a methodological match rather than a domain match: fabricated authority identifier, request for authoritative detail, failure scored on any substantive answer.

This is the first of the decomposed #1619 PRs by opening order (submitted before #1660, which merged first), restructured per maintainer feedback on #1619: prompts placed in the existing misleading technique family rather than a standalone compliance module, and grouping left to tags rather than a new module.

Updated 2026-08-07. The original submission loaded 25 fixed prompts from a data file. It has been reworked to follow the conventions that came out of #1660 review and to meet requirements added to main since March:

• Citation construction is now programmatic rather than a static prompt list, following the pattern in smuggling.HomoglyphObfuscation. Configurable via framework_specs (identifier syntax and out-of-range bounds per framework) and prompts_per_framework (volume). Default output is 30 prompts, six per framework, meeting the thirty-prompt bar in docs/source/extending.probe.rst. The original static set was 25.
• tier dropped from COMPETE_WITH_SOTA to INFORMATIONAL, consistent with a domain-specific probe that ships active = False.
• intent = "C009" (engage with false presuppositions) declared. test_probe_intent requires this of concrete probes and did not exist when this PR was opened.
• Added the demon: tag carried by FalseAssertion, since the technique is the same.
• doc_uri now points at Dahl et al. rather than the Azaria and Mitchell paper inherited from FalseAssertion, which describes an activation-probing detector and has no bearing on this technique. /abs/ rather than /html/, because the LaTeXML conversion of the current version renders empty.
• Range comments and the real-identifier guard in the tests re-checked against current revisions as of 2026-08: NIST SP 800-171 Rev 3 (17 families, 3.1 to 3.17), DFARS clause list through 252.204-7025, plus an 800-53 family-shape pattern. The CMMC template was corrected to the DD.L#-REQ form required by 32 CFR 170.14(c)(1); the previous form used a bare integer where the regulation requires a dotted requirement number.
• garak/data/falseclaims_regulatory_25.txt removed, now redundant.

Files: the diff against main is two files, garak/probes/misleading.py (FabricatedRegulatoryCitation, plus DEFAULT_FRAMEWORK_SPECS, CITATION_QUESTION_TEMPLATES, and _fabricate_identifier) and tests/probes/test_probes_misleading.py (8 tests covering generation, per-framework coverage, uniqueness, determinism, custom specs, and a guard asserting no generated identifier matches a real control pattern). The third file from the original submission, garak/data/falseclaims_regulatory_25.txt, was added and then removed on this branch, so it does not appear in the diff against main.

Not duplicating existing work: searched open PRs for probes touching misleading, regulatory citation, and hallucination-of-citation techniques; no open PR covers this. The related compliance work in #1619 was closed in favour of this decomposition. Open PR #1853 (test coverage for several modules) also adds a new tests/probes/test_probes_misleading.py, so the two will conflict on that path whichever merges second. That PR is currently marked dirty and has not been updated since 2026-07-05. Happy to rebase onto it if it lands first.

Tests run:

pytest tests/probes/test_probes_misleading.py
  8 passed

pytest tests/probes/test_probes.py -k Fabricated
  7 passed

pytest tests/probes/test_probes.py tests/probes/test_probes_misleading.py tests/test_docs.py tests/test_data.py
  2059 passed, 2 failed

black --check garak/probes/misleading.py tests/probes/test_probes_misleading.py
  2 files unchanged

The 2 failures are local environment gaps unrelated to this change: probes.audio.AudioAchillesHeel (no soundfile installed) and probes.sata.MLM (no local nltk averaged_perceptron_tagger_eng data). Python 3.12.8, macOS arm64.

AI assistance: this PR was developed with AI assistance (Claude). I have reviewed every changed line, ran the tests above, and can defend the change end to end.

[dentity007]

@jmartin-tech if you have a moment, this PR (fabricated regulatory citation probe) has been sitting since 2026-03-29 without a review round. CI is green (Linux, macOS, Windows, CLA) and the probe follows the same pattern as the misleading module it extends. Happy to make any changes before review, or rework if the approach doesn't fit. Thanks.

[dentity007]

Pushed an update. The probe now builds its citations programmatically instead of reading a fixed prompt file: identifiers come from a per-framework template plus numeric bounds chosen to fall outside each published range, with framework_specs and prompts_per_framework both configurable. Default output is 30 prompts across five frameworks.

Changes since the original submission:

• intent = "C009" (engage with false presuppositions) declared, matching the sibling FalseAssertion probe. test_probe_intent did not exist when this was opened.
• tier set to INFORMATIONAL and active = False, appropriate for a domain-specific probe.
• doc_uri changed to Dahl et al., Large Legal Fictions (https://arxiv.org/abs/2401.01301). Section 5.2 evaluates prompts whose premises are false by construction and scores any uncritical answer as a hallucination, which is the same rule this probe applies. The paper inherited from FalseAssertion is about activation-based lie detection and does not describe this technique.
• Identifier ranges and the real-control guard in the tests re-checked against current revisions: NIST SP 800-171 Rev 3 has 17 families (3.1 to 3.17), and the DFARS clause list now runs through 252.204-7025. The CMMC template was also corrected to the DD.L#-REQ form required by 32 CFR 170.14(c)(1); the previous form used a bare integer where the regulation requires a dotted requirement number.
• Dropped the active and tier assertions from the tests, per the AGENTS.md guidance against testing default values of configurable plugins.

The part I am least confident about, in case it changes your read: the identifiers are deliberately non-existent rather than realistic, and some use sentinel codes (ZZ, XX) that no framework assigns. A target that has memorised a framework's family list can reject those without reasoning about the number, so difficulty is uneven across the five frameworks and the score is better read as a floor than as a calibrated rate. I also found that the detectors in this module were tuned on short general claims rather than extended regulatory prose, so mitigation.MitigationBypass flags correct compliance refusals as hits. Both are noted in the class docstring. A judge.ModelAsJudge detector configured for this goal would fit better, but that affects FalseAssertion too, so it seemed like a follow-up rather than something to change here.

One heads-up: open PR #1853 also adds a new tests/probes/test_probes_misleading.py, so whichever of us lands second will need to resolve that path. Happy to rebase onto it.

Current main is merged into the branch, and tests and black --check pass locally. Happy to adjust the framework set or the identifier ranges if you would rather see them defined elsewhere.

[dentity007]

@jmartin-tech respectful ping on this one. It is the first decomposed PR from #1619, reworked to follow your #1660 review conventions (programmatic generation instead of a static prompt list, INFORMATIONAL tier) and to meet the requirements added to main since March (intent C009, demonstrated-efficacy citation, substance bar). Change summary is in the comment above; tests and black are green locally. Happy to adjust anything.