[feature] Opt-in leetspeak / ASCII-substitution matching via a surface-form Aho-Corasick corpus

[raeq]

What problem would this solve?

Labels: enhancement, needs-triage

What problem would this solve?

disarm deliberately does not do leet/digit remapping — docs/user-guide/llm-pipelines.md
spells out why: remapping 4→a, 3→e, 0→o, 1→i corrupts the numeric text that pervades
an LLM/catalog/ETL stack (model names, versions, quantities — GPT-4, llama 3.1, 2024).
That refusal is correct for the default mission and should not change.

But for the content-moderation / profanity-filter persona, leetspeak (f4ck, $h1t,
ph uck, |3itch) is a primary evasion vector, and it is precisely the class disarm's
Unicode-confusable layer does not touch. Today there is no first-class way to catch
ASCII-substitution evasion, so a moderation user who otherwise wants disarm's Unicode
canonicalization has to bolt on a second tool. The gap is real; the constraint is that
closing it must not weaken the deterministic, lossless default or corrupt alphanumerics.

Proposed solution

Reframe leet handling as matching, not rewriting. Add an opt-in, off-by-default detector
that compiles a caller-supplied lexicon (banned/target terms) into a leftmost-longest
Aho-Corasick automaton over an enumerated corpus of leet surface forms, scans
canonicalized input once (O(n)), and returns match spans for flag/redaction. It never
mutates the user's text, so it structurally cannot corrupt numeric data — GPT-4 has no
banned reading and passes through untouched, while 4ss is caught.

Why surface-form AC specifically (and why it fits this repo):

• AC match cost is independent of pattern count and size — O(input + matches) whether
  the corpus holds 10 forms or 10⁶. So eager enumeration of spellings is affordable, and the
  whole cost moves to build time where it is controllable.
• Reuses the leftmost-longest aho-corasick machinery already added in Perf cluster H: satellite algorithms (Aho-Corasick, UniqueSlugifier) #242 — no bespoke
  class-NFA and no runtime regex alternation.
• The corpus is an explicit, diffable, auditable spec of exactly what is matched, which
  suits disarm's verification posture (it can be exhaustively tested).
  API sketch — namespaced, off the top level, not in the deterministic pipelines:

from disarm.security import LeetMatcher   # name TBD; see open questions
 
m = LeetMatcher(["fuck", "shit"], precision="symbols")   # or precision="digits" (lossy, higher recall)
m.find("ph uck this $h1t")    # -> [Match(term="fuck", start=0, end=6), Match(term="shit", ...)]
m.contains("f4ck")            # -> True  (only when precision includes the digit tier)
m.censor("f4ck off", "*")     # -> "**** off"   (redaction by span; text elsewhere untouched)

Architecture (along the existing grain)

1. Variant map = source of truth. A TSV (src/tables/data/leet_variants.tsv) mapping each
   target letter to its leet variants, each tagged with a precision tier: a high-precision
   symbol tier ($ @ | ! () and multi-glyph ph, |3, \/\/ — punctuation shapes almost
   never legitimate letters) and a lossy digit tier (4 3 1 0 5 7). Compiled like every other
   disarm table.
2. Offline generator (sibling of scripts/gen_confusables.py): cross-products the variant
   map against the lexicon under the bounds below, emitting the surface-form corpus + per-form
   metadata (term id, severity, precision tier). Serialized as a build artifact; a default
   corpus can be shippable, caller lexicons compiled on demand (AC build is fast).
3. Runtime: leftmost-longest AC scan over the canonicalized input → match spans.
   Implementable in the pure Rust core (no pyo3 needed for the matcher itself; exposed via
   extension-module like the rest — respects Split into pure-Rust translit-core + PyO3 wrapper (standalone Rust + other-language bindings) #38/Package and release translit-core on crates.io (idiomatic Rust) #42, no pyo3 leak into the pure tree).

Generation-time disciplines (what makes it both safe and finite)

• Bound the cross-product. Cap variants/position and max-surface-forms/term (top-N by a
  likelihood weight); beyond the cap, drop the digit tier and keep symbols-only for that term.
  Without this an 8-char term × 4 variants/position ≈ 65k forms and grows unbounded.
• Prune real-word collisions (Scunthorpe) at generation time. Drop any surface form that is
  itself a dictionary word / known false-positive trigger — far easier over a static corpus
  than at runtime, and it keeps precision honest.
• Keep the Unicode pre-pass; do not enumerate noise. Run the existing canonicalizer first
  (fold confusables, strip bidi/zero-width/invisibles, collapse repeated runs fuuuck→fuck,
  optionally strip interspersed separators f.u.c.k). Then the corpus only enumerates genuine
  substitution spellings, not evasion noise — orders of magnitude smaller. Ordering matters:
  confusables must fold before leet (or the variant classes would have to subsume the
  Unicode lookalikes — prefer letting the existing layer own that).
• Digit gating falls out for free. Because matching is goal-directed against the lexicon,
  digits are never substituted globally — only readings that land on a target term match.

Alternatives or workarounds you've tried

• Character-class / generalized automaton (one pattern per term; transitions match a variant
  class instead of a literal byte): compact — O(Σ term length), no cross-product — but needs a
  custom class-NFA or regex alternation, makes per-spelling metadata awkward, and doesn't reuse
  the literal AC path. Wins only for very large lexicons with long terms. Note the variant TSV is
  shared between both designs, so this stays available as a future lazy-expansion option
  behind the same table.
• Destructive char-by-char rewrite (4→a etc. in the pipeline): rejected — corrupts numeric
  text and breaks idempotence/determinism; this is exactly what llm-pipelines.md refuses. A
  non-goal.
• Status quo / external tool: leaves the moderation persona without ASCII-substitution
  coverage and forces a second dependency alongside disarm's Unicode layer.

Scope / non-goals

• Off by default; never in the deterministic default pipelines (security_clean,
  normalize_user_input, …); not a top-level convenience export. Framed like context=True
  abjad mode: opt-in, best-effort, clearly documented.
• Detection/redaction (matching), not canonical rewriting. An optional lexicon-validated
  rewrite mode could follow later (commit a digit substitution only when the result is a
  dictionary word), documented as lossy/non-idempotent.
• English/Latin-centric — gate the variant set by script/lexicon language, don't apply globally.
• Not an output sanitizer; the standard disarm scope disclaimer applies.

Open questions

• Naming/namespace: disarm.security.LeetMatcher vs disarm.adversarial vs a profanity_key
  preset?
• Ship a default lexicon, or require caller-supplied? (Maintenance + locale burden of a bundled
  blocklist, plus the optics of shipping one.)
• Memory ceiling guidance and corpus-size diagnostics for large lexicons.
• Overlap/redaction semantics for nested or adjacent matches.

Acceptance criteria (sketch)

• Opt-in matcher with find / contains / censor over a caller lexicon; symbols vs digits
  precision tiers.
• Property/exhaustive tests: never mutates input outside reported spans; idempotent match output;
  no false matches on benign alphanumerics (GPT-4, 2024, llama3); all known leet
  spellings of a test lexicon are matched; corpus size stays within the configured caps.
• Reuses the Perf cluster H: satellite algorithms (Aho-Corasick, UniqueSlugifier) #242 leftmost-longest AC path; pure-core build carries no pyo3 (Split into pure-Rust translit-core + PyO3 wrapper (standalone Rust + other-language bindings) #38/Package and release translit-core on crates.io (idiomatic Rust) #42 gate).
• Docs: a user-guide page positioning it as opt-in moderation tooling, plus a Threat-Model note.

Proposed solution

Alternatives or workarounds you've tried

No response