fix: mid-word matches, empty-quote wildcards, case-sensitive misses

Third round of review findings on the Article 9 recognizer.

_whole_words grew any substring match outward rather than dropping ones that
started mid-word: a quote of "he" turned "When the interviewer asked, he
said..." into "<SEXUAL_ORIENTATION> interviewer asked, <SEXUAL_ORIENTATION>
said...". Replaced with _span_for_match, which drops mid-word starts and keeps
rightward growth for inflections (Muslim -> Muslims).

The empty-quote guard only checked for `not item.text`, so a quote of a single
space matched every word boundary in the transcript -- reproduced as
"I<HEALTH>manage<HEALTH>my<HEALTH>condition...". Now requires at least one
alphanumeric character.

text.find() was also case-sensitive, so a re-capitalized quote ("Diabetes" vs
the transcript's "diabetes") silently dropped the finding -- undetected Article
9 leakage with no signal. Now case-insensitive matching via re.finditer, offsets
unchanged, plus a warning log on an unmatched quote.

guard_nemo read response["content"], which could KeyError past all four
fail-closed checks on an unexpected response shape. Now .get("content"), which
routes into the existing empty-content failure path.

.env.example did not document the three new SCREENING_LLM_GUARDRAIL_* settings;
an unset var defaults to localhost and fails closed, so a deployed environment
missing it would 502 every request with no obvious cause.

Two items reported, not fixed, because the real fix is a design change: the
900s guardrail timeout runs inside run_in_threadpool and can pin AnyIO's
threadpool for the duration of a cold start (real fix is 202 + polling, already
named in infra/gemma/README.md); and a guardrail RuntimeError surfaces as the
same 502 as an LLM failure, pointing operators at the wrong system during a
Gemma outage.

Author: inesaranab

.env.example

@@ -10,6 +10,16 @@ SCREENING_LLM_API_KEY=ollama
 SCREENING_LLM_MODEL=qwen2.5:3b
 SCREENING_LLM_TIMEOUT_S=60
 
+# Article 9 detector — the self-hosted Gemma-4 endpoint the guardrail calls.
+# The default points at a local vLLM. In any deployed environment this MUST be
+# set to the detector's internal FQDN: the recognizer fails closed, so leaving
+# the localhost default in place makes every /screen request return 502.
+SCREENING_LLM_GUARDRAIL_BASE_URL=http://localhost:8001/v1
+SCREENING_LLM_GUARDRAIL_MODEL=google/gemma-4-31B-it
+# Much larger than the assessment timeout: the endpoint scales to zero, so the
+# first request after an idle period waits for a GPU to boot and load weights.
+SCREENING_LLM_GUARDRAIL_TIMEOUT_S=900
+
 # Confident AI (DeepEval online evals). 
 CONFIDENT_API_KEY=
 CONFIDENT_BASE_URL=https://eu.api.confident-ai.com

app/adapters/guard_nemo.py

@@ -154,7 +154,13 @@ async def scrub(self, text: str) -> ScrubResult:
         response = await self._rails.generate_async(
             messages=[{"role": "user", "content": _encode(text)}]
         )
-        content = response["content"] if isinstance(response, dict) else str(response)
+        # `.get`, not `["content"]`: a dict-shaped response without that key
+        # would raise KeyError straight past every fail-closed check below and
+        # surface as an unclassified 500. Missing content is just another way
+        # the rail failed to produce scrubbed text.
+        content = (
+            response.get("content") if isinstance(response, dict) else str(response)
+        )
         content = "" if content is None else str(content)
         content = content.strip()
 

app/adapters/llm_guardrail_recognizer.py

@@ -5,13 +5,18 @@
 GLiNER's 0.552, and beat the frontier cloud model too (see INE-16).
 """
 
+import logging
+import re
+
 import instructor
 from openai import OpenAI
 from presidio_analyzer import EntityRecognizer, RecognizerResult
 from pydantic import BaseModel
 
 from app.config import settings
 
+logger = logging.getLogger("screen")
+
 _ARTICLE9_ENTITIES = [
     "RELIGION",
     "HEALTH",
@@ -33,28 +38,32 @@ def _is_word_char(char: str) -> bool:
     return char.isalnum() or char == "_"
 
 
-def _whole_words(text: str, start: int, end: int) -> tuple[int, int]:
-    """Grow a span outwards until neither edge cuts a word in half.
+def _span_for_match(text: str, start: int, end: int) -> tuple[int, int] | None:
+    """Turn a raw substring hit into a span that does not cut a word in half.
+
+    Substring matching is blind to word boundaries, and the two ways it can
+    straddle one need opposite treatment:
 
-    `str.find` is plain substring matching, so a quote of "Black" as an
-    ETHNICITY also lands inside "BlackRock" -- and redacting that span alone
-    leaves "<ETHNICITY>Rock", corrupting the employer the candidate is scored
-    on. Growing (rather than dropping the occurrence) is the safe direction:
-    the span never shrinks, so this can only ever redact more, never less. It
-    also catches the inflected form -- a model that quotes "Muslim" against a
-    transcript saying "Muslims" would otherwise leave the trailing "s" behind.
+    * The hit STARTS mid-word -- "he" inside "the"/"When", "MS" inside "CMS".
+      There is no reading of that occurrence under which the candidate
+      disclosed anything, so it is dropped. Growing it leftwards instead (the
+      previous behaviour) redacted whole innocent words: a quote of "he"
+      turned "When the interviewer" into "<SEXUAL_ORIENTATION> interviewer".
+    * The hit ENDS mid-word -- "Muslim" against a transcript saying "Muslims".
+      That is the same disclosure inflected, so the span is grown rightwards.
+      Stopping at the raw end would leave "<RELIGION>s" behind.
 
     Args:
         text: The transcript the offsets refer to.
         start: Start offset of the raw substring match.
         end: End offset (exclusive) of the raw substring match.
 
     Returns:
-        The widened ``(start, end)``. Unchanged when both edges already sit on
-        a word boundary, or when the quote itself begins/ends with punctuation.
+        The span to redact, or None when the hit began mid-word and should be
+        discarded.
     """
-    while start > 0 and _is_word_char(text[start]) and _is_word_char(text[start - 1]):
-        start -= 1
+    if start > 0 and _is_word_char(text[start]) and _is_word_char(text[start - 1]):
+        return None
     while end < len(text) and _is_word_char(text[end - 1]) and _is_word_char(text[end]):
         end += 1
     return start, end
@@ -142,30 +151,61 @@ def analyze(self, text, entities, nlp_artifacts=None):
         seen: set[tuple[str, int, int]] = set()
         for item in detected.entities:
             entity_type = item.entity_type.strip().upper()
-            if entity_type not in requested or not item.text:
+            if entity_type not in requested:
+                continue
+            # A quote with no alphanumeric character in it -- "", " ", "." --
+            # occurs almost everywhere in the transcript. Redacting every hit
+            # replaces the separators between words and destroys the text the
+            # model is scored on ("I<HEALTH>manage<HEALTH>my..."), so such a
+            # quote is never actionable.
+            if not any(char.isalnum() for char in item.text):
                 continue
             # Offsets are computed here, never asked of the model -- LLMs are
-            # unreliable at character arithmetic. No exact match means no
-            # trustworthy span, so the finding is dropped rather than guessed.
+            # unreliable at character arithmetic. No match means no trustworthy
+            # span, so the finding is dropped rather than guessed.
+            #
+            # Case-insensitive on purpose: a model that returns "Diabetes"
+            # against a transcript saying "diabetes" is pointing at a real
+            # disclosure, and the offsets it produces are still exact. Matching
+            # case-sensitively dropped it silently -- under-redaction of an
+            # Article 9 category is the one failure this recognizer exists to
+            # prevent.
             #
             # Every occurrence, not just the first: the same disclosure often
             # appears more than once ("I have diabetes ... my diabetes"), and
             # redacting only the first mention leaks the rest to the model.
             # `seen` absorbs the duplicates a model asked for "every occurrence"
             # tends to return.
-            match_start = text.find(item.text)
-            while match_start != -1:
-                match_end = match_start + len(item.text)
-                start, end = _whole_words(text, match_start, match_end)
-                if (entity_type, start, end) not in seen:
-                    seen.add((entity_type, start, end))
-                    results.append(
-                        RecognizerResult(
-                            entity_type=entity_type,
-                            start=start,
-                            end=end,
-                            score=_SCORE,
-                        )
+            matched = False
+            for match in re.finditer(re.escape(item.text), text, re.IGNORECASE):
+                span = _span_for_match(text, match.start(), match.end())
+                if span is None:
+                    continue
+                matched = True
+                key = (entity_type, span[0], span[1])
+                if key in seen:
+                    continue
+                seen.add(key)
+                results.append(
+                    RecognizerResult(
+                        entity_type=entity_type,
+                        start=span[0],
+                        end=span[1],
+                        score=_SCORE,
                     )
-                match_start = text.find(item.text, match_end)
+                )
+            if not matched:
+                # Logged, not silent: a dropped quote is an Article 9
+                # disclosure the model saw and we did not redact. The quote
+                # itself is special-category data, so only its length is
+                # recorded.
+                logger.warning(
+                    "article9_quote_unmatched",
+                    extra={
+                        "context": {
+                            "entity_type": entity_type,
+                            "quote_length": len(item.text),
+                        }
+                    },
+                )
         return results

tests/unit/test_llm_guardrail_recognizer.py

@@ -77,6 +77,62 @@ def test_drops_quotes_that_are_not_verbatim(recognizer, monkeypatch):
     assert recognizer.analyze("She said she is a Quaker.", ["RELIGION"], None) == []
 
 
+def test_ignores_hits_that_start_inside_another_word(recognizer, monkeypatch):
+    """Substring matching is blind to word boundaries: a short quote like "he"
+    also lands inside "the" and "When". Growing those hits leftwards to the word
+    edge redacted the innocent word whole -- "When the interviewer" came back as
+    "<SEXUAL_ORIENTATION> interviewer" -- destroying the transcript the model is
+    scored on. A hit that begins mid-word is not a disclosure; drop it."""
+    _model_returns(
+        monkeypatch,
+        recognizer,
+        [DetectedEntity(entity_type="SEXUAL_ORIENTATION", text="he")],
+    )
+
+    text = "When the interviewer asked, he said he is out at work."
+    results = recognizer.analyze(text, ["SEXUAL_ORIENTATION"], None)
+    spans = sorted((r.start, r.end) for r in results)
+
+    assert spans == [(28, 30), (36, 38)]
+    assert all(text[s:e] == "he" for s, e in spans)
+
+
+def test_ignores_quotes_with_no_alphanumeric_characters(recognizer, monkeypatch):
+    """A quote of " " or "." matches between every word. Redacting each hit
+    replaces the separators and shreds the transcript
+    ("I<HEALTH>manage<HEALTH>my..."), so it can never be actionable."""
+    _model_returns(
+        monkeypatch, recognizer, [DetectedEntity(entity_type="HEALTH", text=" ")]
+    )
+
+    assert recognizer.analyze("I manage my condition well.", ["HEALTH"], None) == []
+
+
+def test_matches_regardless_of_case(recognizer, monkeypatch):
+    """Models routinely re-capitalise what they quote. The offsets are still
+    exact, so dropping the finding is pure under-redaction -- Article 9 data
+    reaching the model, which is the one failure this recognizer prevents."""
+    _model_returns(
+        monkeypatch, recognizer, [DetectedEntity(entity_type="HEALTH", text="Diabetes")]
+    )
+
+    results = recognizer.analyze("I have diabetes.", ["HEALTH"], None)
+
+    assert [(r.start, r.end) for r in results] == [(7, 15)]
+
+
+def test_grows_a_hit_that_ends_inside_the_same_word(recognizer, monkeypatch):
+    """ "Muslim" against a transcript saying "Muslims" is the same disclosure
+    inflected. Stopping at the raw match end leaves "<RELIGION>s" behind."""
+    _model_returns(
+        monkeypatch, recognizer, [DetectedEntity(entity_type="RELIGION", text="Muslim")]
+    )
+
+    results = recognizer.analyze("Two Muslims on the team.", ["RELIGION"], None)
+
+    assert [(r.start, r.end) for r in results] == [(4, 11)]
+
+
 def test_makes_no_call_when_no_supported_entity_is_requested(recognizer):
     # No stub: a real call would try to reach the endpoint and fail, so passing
     # proves we short-circuit before touching the network.