feat: mask lower-case names with an uncased NER model beside the cased one

.github/workflows/model-smoke.yml

@@ -7,7 +7,7 @@ on:
   schedule:
     - cron: "17 6 * * 1"
   # Any change that can alter model acquisition or inference. Not a required check: it downloads
-  # about 215 MB and needs the network, which the deterministic gate deliberately does not.
+  # about 330 MB and needs the network, which the deterministic gate deliberately does not.
   pull_request:
     paths:
       - scripts/fetch_model.py

README.md

@@ -9,12 +9,12 @@ Includes a FrankenPHP worker-mode example, deterministic fault tests, and an opt
 - PHP 8.2–8.5 client using cURL, with validated job objects and typed rerank, embedding and redaction results. Rerank accepts up to 512 candidate documents and an optional `top_k` so a large set narrows to a small result; the ONNX backend scores them 32 pairs per forward pass.
 - Python 3.12 service with an explicit task allowlist and a separate process per task.
 - Text embedding: up to 32 texts per job return unit-length 384-dimensional vectors for an index the caller owns.
-- Redaction: checksum and pattern rules for card numbers, IBANs, emails, IPv4 addresses and phone numbers on every backend, plus named-entity masking with int8 `bert-base-NER` on the ONNX profile. Assistive, not a compliance control.
+- Redaction: checksum and pattern rules for card numbers, IBANs, emails, IPv4 and IPv6 addresses and phone numbers on every backend, plus named-entity masking with the int8 cased and uncased `bert-base-NER` models on the ONNX profile. Assistive, not a compliance control.
 - Bounded concurrency, queue/result capacity, retention, request sizes and execution deadlines.
 - Cancellation, progress polling, crash detection and generic errors that omit internal exception text.
 - A FrankenPHP example that submits work and returns a job ID immediately.
 - A stdio MCP server with one tool per task, a second client of the same protocol beside the PHP client, plus an index builder over the protocol and a `bridge_search` tool that searches what it built, returning offsets and snippets rather than vectors, a compose service that puts the pinned ONNX models behind it, and a ten-question evaluation for LLM hosts with a runner for the Claude Code CLI; see [the MCP guide](docs/mcp.md).
-- CPU inference with `cross-encoder/ms-marco-TinyBERT-L2-v2`, `sentence-transformers/all-MiniLM-L6-v2` and int8 `dslim/bert-base-NER` through ONNX Runtime, using pinned model files and wheel hashes.
+- CPU inference with `cross-encoder/ms-marco-TinyBERT-L2-v2`, `sentence-transformers/all-MiniLM-L6-v2` and int8 `dslim/bert-base-NER` and `dslim/bert-base-NER-uncased` through ONNX Runtime, using pinned model files and wheel hashes.
 
 The default backends are **`lexical-demo-not-a-model`** for reranking and **`hashing-bow-not-a-model`** for embedding, deterministic word overlap and feature hashing for testing the integration, and **`rules-only-not-a-model`** for redaction, which runs the real checksum and pattern rules but no model. None of them is described as AI. Enable the optional model profile below for real inference.
 
@@ -57,7 +57,7 @@ Acquire dependencies and model data in the bounded, network-enabled fetcher:
 docker compose -f docker/compose.yaml run --rm --no-deps fetcher
 ```
 
-This downloads wheels and about 215 MB of model and tokenizer data into ignored `.cache/`, one directory per task under `models/`; the fetcher refuses any single artifact over 128 MiB. A cache created before the embed task used a flat layout that the fetcher does not delete: remove `.cache/models` and fetch again after upgrading. Wheels are checked against `requirements-model.lock`; the model revision and SHA-256 hashes are fixed in `scripts/fetch_model.py`. On Linux, ensure `.cache/` exists and is writable by container UID 65532 before acquisition. For a disposable local cache, `mkdir -p .cache && chmod 777 .cache` is sufficient; never apply that permission to the repository or another directory.
+This downloads wheels and about 330 MB of model and tokenizer data into ignored `.cache/`, one directory per model under `models/`; the fetcher refuses any single artifact over 128 MiB. Fetch again after an upgrade that adds a model: an ONNX worker refuses to start while a model directory lacks its files, and names it. A cache created before the embed task used a flat layout that the fetcher does not delete: remove `.cache/models` and fetch again after upgrading. Wheels are checked against `requirements-model.lock`; the model revision and SHA-256 hashes are fixed in `scripts/fetch_model.py`. On Linux, ensure `.cache/` exists and is writable by container UID 65532 before acquisition. For a disposable local cache, `mkdir -p .cache && chmod 777 .cache` is sufficient; never apply that permission to the repository or another directory.
 
 Then run **without internet access**:
 
@@ -141,4 +141,4 @@ See [the integration guide](docs/integration.md), [test evidence and limits](doc
 
 ## License
 
-Code: [MIT](LICENSE). The optional [TinyBERT reranker](https://huggingface.co/cross-encoder/ms-marco-TinyBERT-L2-v2) and [MiniLM embedding model](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) are Apache-2.0 at their pinned revisions; the [BERT NER model](https://huggingface.co/dslim/bert-base-NER) is MIT and is fetched as the int8 ONNX conversion published by [Xenova](https://huggingface.co/Xenova/bert-base-NER), pinned by revision and hash. Model weights and dependency wheels are downloaded separately and retain their own licenses.
+Code: [MIT](LICENSE). The optional [TinyBERT reranker](https://huggingface.co/cross-encoder/ms-marco-TinyBERT-L2-v2) and [MiniLM embedding model](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) are Apache-2.0 at their pinned revisions; the cased and uncased BERT NER models, [dslim/bert-base-NER](https://huggingface.co/dslim/bert-base-NER) and [dslim/bert-base-NER-uncased](https://huggingface.co/dslim/bert-base-NER-uncased), are MIT and are fetched as the int8 ONNX conversions published by Xenova ([cased](https://huggingface.co/Xenova/bert-base-NER), [uncased](https://huggingface.co/Xenova/bert-base-NER-uncased)), pinned by revision and hash. Model weights and dependency wheels are downloaded separately and retain their own licenses.

docs/mcp.md

@@ -19,7 +19,7 @@ Every description states what the backend actually is, in the worker's own words
 | `bridge_health` | none | `backend`, `protocol`, the models by task | instant |
 | `bridge_rerank` | `query`, then `documents` (≤64) or `documents_path` (≤512 in a file), `top_k` | `model`, `considered`, `results[]` of `index` (the document's zero-based position), `score`, `snippet` | 0.4 s on the demo backend; with the ONNX model 0.7-0.9 s for 32 documents and 1.0-1.2 s for 512 |
 | `bridge_embed_similarity` | `texts` (2–32) | `model`, `matrix`, `pairs[]` of `a`, `b`, `similarity` | 0.4 s on the demo backend; 1.1-1.3 s for 32 texts with the ONNX model |
-| `bridge_redact` | `text`, `entities` (subset of PER, ORG, LOC), `min_score` | `model`, masked `text`, `spans[]` with `source` | 0.4 s on the demo backend; 1.6-2.4 s for 2000 characters with the ONNX model |
+| `bridge_redact` | `text`, `entities` (subset of PER, ORG, LOC), `min_score` | `model`, masked `text`, `spans[]` with `source` | 0.4 s on the demo backend; 2.8-3.4 s for 2000 characters with the ONNX models |
 | `bridge_search` | `query`, `index_path` (an index directory under the root), `top_k` (1-50), `rerank` (default true) | `embed_model`, `rerank_model`, `chunks`, `considered`, `results[]` of `source`, `chunk`, `start`, `end`, `similarity`, `rerank_score`, `snippet` | one embed job, plus one rerank job unless `rerank` is false |
 
 The times are the worker's, from [the measured table](testing.md#measured-latency-informational), and the ranges are the difference between two runners' CPUs; the model loads afresh for every call, and the MCP server adds its polling interval and HTTP overhead on top. A `documents_path` file is either a JSON array of strings or one document per non-blank line; lines holding only whitespace are skipped and not counted, so a result's `index` is its position among the documents, not a line number. A file over 4 MiB is refused before it is parsed, but the limits that bind are the worker's: at most 512 documents, and the query plus all documents within 200000 characters (the encoded request within 262144 bytes), so a usable file is roughly 200 KB of text. Relative paths resolve under the root, and every path, relative or absolute, must resolve inside it after following symlinks. Snippets are the first 160 characters of a document with whitespace collapsed; they are content from the agent's own files and enter its context like any other file it reads.

docs/protocol.md

@@ -25,7 +25,7 @@ Successful reranking returns `{"model":"...","rankings":[{"index":0,"score":1.0}
 
 Submit `{"task":"embed","input":{"texts":["..."]}}` with 1–32 texts of 1–8192 characters each, totalling at most 200000 characters. A successful result is `{"model":"...","dimensions":384,"vectors":[[...]]}` with one unit-length vector per text in input order; components are rounded to six decimals, so the dot product of two vectors is their cosine similarity. The service returns vectors and stores nothing; the caller owns any index built from them. The deterministic backend identifies itself as `hashing-bow-not-a-model`: a signed feature hash of word tokens that is stable across processes and ignores word order. The ONNX backend is `sentence-transformers/all-MiniLM-L6-v2` with mean pooling; it truncates input beyond 256 word pieces and is trained on English text.
 
-Submit `{"task":"redact","input":{"text":"...","entities":["PER"],"min_score":0.85}}` with 1–8192 characters of valid Unicode text. `entities` is an optional list of distinct labels among `PER`, `ORG` and `LOC` (default `["PER"]`; `MISC` is never offered) and `min_score` an optional number from 0 to 1 (default 0.85); both shape only the model pass. A successful result is `{"model":"...","text":"...","spans":[{"start":0,"end":5,"label":"EMAIL","source":"rule:email","score":1.0}]}`: the text with each span replaced by `[LABEL]`, and the spans sorted and disjoint, with offsets counted in Unicode code points of the original text. Deterministic rules run on every backend and are tagged `rule:`: `CARD` (any 13–19 digit window of a digit run that passes Luhn, longest first), `IBAN` (mod 97, trimming trailing groups the pattern swallowed; a lower-case one also needs a country code from the IBAN registry, and mixed case is never taken), `EMAIL` (letters of any script in the local part and the domain, and a letter or punycode top-level label), `IPV4` and `IPV6` (any form Python's `ipaddress` parser accepts that holds a digit, not starting inside a word, an IPv4-mapped tail and a `%zone` identifier included, the colon of `address: message` excluded) at score 1.0, and `PHONE` at 0.8 because it is a pattern, not a proof: 7–15 digits with a leading plus or separators, or at least ten bare digits, never a date shape. Known pattern false positives remain: a dotted quadruple of small numbers such as a version string matches `IPV4`, hex words joined by colons that the parser accepts, such as `dead::beef`, and eight colon-separated groups of up to four digits, such as `10:20:30:40:50:60:70:80`, match `IPV6`, and separated digit groups such as amounts can match `PHONE`; filter by `source` and `score` where that matters. On the default backend, or when `entities` is empty, the model name is `rules-only-not-a-model` and no model runs. The ONNX backend adds spans tagged `model:PER`, `model:ORG` or `model:LOC` carrying the model's mean word confidence, from `Xenova/bert-base-NER:int8`, the int8 ONNX conversion of `dslim/bert-base-NER` (CoNLL-2003, English news). The model was trained on cased text, so a sentence written all in lower case or all in capitals, and a surname in capitals before a comma and a given name, are read a second time from a title-cased copy with the same offsets. Pieces of one name that touch or meet at a hyphen, full stop or apostrophe are joined before the threshold and carry the score of their most confident piece; any other model span carries its mean word confidence. Overlapping candidates are merged into one span carrying the leftmost, then longest, then strongest label, then the more confident of two model readings, so no part of any candidate stays visible. Redaction is assistive: rules cannot see unstructured personal data, and the model misses names unlike its news training data: on an invented sample it masked every entity in news-style sentences and nearly every one in terse records, but about three in five in informal messages ([the measurement](testing.md#measured-ner-precision-and-recall-informational)), so it is not a compliance control.
+Submit `{"task":"redact","input":{"text":"...","entities":["PER"],"min_score":0.85}}` with 1–8192 characters of valid Unicode text. `entities` is an optional list of distinct labels among `PER`, `ORG` and `LOC` (default `["PER"]`; `MISC` is never offered) and `min_score` an optional number from 0 to 1 (default 0.85); both shape only the model pass. A successful result is `{"model":"...","text":"...","spans":[{"start":0,"end":5,"label":"EMAIL","source":"rule:email","score":1.0}]}`: the text with each span replaced by `[LABEL]`, and the spans sorted and disjoint, with offsets counted in Unicode code points of the original text. Deterministic rules run on every backend and are tagged `rule:`: `CARD` (any 13–19 digit window of a digit run that passes Luhn, longest first), `IBAN` (mod 97, trimming trailing groups the pattern swallowed; a lower-case one also needs a country code from the IBAN registry, and mixed case is never taken), `EMAIL` (letters of any script in the local part and the domain, and a letter or punycode top-level label), `IPV4` and `IPV6` (any form Python's `ipaddress` parser accepts that holds a digit, not starting inside a word, an IPv4-mapped tail and a `%zone` identifier included, the colon of `address: message` excluded) at score 1.0, and `PHONE` at 0.8 because it is a pattern, not a proof: 7–15 digits with a leading plus or separators, or at least ten bare digits, never a date shape. Known pattern false positives remain: a dotted quadruple of small numbers such as a version string matches `IPV4`, hex words joined by colons that the parser accepts, such as `dead::beef`, and eight colon-separated groups of up to four digits, such as `10:20:30:40:50:60:70:80`, match `IPV6`, and separated digit groups such as amounts can match `PHONE`; filter by `source` and `score` where that matters. On the default backend, or when `entities` is empty, the model name is `rules-only-not-a-model` and no model runs. The ONNX backend adds spans tagged `model:PER`, `model:ORG` or `model:LOC` carrying the model's mean word confidence, from two models trained on CoNLL-2003 English news, in the int8 ONNX conversions `Xenova/bert-base-NER` and `Xenova/bert-base-NER-uncased`; its results name `Xenova/bert-base-NER:int8+Xenova/bert-base-NER-uncased:int8`. The cased `dslim/bert-base-NER` reads the text, and reads a sentence written all in lower case or all in capitals, and a surname in capitals before a comma and a given name, a second time from a title-cased copy with the same offsets. The uncased `dslim/bert-base-NER-uncased` then reads the whole text ignoring case, which finds lower-case names the cased model misses, such as one in a sentence that holds a capital; it also tags a domain name such as `example.com` as `ORG`, which inside an email address the `EMAIL` span covers. Pieces of one name that touch or meet at a hyphen, full stop or apostrophe are joined before the threshold and carry the score of their most confident piece; any other model span carries its mean word confidence. Overlapping candidates are merged into one span carrying the leftmost, then longest, then strongest label, then the more confident of two model readings, so no part of any candidate stays visible. Redaction is assistive: rules cannot see unstructured personal data, and the models miss names unlike their news training data: on an invented sample they masked every entity in news-style sentences and terse records, but four in five in informal messages ([the measurement](testing.md#measured-ner-precision-and-recall-informational)), so it is not a compliance control.
 
 Failures use generic error objects with `code` and `message`. Task errors include `task_failed`, `worker_crashed`, `worker_start_failed` and `deadline_exceeded`. Internal exception text is not sent to clients. HTTP contract failures return 400, missing auth 401, missing routes/jobs 404, a connection idle for five seconds while its body is read 408 (a stall while reading headers closes the connection without a response), oversized bodies 413 and capacity exhaustion 429.