Skip to content

Commit c1eba32

Browse files
jackghxjackghx
committed
v0.3.0: OpenRouter recogniser, custom recognisers, and review-UI upgrades
Detection: - Add OPENROUTER_KEY recogniser (sk-or-v1- prefix). Body bounded to [A-Za-z0-9]{40,100} so it stays robust to key variants while the long, near-unique prefix keeps false positives close to zero. Pack is now 33 entity types. - Add runtime custom recognisers: Scrubber add/remove/list plus the /recognizers API (GET/POST/DELETE), held in memory only. Review UI: - Multi-file upload with tabs and a load-sample button. - Line-numbered review with click-a-detection-to-locate (smooth scroll plus line-number flash) and a scrubbed-position minimap. - Side-by-side diff: the Original replaces the Input pane when toggled, so each side gets a full column. - Detection filter/search and a custom-recogniser panel. Docs: - README and SECURITY.md refreshed to the true current state (full 33-entity list, all CLI flags, features, limitations, version 0.3.0). - PROJECT.md history and counts updated; em dashes removed from the docs. Tests: 127 passing.
1 parent fcfa182 commit c1eba32

25 files changed

Lines changed: 1819 additions & 518 deletions

‎README.md‎

Lines changed: 205 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -1,102 +1,237 @@
1-
![CI](https://github.com/jackghx/scrub/actions/workflows/ci.yml/badge.svg)
2-
# Scrub, the local-first security-artefact sanitiser
1+
# Scrub, local-first security-artefact sanitiser
32

4-
Strip infrastructure identifiers and secrets out of logs, configs, and terminal output
5-
**before** you paste them into a public GitHub issue, a vendor ticket, a forum, or a
6-
blog post, and then **review what was detected and decide what to scrub** before you copy
7-
it out.
3+
**Strip secrets and infrastructure identifiers out of logs, configs, and terminal
4+
output before you share them, with a human review step so you decide what leaves
5+
your machine.**
86

9-
> **The data never leaves your machine.** That's the whole pitch, not a preference.
10-
> There are no network calls in the scrubbing path, no telemetry, no phone-home. The API
11-
> runs on `127.0.0.1`; the UI's browser talks only to that local API.
7+
Scrub runs entirely on your machine. Paste or pipe in an artefact; it detects
8+
internal IPs, hostnames, cloud keys, tokens, private keys, and connection strings,
9+
replaces each with a stable placeholder (e.g. `<INTERNAL_IP_1>`), and gives you a
10+
diff to review before anything is exported. Nothing is ever sent anywhere.
1211

13-
Scrub is not a general PII tool. It targets what actually leaks in security work,
14-
internal IPs, hostnames, MACs, cloud keys, tokens, private keys, connection strings,
15-
and adds the piece general tools don't: **consistent, reversible pseudonymisation**
16-
(`<INTERNAL_IP_1>`), so the scrubbed artefact stays analytically useful and can be
17-
restored byte-for-byte.
12+
---
1813

19-
## Human-in-the-loop, by design
14+
## Why
2015

21-
Detection is **recall-favoured**, it surfaces everything, because under-scrubbing leaks
22-
a secret. That's only safe if a human can see and correct what was flagged. So Scrub
23-
**surfaces and applies; you decide**. The UI shows every detection, lets you toggle each
24-
one, and **never claims the output is "safe" or "clean"**. Always eyeball the result
25-
before sharing.
16+
Engineers leak secrets every day by pasting logs into GitHub issues, vendor
17+
tickets, forums, and chat. General PII tools target names and emails; they miss
18+
the things that actually leak in security and infrastructure work: internal IPs,
19+
MAC addresses, cloud keys, JWTs, private-key blocks, DB connection strings.
2620

27-
## Three ways to use it
21+
Scrub fills that gap, and keeps a reversible mapping so the scrubbed artefact
22+
stays useful (you can follow which host talked to which) without exposing the real
23+
identifiers.
2824

29-
| Dir | What | Stack |
30-
|-----|------|-------|
31-
| [`scrub/`](scrub) | The engine: detectors + consistent pseudonymiser + FastAPI service (`/scrub`, `/restore`, `/entities`, `/health`) **and the `scrub` CLI** | Python, Presidio |
32-
| [`ui/`](ui) | The review front end: two-pane paste → review → export, with per-detection toggles | Next.js, TypeScript, Tailwind |
33-
| [`hooks/`](hooks) | A git **pre-commit hook** that blocks commits containing secrets (built on `scrub --check`) | bash |
25+
---
3426

35-
## Quick start (run both)
27+
## What it detects
28+
29+
The detection pack ships **33 entity types**. Two are validated in code rather than
30+
trusted from a regex, IP addresses via the stdlib `ipaddress` module, and credit
31+
cards via the Luhn checksum, and several FP-prone patterns are deliberately scored
32+
low and only cross the threshold when supporting context words are nearby.
33+
34+
- **Network identifiers:** `INTERNAL_IP`, `PUBLIC_IP` (IPv4 + IPv6, validated),
35+
`MAC_ADDRESS`, `HOSTNAME`
36+
- **Cloud / provider keys:** `AWS_ACCESS_KEY`, `AWS_SECRET_KEY`, `AWS_ACCOUNT_ID`,
37+
`GOOGLE_API_KEY`, `STRIPE_KEY`, `OPENAI_KEY`, `OPENROUTER_KEY`,
38+
`FCM_SERVER_KEY`, `SENDGRID_KEY`, `TWILIO_SID`
39+
- **Source-control / package tokens:** `GITHUB_TOKEN`, `GITLAB_TOKEN`,
40+
`NPM_TOKEN`, `SHOPIFY_TOKEN`
41+
- **Chat tokens & webhooks:** `SLACK_TOKEN`, `SLACK_WEBHOOK`, `DISCORD_TOKEN`,
42+
`DISCORD_WEBHOOK`, `TELEGRAM_BOT_TOKEN`
43+
- **Auth tokens / generic secrets:** `JWT`, `BEARER_TOKEN`, `GENERIC_API_KEY`
44+
- **Crypto / connection strings:** `PRIVATE_KEY_BLOCK` (PEM), `SSH_PUBLIC_KEY`,
45+
`DB_CONNECTION_STRING`, `URL_WITH_CREDENTIALS`
46+
- **PII:** `EMAIL_ADDRESS`, `CREDIT_CARD` (Luhn-validated)
47+
- **Filesystem:** `UNIX_HOME_PATH` (username leak)
48+
49+
The live list is always available from the API (`GET /entities`) or
50+
`Scrubber().entities()`. You can also add your own regex recognisers at runtime
51+
from the review UI (see below), they run alongside the built-in pack.
52+
53+
---
54+
55+
## Install
3656

3757
```bash
38-
# terminal 1 – the API (no spaCy model needed for the default mode)
39-
cd scrub
40-
python -m venv .venv && . .venv/Scripts/activate # or: source .venv/bin/activate
41-
pip install -r requirements.txt
42-
uvicorn main:app --host 127.0.0.1 --port 8000
58+
pip install -e .
59+
```
4360

44-
# terminal 2 – the UI
45-
cd ui
46-
npm install
47-
npm run dev # open http://localhost:3000
61+
This puts a `scrub` command on your PATH. The only runtime dependency for the CLI
62+
is `presidio-analyzer`; the API service and full-Presidio mode are optional extras
63+
(`pip install -e ".[api]"`, and a spaCy model for full mode).
64+
65+
---
66+
67+
## CLI usage
68+
69+
```bash
70+
scrub app.log # scrubbed text -> stdout
71+
scrub app.log --mapping m.json # also save the reversible mapping
72+
cat app.log | scrub # read stdin, scrub, write stdout
73+
scrub --restore m.json s.txt # reconstruct the original
74+
scrub --check app.log # report secrets (stderr); exit 1 if any
4875
```
4976

50-
Paste an artefact (try [`scrub/sample.log`](scrub/sample.log)), hit **Scrub**, review the
51-
colour-coded detections, toggle anything you want to keep in the clear, and copy the
52-
result. See [`scrub/README.md`](scrub/README.md) and [`ui/README.md`](ui/README.md) for
53-
details, the optional full-Presidio mode, and tests.
77+
| Flag | Mode | Meaning |
78+
| --- | --- | --- |
79+
| `FILE…` | all | Input file(s). Omit or use `-` to read stdin. Multiple files are only meaningful with `--check`. |
80+
| `--check` | check | Detection only: print a masked report to stderr, exit 1 if any finding is at/above the threshold, else 0. Built for pre-commit hooks. |
81+
| `--restore MAPPING.json` | restore | Reconstruct the original from scrubbed input + this mapping. |
82+
| `--mapping PATH` | transform | Also write the reversible `{placeholder: original}` mapping as JSON to `PATH`. |
83+
| `-t`, `--threshold N` | transform / check | Only act on detections scoring ≥ N (default `0.6`; use `0` to scrub everything, recall over precision). |
84+
| `--allow ENTITY_TYPE` | check | Suppress an entity type, e.g. `--allow PUBLIC_IP`. Repeatable. |
85+
| `--label NAME` | check | Name to use for stdin in the report (default `<stdin>`). |
86+
| `--no-near-miss` | check | Don't print the non-blocking notice about sub-threshold detections on a passing commit. |
87+
| `--no-color` | all | Disable coloured output (colour is stderr-only and already auto-off when stderr isn't a TTY or `NO_COLOR` is set). |
88+
| `--quiet` | all | Suppress the progress spinner on slow interactive runs. |
89+
| `-V`, `--version` | n/a | Print version and exit. |
90+
91+
Running `scrub` with no input and an interactive terminal prints usage and exits 2
92+
(it won't hang waiting on stdin).
93+
94+
The contract that keeps it pipeable and safe:
95+
96+
- Scrubbed / restored text is the **only** thing on **stdout**.
97+
- Diagnostics, reports, and errors go to **stderr**, including a masked summary of
98+
low-confidence detections that were *not* applied, so nothing is silently dropped.
99+
- The reversible mapping holds the real secrets, so it is written **only** to the
100+
`--mapping` path you ask for, never to stdout.
101+
- `--check` and every report **mask** each value, so secrets never hit your
102+
scrollback or CI logs.
103+
104+
---
105+
106+
## Git pre-commit hook
107+
108+
`--check` is designed to drop into a git pre-commit hook so secrets are caught
109+
before they are committed. A ready-made hook and installers live in
110+
[`hooks/`](hooks/):
111+
112+
```bash
113+
hooks/install.sh # bash / macOS / Linux
114+
hooks/install.ps1 # Windows PowerShell
115+
```
54116

55-
## Command line + git pre-commit hook
117+
The hook blocks a commit (exit 1) on any high-confidence finding and prints a
118+
masked report; sub-threshold near-misses are noted without blocking.
56119

57-
Install the CLI (puts `scrub` on PATH):
120+
---
121+
122+
## The review UI
123+
124+
A local web UI (Next.js) for the paste → review → export flow:
58125

59126
```bash
60-
pip install -e . # from the repo root
127+
cd ui
128+
npm install
129+
npm run dev # http://localhost:3000
61130
```
62131

132+
It talks only to the local API (`http://127.0.0.1:8000`); start that with:
133+
63134
```bash
64-
scrub app.log # scrubbed text -> stdout
65-
cat app.log | scrub # pipeable
66-
scrub app.log --mapping m.json # also save the reversible mapping (holds secrets)
67-
scrub --restore m.json scrubbed.txt # reconstruct the original, byte-for-byte
68-
scrub --check app.log # masked report -> stderr; exit 1 if secrets found
135+
cd scrub
136+
uvicorn main:app --host 127.0.0.1 --port 8000
69137
```
70138

71-
**Block secrets at commit time.** The `scrub --check` mode exits non-zero when it finds
72-
a secret, which lets a git pre-commit hook abort the commit:
139+
What the UI gives you:
140+
141+
- **Paste or load files:** drag-and-drop or pick multiple files; each opens in its
142+
own tab. Files are read in the browser; only the text reaches the localhost API.
143+
- **Review with line numbers:** the scrubbed output is line-numbered; click a
144+
detection to scroll to and highlight that line. A gutter "minimap" shows where in
145+
the artefact data was scrubbed.
146+
- **Diff view:** toggle to see the original (with highlights) beside the scrubbed
147+
output.
148+
- **Per-detection control:** keep/dismiss individual detections, filter/search the
149+
list, and move a confidence-threshold slider; the export updates instantly.
150+
- **Custom recognisers:** add your own regex patterns at runtime (in memory only).
151+
- **Restore round-trip:** reconstruct the original from the in-memory mapping to
152+
confirm the scrub is reversible.
153+
154+
API endpoints (all localhost-only): `POST /scrub`, `POST /restore`,
155+
`GET /entities`, `GET /health`, and `GET/POST/DELETE /recognizers` for the custom
156+
recogniser set.
157+
158+
---
159+
160+
## How it works
161+
162+
- **Stable, reversible pseudonymisation.** Every occurrence of the same value gets
163+
the same placeholder, so the scrubbed artefact still reads as a coherent trace;
164+
the kept mapping reconstructs the original byte-for-byte.
165+
- **Context-aware scoring.** In the default (custom-pack-only, no-spaCy) mode, a
166+
lightweight character-window check raises a detection's score when the
167+
recogniser's context words are nearby, so a real AWS secret next to
168+
`aws_secret_access_key` clears the threshold while a bare 40-char blob does not.
169+
- **Validated, not just matched.** IPs and credit-card numbers are validated in
170+
code (stdlib `ipaddress`, Luhn) rather than trusted from a regex.
171+
- **Local-first.** No network calls in the scrub path; no telemetry.
172+
173+
---
174+
175+
## Optional: full Presidio mode
176+
177+
The default mode runs the custom pack only, no spaCy model needed. To also get
178+
Presidio's built-in human-PII recognisers (`PERSON`, etc.) on top:
73179

74180
```bash
75-
./hooks/install.sh # copies hooks/pre-commit into this repo's .git/hooks
181+
python -m spacy download en_core_web_lg
76182
```
77183

78-
…or, for [pre-commit](https://pre-commit.com) framework users, the repo ships a
79-
[`.pre-commit-config.yaml`](.pre-commit-config.yaml) (`pre-commit install`). Reports are
80-
**masked** so secrets never hit your scrollback, and the hook is an honest safety net which is
81-
bypassable with `git commit --no-verify`. Full CLI + hook docs in
82-
[`scrub/README.md`](scrub/README.md).
184+
```python
185+
Scrubber(use_nlp_engine=True) # security pack + built-in PII
186+
```
83187

84-
## Security & CI
188+
For the API, set `SCRUB_USE_NLP=1` before launching `uvicorn`. If the model isn't
189+
installed, Scrub fails fast with the exact `spacy download` command, it never
190+
silently degrades.
191+
192+
---
193+
194+
## Architecture
195+
196+
- **`security_recognizers.py`:** the detection pack (Presidio recognisers +
197+
validated IP/credit-card logic).
198+
- **`pseudonymizer.py`:** turns detections into stable placeholders and back.
199+
- **`scrubber.py`:** ties detection + pseudonymisation behind one `Scrubber`
200+
class, and holds any runtime custom recognisers.
201+
- **`context_enhancer.py`:** spaCy-free context scoring for custom-only mode.
202+
- **`cli.py`:** the command-line interface.
203+
- **`main.py`:** the localhost API used by the review UI.
204+
- **`ui/`:** the Next.js review UI.
205+
206+
---
207+
208+
## Known limitations
209+
210+
- Detection is **pattern + context based**, not semantic. Novel, obfuscated, or
211+
malformed secrets may not match, by design, the pack favours precision on the
212+
high-confidence patterns and leans on context for the ambiguous ones.
213+
- At low thresholds, bare values can be mislabelled (the long tail favours recall);
214+
raise `--threshold`, or use the review step, to tighten this.
215+
- The pack is **not exhaustive:** it targets the identifiers that actually leak in
216+
security/infra work, not every possible secret format.
217+
- **Review before export is the safety net.** Scrub surfaces what it found (and what
218+
it nearly found); a human still decides what is safe to share.
219+
220+
---
221+
222+
## Testing
223+
224+
```bash
225+
cd scrub && pytest
226+
```
85227

86-
Dependency advisories are triaged and documented in [`SECURITY.md`](SECURITY.md) (a
87-
security tool shouldn't ship untriaged vulns), and [`.github/workflows/ci.yml`](.github/workflows/ci.yml)
88-
runs the Python tests, the UI build/type-check, and a dependency audit gated at `high` on
89-
every push and PR.
228+
---
90229

91-
## No persistence of secrets
230+
## License
92231

93-
The mapping that reverses a scrub holds the real secrets. It is returned by the API for
94-
the caller to hold and is kept **in memory only** in the UI. It is never written to disk,
95-
`localStorage`, or anywhere else. The repo's `.gitignore` also excludes
96-
`*.mapping.json`. Treat any mapping you do save like the credentials it contains.
232+
MIT
97233

98-
## Roadmap (not built yet)
234+
---
99235

100-
Format-preserving pseudonymisation (fake-but-valid IP/MAC), an encrypted vault for the
101-
mapping at rest, and screenshot OCR + visual redaction. Any future LLM recogniser must be
102-
a **local** model (e.g. Ollama) and opt-in, the local-first promise is non-negotiable.
236+
*This README reflects scrub v0.3.0. The detected-entity list and CLI flags are the
237+
two things most likely to drift; keep this section honest as recognisers are added.*

‎SECURITY.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,11 @@ how each was handled, states the threat model, and explains how to report a prob
1515
in the Python core (`scrub/`), the CLI, the `--check` scanner, or the review UI's
1616
scrubbing flow. No telemetry, no analytics, no "phone home". The browser UI talks only
1717
to the local API; CORS is restricted to `127.0.0.1:3000` / `localhost:3000`.
18+
- **Custom recognisers stay local and in-memory.** The API's `/recognizers` endpoints
19+
let the UI register user-supplied regex recognisers at runtime. These are held in
20+
memory only (never persisted, gone on restart) and the regex is compiled on the
21+
local, single-user service, a pathological pattern is a self-inflicted local cost,
22+
not a remote-exploitable surface, since the API is localhost-bound and CORS-locked.
1823
- **Secrets are never emitted unmasked.** The CLI writes the reversible mapping (which
1924
contains the real secrets) only to the path you pass via `--mapping`, never to stdout.
2025
The `--check` scanner and the git pre-commit hook mask every value in their reports so

0 commit comments

Comments
 (0)