Repository navigation
Add a custom word list to auto redact - #15
Conversation
tw4
left a comment
There was a problem hiding this comment.
Thanks a lot for this, it looks really good and the tests are great. I left a few small comments. The first two would be good to fix before merging, the others are optional. I'll do the tr.ts strings myself, so you can leave those.
CI didn't run yet, I'll approve the workflow.
| ))} | ||
| <label className="field"> | ||
| <span>{t.settings.customWords}</span> | ||
| <textarea |
There was a problem hiding this comment.
Small one, easy to miss since you couldn't run the app: the textarea isn't styled anywhere, styles.css only covers .field input and select. So it'll probably show up as a white default box in dark mode. Could you add it to that rule in styles.css?
input[type="number"], .field input, .field textarea, select {and maybe .field textarea { resize: vertical; } too.
| fn custom_entries(words: &[String]) -> Vec<Vec<String>> { | ||
| words | ||
| .iter() | ||
| .map(|entry| entry.split_whitespace().map(str::to_lowercase).collect::<Vec<_>>()) |
There was a problem hiding this comment.
I think the entries should go through bare() too, like the OCR words do. Right now something like Acme Corp. or @acme wouldn't match anything. Maybe something like this:
.map(|entry| entry.split_whitespace().map(bare).filter(|w| !w.is_empty()).collect::<Vec<_>>())A small test for it would be nice if you have time.
| let mut out = Vec::new(); | ||
| // Longer phrases first, so "Acme Corp" is one box rather than "Acme" and "Corp" apart. | ||
| let mut entries: Vec<&Vec<String>> = entries.iter().collect(); | ||
| entries.sort_by_key(|entry| std::cmp::Reverse(entry.len())); |
There was a problem hiding this comment.
Optional: this sorts the entries again for every line, so it could be done once in custom_entries instead. Not a big deal though, fine either way.
| } | ||
|
|
||
| /// An OCR word without surrounding punctuation, lowercased. | ||
| fn bare(text: &str) -> String { |
There was a problem hiding this comment.
Just a note, nothing to change here: to_lowercase doesn't handle Turkish I/ı (IŞIK vs ışık). I'll open a separate issue for it.
Entries go through bare(), so "Acme Corp." or "@acme" match, and they are sorted once in custom_entries. The settings textarea gets the same style as the other fields.
|
Thanks for the review! Done in b2bb249:
I also made the browser mock clean entries the same way, so |
|
Thanks a lot, this is great! Tested it locally and everything passes. I approved CI, I'll merge once it's green and add the Turkish text after. |
What does this change?
Closes #14
Settings → Text recognition gets a Custom words box (one word or phrase per line), and auto redact covers those words as you described in #14:
-_./@:, and if a piece matches, the whole OCR word is covered:acmecatchesacme-prod-db,db.acme.local,ops@acmeandacme:443, but notpacme,acmesoftoracmeprod-db.Acme corp.matches) and is one box. Longer entries are matched first, so "Acme Corp" and "Acme" in the same list don't split the phrase.acme-prod) matches an OCR word that is exactly that, not one that only contains it (acme-prod-db), since the split is on the OCR word, not the entry. Easy to change if you'd rather have it match there too.How it's wired:
redact.rs: a newCategory::CustomWord(custom-word) andcustom_words: Vec<String>onCategories, with#[serde(default)]so settings saved before the list still load.Categoriesis no longerCopy;detectalready took it by value, solib.rsdidn't change.autoRedact.customWordsinstore.ts(default[]),RedactCategoriesand theSensitiveHitunion inipc.ts, thendetect_sensitive→redact::detect(). The box keeps the lines as typed and the Rust side ignores blank ones.browserMock.ts:detect_sensitivealso returnscustom-wordhits for single words in the fake OCR, so the box can be tried withnpm run dev.en.tsandtr.tsgetcustomWordsandcustomWordsHint.tr.tshas the English text for you to translate, as you suggested.docs/ARCHITECTURE.md: theredact.rsline mentions custom words. To keep the box the same width I shortened "Luhn-checked cards" to "Luhn cards" there.How was it tested?
npx tsc --noEmitnpm test: I couldn't run Vitest on the Linux machine I used (its CPU is too old for the native bundler binary and it stops with "Illegal instruction"). The only frontend logic change is types, the store default and the settings box;tr.tshas the same two keys asen.ts, soi18n.test.tsshould pass, but CI is the real check here.cargo clippy --all-targets -- -D warningsandcargo test(insrc-tauri), on Linux in Docker with the packages fromci.yml: clippy is clean, 41 tests pass, including the two tesseract fixture tests.npm run tauri dev): not done, no desktop on that machine.New tests in
redact.rs: a single word, case-insensitive and with punctuation around it; separator pieces; near misses (pacme,acmesoft,acmeprod-db); a multi-word entry as one box, and the same first word alone not matched; a longer entry winning over a shorter one; blank entries ignored; an email not counted again as a custom word; and settings JSON withoutcustomWordsstill parsing.Screenshots
None: I couldn't run the app with a screen. The new field is a plain
textareainside the existingfieldlabel, under the four toggles.