feat: harden signal analysis with research-informed review

Author: JACKNINES

CHANGELOG.md

@@ -2,6 +2,19 @@
 
 All notable user-facing changes to Signal Sieve are recorded here.
 
+## Unreleased
+
+- Hardened community verdict parsing and distinguished HTTP, authentication
+  and rewrite-configuration errors from connection failures.
+- Added explicit rewrite consent, native post-analysis and integrity checks
+  before accepting a community-engine clean copy.
+- Exposed hidden display controls in bounded external-result previews.
+- Added bounded local regularity windows and repeated-context diagnostics,
+  with research sources and evidence limits in `RESEARCH_BASIS.md`.
+- Rejected oversized surface-probe input and nonfinite display values.
+- Improved defensive handling of obfuscated scam text, ambiguous URL wrappers,
+  byte-preserving URL queries, malformed ZIP records and child-process failures.
+
 ## 0.13.0 — 2026-08-25
 
 - Made automatic Input processing honor the user's selected Safe Clean,

COMMUNITY_ENGINES.md

@@ -25,7 +25,7 @@ project's optional heavyweight harnesses implicitly.
 This is composition, not code ownership: the external project is neither
 forked nor copied into the Signal Sieve source tree.
 
-## Start the compatible service
+## Start the service (inspection first)
 
 Review the upstream source and release notes first. The shortest upstream
 installation uses Python 3.10 or newer:
@@ -52,11 +52,44 @@ replace it with a reviewed release tag and update deliberately. Signal Sieve
 does not run `git pull`, `docker pull`, package installation, or a shell command
 on the user's behalf.
 
-In Signal Sieve, paste text, choose **Community Engines**, confirm that the
-local service is ready, and then choose **Inspect with Engine** or **Create
-Community Clean Copy**. The latter does not replace Input or the clipboard. It
-shows the engine report, performs a native residual-risk analysis, and requires
-**Use as Result**.
+These commands start the service; they do **not** establish text rewrite
+readiness. In Signal Sieve, paste text, choose **Community Engines**, check the
+connection and choose **Inspect with Engine**. **Capabilities** works even
+without Input. Connection health is not a detector or rewrite readiness test.
+
+### Text cleaning requires Layer B configuration
+
+The current upstream text `/clean` route requires a Layer B rewrite strategy
+and its model/backend dependencies. A bare Python service or core image may
+answer health/inspect while rejecting clean. Follow the reviewed version's
+[upstream configuration guide](https://github.com/guillaumemeyer/watermarks-remover#configuration-env-vars-for-docker-compose)
+and configure its strategy/backend before trying **Create Community Clean Copy**.
+Signal Sieve does not install a model or silently select a remote provider.
+
+Review that configuration and acknowledge the rewrite/data-handling warning in
+the panel. The backend may change meaning, download models or send content to
+another service. Numeric loopback restricts Signal Sieve's destination, **not**
+the external engine's subsequent network activity. Use synthetic text first.
+
+Cleaning does not replace Input or the clipboard. The panel previews the
+candidate and runs all deterministic native text checks (Unicode/covert
+channels, Code Guard, binary content, URLs including private rules, UUIDs and
+scam signals), plus rewrite integrity. **Use as Result** is blocked when any
+native finding remains, analysis is incomplete, protected numbers/URLs/quotes
+change, source code is involved, or Input no longer matches the reviewed bytes.
+This conservative gate is not proof of semantic equivalence: always read the
+candidate. No personal pattern history is read or learned from this review.
+
+### Troubleshooting
+
+| Message | Meaning / next step |
+| --- | --- |
+| No service answered | Start your reviewed engine on `127.0.0.1:8765`. |
+| Rewrite configuration required | Inspect can work; configure upstream Layer B before Clean. |
+| Authentication required | This adapter does not yet support the engine's API key mode. |
+| HTTP rejection / engine error | The server answered but rejected the request; inspect its bounded error. |
+| Unknown verdict | The response supplied no supported verdict; it is not shown as green. |
+| Invalid response | Malformed or contradictory evidence was rejected. |
 
 Developers can verify the complete adapter against a running compatible service
 with a synthetic hidden-Unicode fixture:
@@ -65,10 +98,16 @@ with a synthetic hidden-Unicode fixture:
 ./community-engine-smoke-test.sh
 ```
 
-The smoke test exercises health, capabilities, inspect, clean, and Signal
+With a configured rewrite backend, the smoke test exercises health, capabilities, inspect, clean, and Signal
 Sieve's native residual-risk analysis. It fails closed and never uses clipboard
 contents.
 
+For a deliberately unconfigured Layer B service, run
+`./community-engine-smoke-test.sh --expect-rewrite-unconfigured`. This separately
+checks health, capabilities, the suspicious inspection verdict and the explicit
+configuration error. It does **not** certify cleaning or substitute for the full
+smoke test above.
+
 ### Manual GitHub compatibility run
 
 Maintainers can open **Actions → Community Engine Compatibility → Run
@@ -101,7 +140,9 @@ log for seven days and receives only read access to this repository.
 
 ## Result semantics
 
-An external report proves only what its named detector and configuration can
+The adapter supports legacy Boolean and structured `suspicious.verdict`
+responses. Missing/null verdicts remain unknown; invalid or contradictory
+structured evidence fails closed. An external report proves only what its named detector and configuration can
 support. A result is not proof of authorship, AI generation, provider identity,
 or malicious intent. Provider-keyed and learned watermarks remain untestable
 unless the matching detector, keys or validation material, tokenizer, and

FORMAT_SECURITY.md

@@ -5,6 +5,16 @@ structures, writes a new file, reopens it, reanalyzes it, and confirms the
 source remained byte-for-byte unchanged. Unknown, malformed, signed, encrypted,
 or ambiguously linked structures fail closed.
 
+ZIP-based documents validate every local header against its central record
+before inspection or rebuilding, including entries scheduled for removal.
+Names, flags, methods, CRC/size declarations and streaming descriptors must
+agree; local ranges cannot overlap or enter the central directory. Split-disk
+counts, symlink entries, traversal/alias paths and malformed UTF-8 names marked
+as UTF-8 are refused. Both signed and unsigned 32-bit streaming descriptors are
+supported. This conservative policy follows the record structures in the
+[PKWARE ZIP specification](https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT).
+Structural agreement does not prove the document body is benign.
+
 | Format | Structural inspection | Conservative cleaning |
 | --- | --- | --- |
 | WebP | RIFF `C2PA`, `EXIF`, `XMP ` chunks | Rebuild RIFF, clear EXIF/XMP VP8X flags, preserve image/animation/ICC chunks |

PIXEL_MODULES.md

@@ -16,6 +16,12 @@ the contract to preserve the same staging and verification boundary.
 
 ## Module folder
 
+Runtime arguments must be finite and greater than zero, up to 300 seconds.
+The bridge uses monotonic deadlines, discards stderr, bounds response reads,
+and forcibly stops the directly launched process on deadline or output-budget
+violations. This includes a module that ignores SIGTERM. Supervision is not an
+OS sandbox, a hard disk quota, or containment of a module's descendants.
+
 The selected folder must contain `signalsieve-pixel-module.json` and a regular,
 non-symlink executable below that same folder. Manifest schema version 1 is:
 

README.md

@@ -299,9 +299,17 @@ they contain no real account, organization, or third-party message data.
   address `127.0.0.1:8765`. The app supports its documented health,
   capabilities, text inspection, and clean-copy routes; it never downloads,
   launches, or updates the external project. Cleaned text is reanalyzed by
-  Signal Sieve before the user may place it in Result. Signal Sieve remains
+  Signal Sieve, including native risk checks and rewrite integrity, before the
+  user may place it in Result. Current upstream text cleaning requires a
+  configured Layer B rewrite backend; a healthy connection alone is not enough.
+  Missing verdicts remain unknown, never green. Signal Sieve remains
   fully usable without the service. Setup, update, licensing, and trust details
   are in [COMMUNITY_ENGINES.md](COMMUNITY_ENGINES.md).
+
+  Research-informed improvements include escaped community-engine previews,
+  bounded local regularity windows and repeated-context warnings. These are
+  defensive review aids, not provider watermark verdicts. See the three papers,
+  implementation choices and limitations in [RESEARCH_BASIS.md](RESEARCH_BASIS.md).
 - **File Provenance Inspector:** performs a bounded, read-only scan for C2PA
   container markers and common EXIF/XMP or document metadata in PNG, JPEG,
   WebP, AVIF, HEIC/HEIF, BMP, GIF, TIFF, SVG, PDF, DOCX, XLSX, PPTX, EPUB, ODT,

RESEARCH_BASIS.md

@@ -0,0 +1,93 @@
+# Research-informed signal hardening
+
+This note connects three scientific papers to specific Signal Sieve changes.
+These are defensive engineering adaptations, not reproductions of the papers'
+benchmarks or claims of a universal watermark detector. All examples are
+synthetic and native analysis stays offline.
+
+## 1. Invisible text and misleading displays
+
+**Boucher et al., _Bad Characters: Imperceptible NLP Attacks_ (IEEE S&P 2022).**
+[Paper](https://arxiv.org/abs/2106.09898)
+
+The paper studies invisible characters, homoglyphs, reordering and deletion
+controls that can cause differences between displayed and processed text.
+
+**Engineering application:** community reports, errors and candidate previews
+now expose invisible/control scalars as `⟦U+XXXX⟧`. This includes backspace,
+delete, escape and carriage return rather than interpreting their display
+effects. Previews are bounded to 8,000 scalars and visibly indicate truncation;
+analysis and explicit candidate acceptance continue to use the original bytes.
+Ordinary Arabic/Hebrew letters, newlines and tabs are preserved in previews.
+
+This is a display safeguard, not a complete Unicode confusable detector or
+proof that a flagged character was inserted maliciously. Context-sensitive
+emoji/script cleaning policy is unchanged.
+
+## 2. Do not hide local signals inside document averages
+
+**Kirchenbauer et al., _On the Reliability of Watermarks for Large Language
+Models_ (ICLR 2024).** [Paper](https://arxiv.org/abs/2306.04634)
+
+The paper examines edited and mixed-origin documents. Its WinMax detector
+uses keyed token evidence over spans and requires false-positive calibration
+for the multiple-window search.
+
+**Engineering application:** Surface Regularity now examines overlapping
+180-word windows, normally 90 words apart, and shows up to three non-overlapping
+repetitive regions by word offset. It uses existing repetition/diversity
+features. At most 256 windows are examined; longer documents use evenly spaced
+starts including the first and last, with an explicit sampled-coverage notice.
+
+This is **not WinMax**: there are no keyed green-list hits, z-tests or calibrated
+p-values. Local maxima never increase the document's global heuristic score.
+A highlighted region is a review clue; no highlight does not establish absence
+of a watermark. Word offsets refer to the app's lexical segmentation.
+
+## 3. Repeated text is not independent evidence
+
+**Dathathri et al., _Scalable watermarking for identifying large language model
+outputs_ (Nature 2024).**
+[Paper](https://www.nature.com/articles/s41586-024-08025-4)
+
+SynthID-Text uses keyed context-dependent sampling and scoring; the paper also
+describes repeated-context masking. Its measurement conditions cannot be
+replaced by word count or surface regularity alone.
+
+**Engineering application:** the probe reports distinct four-word contexts
+against all four-word context positions. Below 50% distinct contexts it warns
+that repetition dominates the sample. This 50% threshold is a product heuristic,
+not a threshold from the paper or a calibrated estimate of effective sample
+size. Duplicating a paragraph increases length without manufacturing new
+contextual diversity. Words here are not the provider's model tokens.
+
+No SynthID key, tokenizer, detector model, or vendor attribution is supplied by
+this change. Short or repetitive human writing is not labelled AI-generated.
+
+## Defensive validation
+
+`ResearchSignalHardeningTests.swift` covers display controls and bounded
+previews, a repetitive island inside a diverse document, duplicated contexts,
+sampled-window coverage, oversized input and nonfinite display values.
+`CommunityCleanReviewTests.swift` and `CommunityWatermarkServiceTests.swift`
+cover the separate community-adapter trust boundary and acceptance checks.
+
+The probe refuses input above the shared 1 MiB UTF-8 budget before analysis.
+Statistical signals remain separate from exact Unicode findings. Future
+provider detection needs independently reviewed keys/configuration, appropriate
+negative controls, language/domain coverage and measured false-positive rates.
+
+### Validation record — 2026-09-06
+
+- 321 Swift tests passed, including 27 focused community/probe/research tests.
+- All 47 native test groups passed, including Apple Vision OCR.
+- The development quality gate passed packaging, warnings, privacy checks,
+  linkage and ad-hoc signature verification. This is not notarization.
+- Real upstream adapter preflight passed against
+  [`d9e9590d94e19b39eb2794266292324bfec8249a`](https://github.com/guillaumemeyer/watermarks-remover/commit/d9e9590d94e19b39eb2794266292324bfec8249a):
+  health, capabilities, structured suspicious verdict and HTTP 400 missing-
+  rewrite configuration. The service was bound to loopback, used synthetic
+  text and was stopped after the check. No rewrite model was downloaded.
+- A real configured Layer B clean output was **not** tested in this pass.
+  The preflight explicitly reports that limitation; it is not a watermark
+  removal benchmark or an end-to-end rewrite certificate.

SECURITY.md

@@ -10,6 +10,20 @@ public issue. Provide the smallest synthetic reproduction possible.
 
 ## Security boundaries
 
+The [September defensive audit](docs/DEFENSIVE_AUDIT_2026-09-06.md) documents
+reproduced bypasses, corrections, regression coverage, and remaining limits.
+Scam analysis inspects overlapping 64,000-character windows throughout the
+input and retains the strongest window's bounded evidence. It does not combine
+unrelated distant phrases into a synthetic threat. Compatibility normalization
+and default-ignorable removal are comparison-only operations on brand words;
+the original copied text is preserved. These signals are heuristic review aids.
+
+External text and pixel modules require finite positive timeouts, use monotonic
+deadlines, discard stderr, and bound retained response bytes. Deadline or output
+violations forcibly stop the directly launched process with a bounded wait.
+PDF, Ollama, and Community Engine failure paths use the same bounded stop.
+This is process supervision, not a sandbox or a resource quota on descendants.
+
 - Text processing, Code Guard, Binary Guard, Vaccine, and OCR are local.
 - Surface Regularity is a local, deterministic stylometric triage screen. Its score is not a
   probability and must never be presented as confirmation of AI generation,

Sources/SignalSieve/App/ContentView.swift

@@ -137,6 +137,7 @@ struct ContentView: View {
             CommunityEnginesView(
                 text: model.input,
                 language: model.language,
+                customRules: model.privateRules,
                 onUseResult: { model.output = $0 }
             )
         }