fix(dashboard): warn on the first failed Paperless-ngx probe

[admonstrator]

Background

The "Paperless-ngx unreachable" dashboard banner was wired to scanner.degraded alone. isDegraded() requires failureThreshold (default 3) consecutive failed scan runs, so with the default SCAN_INTERVAL of */30 the warning appeared 90 minutes into an outage at the earliest — and never at all in three common cases:

• The startup retry loop (runInitialScanWhenReachable) recorded connectivity but never called recordRunResult(), so giving up on the initial scan left consecutiveFailures at 0.
• With DISABLE_AUTOMATIC_PROCESSING=yes, isDegraded() returns false by definition, so no outage could ever raise the banner.
• checkConnection() only ran inside the scan loop, so an outage starting between two scan ticks stayed invisible until the next tick.

recordConnectivity() additionally collapsed reachable && authorized into a single reachable flag, so a rejected API token was reported as "Paperless-ngx is not reachable" — pointing users at the network instead of their credentials.

Changes

• scanHealthService: store reachable, authorized and usable separately; add clearConnectivity() for the not-yet-configured case. usable carries the old meaning of reachable.
• server.js: add startConnectivityMonitor(), a standalone probe armed next to (not inside) startScanning() so it also runs with automatic processing disabled. It records connectivity only and never touches the failure counter, so a passive probe cannot push /health to 503. Interval via PAPERLESS_PROBE_INTERVAL_SECONDS (default 60, min 10, 0 disables); the timer is unref'd and skips while a scan is running.
• server.js: count an abandoned initial scan as exactly one failed run. Counting every retry would trip the degraded threshold within minutes and break the startup resilience added in [Bug]: Scanner permanently aborts after a single startup connection failure #272.
• dashboard: show the banner as soon as paperless.usable === false, styled as a warning, and escalate to the existing error style once the scanner is degraded. A reachable-but-rejected probe now reads as a credentials problem.
• Add the missing .theme-alert-warning style (already referenced by views/settings.ejs, never defined).
• Expose authorized, usable and status on PaperlessHealth and regenerate OPENAPI/openapi.json.
• Bump PAPERLESS_AI_VERSION to v2026.08.02 and add the changelog entry.

Note for API consumers: paperless.reachable now means "the host answered" and is true for a 401/403. Use paperless.usable for the previous semantics.

Testing

• New offline test paperless-unreachable-banner (13 assertions). It executes the shipped banner code from dashboard-scripts.ejs in a VM against a DOM stub rather than re-implementing it, and covers the outage, rejected-token, automatic-processing-disabled, degraded and never-probed cases.
• Extended scanner-startup-resilience for the new connectivity fields, clearConnectivity() and the guarantee that probes leave the failure counter alone.
• node scripts/run-tests.js --all: 51 passed, 0 failed, 7 skipped (server-dependent / pdftoppm missing).
• Manual: ran the server against a dead port and against a fake Paperless returning 401. /health reported usable:false, reachable:false and usable:false, reachable:true, status:401 respectively — both with DISABLE_AUTOMATIC_PROCESSING=yes and degraded:false, the case that produced no warning before.
• ESLint and Prettier clean on all touched files; OpenAPI regenerated.

Impact

Outages and bad credentials surface within one probe interval instead of three scan intervals or never. /health semantics and the degraded threshold are unchanged, so the Docker healthcheck keeps tolerating transient failures.

Upstream Status

Not applicable — the banner and scanHealthService are next-only.

---

Also in this PR: automatic OCR processing (closes #260)

Independent of the banner fix above, this PR also adds the feature requested in #260.

Background

The scan loop and OCR are coupled only through the ocr_queue table. scanDocuments() fills the queue in two places — server.js:627 (content shorter than MIN_CONTENT_LENGTH) and server.js:680 (AI failure matching one of the OCR-relevant error markers) — but nothing ever empties it. Repo-wide, processQueueItem() was called from exactly two places, both in routes/setup.js: the single-document endpoint and the "Process All Pending" batch endpoint. There was no cron, no worker, no auto-drain.

For users running a local vision model against handwritten documents — where the Paperless-ngx OCR engine produces nothing usable — the OCR path is the only one that works, and it had to be started by hand after every scan.

Changes

• New services/ocrAutoProcessService.js. drainQueue() takes up to OCR_AUTO_PROCESS_BATCH_SIZE pending items and runs each through the existing mistralOcrService.processQueueItem(), which already does download → OCR → write-back → optional AI analysis. A run is skipped while another is in flight, and a per-item try/catch keeps one bad document from aborting the batch. No retry logic: a failed item is no longer pending, so the next run does not pick it up.

• Connectivity probe before touching anything. processQueueItem() records every error as a terminal failure (updateOcrQueueStatus('failed') + addFailedDocument), so an unreachable or unauthorized Paperless-ngx would otherwise burn the whole queue in a single run and lock those documents behind isDocumentFailed() until someone reset them by hand. This is the one behavioural difference to the manual path, where a human simply does not press the button during an outage.

• Armed as a third cron in startScanning(), next to the scan and reconciliation jobs, placed after the DISABLE_AUTOMATIC_PROCESSING return — OCR + AI writes back to Paperless-ngx, which is exactly what that kill-switch is meant to stop. Each tick skips while scanControl.running is set, so the drain never competes with a scan for the same AI backend.

• Four config keys, all editable under Settings → OCR:

  ENV	Default	Meaning
  OCR_AUTO_PROCESS_ENABLED	no	Work through the queue on a schedule
  OCR_AUTO_PROCESS_INTERVAL	*/15 * * * *	Cron expression
  OCR_AUTO_PROCESS_BATCH_SIZE	10	Documents per run (clamped 1–100)
  OCR_AUTO_ANALYZE	yes	Run AI analysis right after OCR

  OCR_AUTO_ANALYZE defaults to yes because a rejected content write-back leaves the OCR text local-only, where a regular scan would never see it.

• Cron validated in both directions: POST /settings answers 400 on an invalid expression, and the service getter falls back to the default with a warning, so a bad value that reached the process some other way cannot crash cron.schedule() during startup.

• Settings UI: new "Automatic Processing" block in the OCR tab mirroring the PDF-render block — a settings-switch plus a sub-container that hides the schedule and batch size while automation is off. Wired into fieldMappings so the "Overwritten" / "Managed by ENV" pills work.

• @swagger properties for the four new POST /settings fields, spec regenerated. README and docker-compose.yml document the keys.

README.md was not Prettier-clean before this change, so touching it required a formatting pass. The table-width and quote-style hunks in it are cosmetic and unrelated to this feature.

Testing

• New offline test ocr-auto-process (10 cases, 24 assertions): the global MISTRAL_OCR_ENABLED switch winning over the automation switch; an unreachable Paperless-ngx and a rejected token both leaving the queue completely untouched; only pending being requested; the batch limit and its clamping; the invalid-cron fallback; a failing document not aborting the batch; autoAnalyze being forwarded; and the concurrency guard.
• node scripts/run-tests.js --all: 52 passed, 0 failed, 7 skipped.
• Smoke-tested the service getters against the real config module; settings.ejs compiles; ESLint and Prettier clean on every changed file.

Impact

Off by default, so nothing changes for existing installs. Once enabled, queued documents are OCR'd and analysed without anyone opening the OCR page. The manual "Process All Pending" flow is untouched. During a Paperless-ngx outage the queue stays pending instead of being marked failed — a strict improvement over what the manual batch endpoint does today.

Upstream Status

Not applicable — the OCR queue and its UI are next-only.

---

Also in this PR: Changelog tab in Settings

Background

config/changelog.js has carried every release block since v2026.05.01, but only the newest one was ever reachable: /api/changelog/status exports latestRelease, and the What's New modal shows it exactly once per release before marking it seen. After dismissing the modal there was no way to read the notes again — and no way at all to see what earlier releases changed without opening the file or GitHub.

Changes

• config/changelog.js additionally exports releases — every block, newest first, with each entry split into a category and its text by the new categorizeEntry(). Entries are authored as New:, Fix:, Improvement:, Removed: or Security:; anything else (for example the prefix-less "Removed RAG features…" line from v2026.05.01) falls back to a note with the text intact. The existing version and entries exports are untouched, so the modal endpoint keeps receiving raw strings.
• New Changelog tab rendering a timeline: one dot per release on a vertical spine, the current release expanded and marked with a Current pill, older ones collapsed. Built on <details>/<summary>, so expanding needs no JavaScript. Each entry gets a colour-coded category badge with an icon. Data is server-rendered from res.render, so there is no new API endpoint and no OpenAPI change.
• New public/css/changelog.css, loaded only by the settings page. Uses the existing CSS custom properties and adds :root[data-theme='dark'] overrides for the badge colours, which are the only hard-coded values. Below 640px the badge moves above the text and the change count is hidden.
• The tab switcher needed no change — it already resolves data-tab generically — but it now hides the save row on the changelog tab, which is read-only.

Entries may contain links (v2026.05.01 and v2026.08.01 both do), so the text renders with <%- %>. The content is authored in-repo and never user-supplied; the What's New modal already assigns it via innerHTML.

Testing

• New offline test changelog-releases (6 cases, 30+ assertions): the modal's version/entries contract still returns raw strings; releases are newest-first; every entry lands in a known category with its prefix stripped and non-empty text; the categorizer handles known prefixes, a prefix without a colon, and an unknown prefix; inline links survive.
• The same test guards that releases[0].version equals PAPERLESS_AI_VERSION, so bumping the version without adding a changelog block (or the reverse) now fails CI instead of silently leaving the modal on a stale release.
• node scripts/run-tests.js --all: 53 passed, 0 failed, 7 skipped.
• Rendered views/settings.ejs against the real changelog data: 9 release blocks, badges distributed across new/fix/improvement/removed/note, exactly one "Current" pill.
• ⚠️ Not verified visually — the browser extension was not connected, so the layout has only been checked as markup and CSS.

Impact

Additive and read-only. The What's New modal, its endpoints and the seen-tracking are unchanged.

Upstream Status

Not applicable — the settings tabs and the changelog module are next-only.

🤖 Generated with Claude Code