Harden Forge incident storage diagnostics and concurrency

README.md

@@ -56,6 +56,18 @@ forge ref <incident-id>
 # Review aggregate patterns
 forge stats
 
+# Validate the corpus; fail automation if corruption exists
+forge validate --strict
+
+# Preview repair candidates without changing malformed files
+forge repair --dry-run
+
+# Preview quarantine candidates without moving malformed files
+forge quarantine --dry-run
+
+# Explicitly move freshly verified corrupt files into quarantine
+forge quarantine --apply
+
 # Prepare a rendered analysis prompt without calling an LLM
 forge analyze --prepare-only
 ```
@@ -89,17 +101,89 @@ You can also set an `organization_name` in `config.yaml` or `config.local.yaml`
 | Command | Description |
 |---------|-------------|
 | `forge log` | Log a new incident with interactive prompts |
-| `forge list` | List incidents with `--project`, `--severity`, `--since`, `--tag`, structured-axis, and `--limit` filters |
+| `forge list` | List incidents with filters and report valid-corpus, corrupt-corpus, matched, and returned counts |
 | `forge show <id>` | Show full details of one incident; suffix matches like `forge show 001` work |
 | `forge ref <id>` | Print a Proofhouse `IncidentRef` compatibility projection as JSON |
 | `forge edit <id>` | Open an incident in your editor |
-| `forge stats` | Show aggregate counts by severity, type, project, platform, and tags |
+| `forge stats` | Show aggregate counts with explicit corpus and matched-count diagnostics |
+| `forge validate [--strict]` | Report safe relative-path corruption diagnostics; `--strict` exits nonzero for corrupt files |
+| `forge repair [--dry-run]` | Preview corrupt-file repair candidates; always read-only and never rewrites files |
+| `forge quarantine [--dry-run\|--apply]` | Preview corrupt-file candidates by default; `--apply` moves freshly revalidated corrupt regular files into quarantine |
 | `forge playbook` | List playbook entries |
 | `forge playbook show <name>` | Show one playbook entry |
 | `forge analyze` | Run LLM-backed pattern analysis |
 | `forge analyze --prepare-only` | Save the fully rendered analysis input without calling an LLM |
 | `forge mcp serve` | Run Forge as a Streamable HTTP MCP server with local-only defaults |
 
+Read commands treat a missing `incidents/` directory as a valid, empty corpus and
+do not create it. `forge validate --strict` therefore succeeds for a missing
+directory; it fails only for corrupt files or operational scan errors. Corpus
+reads reject file and directory symlinks. Incident ordering is deterministic:
+canonical `YYYY-MM-DD-NNN` IDs use their date and numeric sequence, while legacy
+IDs use timestamp and relative-path fallback ordering. Lists are newest-first;
+analysis loads are oldest-first.
+
+`forge repair` remains preview-only. `forge quarantine` is also read-only by
+default, and `forge quarantine --dry-run` performs the same zero-write scan.
+`forge quarantine --apply` is the sole mutating path; it is mutually exclusive
+with `--dry-run`. Apply aborts before creating the quarantine area when the
+initial scan has operational errors, then reopens every candidate without
+following symlinks and moves only files that are still corrupt regular files.
+Destinations mirror incidents-root-relative paths under the data root's
+`quarantine/` directory. Forge durably syncs each newly inserted quarantine
+directory before descending and verifies that each opened directory descriptor
+still names the created or revalidated entry.
+
+Apply uses one descriptor-relative, atomic no-replace rename from the incident
+pathname to its visible quarantine destination: `renameatx_np` with
+`RENAME_EXCL` on Darwin and `renameat2` with `RENAME_NOREPLACE` on Linux.
+Unsupported platforms, unavailable native symbols, and filesystems that reject
+the required flag fail closed; Forge does not fall back to replacement,
+link-and-unlink, descriptor pseudo-paths, or path-based moves. A destination
+collision leaves both entries untouched. Forge verifies the moved inode and
+content, then syncs the destination parent before the source parent. Successful
+moves leave exactly the visible destination and no private recovery namespace.
+
+If the source changes in the final pre-syscall interval, Forge detects the
+replacement after the move and restores it to the source pathname with the same
+no-replace primitive. A successful durable restore reports
+`SourceChangedError`. A restore collision or post-move verification or
+durability failure may leave the recoverable candidate at either its original
+source pathname or the visible quarantine destination, depending on the failure
+sequence, while a concurrent entry may occupy the other path. Operators must
+inspect both sanitized reported paths; Forge's no-replace recovery never
+overwrites either name. If a later candidate fails
+after earlier moves, Forge reports both the moved and failed relative paths.
+
+`forge edit` copies the selected incident into a private regular-file stage and
+never passes a corpus descriptor to the editor. Forge validates the completed
+stage, then reopens the target through its verified parent-directory descriptor
+and compares device/inode identity, metadata, and bytes with the original
+snapshot before descriptor-relative atomic replacement. Any observed conflict
+fails closed and leaves the product file untouched; replacement publishes a new
+inode, so pre-existing hard links are not modified. Conflict detection is
+bounded by POSIX semantics: POSIX has no compare-and-swap rename, so a concurrent
+replacement in the final recheck-to-rename interval cannot be detected
+atomically.
+Operational storage failures during reopening, staging, publication, fsync,
+replacement, rollback, or cleanup exit nonzero with a stable error class and,
+when available, only the sanitized incidents-root-relative target. Raw storage
+exception messages and configured roots are never printed. When cleanup or
+rollback also fails, Forge reports the primary storage error class separately
+from the `RecoverablePartialStateError` marker, so cleanup never masks the
+publication failure.
+
+Publication keeps a descriptor-relative rollback hard link until the
+replacement directory is durably synced. A post-replacement directory-fsync
+failure restores the original inode when rollback succeeds. If rollback fails,
+Forge preserves the backup link and reports the primary publication error plus
+the recoverable-partial-state marker. If publication itself fails and removing
+or syncing the rollback link also fails, Forge likewise preserves or recreates
+the recovery link while retaining the primary publication classification.
+After successful publication, Forge reports success only after removing the
+backup and syncing its parent directory. Backup unlink or post-unlink
+directory-fsync failure exits nonzero because cleanup durability is unknown.
+
 ## MCP Server
 
 Forge exposes tools over the [Model Context Protocol](https://modelcontextprotocol.io) so agent tools can log and query incidents directly.
@@ -115,6 +199,12 @@ Available tools:
 - `forge_playbook_show`
 - `forge_schema`
 
+`forge_list` and `forge_stats` retain safe partial results when files are merely
+corrupt, including skipped-corrupt diagnostics. Operational scan errors instead
+return an explicit incomplete response containing sanitized corruption and scan
+diagnostics only; incident rows, total markers, and aggregate sections are
+withheld.
+
 ### Local stdio MCP
 
 Use stdio when the MCP client runs on the same machine as Forge.
@@ -241,7 +331,12 @@ If you want to inspect or paste the analysis input into another tool yourself:
 forge analyze --prepare-only
 ```
 
-This writes a dated `analysis-input` file into the active Forge data root.
+Both analysis modes require a complete incident corpus. Corrupt files or
+operational scan errors are reported with safe relative-path diagnostics, and
+analysis exits before creating an artifact or invoking a provider. A genuinely
+missing corpus remains a successful empty, read-only result. For a complete
+non-empty corpus, prepare-only writes a dated `analysis-input` file into the
+active Forge data root.
 
 ## Development