Skip to content

Bridge schema: pydantic models + validate-before-write for the 3 report shapes #67

Description

@LegalMarc

Design doc (read in full first): docs/design/bridge_schema_migration.md — especially "Current State Inventory" §4 and "Migration Plan" step 1, plus the "Cancellation/Deadline and Transactional-Write Interaction" section (its constraints are hard requirements, not notes).

This is step 1 of the bridge-schema migration (the only step in scope here). Steps 2–4 in the design doc (Swift Decodable readers, return-tuple schemas, progress-callback schemas) are explicitly out of scope and must not be started here.

Scope

Python-only. Define pydantic models for the three on-disk report shapes and call .model_validate() immediately before each write, so a malformed report fails loudly in Python instead of being silently guessed at on the Swift side.

The three shapes and their current write boundaries (re-anchored to current main):

  1. Audit report — built and written by write_report() (report.py:50, which calls write_json_file() at report.py:83 and :97); invoked from the transactional path at pipeline.py:1641.
  2. Scrub / metadata report — built by _build_scrub_report() (pipeline.py:3376), written via write_json_file(scrub_report_temp_path, report) at pipeline.py:1605 and :1625.
  3. Failure report — built and written by _write_failure_report() (pipeline.py:1782, write at :1803).

Define one pydantic.BaseModel per shape (suggest a new marcut/report_schema.py or a section of report.py). Validation happens on the in-memory dict/model before it is handed to write_json_file() / write_report().

Non-negotiable safety requirements (from the design doc)

  • Validate before the T7 temp write, not after. For the scrub report, the model_validate() call must sit immediately before write_json_file(scrub_report_temp_path, ...) (pipeline.py:1605/:1625) — i.e. before anything reaches _replace_existing_temp() (pipeline.py:1273). Validating a report that has already been atomically promoted to its final path is the same bug with extra ceremony. Same rule for the audit report before write_report(...) at pipeline.py:1641.
  • A validation failure must raise through the existing RedactionError / _cleanup_temp_artifacts() path (pipeline.py:1764, _cleanup_temp_artifacts at :1278), surfacing as the existing ARTIFACT_FINALIZE_FAILED error code (pipeline.py:1670). Do not introduce a new, unclassified error type or let a raw pydantic.ValidationError escape uncaught.
  • The failure report's message and technical_details fields must stay free-form str (not enums, not constrained patterns). The deadline/timeout error classifier greps them for the substrings "timeout"/"deadline" at pipeline.py:2048 to produce AI_PROCESSING_TIMEOUT. Over-constraining these fields would silently break that classifier — add a test that a deadline-style message still classifies correctly after validation.
  • Do NOT touch MARCUT_PROCESSING_DEADLINE_MONOTONIC parsing in cancellation.py. Its "absent or malformed = no deadline" fail-open contract must be preserved; strict pydantic validation would invert it. Env-var JSON blobs (Inventory §1) are out of scope for this issue entirely.
  • Models should mirror the current dict shapes exactly (including optional keys like warnings/suppressed/settings on the audit report). Use extra="forbid" only if you first confirm no caller adds ad-hoc keys; otherwise prefer being permissive on unknown keys but strict on the known ones — the goal is to catch shape drift, not to reject today's valid reports.

Acceptance criteria

  • Three pydantic models defined, one per report shape.
  • .model_validate() called before each of the three write boundaries listed above (audit, scrub-temp, failure).
  • A malformed dict fed through each model is rejected before crossing the write boundary — one schema-violation test per model asserting this (3 tests min).
  • Validation failure on the scrub/audit path raises RedactionErrorARTIFACT_FINALIZE_FAILED, cleans up temp artifacts, and leaves no partial file at the final path (test this).
  • A failure report whose message contains "deadline"/"timeout" still classifies as AI_PROCESSING_TIMEOUT after validation (regression test).
  • Existing tests/test_pipeline.py and tests/test_cli.py stay green with zero edits beyond what a schema-conformant report legitimately requires.

Verification gate (all PRs in this set)

  • PYTHONPATH=src/python python3 -m pytest -q
  • swift build --package-path src/swift/MarcutApp
  • swift test --package-path src/swift/MarcutApp
  • bash scripts/release_preflight.sh

Dependencies

None. This issue blocks the rationale-data-layer issue (which adds a rationale field onto the audit-report model defined here).


Slice of the pre-public-beta refactor campaign. Land as its own PR; do not combine with other issues in this set. The design-spike tickets (#25/#26/#32) were for producing the docs — do not reopen them.

Metadata

Metadata

Assignees

No one assigned

    Labels

    afkAutonomous workflow-loop eligible

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions