SSA pipeline quality hardening (Phases 1-6)

docs/CHANGELOG-SSA.md

@@ -0,0 +1,47 @@
+# CHANGELOG — SSA pipeline
+
+Scoped changelog for the Site Safety Audit pipeline and its quality
+guards. Newest at top.
+
+## 2026-05-14
+
+### Added
+
+- **Deterministic post-pass** on `build_ssa_report_docx` output. Two
+  runs over the same inputs now produce byte-identical docx files.
+  Lifted from rpd-ssa-builder. See `pims/services/ssa_quality/determinism.py`.
+- **OOXML schema validation** on every report docx via Microsoft's
+  OpenXmlValidator (wrapped in a small .NET CLI reused from
+  rpd-ssa-builder's cache at `%LOCALAPPDATA%\rpd-ssa-validator\`).
+  Errors surface on the diagnostics dict and as log warnings; the
+  build does not abort on findings. Soft-skips when dotnet is
+  unreachable.
+- **Headless LibreOffice render smoke test** after each successful
+  full or from-state build (CLI-main level — does not run during
+  build_ssa_report_docx unit tests). Catches "python-docx accepts
+  it but LibreOffice / Word won't render it" defects that the
+  schema validator may miss. Soft-skips when soffice is not installed.
+- **`--check` preflight CLI flag** on `run_ssa_pipeline.py`. Verifies
+  folder name pattern, Evidence_Master.csv presence and shape,
+  images in folder root, template readability, and environment
+  tool availability (LibreOffice, dotnet, ANTHROPIC_API_KEY).
+  Hard failures return exit 1; informational warnings don't block.
+- **Drift checker** at `tools/check_ssa_drift.py` for paired
+  constants across SSA modules. v1 covers the one drift case
+  introduced when preflight duplicated `_REQUIRED_HEADER` as
+  `_EXPECTED_CSV_HEADERS`.
+- **Byte-hash regression test** in `tests/test_ssa_determinism.py`
+  pins the SHA-256 contract for a fixture build so future
+  formatting drift on the docx output is caught by CI.
+
+### Fixed
+
+- **`<w:tblBorders>` schema-order bug** in `_apply_all_cell_borders`
+  (`pims/services/ssa_pipeline.py:1089`). The function was emitting
+  `<w:tblBorders>` at the end of `<w:tblPr>` via plain `.append()`;
+  the OOXML schema places `tblBorders` at position 11, before
+  `<w:tblLook>` (position 15) which every template table already
+  carries. Tripped four `Sch_UnexpectedElementContentExpectingComplex`
+  errors per built docx. Word's recovery prompt tolerates this;
+  OpenXmlValidator does not. Now inserts before the first child
+  whose tag falls in the positions-after-tblBorders set.

docs/plans/sdg-ssa-quality-hardening.md

@@ -0,0 +1,128 @@
+# SDG-SSA Quality Hardening Plan
+
+**Date:** 2026-05-14
+**Status:** PROPOSED — awaiting operator approval
+**Worktree:** `C:\Users\AlanRichardson\gatekeeper\.claude\worktrees\brave-mestorf-8f2596`
+**Branch:** `claude/brave-mestorf-8f2596` (from `main`)
+
+---
+
+## Goal
+
+Apply rpd-ssa-builder's four quality guards (deterministic post-pass, OOXML schema validation, LibreOffice headless smoke, drift checker) plus the `--check` preflight, CHANGELOG discipline, and byte-hash regression to gatekeeper's SSA pipeline. Operator answered **1B / 2**: SDG audits stay in gatekeeper; rpd-ssa-builder's quality patterns are the import.
+
+## Scope
+
+**In scope:**
+
+- Gatekeeper's SSA report-docx output path (`pims/services/ssa_pipeline.py` → `build_ssa_report_docx`), which is shared by RPD and SDG. Hardening SDG necessarily hardens RPD because they share code. State the fact, don't fork it.
+- `pims/scripts/run_ssa_pipeline.py` — add `--check` preflight wiring.
+- Tests under `tests/test_ssa_pipeline.py` — add deterministic-hash and OOXML schema assertions.
+
+**Out of scope (this plan):**
+
+- Rebranding rpd-ssa-builder; it stays a standalone tool at `G:\My Drive\alan_mcxico\SSA-evidence\SSA rules\`.
+- Changing gatekeeper's SDG vs RPD divergence logic (`_BULK_ENDPOINT`, STAGING-NO-BULK-ENDPOINT.txt sentinel).
+- Spinning out a separate `sdg-ssa-builder` private GitHub repo. **Recommendation: don't.** Gatekeeper is the active pipeline; a thin wrapper repo would just be the four guards + a CLI — too thin to justify two repos. If split is wanted later, it's a Phase 7 conversation.
+
+## Repo Structure Decision
+
+**Recommended: harden inside gatekeeper.** New subpackage `pims/services/ssa_quality/`:
+
+```
+pims/services/ssa_quality/
+  __init__.py
+  determinism.py        # lifted as-is from rpd-ssa-builder
+  oxml_validator.py     # lifted, points at shared %LOCALAPPDATA%\rpd-ssa-validator cache
+  libreoffice_smoke.py  # extracted from build_report.py:167-198
+  preflight.py          # parameterised port of rpd-ssa-builder src/preflight.py
+tools/
+  check_ssa_drift.py    # parameterised port of rpd-ssa-builder tools/check_drift.py
+docs/
+  CHANGELOG-SSA.md      # SSA-scoped changelog, not to clobber any existing root file
+```
+
+Wire-ins all live in `pims/services/ssa_pipeline.py` and `pims/scripts/run_ssa_pipeline.py`.
+
+## What Gets Lifted (verified portable)
+
+| Module | Source path | Destination | Self-contained? |
+|---|---|---|---|
+| Determinism post-pass | `G:\My Drive\alan_mcxico\SSA-evidence\SSA rules\src\determinism.py` (186 lines) | `pims/services/ssa_quality/determinism.py` | **Yes** — stdlib only |
+| OOXML validator | `…\SSA rules\src\oxml_validator.py` (200 lines) | `pims/services/ssa_quality/oxml_validator.py` | **Yes** — stdlib only; reuses cached `%LOCALAPPDATA%\rpd-ssa-validator\bin\Release\net8.0\Validate.dll` |
+| LibreOffice smoke | `…\SSA rules\build_report.py:167–198` | `pims/services/ssa_quality/libreoffice_smoke.py` | **Yes** — once candidate paths are passed in |
+| Drift checker | `…\SSA rules\tools\check_drift.py` (203 lines) | `tools/check_ssa_drift.py` | **Medium** — needs injection of constants instead of importing rpd config |
+| `--check` preflight | `…\SSA rules\src\preflight.py` | `pims/services/ssa_quality/preflight.py` | **Medium** — checks are folder-shape and file-presence; need to adapt for SDG/RPD folder naming `YYYY-MM-DD-{RPD,SDG}[-NN]` (per `run_ssa_pipeline.py` line ~102) |
+
+**OOXML validator note:** we do NOT copy the C# project (`_validator/Program.cs`, `Validate.csproj`). The DLL is already built and cached at `%LOCALAPPDATA%\rpd-ssa-validator\bin\Release\net8.0\Validate.dll` by rpd-ssa-builder. Gatekeeper just calls `dotnet <cached-DLL> <docx-path>`. If the cache is missing, fall through to building it via the rpd-ssa-builder source. This avoids duplicating the C# project across two repos.
+
+## Phase Breakdown (checkpoint-driven, ≤3 files per phase)
+
+Each phase: implement → run tests → diff summary → wait for approval → commit. Never push without separate approval.
+
+### Phase 0 — Verify build environment
+- Confirm `%LOCALAPPDATA%\rpd-ssa-validator\bin\Release\net8.0\Validate.dll` exists.
+- Confirm `C:\Program Files\LibreOffice\program\soffice.exe` exists.
+- Confirm `dotnet --version` reports 8.x.
+- Run `python -m pytest tests/test_ssa_pipeline.py -q` to capture baseline pass count.
+- No code changes. Report results.
+
+### Phase 1 — Determinism post-pass
+- Files: `pims/services/ssa_quality/__init__.py`, `pims/services/ssa_quality/determinism.py`, wire in `pims/services/ssa_pipeline.py::build_ssa_report_docx`.
+- Test: build a fixture report twice; assert SHA-256 matches.
+- Stop condition: hash matches across two builds. If not, debug before proceeding.
+
+### Phase 2 — OOXML schema validator
+- Files: `pims/services/ssa_quality/oxml_validator.py`, wire-in in `ssa_pipeline.py` after determinism call.
+- Test: pass a known-good fixture; assert exit 0. Pass a deliberately-broken docx; assert non-zero.
+- Stop condition: current SSA output validates clean.
+
+### Phase 3 — LibreOffice smoke test
+- Files: `pims/services/ssa_quality/libreoffice_smoke.py`, wire-in.
+- Test: build → smoke; assert pdf appears in temp dir and is non-empty.
+- Timeout: 180s (same as rpd-ssa-builder).
+
+### Phase 4 — `--check` preflight
+- Files: `pims/services/ssa_quality/preflight.py`, wire `--check` flag in `pims/scripts/run_ssa_pipeline.py`.
+- Test: known-bad folder fails fast; known-good folder passes.
+
+### Phase 5 — Drift checker
+- Files: `tools/check_ssa_drift.py`, plus a constants surface (probably a small `pims/services/ssa_quality/contract.py` that names the placeholder tokens, status colours, xlsx headers, font, recipients gatekeeper actually uses).
+- Open question: gatekeeper's `ssa_pipeline.py` does token substitution but I haven't yet enumerated its placeholder set the way rpd-ssa-builder's `config.PH_*` does. Phase 5 starts with mapping that surface — possibly 1 read-only investigation commit first.
+- Test: drift checker exits 0 on current state.
+
+### Phase 6 — CHANGELOG + byte-hash regression
+- Files: `docs/CHANGELOG-SSA.md` (new), `tests/test_ssa_determinism.py` (new fixture-driven hash regression).
+- The hash regression test pins the SHA-256 of the fixture build; future drift on output formatting is caught by CI.
+- Stop condition: test passes locally.
+
+### Phase 7 — (Optional, gated) Spin out `sdg-ssa-builder` repo
+- Only if operator wants visible separation. Recommendation: defer.
+
+## Approval Gates
+
+- After each phase: operator approves the diff before commit.
+- No push to `main` (or any branch) without explicit per-push authorisation.
+- If any phase touches more than 3 files, pause and re-scope before continuing.
+
+## Stop Conditions
+
+- OOXML validator reports schema errors on current gatekeeper output → stop, surface findings, do not paper over.
+- Determinism post-pass fails to make builds byte-identical → stop, diagnose, do not commit half-deterministic output.
+- Drift checker turns out to be a poor fit for gatekeeper's token style → drop or rewrite Phase 5; don't force a broken adapter.
+- Any phase reveals an architectural decision (e.g., gatekeeper's SDG output needs to be split across two docs) → stop, write the product decision, do not slice further.
+
+## Open Questions (for operator)
+
+1. **Single repo (recommended) vs. separate `sdg-ssa-builder` repo?** Recommend single repo (hardening inside gatekeeper) for the reasons above. Phase 7 is optional.
+2. **Determinism scope — docx only, or all three deliverables?** rpd-ssa-builder only deterministically rebuilds the docx because that's its only output. Gatekeeper produces two xlsx files too. xlsx is also a zip-archive with embedded timestamps. Should I extend determinism to xlsx outputs as well? Recommend **yes, but in a later phase** — keep Phase 1 docx-only to match rpd-ssa-builder's exact pattern, then a Phase 1b adds xlsx if desired.
+3. **CHANGELOG location.** Recommend `docs/CHANGELOG-SSA.md` to avoid clobbering any existing root-level changelog. Confirm before writing.
+4. **Drift checker scope.** Should it check the SDG report template's placeholder set, the xlsx headers in `build_pims_enriched_xlsx`, the staging xlsx headers, or all of the above? Recommend all of the above but split across two passes — template+report first, xlsx headers second.
+
+## Outstanding (separate from this plan)
+
+- Audit folders `2026-05-13_RPD-01` and `2026-05-13_RPD-02` use underscores; rpd-ssa-builder's regex requires hyphens. Rename to `2026-05-13-RPD-01` and `2026-05-13-RPD-02`. Not blocking this plan.
+
+## Next Step
+
+Operator approves this plan (or redirects), then I execute Phase 0 (environment verification, no code changes) and report back.

pims/scripts/run_ssa_pipeline.py

@@ -1442,6 +1442,13 @@ def run_once(
 def main(argv: list[str] | None = None) -> int:
     ap = argparse.ArgumentParser(prog="run_ssa_pipeline")
     ap.add_argument("folder", type=Path, help="audit evidence folder")
+    ap.add_argument(
+        "--check", action="store_true",
+        help=(
+            "run preflight checks on the audit folder and exit "
+            "without building any deliverables"
+        ),
+    )
     ap.add_argument("--prepared-by", default="Alan Richardson")
     ap.add_argument("--ignore-freeze", action="store_true")
     ap.add_argument(
@@ -1522,6 +1529,20 @@ def main(argv: list[str] | None = None) -> int:
         format="%(asctime)s %(levelname)s %(name)s | %(message)s",
     )
 
+    if args.check:
+        from pims.services.ssa_quality import format_results, run_preflight
+        results, hard_pass = run_preflight(args.folder)
+        print(format_results(results))
+        print()
+        info_fails = [r for r in results if not r.passed and r.informational]
+        hard_fails = [r for r in results if not r.passed and not r.informational]
+        if hard_pass:
+            extra = f" ({len(info_fails)} informational warning(s))" if info_fails else ""
+            print(f"OK - ready to build{extra}")
+            return 0
+        print(f"BLOCKED - {len(hard_fails)} required check(s) failed")
+        return 1
+
     try:
         phase_flags = (
             args.enrich_only, args.from_state, args.from_report, args.resume,
@@ -1608,6 +1629,23 @@ def main(argv: list[str] | None = None) -> int:
     print(f"outputs:        {len(payload['outputs'])} files in {args.folder}")
     for name in payload["outputs"]:
         print(f"  - {name}")
+    # LibreOffice render smoke test on the report docx (soft-skip if
+    # soffice not installed). Runs once per CLI invocation; build_ssa_
+    # report_docx unit tests don't pay this cost.
+    report_name = next(
+        (n for n in payload.get("outputs", [])
+         if n.startswith("Site-Safety-Audit-Report-") and n.endswith(".docx")),
+        None,
+    )
+    if report_name:
+        from pims.services.ssa_quality import smoke_test_docx
+        smoke = smoke_test_docx(args.folder / report_name)
+        if not smoke.available:
+            print("libreoffice smoke: skipped (LibreOffice not found)")
+        elif smoke.success:
+            print("libreoffice smoke: passed")
+        else:
+            print(f"libreoffice smoke: FAILED - {smoke.failure_detail}")
     return 0
 
 

pims/services/ssa_pipeline.py

@@ -30,6 +30,8 @@
 from pathlib import Path
 from typing import Iterable
 
+from pims.services.ssa_quality import make_docx_deterministic, validate_docx
+
 log = logging.getLogger(__name__)
 
 
@@ -1059,6 +1061,28 @@ def _cm_to_twips(cm: float) -> int:
     return int(round(cm * 567))
 
 
+def _insert_tblborders_in_schema_order(tblPr, borders) -> None:
+    """Insert ``<w:tblBorders>`` at the schema-correct position in ``<w:tblPr>``.
+
+    OOXML places tblBorders at position 11; ``<w:shd>``, ``<w:tblLayout>``,
+    ``<w:tblCellMar>``, ``<w:tblLook>`` etc. (positions 12-18) must come
+    after. Template tables always carry ``<w:tblLook>``, so a plain
+    ``tblPr.append(tblBorders)`` violates schema order. Word's recovery
+    prompt is tolerant of this; the OpenXmlValidator is not.
+    """
+    from docx.oxml.ns import qn
+    after_tblborders = {
+        qn("w:shd"), qn("w:tblLayout"), qn("w:tblCellMar"),
+        qn("w:tblLook"), qn("w:tblCaption"),
+        qn("w:tblDescription"), qn("w:tblPrChange"),
+    }
+    for i, child in enumerate(tblPr):
+        if child.tag in after_tblborders:
+            tblPr.insert(i, borders)
+            return
+    tblPr.append(borders)
+
+
 def _apply_all_cell_borders(tbl) -> None:
     """Apply Word's "All" border preset — single-line ½-pt borders on
     every cell edge inside and outside the table. Sets ``<w:tblBorders>``
@@ -1084,7 +1108,7 @@ def _apply_all_cell_borders(tbl) -> None:
         b.set(qn("w:space"), "0")
         b.set(qn("w:color"), "auto")
         borders.append(b)
-    tblPr.append(borders)
+    _insert_tblborders_in_schema_order(tblPr, borders)
 
 
 def _apply_table_format(
@@ -2905,6 +2929,28 @@ def build_ssa_report_docx(
 
     doc.save(tmp)
     os.replace(tmp, output_path)
+    # Deterministic post-pass: stable zip mtimes + strip non-deterministic
+    # OOXML metadata so re-runs over the same inputs are byte-identical.
+    import datetime as _dt
+    try:
+        _audit_date = _dt.datetime.strptime(audit_date_ddmmyyyy, "%d/%m/%Y").date()
+    except (ValueError, TypeError):
+        _audit_date = _dt.date(2000, 1, 1)
+    make_docx_deterministic(output_path, _audit_date)
+    # OOXML schema validation (soft-skip if dotnet missing). Errors are
+    # logged and surfaced via the diagnostics dict; the build does not
+    # abort on findings so callers retain control over hard-fail policy.
+    _validation = validate_docx(output_path)
+    diagnostics["oxml_validation_errors"] = (
+        [e.short() for e in _validation.errors] if _validation.available else None
+    )
+    if _validation.available and _validation.errors:
+        log.warning(
+            "OOXML schema validation: %d error(s) in %s",
+            len(_validation.errors), output_path.name,
+        )
+        for _e in _validation.errors[:5]:
+            log.warning("  %s", _e.short())
     return diagnostics
 
 

pims/services/ssa_quality/__init__.py

@@ -0,0 +1,27 @@
+"""SSA output quality guards.
+
+Subpackage that hardens the SSA report-docx output path. Modules land
+across Phases 1-5 of the SDG-SSA quality hardening plan
+(docs/plans/sdg-ssa-quality-hardening.md):
+
+- determinism       (Phase 1) — byte-stable docx output
+- oxml_validator    (Phase 2) — OOXML schema validation
+- libreoffice_smoke (Phase 3) — headless render smoke test
+- preflight         (Phase 4) — --check folder/inputs validation
+"""
+from .determinism import make_docx_deterministic
+from .libreoffice_smoke import SmokeResult, smoke_test_docx
+from .oxml_validator import ValidationError, ValidationResult, validate_docx
+from .preflight import CheckResult, format_results, run_preflight
+
+__all__ = [
+    "make_docx_deterministic",
+    "ValidationError",
+    "ValidationResult",
+    "validate_docx",
+    "SmokeResult",
+    "smoke_test_docx",
+    "CheckResult",
+    "format_results",
+    "run_preflight",
+]