From 53cb0e517938f63e7721e1451287f83d42841e60 Mon Sep 17 00:00:00 2001 From: DuqueOM Date: Fri, 4 Sep 2026 14:31:21 -0600 Subject: [PATCH] fix(docs,ci): two baseline entries were misclassified, and eight gates had no local hook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reviewing the baseline I filed in #84 showed two of its five "unimplemented" entries were my own misreadings. scripts/smoke_test.py was never missing. The release-checklist skill says in so many words that "the template does not ship a scripts/smoke_test.py script (would compete with deploy-common.yml as SSOT)". It is a deliberate, documented non-existence, and the baseline recorded a correct design decision as a defect. The gate flagged it only because the sentence wrapped the path in a code span; rewritten as plain prose per the convention the governance doc already states. scripts/load_test_services.py was a wrong path, not a missing file. The artefact exists as templates/service/tests/load_test.py, run via `make load-test`. The load-test workflow gave three `locust -f scripts/load_test_services.py` commands that fail as written. Corrected to tests/load_test.py, the form that is right from inside a generated service. Baseline: 7 entries -> 5. templates/service/docs/ADOPTION.md linked ../SECURITY.md, which resolves to templates/service/SECURITY.md here and to the service root in a generated service. Neither exists. It never failed CI because .github/markdown-link-check.json carried a dedicated ignorePatterns entry silencing that exact link — it was suppressed, not unnoticed. The sentence is about this template's disclosure SLA, so the link now points upstream absolutely and the suppression is deleted. The repo-root copy of the same document was always correct and is untouched. The systemic half: eight validate-templates.yml gates had no pre-commit hook at all — doc-coherence, cicd-template-drift, vendored-runtime-drift, common-utils-drift, dashboard-inventory, baselines-expiry, test-clock-isolation and agentic-adapter-sync. There was no way to run the full gate set locally, so the only feedback loop was push-and-read-CI, which is how three breakages reached the remote on #84. All eight are now pre-commit hooks, path-filtered so each fires only on the files it guards. This is deliberately NOT a revival of the pre-push stage retired in R5-L4: that decision was about a 60-second scaffold hook training `--no-verify`, and all thirteen gates together run in 0.9 s, inside the < 5 s budget this config targets. The reasoning is recorded next to the original decision so the two are read together. `make verify` runs the same thirteen in one command for the sweep-before-PR case, reporting every failure rather than stopping at the first. Slow end-to-end stays in `make smoke`. Verified negatively: a dead path fails check_doc_path_refs through `make verify`, and breaking the byte-identity of a vendored runbook — the exact mistake that reached CI on #84 — is now caught by the vendored-runtime-drift hook at commit time. Co-Authored-By: Claude Opus 5 --- .devin/skills/release-checklist/SKILL.md | 2 +- .devin/workflows/load-test.md | 6 +- .doc-path-baseline.yml | 8 -- .github/markdown-link-check.json | 3 +- .pre-commit-config.yaml | 77 +++++++++++++++++++ CHANGELOG.md | 52 +++++++++++++ Makefile | 56 +++++++++++++- agentic/skills/release-checklist/SKILL.md | 2 +- agentic/workflows/load-test.md | 6 +- .../agentic/skills/release-checklist/SKILL.md | 2 +- .../service/agentic/workflows/load-test.md | 6 +- templates/service/docs/ADOPTION.md | 3 +- 12 files changed, 199 insertions(+), 24 deletions(-) diff --git a/.devin/skills/release-checklist/SKILL.md b/.devin/skills/release-checklist/SKILL.md index 8be2713..9149e89 100644 --- a/.devin/skills/release-checklist/SKILL.md +++ b/.devin/skills/release-checklist/SKILL.md @@ -154,7 +154,7 @@ kill %1 For a multi-service / multi-cloud sweep, drive the loop from a runbook page or a CI workflow-dispatch — the template does not ship -a `scripts/smoke_test.py` script (would compete with `deploy-common.yml` +a scripts/smoke_test.py script (would compete with `deploy-common.yml` as SSOT). ## Rollback Plan diff --git a/.devin/workflows/load-test.md b/.devin/workflows/load-test.md index 7b72ec2..d45af45 100644 --- a/.devin/workflows/load-test.md +++ b/.devin/workflows/load-test.md @@ -13,7 +13,7 @@ Determine which service and environment to test: ## 2. Configure Locust -Verify `scripts/load_test_services.py` has the correct endpoints and payloads for the target service. +Verify `tests/load_test.py` has the correct endpoints and payloads for the target service. Before starting load, verify the FastAPI contract with the same auth posture production clients use: @@ -40,7 +40,7 @@ running Locust. Do not load-test a payload that returns 422. ## 3. Run Load Test (GCP) ```bash -locust -f scripts/load_test_services.py \ +locust -f tests/load_test.py \ --host http://${GCP_ENDPOINT} \ --users 100 \ --spawn-rate 10 \ @@ -52,7 +52,7 @@ locust -f scripts/load_test_services.py \ ## 4. Run Load Test (AWS) ```bash -locust -f scripts/load_test_services.py \ +locust -f tests/load_test.py \ --host http://${AWS_ENDPOINT} \ --users 100 \ --spawn-rate 10 \ diff --git a/.doc-path-baseline.yml b/.doc-path-baseline.yml index 0e3d88b..1900d7b 100644 --- a/.doc-path-baseline.yml +++ b/.doc-path-baseline.yml @@ -35,14 +35,6 @@ entries: expiry: 2027-09-04 # --- unimplemented -------------------------------------------------------- - - path: scripts/smoke_test.py - reason: unimplemented — the release-checklist skill instructs running it; no such script exists in either root - expiry: 2026-12-01 - - - path: scripts/load_test_services.py - reason: unimplemented — the load-test workflow instructs running it; no such script exists in either root - expiry: 2026-12-01 - - path: templates/service/tests/test_context_files_hygiene.py reason: unimplemented — docs/agentic/contextualization.md cites it as the enforcing test; the test was never written expiry: 2026-12-01 diff --git a/.github/markdown-link-check.json b/.github/markdown-link-check.json index cc594f0..85cdae6 100644 --- a/.github/markdown-link-check.json +++ b/.github/markdown-link-check.json @@ -8,8 +8,7 @@ { "pattern": "^https://github\\.com/\\{ORG\\}" }, { "pattern": "youtu\\.be/XXX" }, { "pattern": "example\\.com" }, - { "pattern": "^https://ai\\.google\\.dev/" }, - { "pattern": "^\\.\\./SECURITY\\.md$" } + { "pattern": "^https://ai\\.google\\.dev/" } ], "replacementPatterns": [ { diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index e9bff94..e81269d 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -43,6 +43,19 @@ # Single-stage install — pre-push stage was retired in R5-L4 (May 2026); # `make smoke` + CI pr-smoke-lane.yml replace it. +# +# 2026-09-04: eight validate-templates.yml gates had no local hook at all +# (doc-coherence, cicd-template-drift, vendored-runtime-drift, +# common-utils-drift, dashboard-inventory, baselines-expiry, +# test-clock-isolation, agentic-adapter-sync). That gap — not the missing +# pre-push stage — is what let three breakages reach the remote in one +# afternoon. They are added below as pre-commit hooks rather than by +# reviving pre-push: all thirteen gates together run in 0.9 s, well inside +# the < 5 s pre-commit budget this file targets, so R5-L4's reasoning +# (a 60 s hook trains `--no-verify`) does not apply to them. +# +# `make verify` runs the same set in one command, for the sweep-before-PR +# case where you want every gate regardless of which files you touched. default_install_hook_types: [pre-commit] # fail_fast: false is the DEFAULT (shown explicit for intent) — we @@ -195,6 +208,70 @@ repos: language: system pass_filenames: false files: ^(VERSION|README\.md|QUICK_START\.md|docs/TUTORIAL\.md|docs/PROGRESSION\.md|scripts/check_adopter_scaffold_ref\.py)$ + - id: doc-coherence + name: documentation coherence (rule 16, ADR-031) + description: | + C1-C7: VERSION vs CHANGELOG, llms.txt version, anti-pattern + counts, agentic surface counts, ADR traceability, release-note + existence, English-only + private-repo guard. + entry: python3 scripts/check_doc_coherence.py + language: system + pass_filenames: false + files: ^(VERSION|CHANGELOG\.md|llms\.txt|AGENTS\.md|README\.md|CLAUDE\.md|docs/decisions/|releases/|agentic/|scripts/check_doc_coherence\.py) + - id: cicd-template-drift + name: template action versions match runtime + description: | + Dependabot only scans .github/workflows/, so action references + under templates/ age silently and ship to every scaffolded + service. See docs/governance/cicd-templates-drift.md. + entry: python3 scripts/check_cicd_template_drift.py + language: system + pass_filenames: false + files: \.ya?ml$|^scripts/check_cicd_template_drift\.py$ + - id: vendored-runtime-drift + name: vendored copies match their canonical originals + description: | + Files vendored byte-identical into templates/service/ (AGENTS.md, + the agentic surface, several scripts and runbooks) must not + diverge. Editing one side only is anti-pattern Q-06. + entry: python3 scripts/check_vendored_runtime_drift.py + language: system + pass_filenames: false + files: ^(scripts/|docs/runbooks/|agentic/|templates/service/|templates/config/|AGENTS\.md|AGENT_CONTEXT\.md) + - id: common-utils-drift + name: common_utils distribution contract (ADR-025) + entry: python3 scripts/check_common_utils_drift.py + language: system + pass_filenames: false + files: common_utils/ + - id: dashboard-inventory + name: Grafana dashboard inventory is complete + entry: python3 scripts/check_dashboard_inventory.py + language: system + pass_filenames: false + files: ^(templates/service/monitoring/|docs/observability/|scripts/check_dashboard_inventory\.py) + - id: baselines-expiry + name: security baseline entries are dated and in-date + entry: python3 scripts/check_baselines_expiry.py + language: system + pass_filenames: false + files: ^(\.security-baselines/|scripts/check_baselines_expiry\.py) + - id: test-clock-isolation + name: tests do not depend on the wall clock + entry: python3 scripts/check_test_clock_isolation.py + language: system + pass_filenames: false + files: (^|/)tests?/.*\.py$|^scripts/check_test_clock_isolation\.py$ + - id: agentic-adapter-sync + name: generated adapter surfaces are in sync + description: | + .devin/, .cursor/, .claude/ and .codex/ are generated from + agentic/ by sync_agentic_adapters.py. Editing agentic/ without + re-running the sync ships a stale adapter surface. + entry: python3 scripts/sync_agentic_adapters.py --check + language: system + pass_filenames: false + files: ^(agentic/|\.devin/|\.cursor/|\.claude/|\.codex/|templates/config/agentic_manifest\.yaml) - id: doc-path-refs name: documentation path references resolve description: | diff --git a/CHANGELOG.md b/CHANGELOG.md index 2765791..d2d2bd1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,58 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and [Sem ## [Unreleased] +### Fixed — two baseline entries were misclassified, not unimplemented + +- **`scripts/smoke_test.py` was never missing.** The release-checklist + skill says in so many words that *"the template does not ship a + `scripts/smoke_test.py` script (would compete with `deploy-common.yml` + as SSOT)"*. It is a deliberate, documented non-existence. The path + reference gate flagged it only because the sentence wrapped it in a code + span, and the baseline then recorded a correct design decision as a + defect. Rewritten as plain prose per the convention in + `docs/governance/doc-path-references.md`: a code span asserts *this + resolves*, so a path you are talking about is not one. +- **`scripts/load_test_services.py` was a wrong path, not a missing + file.** The artefact exists as `templates/service/tests/load_test.py`, + run via `make load-test`. The load-test workflow gave three + `locust -f scripts/load_test_services.py` commands that fail as written. + Corrected to `tests/load_test.py`, the form that is right from inside a + generated service. +- Baseline down from 7 entries to 5. + +### Fixed — a broken link was suppressed rather than repaired + +- `templates/service/docs/ADOPTION.md` linked `[SECURITY.md](../SECURITY.md)`, + which resolves to `templates/service/SECURITY.md` in this repo and to the + service root in a generated one. Neither exists. +- It never failed CI because `.github/markdown-link-check.json` carried a + dedicated `ignorePatterns` entry, `^\.\./SECURITY\.md$`, silencing that + exact link. The sentence is about *this template's* disclosure SLA, so + the link now points at the upstream `SECURITY.md` absolutely, and the + suppression is deleted. The repo-root copy of the same document was + always correct and is untouched. + +### Added — `make verify`, and the eight CI gates that had no local hook + +- Eight jobs in `validate-templates.yml` had no pre-commit hook at all: + doc-coherence, cicd-template-drift, vendored-runtime-drift, + common-utils-drift, dashboard-inventory, baselines-expiry, + test-clock-isolation and agentic-adapter-sync. There was no way to run + the full gate set locally, so the only feedback loop was push-and-read-CI. +- All eight are now pre-commit hooks, path-filtered so each fires only on + the files it guards. This is deliberately **not** a revival of the + pre-push stage retired in R5-L4: that decision was about a 60-second + scaffold hook training `--no-verify`, and all thirteen gates together + run in 0.9 s — inside the < 5 s budget this config targets. The + reasoning is recorded next to the original decision. +- `make verify` runs the same thirteen in one command for the + sweep-before-PR case, reporting **every** failure rather than stopping at + the first. Slow end-to-end stays in `make smoke`. +- Verified negatively: introducing a dead path fails `check_doc_path_refs` + through `make verify`, and breaking the byte-identity of a vendored + runbook — the exact mistake that reached CI on #84 — is now caught by the + `vendored-runtime-drift` hook at commit time. + ### Fixed — the Copier migration relocated the template tree and the prose never followed - **Root cause, traced.** ADR-030's migration (commit `fe89e92`, diff --git a/Makefile b/Makefile index 8fb3506..609c38d 100644 --- a/Makefile +++ b/Makefile @@ -12,7 +12,7 @@ # make test-examples # Run example regression tests .PHONY: help install-dev lint-all format-all validate-templates \ - validate-agentic bootstrap smoke \ + validate-agentic bootstrap smoke verify \ mcp-check mcp-doctor mcp-render-docs \ report-validate report-example \ demo-minimal test-examples clean @@ -155,6 +155,60 @@ eda-validate: ## Validate EDA pipeline: syntax + run against example dataset validate-templates: lint-all validate-k8s validate-agentic test-scaffold eda-validate ## Validate all templates (lint + K8s + agentic + scaffold + EDA) @echo "$(GREEN)✓ All templates validated$(NC)" +# ═══════════════════════════════════════════════ +# Gate suite — the CI contract, runnable locally +# ═══════════════════════════════════════════════ +# Every check below is a job in validate-templates.yml. Before this target +# existed the only way to run the full set was to push and read CI, which +# is how three separate breakages in one afternoon (Link Check, vendored +# byte-identity, evidence-check) reached the remote instead of dying on the +# contributor's machine. +# +# Deliberately excludes the slow ones: `make smoke` (scaffolder E2E, ~2.5 +# min) and the pytest suites. This target is the fast contract — every gate +# here is sub-second — so it is cheap enough to sit on pre-push. +# +# Runs ALL gates and reports every failure, rather than stopping at the +# first. A contributor who broke three things should learn that in one run. + +GATES := \ + check_doc_coherence \ + check_doc_path_refs \ + check_cicd_template_drift \ + check_vendored_runtime_drift \ + check_common_utils_drift \ + check_dashboard_inventory \ + check_baselines_expiry \ + check_gitleaks_pin \ + check_adopter_scaffold_ref \ + check_service_adr_references \ + check_test_clock_isolation \ + validate_agentic + +verify: ## Run every fast CI gate locally (the pre-push contract). Slow E2E lives in `make smoke`. + @echo "$(GREEN)Running the CI gate suite...$(NC)" + @failed=""; \ + for gate in $(GATES); do \ + if python3 scripts/$$gate.py >/tmp/$$gate.out 2>&1; then \ + printf ' %-34s PASS\n' "$$gate"; \ + else \ + printf ' %-34s FAIL\n' "$$gate"; \ + failed="$$failed $$gate"; \ + fi; \ + done; \ + if python3 scripts/sync_agentic_adapters.py --check >/tmp/adapter_sync.out 2>&1; then \ + printf ' %-34s PASS\n' "adapter-sync"; \ + else \ + printf ' %-34s FAIL\n' "adapter-sync"; \ + failed="$$failed sync_agentic_adapters"; \ + fi; \ + if [ -n "$$failed" ]; then \ + echo "$(RED)✗ failing gates:$$failed$(NC)"; \ + for g in $$failed; do echo "--- $$g ---"; tail -20 /tmp/$$g.out; done; \ + exit 1; \ + fi; \ + echo "$(GREEN)✓ every gate green — safe to push$(NC)" + # ═══════════════════════════════════════════════ # Example (Fraud Detection) # ═══════════════════════════════════════════════ diff --git a/agentic/skills/release-checklist/SKILL.md b/agentic/skills/release-checklist/SKILL.md index 8be2713..9149e89 100644 --- a/agentic/skills/release-checklist/SKILL.md +++ b/agentic/skills/release-checklist/SKILL.md @@ -154,7 +154,7 @@ kill %1 For a multi-service / multi-cloud sweep, drive the loop from a runbook page or a CI workflow-dispatch — the template does not ship -a `scripts/smoke_test.py` script (would compete with `deploy-common.yml` +a scripts/smoke_test.py script (would compete with `deploy-common.yml` as SSOT). ## Rollback Plan diff --git a/agentic/workflows/load-test.md b/agentic/workflows/load-test.md index 7b72ec2..d45af45 100644 --- a/agentic/workflows/load-test.md +++ b/agentic/workflows/load-test.md @@ -13,7 +13,7 @@ Determine which service and environment to test: ## 2. Configure Locust -Verify `scripts/load_test_services.py` has the correct endpoints and payloads for the target service. +Verify `tests/load_test.py` has the correct endpoints and payloads for the target service. Before starting load, verify the FastAPI contract with the same auth posture production clients use: @@ -40,7 +40,7 @@ running Locust. Do not load-test a payload that returns 422. ## 3. Run Load Test (GCP) ```bash -locust -f scripts/load_test_services.py \ +locust -f tests/load_test.py \ --host http://${GCP_ENDPOINT} \ --users 100 \ --spawn-rate 10 \ @@ -52,7 +52,7 @@ locust -f scripts/load_test_services.py \ ## 4. Run Load Test (AWS) ```bash -locust -f scripts/load_test_services.py \ +locust -f tests/load_test.py \ --host http://${AWS_ENDPOINT} \ --users 100 \ --spawn-rate 10 \ diff --git a/templates/service/agentic/skills/release-checklist/SKILL.md b/templates/service/agentic/skills/release-checklist/SKILL.md index 8be2713..9149e89 100644 --- a/templates/service/agentic/skills/release-checklist/SKILL.md +++ b/templates/service/agentic/skills/release-checklist/SKILL.md @@ -154,7 +154,7 @@ kill %1 For a multi-service / multi-cloud sweep, drive the loop from a runbook page or a CI workflow-dispatch — the template does not ship -a `scripts/smoke_test.py` script (would compete with `deploy-common.yml` +a scripts/smoke_test.py script (would compete with `deploy-common.yml` as SSOT). ## Rollback Plan diff --git a/templates/service/agentic/workflows/load-test.md b/templates/service/agentic/workflows/load-test.md index 7b72ec2..d45af45 100644 --- a/templates/service/agentic/workflows/load-test.md +++ b/templates/service/agentic/workflows/load-test.md @@ -13,7 +13,7 @@ Determine which service and environment to test: ## 2. Configure Locust -Verify `scripts/load_test_services.py` has the correct endpoints and payloads for the target service. +Verify `tests/load_test.py` has the correct endpoints and payloads for the target service. Before starting load, verify the FastAPI contract with the same auth posture production clients use: @@ -40,7 +40,7 @@ running Locust. Do not load-test a payload that returns 422. ## 3. Run Load Test (GCP) ```bash -locust -f scripts/load_test_services.py \ +locust -f tests/load_test.py \ --host http://${GCP_ENDPOINT} \ --users 100 \ --spawn-rate 10 \ @@ -52,7 +52,7 @@ locust -f scripts/load_test_services.py \ ## 4. Run Load Test (AWS) ```bash -locust -f scripts/load_test_services.py \ +locust -f tests/load_test.py \ --host http://${AWS_ENDPOINT} \ --users 100 \ --spawn-rate 10 \ diff --git a/templates/service/docs/ADOPTION.md b/templates/service/docs/ADOPTION.md index 6b7e56c..09340b8 100644 --- a/templates/service/docs/ADOPTION.md +++ b/templates/service/docs/ADOPTION.md @@ -283,7 +283,8 @@ The template explicitly does **not** address: ## 7. Disclosure SLA (R4 audit M4 clarification) -The vulnerability response timeline in [`SECURITY.md`](../SECURITY.md) +The vulnerability response timeline in +[`SECURITY.md`](https://github.com/DuqueOM/ml-service-template/blob/main/SECURITY.md) is **operational** for this template: - The maintainer commits to the response times listed (Critical 48h /