Skip to content

fix(security,docs): the baseline gate saw no entries, and Art. 11 cited a missing index - #88

Merged
DuqueOM merged 2 commits into
mainfrom
fix/adr-index-and-baseline-review
Sep 4, 2026
Merged

fix(security,docs): the baseline gate saw no entries, and Art. 11 cited a missing index#88
DuqueOM merged 2 commits into
mainfrom
fix/adr-index-and-baseline-review

Conversation

@DuqueOM

@DuqueOM DuqueOM commented Sep 4, 2026

Copy link
Copy Markdown
Owner

Third of three PRs closing the residue from #84. Three findings, one theme: a control that reports green because its scanner is narrower than the surface it guards.

1. The baseline expiry gate was watching nothing

scripts/check_baselines_expiry.py reported:

[baselines] OK — no expired or unannotated entries (as of 2026-09-04).

while three HIGH-severity GKE checks sat suppressed in .security-baselines/tfsec.yml. It was not lying about the entries it saw — it saw none.

Its yaml_entry pattern required a suppression id to begin with an uppercase letter. That matches checkov ids (CKV_AWS_18) and misses every tfsec id (google-gke-enable-master-networks). Zero matches → zero expired → pass.

Widening to lowercase alone would have been wrong. checkov.yml contains a framework: list (terraform, kubernetes, dockerfile) — a sequence, but not a suppression. So the scanner is now block-aware and treats only items under exclude: / skip-check: as entries — exactly what its own comment had claimed since it was written:

# We look at exclude: / skip-check: blocks.

The three entries also carried Review-by: 2027-01, a format the gate does not parse (it wants # expiry: YYYY-MM-DD on the entry line or the one directly above). Normalised, keeping the original justification prose intact.

Before After
Entries the gate sees in tfsec.yml 0 3
--as-of 2027-06-01 pass fails all three as expired
False positives from framework: n/a none

2. The review record that was promised and never created

.security-baselines/README.md step 4 has instructed reviewers to "update docs/audit/baseline-review.md (next quarterly review)" since the baselines were introduced. The document did not exist, so three HIGH suppressions had a justification in a YAML comment and no review record anywhere. A suppression without a review record is indistinguishable from one nobody remembers making.

The first dated review records all three with their compensating controls. Worth stating plainly: all three are tool limitations, not accepted risks

  • PodSecurityPolicy was removed in Kubernetes 1.25; enabling it would fail the apply. PSS is enforced via namespace labels in every overlay (D-29).
  • master_authorized_networks_config is present, as a dynamic block. tfsec v1.28 does not evaluate dynamic blocks.
  • The metadata attribute lives on the node pools, because the module uses remove_default_node_pool = true; tfsec inspects only the cluster.

In all three the control exists and tfsec cannot see it. That is why the expiries are tied to the tfsec → trivy migration rather than to a risk-acceptance window.

3. An Art. 11 compliance claim an auditor could falsify in one command

docs/COMPLIANCE_MAPPING.md cited an ADR index as EU AI Act Art. 11 evidence — "documentation sufficient to assess compliance". The index did not exist, and the same row claimed "ADRs (37)" against 45 on disk. Two defects in one compliance assertion.

scripts/generate_adr_index.py generates docs/decisions/README.md from the files themselves; --check fails CI when it goes stale, wired as a CI step and a pre-commit hook. It handles both heading conventions in use (# ADR-001: Title and # ADR-045 — Title) and cross-checks the number in the heading against the number in the filename.

The Art. 11 row now links the index instead of restating a count — removing the drift surface rather than correcting one instance of it. Writing the index by hand would have reproduced the defect on the next ADR.

Baseline

Across the three PRs in this series, .doc-path-baseline.yml goes from 7 entries to 2, and no unimplemented entries remain — only the two genuine runtime-artifact ones.


Evidence — Schema / Contract Test

scripts/generate_adr_index.py --check is the contract for the index (verified in CI and pre-commit); scripts/check_baselines_expiry.py --as-of is the contract for the expiry behaviour. Both are exercised below.

Evidence — Real Execution Output

$ # before: the gate saw nothing
$ python3 scripts/check_baselines_expiry.py
[baselines] OK — no expired or unannotated entries (as of 2026-09-04).

$ # after making the scanner block-aware, before normalising the annotations
$ python3 scripts/check_baselines_expiry.py
[baselines] 3 issue(s):
  - .security-baselines/tfsec.yml:20  missing expiry annotation: - google-gke-enforce-pod-security-policy
  - .security-baselines/tfsec.yml:29  missing expiry annotation: - google-gke-enable-master-networks
  - .security-baselines/tfsec.yml:50  missing expiry annotation: - google-gke-metadata-endpoints-disabled

$ # after normalising to `# expiry: 2027-01-01`
$ python3 scripts/check_baselines_expiry.py
[baselines] OK — no expired or unannotated entries (as of 2026-09-04).

$ python3 scripts/check_baselines_expiry.py --as-of 2027-06-01
[baselines] 3 issue(s):
  - .security-baselines/tfsec.yml:21  expired on 2027-01-01: - google-gke-enforce-pod-security-policy
  - .security-baselines/tfsec.yml:31  expired on 2027-01-01: - google-gke-enable-master-networks
  - .security-baselines/tfsec.yml:53  expired on 2027-01-01: - google-gke-metadata-endpoints-disabled

$ python3 scripts/generate_adr_index.py
[adr-index] wrote docs/decisions/README.md

$ python3 scripts/generate_adr_index.py --check
[adr-index] OK — index lists all 45 ADRs and is current.

$ # staleness is detected, not just absence
$ echo '| 999 | [bogus](x.md) |' >> docs/decisions/README.md
$ python3 scripts/generate_adr_index.py --check
::error::docs/decisions/README.md is stale. An ADR was added, renamed or retitled without regenerating the index.
exit=1

Evidence — CI Run Link

See the checks on this PR.

🤖 Generated with Claude Code

DuqueOM and others added 2 commits September 4, 2026 14:52
…ed a missing index

Three findings, one theme: a control that reports green because its
scanner is narrower than the surface it guards.

1. scripts/check_baselines_expiry.py reported "OK — no expired or
   unannotated entries" while three HIGH-severity GKE checks sat
   suppressed in .security-baselines/tfsec.yml. It saw none of them: its
   yaml_entry pattern required a suppression id to start with an uppercase
   letter, which matches checkov ids (CKV_AWS_18) and misses every tfsec
   id (google-gke-enable-master-networks).

   Widening to lowercase alone would have been wrong — `framework:
   [terraform, kubernetes, dockerfile]` in checkov.yml is a sequence too.
   The scanner is now block-aware and treats only items under `exclude:` /
   `skip-check:` as entries, which is what its own comment had claimed
   since it was written.

   The three entries also carried `Review-by: 2027-01`, a format the gate
   does not parse. Normalised to `# expiry: 2027-01-01`, keeping the
   original justification prose. Verified: all three now seen and in-date,
   `--as-of 2027-06-01` fails all three, and `framework:` still yields no
   false positive.

2. .security-baselines/README.md step 4 has instructed reviewers to update
   docs/audit/baseline-review.md since the baselines were introduced. It
   did not exist. The first dated review now records all three
   suppressions with their compensating controls. All three are tool
   limitations rather than accepted risks: PSP was removed in Kubernetes
   1.25 and PSS covers it via namespace labels; master_authorized_networks
   _config exists as a dynamic block tfsec cannot evaluate; the metadata
   attribute lives on the node pools, not the cluster tfsec inspects.

3. docs/COMPLIANCE_MAPPING.md cited an ADR index as EU AI Act Art. 11
   evidence. The index did not exist, and the same row claimed "ADRs (37)"
   against 45 on disk — a compliance mapping pointing at a missing
   artefact and miscounting the one it has, falsifiable in one command.

   scripts/generate_adr_index.py generates docs/decisions/README.md from
   the files and --check fails CI when it goes stale, wired as a CI step
   and a pre-commit hook. It parses both heading conventions in use. The
   Art. 11 row now links the index instead of restating a count, removing
   the drift surface rather than correcting one instance of it.

Baseline: two more entries resolved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The gate suite is the local mirror of the CI contract; a gate that CI runs
and `make verify` does not is the same blind spot in miniature.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@DuqueOM
DuqueOM force-pushed the fix/adr-index-and-baseline-review branch from af8bbfd to a335bac Compare September 4, 2026 20:52
@DuqueOM
DuqueOM merged commit 0b7d35a into main Sep 4, 2026
25 checks passed
@DuqueOM
DuqueOM deleted the fix/adr-index-and-baseline-review branch September 4, 2026 20:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant