feat: add repo-setup audit skill by tonyandrewmeyer · Pull Request #22 · canonical/charm-tech

tonyandrewmeyer · 2026-07-02T01:14:51Z

This PR adds a skill that summarises and automates (deterministically where possible) the repository setup (including SSDLC) work I've done this cycle.

The skill:

Is tier-aware (product / canonical / personal), agent-generic (not Claude-specific).
Audits a repository against the 26.10 baseline: SSDLC compliance, supply-chain hygiene, best-of-class extras.
Ships deterministic check & fix scripts plus consolidated references (SSDLC framework, settled carve-out decisions, tools measured and skipped, sweep history, open investigations).

I've tested this (with Claude Code) against a Canonical Charm Tech repo in the "product" group, a non-product Charm Tech repo, and a personal repo that might end up a Canonical repo some day.

Tier-aware (product / canonical / personal), agent-generic skill that audits a Canonical Charm Tech repository against the 26.10 repo-setup baseline: SSDLC compliance, supply-chain hygiene, and best-of-class extras. Ships deterministic check + fix scripts plus consolidated references covering the SSDLC framework, settled carve-out decisions, tools we measured and skipped, sweep history, and open investigations. Lifted from the canonical-work-queue staging tree where it was iterated during the 26.10 repo-setup work.

tonyandrewmeyer · 2026-07-02T02:30:23Z

+<!--
+For detailed dev-environment setup, build, and test instructions, link here to
+the substantive doc if one exists (HACKING.md, docs/contributing.md, etc.).
+Most Charm Tech repos keep this section inline rather than redirecting.
+-->
+


Suggested change

Co-authored-by: Tony Meyer <tony.meyer@gmail.com>

…cycle The Canonical Security "Repository security" and "How-To: Secure a repo" pages published 2026-07-01 recommend Require signed commits on protected branches. Team decision 2026-07-02: not required this cycle; re-evaluate in 26.10+1. Rationale in decisions.md. Also silences a would-be gap-report entry: a repo without require_signed_commits on its default-branch ruleset is not flagged in this cycle.

… when membership > Charm Tech team Charm Tech maintainer scope. A CODEOWNERS file that maps everything to the same team that already required-reviews every PR adds friction without filtering signal. CODEOWNERS is therefore only required for a charm-tech repo when its maintainer/contributor set is broader than the Charm Tech team — cross-team ownership, external contributors, or per-directory owners mapping to different sub-teams. In 26.10 that's charmlibs only. A check should not flag "no CODEOWNERS" on a Charm-Tech-only repo, and should flag a CODEOWNERS file that maps everything to the Charm Tech team wholesale.

Adds a per-repo secret-handling audit modelled on the Canonical Security "Repository security" reference (Secrets section) and the 2026-07-02 fleet audit recorded in roadmap/26.10/repo-setup/security-docs-gap.md rows #32-#36. Detects, in .github/workflows/*.y*ml: * workflow-level env: with ${{ secrets.* }} * job-level env: with ${{ secrets.* }} Both over-scope: every job or step in reach can read the secret. Step-level env: is the correct scope. * run: lines that echo / printf / cat a secret expression. The reference page warns "log masking is not guaranteed for every transformation" — keep secrets out of shell-visible strings. * secrets: inherit on reusable-workflow calls. Pass named secrets so the callee's surface is auditable. Env-scope checks need python3+PyYAML for reliable scope resolution; textual checks (echo/inherit) run without either. Verified against: * canonical/charmhub-listing-review @ main → fail (1 echo hit on review-request.yaml:31 — same finding as the fleet audit) * canonical/charmhub-listing-review @ security/drop-redundant-gh-auth-echo → pass * canonical/charmlibs → pass

…config Policy change: every third-party GitHub Action pins to a SHA — no allowlist exceptions for actions/, github/, pypa/, canonical/. The pypa/* open investigation resolved into the uniform posture rather than a per-org carve-out. .github/zizmor.yaml is no longer needed (the default unpinned-uses rule enforces the policy), so the config template and its fix script are removed. The zizmor-config check now only requires that a workflow invokes zizmor. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

…ss-stale not required The Canonical Security "How-To: Secure a repo" page recommends both Require conversation resolution and dismiss stale reviews on new commits on protected branches. Team decision 2026-07-02: not required this cycle. Conversation resolution required: the team already treats unresolved threads as a review-blocker socially; a merge gate adds friction (chase every "nit" the reviewer intended as advisory) and every follow-up commit re-triggers the resolution round. Net-negative for Charm-Tech-sized PRs with a small consistent reviewer set. Dismiss stale reviews on new commits: most Charm Tech PRs iterate quickly; auto-dismissing on every push forces a full re-review even for a rebase or typo fix. The Require last push approval flag (already in the CRA baseline) covers the substantive risk without that churn. Both re-evaluated in a future cycle if the team scales or an incident ties to their absence.

…refix Match the resolved rule from the dependabot-conventions spec (canonical#17): 'chore:' with no scope, on both the pip and github-actions ecosystems. Setting only 'prefix: chore' produces 'chore: bump ...' automatically — Dependabot inserts the colon-space when the prefix ends in a letter — so 'include: scope' is unnecessary and would produce the non-conforming 'chore(pip): ...' / 'ci(github-actions): ...' shape. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Adds a per-repo check for the "Lockfile(s)" section of the Canonical Security "How-To: Secure a repo" page: every uv run / uv sync invocation in CI must pass --locked (or --frozen) so that a stale uv.lock fails the build rather than silently re-resolving. Prefers --locked over --frozen. --locked asserts uv.lock is fresh vs pyproject.toml and fails otherwise, forcing the PR author to commit a regenerated lockfile — the resolution delta becomes reviewable. --frozen skips the freshness check entirely, so drift is silently masked. Scope: * Reads .github/workflows/*.y*ml AND the top-level Makefile (api_demo_server calls uv run through `make lint` / `make integration` — the CI-shell-out path needs to be covered). * Classifies each invocation: locked / frozen / bare / skipped-*. * Skips patterns that don't touch the project lockfile: uv run --no-project --script / --with-requirements, uvx, and uv tool install/run. * na when there's no pyproject.toml / uv.lock (Go repos etc.) or no project-scoped invocations found. Verified 2026-07-02 against: * canonical/jubilant → fail (9 bare in Makefile) * canonical/charmhub-listing-review → fail (3 frozen, 2 bare — the "frozen exemplar" designation was overstated; two bare uv run invocations remain) * canonical/pebble → na (Go, no uv) * canonical/api_demo_server → fail (6 bare across security.yaml and Makefile) Matches the pattern of the workflow-secrets check: bash script that shells out to python3 for line-level classification, emits JSON per scripts/lib/common.sh conventions.

Adds a per-repo check that pyproject.toml sets [tool.uv].exclude-newer to a rolling ≥7-day quarantine. Rationale (Canonical Security "How-To: Secure a repo" — Minimum release age section): Dependabot cooldown protects PRs Dependabot opens; exclude-newer protects every OTHER dep-resolution path (manual uv add, uv lock regens, uvx bootstraps, CI re-resolves). Per uv docs, exclude-newer accepts three formats: - Friendly duration (rolling window): "7 days", "1 week", "30 days" - ISO 8601 duration (rolling window): "P7D", "P30D", "PT24H" - RFC 3339 timestamp (absolute snapshot): 2026-01-01T00:00:00Z The check accepts all three but only friendly-duration and ISO 8601 values contribute to the ≥7-day computation. Snapshots pass with a note recommending a rolling value instead — snapshots drift silently as time passes and need periodic manual bumping. Verified 2026-07-02 against: * canonical/jubilant → fail (uv.lock but no [tool.uv]) * canonical/pytest-jubilant → fail (same) * canonical/api_demo_server → fail (same) * canonical/charmlibs → fail ([tool.uv] present, exclude-newer missing) * canonical/pebble → na (Go, no pyproject.toml) * canonical/charm-ubuntu → na (no pyproject.toml) * synthesised "7 days" / "P1W" → pass * synthesised "3 days" → fail (below threshold) * synthesised RFC 3339 timestamp → pass with snapshot note Refinement: a project with uv.lock but no [tool.uv] IS a uv project that just hasn't declared the config section — that's a fail (add the section), not na. The na verdict is reserved for projects with no uv.lock and no [tool.uv], where uv might not even be in use.

…v exclude-newer Records the finding that Go's module resolver has no native minimum-release-age concept — no go.mod directive, GOFLAGS value, or env var equivalent to `[tool.uv].exclude-newer = "7 days"`. The residual vector (a dev running `go get -u <mod>@latest` or `go mod tidy` locally, pulling a fresh release into go.sum before Dependabot cools it down) is unaddressed on pebble and concierge. Enumerates four workaround options (custom check, private proxy, PR-diff inspection, accept-and-monitor) and defers to accept-and- monitor for the 26.10 cycle. If the fleet's Go footprint grows or an incident traces to this vector, prefer PR-diff inspection. Explicitly notes: a check should not flag pebble or concierge for missing a Go-side release-age control in the 26.10 cycle.

Converts the skill's 37 shell scripts to Python for reviewability and maintenance. All scripts are stdlib-only except dependabot.py and workflow-secrets.py, which use a PEP 723 uv run shebang to pull in PyYAML (the shell versions already had PyYAML as a soft optional dep via embedded python heredocs). Preserves check ids, summary strings, evidence keys, remediation shapes, and exit codes (0/1/2/3 = pass/fail/na/unknown) so the JSON report contract is unchanged. Umbrella runner verified end-to-end against this repo (27 checks emit valid single-line JSON, aggregate report matches SKILL.md schema). Also fixes a latent bug in add-dependabot: the shell version referenced assets/dependabot.yml.template but the asset is dependabot.yaml.template. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Adds a per-repo check that [tool.uv].no-build = true is set in pyproject.toml. uv refuses any sdist install and only accepts wheels, closing the setup.py / PEP 517-hook arbitrary-code-execution vector at install time. Rationale (Canonical Security "How-To: Secure a repo" — Install scripts): source distributions run setup.py / build hooks during install; wheels don't. Refusing sdist eliminates a well-known supply-chain vector at zero cost for typical Charm Tech repos. A fleet-wide scan (2026-07-02) found 0/571 dependencies across every uv project were sdist-only, so this is a zero-cost policy to adopt. Per-package escape hatches via no-build-package = ["some-pkg"] are recognised and reported in evidence for future one-off exceptions. Written in Python matching the ported skill layout (was initially drafted in bash, then rewritten to match the .py convention after the skill was ported). Verified 2026-07-02 against: * canonical/jubilant → fail (uv.lock present but no [tool.uv]) * canonical/pebble → na (Go, no pyproject.toml) * canonical/charmlibs → fail ([tool.uv] present, no-build missing) * canonical/charm-ubuntu on chore/modernize-charm → pass * synthesised no-build = true → pass * synthesised no-build = false → fail * synthesised no-build = true + per-pkg override → pass with the override reflected in evidence

Replace the `[tool.uv]` block with the canonical fleet comment (rolling quarantine rationale), drop `no-build = true`, and drop `--locked` from CI workflow `uv run` / `uv sync` invocations. See canonical/pytest-jubilant#98 for the exemplar PR and canonical/charm-tech#22 (references/decisions.md) for the recorded deferral of `no-build` and `--locked`: uv has no allow-list to exempt the workspace project from `no-build`, and rolling `exclude-newer` is fundamentally incompatible with `--locked` (uv records the resolved-at-lock-time timestamp in `uv.lock`, so every day CI recomputes a different value and `--locked` errors). Rolling `exclude-newer` is the surviving pattern.

…b-actions template The GitHub Actions ecosystem supports only cooldown.default-days; semver-major-days / semver-minor-days / semver-patch-days are supported on the uv/pip/gomod ecosystems but not github-actions. Leaving the key in fails the .github/dependabot.yaml validation check on every Dependabot PR, which in turn causes Dependabot to fall back to per-package PRs (no groups). Ref: https://docs.github.com/en/code-security/reference/supply-chain-security/dependabot-options-reference

Cover detect-tier, the check.py runner, and dependabot (the check with the most parsing logic). One pass + one fail per exercised check; each test writes a fixture tree and runs the real script via uv, matching production invocation. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

tonyandrewmeyer · 2026-07-21T07:55:32Z

+
+- The change is docs-only or a single bug fix — nothing in this skill applies.
+- The repo isn't a long-lived first-party project (forks of upstream projects, scratch repos, demo recordings).
+- The user is asking about *one* specific tool (e.g. "how do I add zizmor") — answer directly using [`references/decisions.md`](references/decisions.md) rather than running a full audit.


Suggested change

- The user is asking about *one* specific tool (e.g. "how do I add zizmor") — answer directly using [`references/decisions.md`](references/decisions.md) rather than running a full audit.

- The user is asking about *one* specific tool (for example, "how do I add zizmor") — answer directly using [`references/decisions.md`](references/decisions.md) rather than running a full audit.

tonyandrewmeyer · 2026-07-21T07:57:46Z

+| Check ID | Tiers | Mandate | Notes |
+|---|---|---|---|
+| `security-md` | all | SEC0025 / SEC0026 + V2.0 cross-cutting | File presence + disclosure-policy link. |
+| `dependabot` | all | SEC0025 | `.github/dependabot.{yml,yaml}` with ≥1 ecosystem **and** a cooldown of ≥7 days on each (Charm Tech baseline — charmlibs#499). Cooldown values validated via python3+PyYAML; falls back to a presence-only check when those are missing. |


Suggested change

| `dependabot` | all | SEC0025 | `.github/dependabot.{yml,yaml}` with ≥1 ecosystem **and** a cooldown of ≥7 days on each (Charm Tech baseline — charmlibs#499). Cooldown values validated via python3+PyYAML; falls back to a presence-only check when those are missing. |

| `dependabot` | all | SEC0025 | `.github/dependabot.{yml,yaml}` with ≥1 ecosystem **and** a cooldown of ≥7 days on each. Cooldown values validated via python3+PyYAML; falls back to a presence-only check when those are missing. |

tonyandrewmeyer · 2026-07-21T08:01:08Z

+
+```bash
+# audit
+scripts/check.py                                 # auto-detect tier, JSON output


Suggested change

scripts/check.py # auto-detect tier, JSON output

scripts/check.py # auto-detect tier, JSON output

tonyandrewmeyer · 2026-07-21T08:03:36Z

+
+## Tool pinning — `pyproject.toml` is the source of truth
+
+All tool version pins belong in `[dependency-groups]` in


Suggested change

All tool version pins belong in `[dependency-groups]` in

All Python tool version pins belong in `[dependency-groups]` in

tonyandrewmeyer · 2026-07-21T08:04:48Z

+SHA, same as any other third-party action. The prior ref-pin carve-out
+for GitHub-owned and PyPA-owned actions has been retired: the `pypa/*`
+open investigation surfaced `release/v1` as a moving *branch* (not a
+tag), and rather than special-case one org and leave the others on a
+weaker rationale, the whole allowlist is dropped. Uniform SHA-pinning is
+the policy.


Suggested change

SHA, same as any other third-party action. The prior ref-pin carve-out

for GitHub-owned and PyPA-owned actions has been retired: the `pypa/*`

open investigation surfaced `release/v1` as a moving *branch* (not a

tag), and rather than special-case one org and leave the others on a

weaker rationale, the whole allowlist is dropped. Uniform SHA-pinning is

the policy.

SHA, same as any other third-party action.

tonyandrewmeyer · 2026-07-21T08:05:11Z

+the same name. Done for Charm Tech in
+[canonical/canonical-repo-automation#873](https://github.com/canonical/canonical-repo-automation/pull/873)
+(merged 2026-06-09).


Suggested change

the same name. Done for Charm Tech in

[canonical/canonical-repo-automation#873](https://github.com/canonical/canonical-repo-automation/pull/873)

(merged 2026-06-09).

the same name.

tonyandrewmeyer · 2026-07-21T08:05:38Z

+OFF (action needed) for: api_demo_server (no releases yet; toggle when
+first release approaches), charmlibs.
+


Suggested change

OFF (action needed) for: api_demo_server (no releases yet; toggle when

first release approaches), charmlibs.

tonyandrewmeyer · 2026-07-21T08:05:59Z

+Resolved 2026-06-09:
+


Suggested change

Resolved 2026-06-09:

tonyandrewmeyer · 2026-07-21T08:06:23Z

+
+## CODEOWNERS — only required when membership exceeds the Charm Tech team
+
+Resolved 2026-07-02. The Canonical Security "Repository security" and


Suggested change

Resolved 2026-07-02. The Canonical Security "Repository security" and

The Canonical Security "Repository security" and

tonyandrewmeyer · 2026-07-21T08:06:59Z

+A repo without `CODEOWNERS` whose maintainer set is the Charm Tech team
+is **not** a gap. A check should not flag it. A check *should* flag a
+CODEOWNERS file that references only the Charm Tech team wholesale (no
+benefit; adds review friction) — i.e. the anti-pattern is a file that


Suggested change

benefit; adds review friction) — i.e. the anti-pattern is a file that

benefit; adds review friction) — that is, the anti-pattern is a file that

tonyandrewmeyer · 2026-07-21T08:07:13Z

+
+## PR-review ergonomics — conversation resolution and dismiss-stale-on-push not required
+
+Resolved 2026-07-02. The Canonical Security "How-To: Secure a repo"


Suggested change

Resolved 2026-07-02. The Canonical Security "How-To: Secure a repo"

The Canonical Security "How-To: Secure a repo"

tonyandrewmeyer · 2026-07-21T08:08:02Z

+Resolved 2026-07-02, after the Canonical Security "Repository security"
+and "How-To: Secure a repo" pages (published Jul 01, 2026) recommended
+`Require signed commits` on protected branches.
+


Suggested change

Resolved 2026-07-02, after the Canonical Security "Repository security"

and "How-To: Secure a repo" pages (published Jul 01, 2026) recommended

`Require signed commits` on protected branches.

tonyandrewmeyer · 2026-07-21T08:08:20Z

+Resolved 2026-07-03, after the first cross-repo rollout of the uv
+hardening pattern surfaced a fundamental gap in uv itself.
+


Suggested change

Resolved 2026-07-03, after the first cross-repo rollout of the uv

hardening pattern surfaced a fundamental gap in uv itself.

tonyandrewmeyer · 2026-07-21T08:08:59Z

+The fleet-wide scan (2026-07-02) confirmed 0/571 dependencies across the
+canonical/* uv projects are sdist-only, so `no-build` provides zero
+current benefit. The remaining pattern (`exclude-newer = "7 days"`,
+see the `--locked` decision below for why that's the only survivor)
+already gives a week's warning on any new dep before it can enter the
+resolution — the practical supply-chain protection is intact.
+


Suggested change

The fleet-wide scan (2026-07-02) confirmed 0/571 dependencies across the

canonical/* uv projects are sdist-only, so `no-build` provides zero

current benefit. The remaining pattern (`exclude-newer = "7 days"`,

see the `--locked` decision below for why that's the only survivor)

already gives a week's warning on any new dep before it can enter the

resolution — the practical supply-chain protection is intact.

tonyandrewmeyer · 2026-07-21T08:09:15Z

+Resolved 2026-07-03, immediately after the `no-build` deferral above.
+


Suggested change

Resolved 2026-07-03, immediately after the `no-build` deferral above.

tonyandrewmeyer · 2026-07-21T08:10:37Z

+
+    for path in ("CODE_OF_CONDUCT.md", "docs/CODE_OF_CONDUCT.md", ".github/CODE_OF_CONDUCT.md"):
+        p = Path(path)
+        if p.is_file():


This would be cleaner as if not: continue.

…-pin, don't tag-pin)

…mit-config carries rev:

Every check module, the umbrella runner, and detect-tier.py import from lib.common, but scripts/lib/common.py had never been committed: the Python-template `lib/` line in .gitignore has no leading slash, so it matched any directory named lib at any depth, including skills/engineering/charm-tech-baseline/scripts/lib/. Every commit that added files there silently failed to stage them, and git status stayed clean because git had never seen them. Without the file, all ~25 check modules and scripts/check.py fail with ModuleNotFoundError, and 8 of the skill's own tests fail. Nothing caught it because charm-tech has no .github/workflows/, so the suite has never run in CI. Scope `lib/` and `lib64/` to the repo root, which is what the template intended (a venv at the top level), and commit the file. With this change the suite goes from 8 failed to 8 passed.

Implements the five Layer 1 staleness checks from roadmap/26.10/repo-setup/agents-md-validation.md: commands parse and their entry-point tool resolves; safe (lint/format-check/unit-test/build) commands are executed and environment-gated ones (integration needing Docker/LXD/juju, root-only tests, anything side-effecting) are reported verify-manually with the gating dependency named; referenced paths, symbols, and named gocheck test suites resolve; tool-version pins in prose match what .github/workflows actually pin; and harness-shaped content (attribution trailers, tool hints) is scope-linted out. Validated against the four already-swept repos (pebble, pytest-jubilant, charm-ubuntu, api_demo_server) on their chore/agents-md branches, including reconstructing pebble's pre-fix stale PebbleSuite/cmd-pebble reference to confirm check 3 fires on it.

tonyandrewmeyer changed the title ~~feat(charm-tech-baseline): add repo-setup audit skill~~ feat: add repo-setup audit skill Jul 2, 2026

Merge branch 'main' into feat/add-charm-tech-baseline-skill

4817a67