ci: add CI Check 2 — capability matrix drift warning#60
Conversation
…warning - Wrap the drift-comment github-script body in try/catch so a read-only GITHUB_TOKEN on fork PRs (403 on write calls) can't fail the job; the ::warning:: annotations already carry the signal, the sticky comment is best-effort. - Restore contents: read alongside pull-requests: write on the check job's permissions block in all 4 validate-sdk-compliance-*.yml workflows, since any job-level permissions key zeroes out unlisted scopes and checkout needs contents: read. - Add docs/capability-matrix.md so the drift summary's doc link resolves. - Sanitize newlines out of featureId/symbol before building ::warning:: annotation messages to prevent workflow-command injection via YAML.
QuintinWillison
left a comment
QuintinWillison
left a comment
There was a problem hiding this comment.
In general I would aim to avoid anything in CI that doesn't fail hard, forcing immediate correction. I might be missing something in terms of wider context, but I'm not sure it's clear who's going to listen to the event "CI posts a warning". My fear is that this'll be a source of "warnings noise" that becomes unfixed annoyance rather than forced to be fixed due to it blocking CI green state.
We can lift to an error once all SDKs are in a good shape. Wdyt? |
2 participants
Summary
Implements SDK-993: a new non-blocking CI check that warns when a capability marked
implementedin an SDK'ssdk-compliance.yamlhas a registered symbol the parser can no longer find, or has no registered symbols at all to verify against.This is deliberately separate from and additive to Check 1 (
api-check.ts/check-api-symbols.ts), which only catches removal within the diff of the current PR and therefore can't see drift that accumulated silently across past PRs — the actual problem this check targets. Check 1's existing blocking behavior is left untouched.src/drift-check.ts— pure detection logic (checkDrift,formatDriftSummary)src/compliance-source-map.ts— maps YAML symbol/feature lines for annotations, using the existingyamldependencysrc/check-drift.ts— CLI wrapper, never blocks the build (noprocess.exit).github/actions/sdk-compliance-check-drift/action.yml— runs the CLI, then upserts/deletes a sticky PR comment viaactions/github-script(comment created only when drift exists, deleted once resolved)docs/capability-matrix.md— new doc, since the ticket's mandated warning message links to itFindings from the whole-branch review were fixed in a follow-up commit:
contents: readalongside the newpull-requests: writepermission (job-levelpermissions:zeroes unlisted scopes)Design spec and implementation plan:
docs/superpowers/specs/2026-07-10-ci-check-2-drift-warning-design.md,docs/superpowers/plans/2026-07-10-ci-check-2-drift-warning.md(local only —docs/superpowers/is gitignored in this repo).Test plan
npm test— 174/174 passingnpm run typecheck— cleannpm run validate— OKEnd-to-end test against a real SDK repo
Composite-action refs in reusable workflows can't be parameterized with
${{ inputs.* }}(a GitHub Actions restriction — same limitation already applies to Check 1's composite actions), sosdk-compliance-check-drift@maincan't be exercised pre-merge without pointing it at a real ref. Used a disposable branch (supabase/sdk@claude/sdk-993-e2e-test, since deleted) to test the full path against supabase-swift#1113:sdk-compliance.yamland real Swift symbol graph — found genuine drift: 9 registered-but-missing symbols (auth.mfa.enroll,auth.mfa.verify,auth.passkey.*,realtime.subscriptions.broadcast) and 8implementedentries with nosymbolsregistered at all.::warning::annotations printed correctly, pointing at the right compliance file.Test branch reverted to its original state afterward; no residue left in either repo.
🤖 Generated with Claude Code