Skip to content

Machine-actionable quality gates (deterministic, self-explaining CI): raise from 4/5 to 5/5 (enterprise quality audit) #125

Description

@Kravalg

Description

This issue was produced by the automated enterprise quality & NFR audit (2026-07-02), which audited the repository against the quality attributes from Wikipedia's "List of system quality attributes" plus AI-native development readiness. Attribute: machine-actionable quality gates (deterministic, self-explaining CI), cluster: AI-Native Autonomous Development Readiness, current score: 4/5. Audit justification: Determinism is institutionalized and self-enforced: every action is pinned to a full commit SHA and a structural test (test_actions_are_pinned_to_full_commit_shas in tests/pulumi/test_delivery_contracts.py) fails CI otherwise; the Dockerfile digest-pins the base image and sha256-verifies Pulumi/awscli/uv/actionlint/gitleaks/hadolint downloads; uv lock --check gates dependency drift; all jobs have timeout-minutes and concurrency groups. Gates self-explain — the destructive gate prints the exact override label to apply, test-coverage tells you which suites to run first, and preview summaries land in GITHUB_STEP_SUMMARY. Remaining gaps: failures are only log-text (no JUnit/SARIF artifacts or GitHub annotations for pytest/ruff suites an agent could parse), and pip-audit --strict makes the PR gate time-varying with no in-repo vulnerability-exception mechanism.

Test and lint gates emit no machine-readable results — pulumi-unit.yml, pulumi-integration.yml, pulumi-policy.yml, python-quality.yml, and pulumi-structural.yml run pytest/ruff inside Docker and surface failures only as raw job logs. Only Scorecard uploads SARIF (nightly-guardrails.yml). An agent or dashboard triaging a red check must scrape unstructured logs instead of reading a JUnit XML artifact or file/line annotations, which raises the cost of autonomous fix loops across many downstream template consumers.

pip-audit --strict is a time-varying blocker with no documented exception path — make test-deps-security runs pip-audit --strict against the locked environment; a newly published CVE turns unrelated PRs red with no code change, undermining the 'deterministic CI' property. There is no --ignore-vuln allowlist file, no documented triage/waiver workflow in docs/ci-guardrails.md or docs/security-baseline.md, so the only unblocking options are an emergency dependency bump or weakening the gate ad hoc.

Tasks

  • Add --junitxml=.artifacts/test-results/.xml to the pytest invocations in the Makefile (via PYTEST_ADDOPTS) and upload .artifacts/test-results as workflow artifacts in the corresponding workflows
  • Enable ruff's GitHub output format (ruff check --output-format=github) in CI so violations become file/line annotations
  • Extend the structural tests to assert the workflows upload the results artifact
  • Add a version-controlled exception mechanism (e.g., a pip-audit ignore list file passed via --ignore-vuln, with mandatory expiry comments) wired into make test-deps-security
  • Document the waiver workflow (who approves, expiry, follow-up issue) in docs/security-baseline.md
  • Add a structural test asserting every ignore entry carries an expiry/justification

Acceptance Criteria

  • Every failed PR test workflow exposes a JUnit XML artifact naming the failing test and file
  • Ruff violations appear as inline PR annotations
  • Structural tests fail if a test workflow stops uploading results
  • A new upstream advisory can be waived by a reviewed one-line PR without disabling the gate
  • Expired waivers fail CI automatically
  • The waiver policy is documented and cross-linked from docs/ci-guardrails.md
  • Attribute re-scores 5/5 in a follow-up enterprise quality audit

Metadata

Metadata

Assignees

No one assigned

    Labels

    devopsdocumentationImprovements or additions to documentationenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions