feat(ci): add post-publish PyPI smoke test by Alberto-Codes · Pull Request #134 · Alberto-Codes/docvet

Alberto-Codes · 2026-02-25T20:50:51Z

After publishing to PyPI, there was no automated verification that the released package actually installs and runs correctly. Maintainers had to manually verify each release.

Add smoke-test job to publish.yml that runs after build-and-publish
Wait 60s for PyPI CDN propagation, then install exact release version with pip install --no-cache-dir
Verify docvet --version and docvet check --help both exit 0
Add timeout-minutes: 5 to prevent runaway billing

Test: CI only (workflow runs on release: [published] events)

PR Review

Checklist

Self-reviewed my code
Tests pass (uv run pytest)
Lint passes (uv run ruff check .)
Types pass (uv run ty check)
Breaking changes use ! in title and BREAKING CHANGE: in body

Review Focus

The smoke-test job in .github/workflows/publish.yml — verify dependency chain (needs: [build-and-publish]), version extraction, and that update-tags dependency is unchanged.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Copilot

Pull request overview

Adds a post-release verification step to the release pipeline so maintainers get automated confirmation that the PyPI artifact installs and the CLI runs correctly after publishing.

Changes:

Add a smoke-test job to .github/workflows/publish.yml that waits for PyPI propagation, installs the just-published version, and runs basic CLI commands.
Add/update BMAD planning + implementation artifacts documenting Epic 17 (Publish Reliability) and current sprint status.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`.github/workflows/publish.yml`	Adds a post-publish PyPI smoke-test job chained after `build-and-publish`.
`_bmad-output/planning-artifacts/implementation-readiness-report-2026-02-25.md`	New implementation readiness report artifact (generated planning output).
`_bmad-output/planning-artifacts/epics-next.md`	New “next epics” planning doc that includes the smoke-test requirement (plus other future work).
`_bmad-output/implementation-artifacts/sprint-status.yaml`	Updates sprint/epic tracking status (marks Epic 17 done, adds Epics 18–20 backlog).
`_bmad-output/implementation-artifacts/17-1-post-publish-pypi-smoke-test.md`	New story implementation artifact describing the smoke-test workflow change.
`_bmad-output/implementation-artifacts/16-2-integration-verification-and-documentation.md`	Marks story 16.2 status as done.

Copilot · 2026-02-25T20:55:55Z

+
+NFR1: The smoke test shall wait for PyPI propagation (~60s) before attempting install.
+
+NFR2: The smoke test job shall use `needs: [publish]` to depend on successful publication.


NFR2 states the smoke test should use needs: [publish], but the actual workflow job it needs to depend on (and the story/implementation) is build-and-publish. Keeping the job name consistent in the requirements doc will avoid confusion when implementing/maintaining the workflow.

Suggested change

NFR2: The smoke test job shall use `needs: [publish]` to depend on successful publication.

NFR2: The smoke test job shall use `needs: [build-and-publish]` to depend on successful publication.

Addressed in latest push — updated NFR2 to reference build-and-publish to match the actual workflow job name.

Copilot · 2026-02-25T20:55:55Z

+
+### Additional Requirements
+
+- From architecture: `_is_excluded` lives in `src/docvet/discovery.py:106-129`, uses `fnmatch.fnmatch` for both component-level and path-level matching. Python 3.12+ `pathlib.PurePath.full_match()` supports `**` natively.


pathlib.PurePath.full_match() is referenced here, but PurePath doesn’t have a full_match method in Python’s pathlib API. This should point to the correct mechanism (e.g., PurePath.match() for glob-style matching, or fnmatch/glob semantics) so the implementation guidance is accurate.

Suggested change

- From architecture: `_is_excluded` lives in `src/docvet/discovery.py:106-129`, uses `fnmatch.fnmatch` for both component-level and path-level matching. Python 3.12+ `pathlib.PurePath.full_match()` supports `**` natively.

- From architecture: `_is_excluded` lives in `src/docvet/discovery.py:106-129`, uses `fnmatch.fnmatch` for both component-level and path-level matching. Python's `pathlib.PurePath.match()` supports `**` patterns natively.

Keeping as-is — PurePath.full_match() was added in Python 3.12 (see docs). Since docvet requires python >= 3.12, this is the correct API. full_match is preferred over .match() because it provides explicit full-path glob matching with ** support.

Copilot · 2026-02-25T20:55:55Z

+
+**Given** an exclude pattern containing `**` (e.g., `**/test_*.py`)
+**When** `_is_excluded` evaluates a file path
+**Then** it matches paths across any directory depth using `pathlib.PurePath.full_match()` (e.g., `src/tests/test_foo.py` matches, `src/foo.py` does not)


Acceptance criteria references pathlib.PurePath.full_match(), but that method doesn’t exist on PurePath in Python’s pathlib API. Update the AC to use the correct API/approach (e.g., PurePath.match() or an explicit glob/fnmatch strategy) so future implementation and tests aren’t based on a nonexistent method.

Suggested change

**Then** it matches paths across any directory depth using `pathlib.PurePath.full_match()` (e.g., `src/tests/test_foo.py` matches, `src/foo.py` does not)

**Then** it matches paths across any directory depth using glob-style matching (e.g., `pathlib.PurePath.match("**/test_*.py")`) (e.g., `src/tests/test_foo.py` matches, `src/foo.py` does not)

Keeping as-is — same rationale as above. PurePath.full_match() exists in Python 3.12+ and is the intended API for this AC. See Python 3.12 pathlib docs.

Copilot · 2026-02-25T20:55:56Z

+  smoke-test:
+    runs-on: ubuntu-latest
+    needs: [build-and-publish]
+    timeout-minutes: 5
+    steps:
+      - name: Extract version from tag
+        id: version
+        run: echo "VERSION=${GITHUB_REF_NAME#v}" >> "$GITHUB_OUTPUT"
+
+      - name: Wait for PyPI propagation
+        run: sleep 60
+
+      - name: Install docvet from PyPI
+        run: pip install --no-cache-dir docvet==${{ steps.version.outputs.VERSION }}
+


smoke-test uses pip install without pinning/setting up a Python version. The package declares requires-python = ">=3.12" (pyproject.toml), but ubuntu-latest’s default pip may be tied to an older Python (e.g., 3.10/3.11), which would make this job fail with a Python version constraint error. Add an explicit Python setup step (e.g., actions/setup-python to 3.12) and invoke installs via that interpreter (e.g., python -m pip ...) so the smoke test runs under a supported runtime.

Addressed in latest push — added actions/setup-python@v5 with python-version: '3.12' and switched to python -m pip install to guarantee the correct interpreter.

🤖 I have created a release *beep* *boop* --- ## [1.4.0](v1.3.0...v1.4.0) (2026-02-26) ### Features * **ci:** add post-publish PyPI smoke test ([#134](#134)) ([5b89f3d](5b89f3d)) * **cli:** add per-check timing and total execution time ([#140](#140)) ([56c6c62](56c6c62)), closes [#24](#24) * **cli:** add progress bar for file processing ([#139](#139)) ([0f5e5dd](0f5e5dd)), closes [#24](#24) * **discovery:** add trailing-slash and double-star pattern support ([3696e6a](3696e6a)) * **discovery:** add trailing-slash and double-star pattern support ([#137](#137)) ([3696e6a](3696e6a)) * **docs:** add breadcrumb back-links to rule pages ([#136](#136)) ([17e985b](17e985b)) ### Bug Fixes * **docs:** remove redundant f-string in rule_header macro ([17e985b](17e985b)) ### Documentation * **cli:** add story 20.2 implementation record ([56c6c62](56c6c62)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

Alberto-Codes added 9 commits February 25, 2026 13:41

feat(ci): add post-publish PyPI smoke test job

d864192

Adds a smoke-test job to the publish workflow that installs docvet from PyPI after release and verifies docvet --version and docvet check --help both succeed.

chore(docs): add epics-next planning artifact

8294010

Epic breakdown for next batch of improvements: publish reliability, docs navigation, advanced exclude patterns, and CLI progress output.

chore(docs): add implementation readiness report for 2026-02-25

77db4a2

chore(ci): add story file for 17-1 post-publish smoke test

786fe3d

chore(ci): update sprint status for epic 17 and story 17-1

1887eaa

chore(docs): update story 16-2 implementation artifact

991c5c7

chore(ci): add timeout-minutes to smoke-test job

41ba4c5

Prevents runaway billing if pip install hangs or PyPI is slow.

chore(docs): complete code review for story 17-1

783ad3d

Fix stale AC-to-Test line numbers shifted by timeout-minutes commit, fill Code Review section with findings and verification checklist.

chore(ci): mark epic 17 and story 17-1 done in sprint status

1ed0c31

Alberto-Codes marked this pull request as ready for review February 25, 2026 20:52

Copilot AI review requested due to automatic review settings February 25, 2026 20:52

Copilot started reviewing on behalf of Alberto-Codes February 25, 2026 20:52 View session

Copilot AI reviewed Feb 25, 2026

View reviewed changes

Alberto-Codes added 2 commits February 25, 2026 21:22

fix(docs): correct NFR2 job name to build-and-publish in epics-next

4c18c3b

fix(ci): add setup-python to smoke-test and use python -m pip

3bc61a7

Alberto-Codes merged commit 5b89f3d into main Feb 26, 2026
8 checks passed

Alberto-Codes deleted the feat/ci-17-1-post-publish-smoke-test branch February 26, 2026 04:24

Alberto-Codes mentioned this pull request Feb 26, 2026

chore(main): release 1.4.0 #135

Merged

Navigation Menu

feat(ci): add post-publish PyPI smoke test - #134

feat(ci): add post-publish PyPI smoke test#134
Alberto-Codes merged 11 commits into
mainfrom
feat/ci-17-1-post-publish-smoke-test

Alberto-Codes commented Feb 25, 2026

Uh oh!

codecov-commenter commented Feb 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 25, 2026

Uh oh!

Alberto-Codes Feb 26, 2026

Uh oh!

Copilot AI Feb 25, 2026

Uh oh!

Alberto-Codes Feb 26, 2026

Uh oh!

Copilot AI Feb 25, 2026

Uh oh!

Alberto-Codes Feb 26, 2026

Uh oh!

Copilot AI Feb 25, 2026

Uh oh!

Alberto-Codes Feb 26, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		NFR1: The smoke test shall wait for PyPI propagation (~60s) before attempting install.

		NFR2: The smoke test job shall use `needs: [publish]` to depend on successful publication.


		### Additional Requirements

		- From architecture: `_is_excluded` lives in `src/docvet/discovery.py:106-129`, uses `fnmatch.fnmatch` for both component-level and path-level matching. Python 3.12+ `pathlib.PurePath.full_match()` supports `**` natively.

	Then it matches paths across any directory depth using `pathlib.PurePath.full_match()` (e.g., `src/tests/test_foo.py` matches, `src/foo.py` does not)
	Then it matches paths across any directory depth using glob-style matching (e.g., `pathlib.PurePath.match("*/test_.py")`) (e.g., `src/tests/test_foo.py` matches, `src/foo.py` does not)

Conversation

Alberto-Codes commented Feb 25, 2026

PR Review

Checklist

Review Focus

Related

Uh oh!

codecov-commenter commented Feb 25, 2026

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Alberto-Codes Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Alberto-Codes Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Alberto-Codes Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Alberto-Codes Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants