From 4c2ff7f8ca0011d7cbb02e1668563316608a1e2b Mon Sep 17 00:00:00 2001 From: edavidaja Date: Wed, 1 Jul 2026 09:11:15 -0400 Subject: [PATCH 01/19] docs: add great-docs migration design spec --- .../2026-07-01-great-docs-migration-design.md | 164 ++++++++++++++++++ 1 file changed, 164 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-01-great-docs-migration-design.md diff --git a/docs/superpowers/specs/2026-07-01-great-docs-migration-design.md b/docs/superpowers/specs/2026-07-01-great-docs-migration-design.md new file mode 100644 index 000000000..ff09712c2 --- /dev/null +++ b/docs/superpowers/specs/2026-07-01-great-docs-migration-design.md @@ -0,0 +1,164 @@ +# Migrate rsconnect-python docs from mkdocs to great-docs + +**Date:** 2026-07-01 +**Status:** Approved for planning + +## Goal + +Replace the mkdocs/Material documentation toolchain with +[great-docs](https://posit-dev.github.io/great-docs/) (a Quarto-based framework) +while preserving the site's current audience-facing behavior. + +Preserve: + +- Canonical URL `docs.posit.co/rsconnect-python` via **S3 hosting** with + tag-based promotion (great-docs replaces only the build step; CI keeps syncing + the built directory to S3). +- Current site **scope**: CLI reference + narrative guides + changelog. **No + Python API reference** — the public contract is the CLI, not rsconnect's + importable internals. +- **Single-version** behavior (one "latest" site), matching today's + promotion-on-tag model. +- **PR previews**. + +Change: + +- Changelog moves to great-docs' **auto-generated-from-GitHub-Releases** model, + after a one-time backfill of Release bodies from the existing `CHANGELOG.md`. + +Strategy: **spike to de-risk, then a single cutover PR** (chosen over big-bang +and parallel dual-build). + +## Current state (baseline) + +- `mkdocs.yml` — Material theme, `docs/overrides/` header/footer partials, custom + CSS, Posit logos, Google Tag Manager analytics (`GTM-KHBDBW7`). +- Narrative content (markdown): `docs/index.md`, + `docs/programmatic-provisioning.md`, `docs/deploying.md`, + `docs/server-administration.md`. +- CLI reference: 16 stub files in `docs/commands/*.md`, each a + `::: mkdocs-click` directive pointing at command objects in `rsconnect.main`. +- `mkdocs-macros` injects `{{ rsconnect_python.version }}` (from a `VERSION` env + var) into `deploying.md`. +- `docs/CHANGELOG.md` maintained by hand (Keep-a-Changelog format, 40+ versioned + sections plus an `## Unreleased` section); treated as release source of truth. +- Builds to `site/`, deployed to S3; promoted on tag. PR previews via + `pr-preview-action` publishing `./site/`. +- Markdown extensions in use: admonitions, `pymdownx.tabbed`, snippets, mermaid, + footnotes, magiclink, keys. + +### great-docs facts established during research + +- Quarto-based, Python ≥3.11, **requires the Quarto CLI installed**. +- Config in `great-docs.yml` at project root; **auto-generates `_quarto.yml`** + (do not hand-edit — overwritten each build). +- CLI docs: `cli: enabled: true`; auto-discovers the Click group. + `rsconnect.main` is a standard discovery location. +- Changelog auto-generated from GitHub Releases (`changelog.enabled`, default on); + reads repo URL from `pyproject.toml` `[project.urls] Repository`. +- Output directory is `great-docs/_site/`. Deployment to **S3 is explicitly + supported** (sync that directory). +- Build/preview via the `great-docs` CLI (`great-docs build`, + `great-docs setup-github-pages`; preview via build + open). +- **Empty Release bodies today:** all ~89 tags (back to `1.5.0b1`, 2020) have no + notes. The real history lives entirely in `CHANGELOG.md`. + +## Phases + +### Phase 0 — Spike (throwaway branch) + +Validate load-bearing unknowns against the real package before committing to the +rewrite. Produces a short findings note that either greenlights Phase 2 or flags +required design changes. + +1. Can great-docs build **CLI-only** with the API reference disabled/absent? + (Biggest unknown — great-docs is primarily an API-doc generator.) Fallback: + accept a minimal auto-generated reference, or revisit scope. +2. Does CLI auto-discovery of `rsconnect.main`'s Click group produce acceptable + output for all 16 commands? +3. Can GTM analytics (`GTM-KHBDBW7`) be injected given great-docs owns + `_quarto.yml`? +4. What is the Quarto/great-docs equivalent for the `{{ rsconnect_python.version }}` + substitution used in `deploying.md`? +5. Confirm `great-docs/_site/` output syncs to S3 cleanly. + +### Phase 1 — Changelog backfill (independent, reversible) + +One-time script: + +- Parse `docs/CHANGELOG.md` by `## [X.Y.Z] - YYYY-MM-DD` section. +- Map each section to its matching git tag (`X.Y.Z`); handle `bN` prereleases. +- Populate each empty GitHub Release body via `gh release edit --notes-file -`. +- Verify a sample of releases renders correctly. + +Leaves the `## Unreleased` block in `CHANGELOG.md` (GitHub Releases can't +represent an untagged section). Going forward, release notes are authored in the +GitHub Release; update `CLAUDE.md`'s release section to reflect the shifted +source of truth. This phase is reversible (touches only Release notes, not +artifacts) and can run before or in parallel with the site work. + +### Phase 2 — great-docs scaffolding + +- Add `great-docs.yml`: CLI enabled, changelog enabled (auto), API reference + disabled, homepage + user-guide layout, Posit branding. +- Swap the `docs` dependency group in `pyproject.toml`: remove mkdocs packages, + add `great-docs`. +- Add Quarto CLI to the docs CI job. +- Confirm `_quarto.yml` is generated and git-ignored. +- Resolve exact source-content layout great-docs expects (`great-docs/` dir vs + root `user_guide/`) — informed by Phase 0. + +### Phase 3 — Content migration + +- **Narrative → `.qmd` (all pages, uniformly):** `index.qmd`, + `programmatic-provisioning.qmd`, `deploying.qmd`, `server-administration.qmd`. + No `.md` carve-out — uniform `.qmd` enables executable examples, version + substitution, and future version fences without a later format migration. + Convert mkdocs-isms to Quarto equivalents: admonitions → callouts, + `pymdownx.tabbed` → tabsets, snippets → includes, mermaid (native), + footnotes/magiclink. +- **CLI reference:** delete the 16 `docs/commands/*.md` mkdocs-click stubs; + great-docs generates this section. +- **Branding:** port Posit logos, favicon, custom CSS, and footer/header intent + into great-docs theming config; drop `docs/overrides/`. +- **Changelog:** rendered via auto-changelog (Phase 1 makes the history + complete). + +### Phase 4 — Build tooling & CI + +- `justfile`: `docs` / `docs-serve` → `great-docs build` / preview; S3 sync + recipes retarget `site/` → `great-docs/_site/`. +- `.github/workflows/main.yml` docs job: install Quarto, run `just docs`; + sync/promote steps unchanged in intent, pointed at the new path. +- `.github/workflows/preview-docs.yml`: point `source-dir` at + `great-docs/_site/`. +- Update or remove `docs/requirements.txt` (superseded by the dependency group). + +### Phase 5 — Cutover & cleanup + +- Remove `mkdocs.yml`, mkdocs dependencies, `docs/overrides/`, and the + mkdocs-click command stubs. +- Update `CLAUDE.md` (docs commands, changelog/release process). +- Verify built-site parity before merge (see Validation). + +## Risks / open questions carried into the plan + +- **API-reference-disable feasibility** (Phase 0 gate) — the biggest unknown, + given great-docs' purpose. +- **GTM injection** — if unsupported, decide between Quarto-native Google + Analytics or a custom head include if great-docs exposes one. +- **Content dir layout** — exactly where great-docs expects source, resolved in + Phase 0/2. +- **URL/anchor stability** — great-docs' CLI page structure differs from the + current per-command nav; existing deep links (e.g. `commands/deploy/`) may + change. Enumerate any inbound links that break and decide whether redirects + are needed. + +## Validation / testing + +- Phase 0 spike findings note. +- Local `great-docs build` producing a complete `great-docs/_site/`. +- Manual parity checklist: all narrative pages present; all 16 CLI commands + documented; changelog history complete; `{{ version }}` substitution resolves; + Posit branding present; GTM/analytics firing. +- PR-preview visual check before the S3 cutover. From a20f860a63e1ed66fad20185b9c6ab17b9c08b84 Mon Sep 17 00:00:00 2001 From: edavidaja Date: Wed, 1 Jul 2026 09:16:38 -0400 Subject: [PATCH 02/19] docs: add great-docs migration implementation plan --- .../plans/2026-07-01-great-docs-migration.md | 677 ++++++++++++++++++ 1 file changed, 677 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-01-great-docs-migration.md diff --git a/docs/superpowers/plans/2026-07-01-great-docs-migration.md b/docs/superpowers/plans/2026-07-01-great-docs-migration.md new file mode 100644 index 000000000..a680e5771 --- /dev/null +++ b/docs/superpowers/plans/2026-07-01-great-docs-migration.md @@ -0,0 +1,677 @@ +# great-docs Migration Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Replace the mkdocs/Material documentation toolchain with great-docs (Quarto-based) while preserving the `docs.posit.co/rsconnect-python` S3 site, CLI-reference + narrative + changelog scope, and PR previews. + +**Architecture:** great-docs reads `great-docs.yml`, auto-discovers the Click CLI from `rsconnect.main`, auto-generates a changelog from GitHub Releases, and builds a static site to `great-docs/_site/`. CI keeps syncing that directory to the existing S3 buckets. A one-time script backfills the empty GitHub Release bodies from the existing `CHANGELOG.md` so the auto-changelog has full history. Narrative pages become `.qmd`. + +**Tech Stack:** great-docs, Quarto CLI, uv, GitHub Actions, AWS S3, `gh` CLI, pytest. + +## Global Constraints + +- Hosting stays on **S3** (`s3://rstudio-connect-downloads/connect/rsconnect-python/latest/docs/` and `s3://docs.rstudio.com/rsconnect-python/`); great-docs replaces only the build step. +- Site scope: CLI reference + narrative guides + changelog. **No Python API reference.** +- **Single-version** site (one "latest"), matching current promotion-on-tag behavior. +- great-docs output directory is `great-docs/_site/` (all S3/preview paths point here, not `site/`). +- great-docs requires **Python ≥3.11 and the Quarto CLI**; the package itself still supports Python ≥3.8. Run great-docs via `uv run --python 3.12 --with great-docs ...` so its Python floor never conflicts with the project's `requires-python = ">=3.8"`. +- Repository URL for the auto-changelog comes from `pyproject.toml` `[project.urls] Repository`. +- Git tags have **no `v` prefix** (e.g. `1.29.0`). CHANGELOG version headers are `## [X.Y.Z] - YYYY-MM-DD`. + +--- + +## File Structure + +- `docs/superpowers/spikes/2026-07-01-great-docs-findings.md` — spike outcomes (Task 1). +- `scripts/backfill_release_notes.py` — CHANGELOG → GitHub Releases backfill (Tasks 2–3). +- `tests/test_backfill_release_notes.py` — parser unit tests (Task 2). +- `great-docs.yml` — great-docs config (Task 4). +- `great-docs/user_guide/*.qmd` — migrated narrative pages (Task 5). Exact directory confirmed by Task 1. +- `great-docs/_variables.yml` (generated at build) — version substitution (Tasks 5, 7). +- `pyproject.toml` — drop the `docs` dependency group (Task 4). +- `justfile` — `docs`/`docs-serve`/`clean` and S3 recipes retargeted (Task 7). +- `.github/workflows/main.yml`, `.github/workflows/preview-docs.yml` — Quarto install + path changes (Task 7). +- Removed in Task 8: `mkdocs.yml`, `docs/commands/`, `docs/overrides/`, `docs/css/`, `docs/requirements.txt`, `docs/index.md`, `docs/deploying.md`, `docs/programmatic-provisioning.md`, `docs/server-administration.md`, `docs/.gitignore`. + +--- + +## Task 1: Spike — validate great-docs feasibility + +Exploratory task. Produces a findings note that greenlights Tasks 4–8 or records required deviations. Work on a throwaway branch; nothing here needs to merge except the findings note. + +**Files:** +- Create: `docs/superpowers/spikes/2026-07-01-great-docs-findings.md` + +**Interfaces:** +- Produces: confirmed values for downstream tasks — (a) how to disable the API reference in `great-docs.yml`; (b) the source-content directory great-docs expects (assumed `great-docs/user_guide/`); (c) the working version-substitution mechanism (assumed Quarto `_variables.yml` + `{{< var rsconnect_python.version >}}`); (d) whether GTM can be injected; (e) that `great-docs build` emits to `great-docs/_site/`. + +- [ ] **Step 1: Scaffold a throwaway great-docs project in a scratch dir** + +```bash +git checkout -b spike/great-docs +uv run --python 3.12 --with great-docs great-docs config +``` +Expected: a starter `great-docs.yml` is written listing all documented options. Read it to locate the keys for CLI, changelog, reference, user-guide dir, and analytics. + +- [ ] **Step 2: Probe CLI-only build with the API reference disabled** + +Edit `great-docs.yml` to enable the CLI, enable the changelog, and disable/omit the API reference. Then: +```bash +uv run --python 3.12 --with great-docs great-docs build +ls great-docs/_site/ +``` +Record in the findings note: the exact YAML that disables the reference, whether the build succeeds with no reference, and whether the 16 `rsconnect` subcommands are auto-discovered from `rsconnect.main`. If the reference cannot be disabled, record the fallback (accept a minimal auto-reference) per the spec. + +- [ ] **Step 3: Probe version substitution** + +Create `great-docs/_variables.yml` with `rsconnect_python:\n version: "9.9.9"` and put `{{< var rsconnect_python.version >}}` in a test page under the user-guide dir. Build and grep the output HTML for `9.9.9`. Record whether Quarto var substitution works within the great-docs source layout; if not, record the fallback (build-time `sed` replacement in the `docs` recipe). + +- [ ] **Step 4: Probe GTM/analytics injection** + +Search the `great-docs config` output and great-docs docs for an analytics/head-include key. Attempt to inject `GTM-KHBDBW7`. Record whether it is supported, and if not, the fallback decision (Quarto-native `google-analytics` vs. a custom head include vs. dropping GTM). + +- [ ] **Step 5: Write the findings note and clean up** + +Write `docs/superpowers/spikes/2026-07-01-great-docs-findings.md` capturing the confirmed values (a)–(e) above with the exact YAML snippets. Then: +```bash +git checkout . # discard scratch config +git switch -c feat/great-docs-migration main +git add docs/superpowers/spikes/2026-07-01-great-docs-findings.md +git commit -m "docs: record great-docs spike findings" +``` +Expected: findings note committed on the migration branch; scratch artifacts discarded. + +--- + +## Task 2: Changelog backfill parser + +**Files:** +- Create: `scripts/backfill_release_notes.py` +- Test: `tests/test_backfill_release_notes.py` + +**Interfaces:** +- Produces: `parse_changelog(text: str) -> dict[str, str]` — maps each released version (e.g. `"1.29.0"`) to its release-notes body (markdown). Excludes the `Unreleased` section and the trailing link-reference definitions. Consumed by Task 3. + +- [ ] **Step 1: Write the failing test** + +```python +# tests/test_backfill_release_notes.py +from scripts.backfill_release_notes import parse_changelog + +SAMPLE = """# Changelog + +Some preamble. + +## Unreleased + +- unreleased thing + +## [1.29.0] - 2026-04-29 + +- Added `rsconnect deploy nodejs` command. + +### Added + +- `rsconnect content get-lockfile` command. + +## [1.28.2] - 2025-12-05 + +### Fixed + +- Corrected Changelog. + +[Unreleased]: https://github.com/posit-dev/rsconnect-python/compare/1.5.0...HEAD +[1.5.0]: https://github.com/posit-dev/rsconnect-python/releases/tag/1.5.0 +""" + + +def test_parse_changelog_extracts_versions_only(): + entries = parse_changelog(SAMPLE) + assert set(entries) == {"1.29.0", "1.28.2"} + + +def test_parse_changelog_excludes_unreleased_and_link_refs(): + entries = parse_changelog(SAMPLE) + assert "unreleased thing" not in "".join(entries.values()) + assert "compare/1.5.0" not in "".join(entries.values()) + + +def test_parse_changelog_keeps_body_and_subsections(): + entries = parse_changelog(SAMPLE) + assert "deploy nodejs" in entries["1.29.0"] + assert "### Added" in entries["1.29.0"] + assert "get-lockfile" in entries["1.29.0"] + assert entries["1.28.2"].startswith("### Fixed") + + +def test_parse_changelog_strips_surrounding_blank_lines(): + entries = parse_changelog(SAMPLE) + assert not entries["1.29.0"].startswith("\n") + assert not entries["1.29.0"].endswith("\n") +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run --group test pytest tests/test_backfill_release_notes.py -v` +Expected: FAIL — `ModuleNotFoundError` / `parse_changelog` not defined. + +- [ ] **Step 3: Write minimal implementation** + +```python +# scripts/backfill_release_notes.py +"""Backfill empty GitHub Release bodies from docs/CHANGELOG.md. + +The GitHub Releases for this project have empty bodies; the real history +lives in docs/CHANGELOG.md (Keep a Changelog format). This one-time script +parses that file and populates each matching Release so great-docs can +generate its Changelog page from Releases. +""" +from __future__ import annotations + +import argparse +import re +import subprocess +import sys +from typing import Dict + +VERSION_HEADER = re.compile( + r"^## \[(?P\d+\.\d+\.\d+(?:[a-z]+\d+)?)\](?: - \S+)?\s*$" +) +LINK_REF = re.compile(r"^\[[^\]]+\]:\s+https?://") + + +def parse_changelog(text: str) -> Dict[str, str]: + """Split a Keep-a-Changelog document into {version: body}. + + Skips the 'Unreleased' section and trailing link-reference + definitions. Body is the markdown between a version header and the + next '## ' header, with surrounding blank lines stripped. + """ + entries: Dict[str, str] = {} + current: str | None = None + buffer: list[str] = [] + + def flush() -> None: + if current is not None: + entries[current] = "\n".join(buffer).strip() + + for line in text.splitlines(): + if line.startswith("## "): + flush() + match = VERSION_HEADER.match(line) + current = match.group("version") if match else None + buffer = [] + continue + if LINK_REF.match(line): + continue + if current is not None: + buffer.append(line) + + flush() + return entries +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `uv run --group test pytest tests/test_backfill_release_notes.py -v` +Expected: PASS (all four tests). + +- [ ] **Step 5: Commit** + +```bash +git add scripts/backfill_release_notes.py tests/test_backfill_release_notes.py +git commit -m "feat: parse CHANGELOG.md into per-version release notes" +``` + +--- + +## Task 3: Changelog backfill execution + +**Files:** +- Modify: `scripts/backfill_release_notes.py` (add CLI entry point around `parse_changelog`) +- Test: `tests/test_backfill_release_notes.py` (add plan-building test) + +**Interfaces:** +- Consumes: `parse_changelog` from Task 2. +- Produces: `build_backfill_plan(entries, existing_tags, release_has_body) -> list[tuple[str, str]]` — the ordered `(tag, body)` edits to apply, limited to releases that exist and currently have an empty body. `main()` runs a dry-run by default and applies with `--apply`. + +- [ ] **Step 1: Write the failing test** + +```python +# add to tests/test_backfill_release_notes.py +from scripts.backfill_release_notes import build_backfill_plan + + +def test_build_backfill_plan_skips_missing_and_nonempty(): + entries = {"1.29.0": "body A", "1.28.2": "body B", "1.0.0": "old"} + existing_tags = {"1.29.0", "1.28.2"} # 1.0.0 has no release + release_has_body = {"1.29.0": False, "1.28.2": True} # 1.28.2 already has notes + plan = build_backfill_plan(entries, existing_tags, release_has_body) + assert plan == [("1.29.0", "body A")] +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run --group test pytest tests/test_backfill_release_notes.py::test_build_backfill_plan_skips_missing_and_nonempty -v` +Expected: FAIL — `build_backfill_plan` not defined. + +- [ ] **Step 3: Write minimal implementation** + +```python +# add to scripts/backfill_release_notes.py + +def build_backfill_plan( + entries: Dict[str, str], + existing_tags: set[str], + release_has_body: Dict[str, bool], +) -> list[tuple[str, str]]: + """Return (tag, body) edits for releases that exist and are empty.""" + plan: list[tuple[str, str]] = [] + for version, body in entries.items(): + if version not in existing_tags: + continue + if release_has_body.get(version, False): + continue + plan.append((version, body)) + return plan + + +def _gh_releases() -> Dict[str, bool]: + """Map existing release tag -> whether its body is non-empty.""" + out = subprocess.run( + ["gh", "release", "list", "--limit", "500", "--json", "tagName,name"], + check=True, capture_output=True, text=True, + ).stdout + import json + result: Dict[str, bool] = {} + for rel in json.loads(out): + tag = rel["tagName"] + body = subprocess.run( + ["gh", "release", "view", tag, "--json", "body", "-q", ".body"], + check=True, capture_output=True, text=True, + ).stdout.strip() + result[tag] = bool(body) + return result + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("changelog", nargs="?", default="docs/CHANGELOG.md") + parser.add_argument("--apply", action="store_true", help="Actually edit releases") + args = parser.parse_args(argv) + + with open(args.changelog, encoding="utf-8") as handle: + entries = parse_changelog(handle.read()) + + body_by_tag = _gh_releases() + existing = set(body_by_tag) + plan = build_backfill_plan(entries, existing, body_by_tag) + + for tag, body in plan: + print(f"{'APPLY' if args.apply else 'DRY-RUN'}: {tag} ({len(body)} chars)") + if args.apply: + subprocess.run( + ["gh", "release", "edit", tag, "--notes-file", "-"], + input=body, text=True, check=True, + ) + print(f"{len(plan)} release(s) {'updated' if args.apply else 'would be updated'}.") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `uv run --group test pytest tests/test_backfill_release_notes.py -v` +Expected: PASS (all tests). + +- [ ] **Step 5: Dry-run against the real repo and spot-check** + +Run: `uv run python scripts/backfill_release_notes.py` +Expected: prints `DRY-RUN: 1.29.0 (...)` etc. for the released versions present in CHANGELOG that currently have empty bodies; ends with `N release(s) would be updated.` Sanity-check N against the ~40 versioned CHANGELOG sections. + +- [ ] **Step 6: Apply the backfill and verify one release** + +Run: +```bash +uv run python scripts/backfill_release_notes.py --apply +gh release view 1.29.0 --json body -q '.body' +``` +Expected: the apply run reports the releases updated; `gh release view 1.29.0` now shows the migrated notes (matching the `## [1.29.0]` CHANGELOG section). + +- [ ] **Step 7: Commit** + +```bash +git add scripts/backfill_release_notes.py tests/test_backfill_release_notes.py +git commit -m "feat: backfill GitHub Release notes from CHANGELOG" +``` + +--- + +## Task 4: great-docs scaffolding & config + +**Files:** +- Create: `great-docs.yml` +- Modify: `pyproject.toml` (remove the `docs` dependency group) + +**Interfaces:** +- Consumes: spike-confirmed keys from Task 1 (reference-disable YAML, user-guide dir, analytics support). +- Produces: a `great-docs.yml` that builds a CLI + changelog site (no API reference) to `great-docs/_site/`, consumed by Tasks 5–7. + +- [ ] **Step 1: Write `great-docs.yml`** + +Use the keys confirmed in Task 1's findings note. Baseline (adjust key names to the findings): + +```yaml +# great-docs.yml +site: + title: rsconnect-python +cli: + enabled: true + module: rsconnect.main +changelog: + enabled: true + max_releases: 100 +reference: + enabled: false # per Task 1 findings; if unsupported, use the recorded fallback +user_guide: great-docs/user_guide +homepage: user_guide +``` + +- [ ] **Step 2: Remove the mkdocs `docs` dependency group** + +Delete the `docs = [...]` block from `[dependency-groups]` in `pyproject.toml` (great-docs runs via `uv run --with great-docs`, not the group). Leave `[project.urls] Repository` intact — the auto-changelog needs it. + +Confirm the block removed: +```bash +grep -n "mkdocs" pyproject.toml +``` +Expected: no output. + +- [ ] **Step 3: Build to verify the config is valid** + +Run: +```bash +uv run --python 3.12 --with great-docs great-docs build +``` +Expected: build succeeds; `great-docs/_site/index.html` exists, a CLI reference page exists, and a Changelog page renders with the backfilled history from Task 3. + +- [ ] **Step 4: Ignore generated build artifacts** + +Create/replace `.gitignore` entries so the generated site and Quarto internals are not committed: +```bash +printf '_site/\n.quarto/\n_quarto.yml\n_variables.yml\n' > great-docs/.gitignore +``` +(Adjust to the actual generated-file set observed in Step 3.) + +- [ ] **Step 5: Commit** + +```bash +git add great-docs.yml great-docs/.gitignore pyproject.toml +git commit -m "build: add great-docs config, drop mkdocs deps" +``` + +--- + +## Task 5: Migrate narrative content to `.qmd` + +The four narrative pages are plain markdown; the only mkdocs-specific construct across all of them is the single `{{ rsconnect_python.version }}` macro in `deploying.md`. Migration is: move to the great-docs user-guide dir as `.qmd`, and convert that one macro to the spike-confirmed substitution mechanism. Page prose is preserved verbatim otherwise. + +**Files:** +- Create: `great-docs/user_guide/index.qmd`, `deploying.qmd`, `programmatic-provisioning.qmd`, `server-administration.qmd` (from the matching `docs/*.md`) +- Create: `great-docs/_variables.yml` (checked-in placeholder; regenerated at build in Task 7) + +**Interfaces:** +- Consumes: `great-docs.yml` `user_guide` path (Task 4); version-substitution mechanism (Task 1). +- Produces: rendered user-guide pages consumed by the build in Tasks 7–8. + +- [ ] **Step 1: Copy the four pages into the user-guide dir as `.qmd`** + +```bash +mkdir -p great-docs/user_guide +for f in index deploying programmatic-provisioning server-administration; do + cp "docs/$f.md" "great-docs/user_guide/$f.qmd" +done +``` + +- [ ] **Step 2: Convert the version macro in `deploying.qmd`** + +Replace the mkdocs macro with the Quarto var (per Task 1 findings). In `great-docs/user_guide/deploying.qmd`, change: +``` +Generated from rsconnect-python {{ rsconnect_python.version }} +``` +to: +``` +Generated from rsconnect-python {{< var rsconnect_python.version >}} +``` +(If Task 1 recorded the `sed` fallback instead, leave a literal placeholder token here and note it for Task 7's build recipe.) + +- [ ] **Step 3: Add a checked-in `_variables.yml`** + +```bash +printf 'rsconnect_python:\n version: "dev"\n' > great-docs/_variables.yml +``` +(Task 7 overwrites this at build time with the real version. `dev` is the local-build fallback.) + +- [ ] **Step 4: Add page ordering/titles if required by great-docs** + +If the findings note shows great-docs orders user-guide pages by front matter or a nav key, add a `title:` (and `order:`) front-matter block to each `.qmd` matching the current mkdocs nav labels: `Getting Started` (index), `Programmatic Provisioning`, `Deploying Content`, `Server Administration`. Otherwise skip. + +- [ ] **Step 5: Build and verify pages + substitution render** + +Run: +```bash +uv run --python 3.12 --with great-docs great-docs build +grep -rl "rsconnect-python dev" great-docs/_site/ || echo "MISSING version substitution" +``` +Expected: build succeeds; all four pages present in `great-docs/_site/`; the version string resolved to `dev` (not the literal `{{ ... }}`). + +- [ ] **Step 6: Commit** + +```bash +git add great-docs/user_guide great-docs/_variables.yml +git commit -m "docs: migrate narrative pages to great-docs qmd" +``` + +--- + +## Task 6: Branding & analytics + +**Files:** +- Create: `great-docs/` asset files (logo, favicon, custom CSS) as the findings note dictates +- Modify: `great-docs.yml` (theme/branding/analytics keys) + +**Interfaces:** +- Consumes: analytics support finding (Task 1); existing assets under `docs/images/` and `docs/css/custom.css`. +- Produces: a branded site with GTM (or the recorded analytics fallback). + +- [ ] **Step 1: Copy brand assets into the great-docs project** + +```bash +mkdir -p great-docs/assets +cp docs/images/iconPositConnect.svg docs/images/favicon.ico great-docs/assets/ +cp docs/css/custom.css great-docs/assets/custom.css +``` + +- [ ] **Step 2: Wire logo, favicon, and CSS into `great-docs.yml`** + +Add the theme keys confirmed in Task 1 (logo, favicon, extra CSS) pointing at `great-docs/assets/`. Keep it minimal — only what maps to the current Material config (Posit logo, favicon, custom CSS). + +- [ ] **Step 3: Configure analytics** + +Add GTM (`GTM-KHBDBW7`) via the analytics/head-include key from Task 1. If GTM injection is unsupported, apply the recorded fallback (Quarto-native `google-analytics`, or omit with a note). + +- [ ] **Step 4: Build and verify branding + analytics** + +Run: +```bash +uv run --python 3.12 --with great-docs great-docs build +grep -rl "GTM-KHBDBW7" great-docs/_site/ || echo "analytics not injected (check fallback)" +``` +Expected: build succeeds; logo/favicon present in output; analytics snippet present (or fallback confirmed per findings). + +- [ ] **Step 5: Commit** + +```bash +git add great-docs.yml great-docs/assets +git commit -m "docs: port Posit branding and analytics to great-docs" +``` + +--- + +## Task 7: Build tooling & CI + +**Files:** +- Modify: `justfile` (`docs`, `docs-serve`, `clean`, `sync-latest-docs-to-s3`, `promote-docs-in-s3`) +- Modify: `.github/workflows/main.yml` (docs job) +- Modify: `.github/workflows/preview-docs.yml` + +**Interfaces:** +- Consumes: build command and `_variables.yml` mechanism (Tasks 1, 5); output dir `great-docs/_site/`. +- Produces: CI that builds with great-docs and syncs `great-docs/_site/` to the existing S3 buckets and PR previews. + +- [ ] **Step 1: Update the `docs` and `docs-serve` recipes** + +Replace the mkdocs recipes in `justfile`: +```make +# Build the documentation site +docs: + printf 'rsconnect_python:\n version: "%s"\n' "$(uv version --short)" > great-docs/_variables.yml + uv run --python 3.12 --with great-docs great-docs build + +# Serve the documentation with live reload +docs-serve: + printf 'rsconnect_python:\n version: "%s"\n' "$(uv version --short)" > great-docs/_variables.yml + uv run --python 3.12 --with great-docs great-docs preview +``` +(If Task 1 recorded the `sed` fallback for version substitution, replace the `_variables.yml` line with the recorded substitution command. If great-docs has no `preview` subcommand, use `great-docs build` + a static server per the findings.) + +- [ ] **Step 2: Retarget the S3 and clean recipes** + +In `justfile`, change `site/` → `great-docs/_site/` in both S3 recipes, and update `clean` to remove `great-docs/_site`: +```make +clean: + rm -rf .coverage .pytest_cache build dist htmlcov rsconnect_python.egg-info rsconnect.egg-info great-docs/_site + +sync-latest-docs-to-s3: + aws s3 sync --acl bucket-owner-full-control --cache-control max-age=0 great-docs/_site/ s3://rstudio-connect-downloads/connect/rsconnect-python/latest/docs/ + +promote-docs-in-s3: + aws s3 sync --delete --acl bucket-owner-full-control --cache-control max-age=300 great-docs/_site/ s3://docs.rstudio.com/rsconnect-python/ +``` + +- [ ] **Step 3: Verify the recipe end-to-end locally** + +Run: +```bash +just docs +ls great-docs/_site/index.html +grep -rl "rsconnect-python $(uv version --short)" great-docs/_site/ || echo "MISSING version" +``` +Expected: build succeeds; index exists; the real project version (not `dev`) appears in the output. + +- [ ] **Step 4: Add Quarto to the `docs` CI job** + +In `.github/workflows/main.yml`, in the `docs` job, add the Quarto setup action before `build docs` (after the `setup-just` step): +```yaml + - uses: quarto-dev/quarto-actions/setup@v2 +``` +The `run: just docs` step is unchanged; the S3 sync/promote steps now push `great-docs/_site/` via the updated recipes (no YAML path change needed since they call `just`). + +- [ ] **Step 5: Update the PR preview workflow** + +In `.github/workflows/preview-docs.yml`: add the Quarto setup step before `Install and Build`, and change `source-dir: ./site/` to `source-dir: ./great-docs/_site/`: +```yaml + - uses: quarto-dev/quarto-actions/setup@v2 +``` +```yaml + with: + source-dir: ./great-docs/_site/ +``` + +- [ ] **Step 6: Commit** + +```bash +git add justfile .github/workflows/main.yml .github/workflows/preview-docs.yml +git commit -m "build: run docs via great-docs in CI and just recipes" +``` + +--- + +## Task 8: Remove mkdocs and finalize + +**Files:** +- Delete: `mkdocs.yml`, `docs/commands/`, `docs/overrides/`, `docs/css/`, `docs/requirements.txt`, `docs/.gitignore`, and the four migrated `docs/*.md` pages +- Keep: `docs/CHANGELOG.md` (retained for the `Unreleased` section and as the notes source for future releases), `docs/images/` +- Modify: `CLAUDE.md` (docs commands + changelog/release process), `docs/superpowers/spikes/2026-07-01-great-docs-findings.md` if any notes changed + +**Interfaces:** +- Consumes: everything from Tasks 4–7. +- Produces: a repo with no mkdocs residue and an accurate `CLAUDE.md`. + +- [ ] **Step 1: Delete mkdocs files and migrated originals** + +```bash +git rm mkdocs.yml docs/requirements.txt docs/.gitignore +git rm -r docs/commands docs/overrides docs/css +git rm docs/index.md docs/deploying.md docs/programmatic-provisioning.md docs/server-administration.md +``` + +- [ ] **Step 2: Confirm no mkdocs references remain** + +Run: +```bash +grep -rniE "mkdocs|pymdownx|mkdocs-click|mkdocs-macros" . --exclude-dir=.git --exclude-dir=great-docs/_site --exclude-dir=docs/superpowers +``` +Expected: no output (or only historical mentions inside `docs/CHANGELOG.md`, which are fine). + +- [ ] **Step 3: Update `CLAUDE.md`** + +- In the Documentation commands section, replace the mkdocs `just docs`/`docs-serve` descriptions with the great-docs equivalents (note: requires Quarto CLI; runs via `uv run --with great-docs`). +- In the Releasing section, update the changelog guidance: release notes are now authored in the GitHub Release (source of truth for the published changelog); `docs/CHANGELOG.md` retains only the `Unreleased` section for in-flight work. Reference `scripts/backfill_release_notes.py` as the one-time migration. + +- [ ] **Step 4: Full build + parity check** + +Run: +```bash +just docs +``` +Then verify against this checklist (manually open `great-docs/_site/`): +- All four narrative pages present with correct titles. +- All 16 CLI commands documented (add, bootstrap, content, deploy, details, environment, info, integration, list, login, logout, quickstart, remove, system, version, write-manifest). +- Changelog page shows full history from the backfilled Releases. +- Version string resolves to the real version in `deploying`. +- Posit logo/favicon present; analytics present (or fallback confirmed). + +- [ ] **Step 5: Commit** + +```bash +git add -A +git commit -m "build: remove mkdocs toolchain, update CLAUDE.md" +``` + +--- + +## Self-Review + +**Spec coverage:** +- S3 hosting preserved → Task 7 (S3 recipes retargeted, CI unchanged in intent). ✓ +- CLI + narrative + changelog scope, no API reference → Tasks 4 (`reference.enabled: false`), 5 (narrative), 3/4 (changelog). ✓ +- Single-version → no `versions:` key introduced. ✓ +- Backfill-then-auto changelog → Tasks 2, 3, then 4 (`changelog.enabled`). ✓ +- Spike-then-migrate strategy → Task 1 gates Tasks 4–8. ✓ +- All narrative pages to `.qmd` → Task 5. ✓ +- CLI auto-discovery from `rsconnect.main` → Tasks 1, 4. ✓ +- Version substitution → Tasks 1, 5, 7. ✓ +- GTM/analytics → Tasks 1, 6. ✓ +- Build tooling + PR previews → Task 7. ✓ +- Cutover/cleanup + CLAUDE.md → Task 8. ✓ +- Risks (reference-disable, GTM, content-dir layout, URL stability) → carried into Task 1 findings with fallbacks. + +**Placeholder scan:** No "TBD/TODO". The intentional spike-dependent values (exact YAML keys, analytics mechanism) are resolved in Task 1 and referenced with documented fallbacks — not silent gaps. + +**Type consistency:** `parse_changelog(text) -> dict[str, str]` and `build_backfill_plan(entries, existing_tags, release_has_body) -> list[tuple[str, str]]` are used consistently across Tasks 2–3. + +**Open item — URL stability:** great-docs' CLI page structure differs from the current `commands//` paths; inbound deep links may break. Enumerate broken paths during Task 8's parity check and decide whether S3 redirects are warranted (out of scope for this plan unless the check surfaces critical links). From 8c5d2cc39e61f8d57c133370e0016f60b51e97ed Mon Sep 17 00:00:00 2001 From: edavidaja Date: Wed, 1 Jul 2026 09:33:38 -0400 Subject: [PATCH 03/19] docs: record great-docs spike findings --- .../spikes/2026-07-01-great-docs-findings.md | 84 +++++++++++++++++++ 1 file changed, 84 insertions(+) create mode 100644 docs/superpowers/spikes/2026-07-01-great-docs-findings.md diff --git a/docs/superpowers/spikes/2026-07-01-great-docs-findings.md b/docs/superpowers/spikes/2026-07-01-great-docs-findings.md new file mode 100644 index 000000000..8ca27b42f --- /dev/null +++ b/docs/superpowers/spikes/2026-07-01-great-docs-findings.md @@ -0,0 +1,84 @@ +# great-docs spike findings — 2026-07-01 + +Validated great-docs against the real rsconnect-python package (via a scratch project + the +installed package). Results gate/adjust plan Tasks 4–8. + +## ✅ CLI-only build works; Python API reference can be suppressed + +Config that builds CLI + user-guide + changelog with **no Python API reference**: + +```yaml +display_name: rsconnect-python +cli: + enabled: true + module: rsconnect.main +reference: [] # empty explicit sections => no auto Python API reference +changelog: + enabled: true +``` + +`reference: []` produced **zero** non-CLI reference pages. Build emitted 87–88 HTML pages. + +## ✅ CLI auto-discovery from `rsconnect.main` is rich + +great-docs generated `reference/cli/.qmd` for every command **plus per-content-type +subcommands** (e.g. `reference/cli/deploy/{quarto,notebook,...}.qmd`, +`reference/cli/write_manifest/{flask,dash,voila,api,nodejs,gradio,fastapi,pyproject,notebook,panel}.qmd`). +Cosmetic: Click warns "parameter --verbose/-v used more than once" — originates in rsconnect's +CLI, not great-docs; harmless. + +## Content location (corrects the plan) + +- `user_guide/` lives at the **project root** by default (NOT `great-docs/user_guide/`). Override + with the `user_guide:` key if needed. +- great-docs renders into a **managed `great-docs/` working directory** and writes output to + **`great-docs/_site/`**. It also generates `great-docs/_quarto.yml`, `great-docs/index.qmd`, + `great-docs/scripts/post-render.py`, `great-docs/_package_meta.json`, `skill.md`, etc. + All generated/managed paths must be git-ignored. + +## Build recipe (corrects the plan's `uv run --with` approach) + +The `uv run --python 3.12 --with great-docs` ephemeral approach **fails**: Quarto's post-render +hook (`great-docs/scripts/post-render.py`) runs under an interpreter that must have `pygments` +AND `great_docs` importable, and the ephemeral env is not the interpreter Quarto uses. + +**Working recipe:** a dedicated venv (Python 3.12) with `great-docs`, `pygments`, and the project +installed, run **activated** so `python3` on PATH is that venv: + +```bash +uv venv --python 3.12 .venv-docs +uv pip install --python .venv-docs great-docs pygments . # "." installs rsconnect +source .venv-docs/bin/activate +great-docs build # -> great-docs/_site/ +``` + +- `pygments` must be installed explicitly (declared great-docs dep, but the post-render subprocess + needs it on the active interpreter). +- A non-`.venv` name (`.venv-docs`) works when **activated**, so it won't clobber the dev/test + `.venv`. (great-docs only auto-detects `.venv`/`venv` in the project root for `QUARTO_PYTHON`; + otherwise it uses its own `sys.executable`, which the activated venv satisfies.) +- Requires-python note: installing into a 3.12 venv via `uv pip install` avoids the dependency-group + resolution conflict between the project (`>=3.8`) and great-docs (`>=3.11`). + +## ❌ Version substitution has no clean mechanism + +`_variables.yml` at the project root is NOT picked up (Quarto renders in the managed `great-docs/` +dir; the root variables file never reaches it). The literal `{{< var ... >}}` token survives in +output. **great-docs auto-detects the package version (1.29.0) and injects version badges**, so the +version is already displayed. Options for the inline "Generated from rsconnect-python X" line: +(a) drop it (redundant with the auto version badge), or (b) post-build `sed` over +`great-docs/_site/**/*.html` replacing a sentinel token. + +## ❌ No analytics/GTM hook + +great-docs has **no** analytics/GA/GTM support. `site:` keys are whitelisted (theme, toc, +toc-depth, toc-title, show_dates, date_format, show_author, show_security) — arbitrary Quarto keys +like `include-in-header` are NOT forwarded. Options to preserve `GTM-KHBDBW7`: +(a) post-build injection of the GTM snippet into every `great-docs/_site/**/*.html` ``, +(b) drop analytics, or (c) request an analytics feature from great-docs maintainers (Posit-internal). + +## Net gate result + +Feasible. The scope-defining assumptions hold (CLI-only, S3 output dir, single version). Two +cosmetic/outward-facing items (version line, GTM) lack config hooks and need a post-build step or a +drop decision — escalated to the user before Tasks 5–7. From 45a88f5a7a34e852cb01e6a06e520f54835078a6 Mon Sep 17 00:00:00 2001 From: edavidaja Date: Wed, 1 Jul 2026 09:37:32 -0400 Subject: [PATCH 04/19] docs: GTM works via include_in_header (spike finding) --- .../spikes/2026-07-01-great-docs-findings.md | 24 ++++++++++++++----- 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/docs/superpowers/spikes/2026-07-01-great-docs-findings.md b/docs/superpowers/spikes/2026-07-01-great-docs-findings.md index 8ca27b42f..c2cf4ceee 100644 --- a/docs/superpowers/spikes/2026-07-01-great-docs-findings.md +++ b/docs/superpowers/spikes/2026-07-01-great-docs-findings.md @@ -69,13 +69,25 @@ version is already displayed. Options for the inline "Generated from rsconnect-p (a) drop it (redundant with the auto version badge), or (b) post-build `sed` over `great-docs/_site/**/*.html` replacing a sentinel token. -## ❌ No analytics/GTM hook +## ✅ GTM/analytics via `include_in_header` (Quarto includes) -great-docs has **no** analytics/GA/GTM support. `site:` keys are whitelisted (theme, toc, -toc-depth, toc-title, show_dates, date_format, show_author, show_security) — arbitrary Quarto keys -like `include-in-header` are NOT forwarded. Options to preserve `GTM-KHBDBW7`: -(a) post-build injection of the GTM snippet into every `great-docs/_site/**/*.html` ``, -(b) drop analytics, or (c) request an analytics feature from great-docs maintainers (Posit-internal). +great-docs has no dedicated analytics key, and `site:` keys are whitelisted (theme, toc, toc-depth, +toc-title, show_dates, date_format, show_author, show_security) so nesting includes under `site:` +does NOT work. BUT great-docs supports a **top-level `include_in_header`** key that it merges into +Quarto's `format.html.include-in-header` (core.py:11247–11250; config.py:1205). Accepts an inline +`text:` block or a `file:` entry. Verified: an inline GTM ` +``` + +This is the clean, config-based path — no post-build HTML mutation needed for analytics. (Note: +`include_in_header` is head-only; the GTM `