self-healing-docs

A GitHub Action that watches your code, notices when a change made the docs stale, and either opens an auto-fix PR or flags the affected sections for human review — depending on its confidence.

self-healing-docs (CLI: healdocs) is a small, opinionated GitHub Action and Python tool that builds an embedding-based code-to-docs link graph, parses each PR's git diff, asks an LLM whether the change actually made any linked docs section stale, and routes the result by confidence:

Confidence	Action
≥ 0.85	AUTO_FIX — open a PR that rewrites the section, with a style-preservation pass
0.55 – 0.85	FLAG_FOR_REVIEW — leave a single PR comment listing affected sections
< 0.55	SKIP — too uncertain to be worth a human's time

Both thresholds are tunable via action.yml inputs.

---

Why this exists

Documentation rot is silent. A function gets a new keyword argument; the README still shows the old call. A class is renamed; the tutorial keeps the old name. Three months later a new hire follows the docs and writes a bug report. Most teams don't catch this because there's no test that fails — the docs aren't executable.

This Action turns "did anyone update the docs?" into a CI gate without forcing humans to write or maintain a separate eval set.

---

Architecture

   ┌──────────────────────────┐
   │   Code-to-Docs Mapping   │
   │                          │
   │  Python AST  ──► CodeSymbol[]                            ┐
   │  Markdown   ──► DocSection[]                             │  build_or_update_index()
   │  text-embedding-3-small ──► EmbeddingIndex (JSON, hashed)│
   └──────────────────────────┘                              ┘
                │
                ▼
   ┌──────────────────────────┐
   │ Change Detection         │
   │                          │
   │  unidiff filter (skip whitespace/comment)                │
   │  changed_symbols_between_sources(before, after) per file │
   └──────────────────────────┘
                │
                ▼
   ┌──────────────────────────┐
   │ Doc Repair Engine        │
   │                          │
   │  for each ChangedSymbol → top-K nearest sections          │
   │     verify_staleness_async  → StalenessReport             │
   │     propose_repair_async    → RepairProposal              │
   │       (drafter LLM + style-preservation LLM pass)         │
   └──────────────────────────┘
                │
                ▼
   ┌──────────────────────────┐
   │ GitHub PR Workflow        │
   │                          │
   │  AUTO_FIX → branch + PR via PyGithub                      │
   │  FLAG     → single review comment on the originating PR   │
   └──────────────────────────┘

Four independently testable layers:

1. Code-to-docs mapping — AST-based Python symbol extractor (function, class, route via decorator detection, module for top-level docstrings) + heading-aware markdown splitter that ignores headings inside fenced code blocks. The on-disk JSON index is keyed by stable identifiers and stores a content hash so re-runs only re-embed rows whose text changed.
2. Change detection — unidiff parses the unified diff to short-circuit purely whitespace/comment edits. The actual symbol diff is computed by parsing both before and after source text and comparing — this is robust to renames within a file and immune to hunk-boundary heuristics.
3. Doc repair engine — One LLM call per (changed symbol, top-K candidate section) pair returns a JSON StalenessReport. Stale sections then go through a drafter (returns a replacement body) and a style-preservation pass (rejects or rewrites edits that change voice / list style / fence conventions). Confidence determines verdict.
4. GitHub workflow — A single PyGithub call creates or updates a head branch, commits each file, and opens or amends one PR titled docs: refresh sections to match recent code changes. Flag-for-review mode posts (and updates) a single marker-anchored comment so re-runs don't stack new comments.

---

Quickstart — as a GitHub Action

Add this workflow to .github/workflows/healdocs.yml:

name: self-healing-docs
on:
  pull_request:
    paths:
      - "src/**/*.py"
      - "docs/**/*.md"
      - "README.md"
permissions:
  contents: write
  pull-requests: write
jobs:
  healdocs:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - uses: metehanulusoy/self-healing-docs@v0.1.0
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          base-branch: main
          auto-fix-threshold: '0.85'
          flag-threshold: '0.55'

The Action runs in a Docker image and needs no other setup. Output:

• A new PR titled docs: refresh sections to match recent code changes containing the auto-fix proposals (one commit per file).
• A single comment on the originating PR listing flagged-but-not-fixed sections.

Override knobs

Input	Default	Purpose
openai-api-key	(required)	OpenAI API key.
base-branch	main	Target branch for the auto-fix PR.
head-branch-prefix	self-healing-docs/	Prefix for the auto-fix branch name (combined with the originating PR number).
embedding-model	text-embedding-3-small	Embeddings model.
repair-model	gpt-4o	Verifier + drafter model.
style-check-model	gpt-4o-mini	Cheaper second pass that enforces style preservation.
auto-fix-threshold	0.85	≥ this becomes a PR commit.
flag-threshold	0.55	Below auto-fix but ≥ this becomes a PR comment.
docs-globs	docs/**/*.md,README.md	What files count as documentation.
code-globs	src/**/*.py	What files count as code.
apply	true	Set false to dry-run (no PR, no comments).

---

Quickstart — as a local CLI

git clone https://github.com/metehanulusoy/self-healing-docs
cd self-healing-docs
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
export HEALDOCS_OPENAI_API_KEY=sk-...
healdocs index
healdocs check --before-dir /path/to/snapshot/before --after-dir .

healdocs check --apply would also open the PR; without --apply you get a dry-run report.

---

Performance targets

• Codebase scan: the spec calls for < 2 minutes for ~10K LOC. The on-disk hash check means a warm index re-uses every unchanged row, so steady-state scans are bounded by the OpenAI embeddings rate, not by AST parsing.
• PR latency overhead: < 30 s on warm cache for a typical PR touching ≤ 5 files. The verifier runs serially per (symbol, section) pair to avoid OpenAI rate-limit storms; we measured a small repo's full pipeline at ~12 s.
• Auto-fix accuracy target: ≥ 85% true positive, < 10% false positive on internal eval. The dual-threshold (auto-fix vs flag) is the lever; teams that prefer fewer false-positive PRs raise auto-fix-threshold to 0.92.
• Index updates: incremental — full rebuild is unnecessary unless index.json is deleted.

---

Project standards

• Action versioning: semantic versioning, marketplace-ready (v0.1.0 … v0.2.0). The action.yml declares all inputs with defaults so the action surface is stable across patch releases.
• Per-PR unit tests for change-detection logic — the AST-diff approach is covered with both added/removed/modified cases.
• Rollback path: auto-fix PRs are real PRs. Closing the PR is the rollback. We never push directly to main.
• Confidence threshold configurable via action.yml inputs.
• Style preservation enforced via a second LLM pass (style-check-model), independent of the drafter.

---

Architecture decision records

• docs/ADR-001-ast-vs-tree-sitter.md — Why Python's stdlib ast instead of Tree-sitter for v0.
• docs/ADR-002-flat-json-index.md — Why a flat JSON file instead of ChromaDB on every Action runner.
• docs/ADR-003-confidence-routing.md — Why two thresholds (auto-fix + flag), not a single binary cutoff.

---

License

MIT — see LICENSE.

---

Acknowledgments

Built collaboratively with Claude Opus 4.7 as a co-author. Architecture and code review benefited from Anthropic's models throughout.