Skip to content

Docs-as-context (documentation optimized for LLM consumption): raise from 4/5 to 5/5 (enterprise quality audit) #130

Description

@Kravalg

Description

This issue was produced by the automated enterprise quality & NFR audit (2026-07-02), which audited the repository against the quality attributes from Wikipedia's "List of system quality attributes" plus AI-native development readiness. Attribute: docs-as-context (documentation optimized for LLM consumption), cluster: AI-Native Autonomous Development Readiness, current score: 4/5. Audit justification: The docs/ handbook is compact (~2,000 lines across 11 focused files), heavily tabular (workflow-to-make-target matrices in docs/ci-architecture.md and docs/ci-guardrails.md), cross-linked from a single index (docs/README.md), and — unusually — kept truthful by automation: structural tests in tests/pulumi/test_delivery_contracts.py assert docs/sre-operations.md maps every blocking check to its local command and that ci-architecture/quality-gates wording matches the current make surface. That is close to ideal LLM context. The alignment mechanism is brittle substring assertions rather than generated references, and there is no markdown link-check or lint gate anywhere in CI (only yamllint/actionlint/hadolint), so cross-link rot is undetectable.

No markdown link-check or lint gate in CI — make test-repo-hygiene covers actionlint, yamllint, and hadolint but nothing validates the ~11 docs/*.md files, README.md, or AGENTS.md. Broken relative links (e.g., docs/README.md -> github-actions-secrets.md, ../SECURITY.md) or malformed markdown would ship silently, and stale links are exactly what degrades LLM retrieval of the handbook.

Docs-code alignment relies on brittle substring assertions instead of generated references — Tests like test_sre_docs_map_blocking_ci_checks_back_to_local_commands assert exact prose fragments (e.g., "Maintainability -> make test-maintainability"), and docs/README.md hand-copies the make help output (lines 53-84). This catches some drift but inverts the maintenance burden: reworded-but-correct docs fail tests, while new make targets or workflows that nobody adds an assertion for drift silently (the docs/README.md target list already paraphrases several help strings).

Tasks

  • Add a make test-docs target running a link checker (lychee or markdown-link-check) plus markdownlint over docs/, README.md, AGENTS.md, CONTRIBUTING.md
  • Wire it into make test-repo-hygiene and the security-scans or a docs workflow, and add it to the docs/ci-architecture.md matrix
  • Cover the new target in the make-target structural tests and bats suite
  • Replace the hand-maintained command table in docs/README.md with content generated from make help (a scripts/generate_docs_index.py check that fails when the committed table differs from generated output)
  • Add a completeness test asserting every workflow file in .github/workflows/ appears in the docs/ci-architecture.md matrix and every public make target appears in the docs, instead of spot-check substrings

Acceptance Criteria

  • A PR introducing a broken relative link in docs/ fails CI with the offending file and link named
  • make test-docs runs locally inside the existing Docker workspace with no network-dependent flakiness for internal links
  • The target appears in the workflow matrix docs and CLI tests
  • Adding a make target or workflow without documenting it fails make test-pulumi
  • The docs command table is verifiably identical to make help output
  • Rewording doc prose no longer breaks tests unless a command mapping actually changed
  • Attribute re-scores 5/5 in a follow-up enterprise quality audit

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions