docs: full technical documentation suite indexed in docs/README.md

[zachtheyek]

Closes #131

Stacking

Based on integration/pr11-base (the all-code base: training pipeline + retry overhaul, inference energy-detection + PFB + retry/viz, benchmarking suite #134, HuggingFace integration #130, release engineering #133). Being docs-only, this PR documents everything before it and carries no code changes. Merge after the code PRs it describes.

What this adds

The full docs/ technical suite — one document per pipeline surface — indexed in docs/README.md, replacing docs/placeholder:

Document	Surface
ARCHITECTURE.md	System map, data model, module/data-flow, init order, singletons, artifact layout
TRAINING_PIPELINE.md	Curriculum, round lifecycle, memmap round data + producer, retries, every training plot
INFERENCE_PIPELINE.md	Catalog grouping, streaming flow, HF model resolution, tag guards, manifest retries, viz
PREPROCESSING.md	Energy detection math, spline vs PFB, vectorized D'Agostino–Pearson, dedup, downsample-at-extraction, injection
MODELS.md	Beta-VAE architecture + composite loss, RF feature layout, threshold/SHAP semantics
DATABASE.md	Every table incl. pipeline_stages, writer thread, flush/supersede, migration mechanism
RUNTIME_SERVICES.md	Logger/Slack, ResourceManager lifecycle, monitor + stage annotations
TESTING.md	Suite layout, markers, isolation fixtures, CI, cluster smokes
BENCHMARKING.md	New — stage timers, pipeline_stages, report tool, micro-benchmarks, baselines
RELEASE.md	Version-coupling contract, CD gates (verify-don't-create), trusted publishing, runbook
GITHUB_AUTOMATION.md	Workflow catalog, dedup markers, assistant-handle rules

Also refreshes CLAUDE.md, the repo-context skill, and CONTRIBUTING.md, and extends CONFIG_AND_CLI.md / GPU_RUNTIME_GUIDE.md cross-references.

Phase B (this session) — the deferred, code-dependent pieces

Written against the landed modules (benchmark.py, hf_hub.py, tag_guards.py, db/db.py, utils/benchmark_report.py, benchmarks/, .github/workflows/release.yml, utils/hf_tag_release.py):

• docs/BENCHMARKING.md (new) — stage_timer/record_stage with thread-local dot-name nesting, the instrumented stage map, the pipeline_stages table (schema v3), monitor.annotate_stages overlay, benchmark_report.py (stage tree / flame PNG / suggestion rules), the benchmarks/ micro-benchmarks, and baseline numbers (123× vectorized normality and 11.9× PFB-vs-spline on blpc3; end-to-end inference ~6.2× and ED ~45 min → ~2 min/cadence; training RSS held ~26% vs the old round-2 OOM).
• INFERENCE_PIPELINE.md — filled the <!-- PHASE-B: HF model resolution --> gap: the full artifact-resolution chain (explicit paths → --hf-revision → installed-release v{__version__} with the strict dev-version guard → latest release/final_vX tag), the all-or-none validation, and the inference-scope tag-dedup guards. Corrected the now-stale "all three paths required" text.
• DATABASE.md — added the pipeline_stages table and a user_version → feature map (v1 superseded columns, v2 inference_cadences, v3 pipeline_stages), verified against _migrate_schema; bumped the version/table-count references.
• RUNTIME_SERVICES.md — documented the monitor.annotate_stages CPU-panel overlay (config knob, depth≤2 rule, flush-first).
• docs/README.md — real BENCHMARKING.md index row replacing the placeholder comment.
• RELEASE.md — cross-checked every CD gate against the implemented release.yml/hf_tag_release.py; added the build-stage wheel smoke and the GitHub Release artifact attachment for fidelity.

Verification

Docs-only; no runtime surface. Checks performed:

• Link + anchor check (scripted: every relative markdown link and #anchor across all docs resolved against the filesystem / target headings):

  Checked 243 relative links across 17 markdown files.
  All relative links and anchors resolve. 0 broken.
  

• Code fidelity — every cited symbol/flag confirmed against source: stage_timer/record_stage/current_stage/_ACTIVE_STAGES (benchmark.py), write_pipeline_stage/query_pipeline_stages/_SCHEMA_VERSION=3/_migrate_schema (db/db.py), select_annotation_spans/_ANNOTATION_MAX_DEPTH/annotate_stages (monitor.py + config default), build_stage_tree/top_slowest/suggestion thresholds (benchmark_report.py), version_default_revision/_SEMVER_TAG_PATTERN/resolve_hf_revision/resolve_inference_artifacts (hf_hub.py), enforce_tag_guards/find_inference_tag_collisions/--force-tag (tag_guards.py), and the release-gate ordering in release.yml.
• pre-commit run --all-files — green.
• Unit suite unaffected (no test fixture reads docs/; CLAUDE.md/CONTRIBUTING structural fixtures untouched by content-only edits). CI will exercise it.
• Both commits carry verified GPG signatures.

Note for a follow-up (not in scope here): utils/benchmark_report.py's encode-input-bound suggestion prints --inf-batch-size, but the real flag is --per-replica-batch-size; BENCHMARKING.md documents the correct flag.

AI assistance

Authored with Claude Code (Anthropic) — Phase B doc drafting against the landed code, plus the scripted link/anchor check and code-symbol verification. Reviewed for factual accuracy against the source. Disclosed per AI_POLICY.md.

[claude]

Claude finished @zachtheyek's task in 1m 33s —— View job

---

Review of PR #135: docs suite

• Gather context and understand changes
• Review docs/README.md index and cross-references
• Review ARCHITECTURE.md
• Review TRAINING_PIPELINE.md
• Review INFERENCE_PIPELINE.md
• Review PREPROCESSING.md
• Review MODELS.md
• Review DATABASE.md
• Review RUNTIME_SERVICES.md
• Review TESTING.md and BENCHMARKING.md
• Review RELEASE.md and GITHUB_AUTOMATION.md
• Review CLAUDE.md, CONTRIBUTING.md, and skill updates
• Post final review

---

Overall assessment

This is an excellent, comprehensive documentation suite. The 11 new documents cover every major pipeline surface with consistent depth, well-structured tables, and precise code references. The writing is technically rigorous — equations, architecture tables, schema definitions, and diagnostic interpretation guides are all present. The PR description's verification claims (243 links checked, symbol fidelity against source) are credible given the level of detail.

Verdict: Approve. This is docs-only, well-verified, and significantly raises the project's documentation quality. The notes below are minor suggestions, none blocking.

---

Strengths

1. Consistent structure. Every document follows the same pattern: TL;DR → detailed sections → config quick reference / "where to go next". Makes navigation predictable.

2. Actionable diagnostic guides. The training plot catalog (TRAINING_PIPELINE.md) and inference viz catalog (INFERENCE_PIPELINE.md) don't just describe figures — they tell you what to look for and what it means when something's wrong. This is the most valuable part of the suite for new contributors.

3. Cross-reference discipline. Documents link to each other at seam boundaries (e.g., MODELS.md defers training to TRAINING_PIPELINE.md, serving to INFERENCE_PIPELINE.md) rather than duplicating. The README.md index provides the central hub.

4. Phase B fills. The HF resolution chain in INFERENCE_PIPELINE.md:66-104, the pipeline_stages table in DATABASE.md:207-224, and the annotate_stages section in RUNTIME_SERVICES.md:179-200 are thorough and well-verified against source.

5. The BENCHMARKING.md document is a particularly strong addition — it ties together the timer API, the DB table, the report tool, the micro-benchmarks, and the baseline numbers into a coherent story with the annotated resource plot reading guide.

---

Minor notes

1. ARCHITECTURE.md:27 — stray backtick in path reference

[`src/aetherscan/main.py`](../src/aetherscan/main.py)`:setup_gpu_strategy()`.

The backtick before :setup_gpu_strategy() creates a visual break in rendered markdown. Consider:

[`src/aetherscan/main.py`](../src/aetherscan/main.py) `setup_gpu_strategy()`.

or inline the method name inside the link text.

2. TRAINING_PIPELINE.md:346 — filename reference

`rf_oob_accuracy_curve_{tag}.png` (from `plot_rf_ensemble_accuracy_curve`)

The PR description notes that a follow-up will address the --inf-batch-size vs --per-replica-batch-size flag name mismatch in benchmark_report.py. Good that BENCHMARKING.md documents the correct flag — just flagging that this is tracked.

3. DATABASE.md — row-count growth math clarity

At line 282, the injection_stats growth estimate:

3 × 499 200 × ~24 ≈ 36 M rows per round

The ~24 is the sum of 18 intensity rows + average 6 signal-characteristic rows. The math checks out, but the "average 6" derivation (0/6/6/12 across the four types, each at 25% of main) could be made explicit for readers who don't know the type distribution. Very minor — the current text is fine for the target audience.

4. GITHUB_AUTOMATION.md:90 — long table row

The claude-update-docs.yml row in the workflow table (line 90) is quite dense — it packs trigger, behavior, dedup marker, and the CLI regeneration detail into one cell. This is a formatting preference, not a correctness issue, but a note/callout box below the table could improve readability.

5. Cross-reference to KNOWN_ISSUES.md

RUNTIME_SERVICES.md references KNOWN_ISSUES.md #6, #10, and #11. These are good pointers — just worth confirming on merge that the issue numbers still match (known-issues lists can shift with edits).

6. docs/README.md — GPU_RUNTIME_GUIDE.md and CONFIG_AND_CLI.md summaries

These two pre-existing documents get updated summary cells in the index table. The summaries are accurate to the current content of those documents — no issues.

---

Security review

No secrets, tokens, or sensitive paths appear in any document. The Slack webhook/token handling guidance in RUNTIME_SERVICES.md correctly references SECURITY.md. The RELEASE.md document's treatment of PyPI trusted publishing and HF token handling is sound — no long-lived secrets stored, OIDC-based CD auth properly described.

---

No issues found in

• Schema and table definitions (DATABASE.md) — verified against the described _SCHEMA_VERSION=3 and migration steps
• Model architecture tables (MODELS.md) — layer dimensions and stride patterns are internally consistent
• Preprocessing math (PREPROCESSING.md) — D'Agostino–Pearson derivation, PFB response generation, and dedup algorithm descriptions are precise
• Release workflow gates (RELEASE.md) — the 6-step CD sequence matches the described release.yml behavior
• Testing documentation (TESTING.md) — marker semantics, fixture descriptions, and CI configuration are accurate

Nice work on the suite. The baseline documentation gap is now comprehensively filled.

[claude]

Hi @zachtheyek — the PR body mentions Closes #131, but GitHub isn't registering it as a formal closing reference (verified via the GraphQL closingIssuesReferences API). This can happen if the keyword is inside a collapsed section, blockquote, or certain formatting contexts. Could you re-link #131 using the Development sidebar on the right, or ensure Closes #131 appears as plain text at the top level of the PR body? Per CONTRIBUTING.md, every PR should have a formally linked issue. Removing the needs-issue label once the link registers.

[zachtheyek]

Applied the review polish (3b2b044, GPG-signed):

1. ARCHITECTURE.md — moved setup_gpu_strategy() into prose so the main.py link no longer renders a stray trailing backtick span.
2. DATABASE.md — made the ~24 rows/cadence derivation explicit: 18 intensity rows (6 stats × 3 stages A/B/C) + mean 6 signal-characteristic rows across the 0/6/6/12 equal-weighted types.
3. GITHUB_AUTOMATION.md — added a callout unpacking the dense claude-update-docs row (deterministic shell regen → handle-mention issue → assistant PR, split because the follow-up run has no python in its allowlist).
4. RUNTIME_SERVICES.md — verified the KNOWN_ISSUES.md references against the current file: Fix exponential SNR schedule off-by-1 error #6 = Resource Monitor CPU Undercounting, Deprecate TensorBoard support until further notice #10 = Missing Sections in Resource Monitor Plots, Deprecate TensorBoard support #11 = Negative Memory Freed Calculation. No drift; no change needed.
5. The benchmark_report.py --inf-batch-size string is being fixed in Benchmarking suite: always-on stage timing, annotated resource plots, report tooling, micro-benchmarks #134 by another agent; this doc already documents the correct flag (--per-replica-batch-size), so no doc change.

Verification: pre-commit run green on the touched files; scripted link+anchor check — Checked 243 relative links across 17 markdown files. All resolve. 0 broken.

[zachtheyek]

@claude please perform a second-pass code review of this PR. Re-review the current diff for any remaining correctness bugs, security issues, or significant quality problems, and post your findings. If the diff is clean, please say so explicitly. (Final verification round before merge.)

[claude]

Second-pass code review — clean

I've reviewed the full diff (2,891 lines across 17 files) for correctness bugs, security issues, and significant quality problems. The diff is clean.

What I checked

1. No security issues. No secrets, tokens, or internal URLs exposed in any document. Slack/HF token handling is described correctly (gitignored .env, never logged/committed).

2. Internal consistency. Cross-references between documents are bidirectional and coherent (e.g. DATABASE.md ↔ BENCHMARKING.md on pipeline_stages, MODELS.md ↔ TRAINING_PIPELINE.md on loss terms, INFERENCE_PIPELINE.md ↔ RELEASE.md on weight resolution). The schema version map (v0→v1→v2→v3) in DATABASE.md is consistent with what's described in BENCHMARKING.md.

3. Technical accuracy (spot-checks).

   • The Beta-VAE encoder shape table in MODELS.md is internally consistent: each stride-2 layer halves both spatial dimensions, arriving at (1, 32, 256) → flatten 8192 → dense 512 → latent 8. Correct.
   • The D'Agostino–Pearson derivation in PREPROCESSING.md correctly describes the central-moment approach with shift-invariance rationale and the Z₁² + Z₂² composition. The window-size calculation (n = time_bins × window_size = 16 × 256 = 4096) matches defaults.
   • The cleanup ordering in RUNTIME_SERVICES.md (processes → pools → SHM → monitor → DB → logger) is consistent with ARCHITECTURE.md's init order (reverse dependency).
   • The retry/resume semantics described in INFERENCE_PIPELINE.md (manifest skip, mark-superseded before fresh writes, failure containment) are consistent with DATABASE.md's flush/supersede protocol description.

4. Structural correctness.

   • docs/placeholder correctly removed and replaced by the full suite.
   • docs/README.md index table covers all 13 documents with accurate one-line summaries.
   • The CONTRIBUTING.md diff correctly replaces the # TODO: add an architecture section? warning block with a pointer to the new docs.
   • Mermaid diagram in ARCHITECTURE.md has valid syntax and correctly represents the train/inference data flow.

5. The noted follow-up (benchmark_report.py printing --inf-batch-size vs the correct --per-replica-batch-size) is correctly called out in the PR body as out-of-scope, and BENCHMARKING.md documents the correct flag.

No findings to report. LGTM for merge (after its code-PR dependencies land, per the stacking note).

[zachtheyek]

Pushed a docs-only accuracy pass to docs/TESTING.md (commit 272bef1):

1. Completed the ## Layout unit-test tree. Added the four tests/unit/ files that exist on this branch but were missing from the fenced tree — test_tag_guards.py, test_hf_hub.py, test_benchmark.py, test_logger.py — placed near their related domains, with terse one-line descriptions read off each file and column alignment / the └── last-entry connector preserved. (test_monitor.py stays correctly absent — none exists.)

2. Added a ## Coverage and deliberate gaps section so a reader can tell deliberate gap from oversight:

   • monitor has no dedicated unit-test module by design — it's dominated by the 1 Hz background sampling thread (PSS process-tree stats) and matplotlib rendering, both low-value to unit-test; behavior is covered by the integration smokes and by manual inspection of the resource-utilization plot uploaded to Slack. (Its one pure helper, select_annotation_spans, is covered in test_benchmark.py.)
   • logger is unit-tested only for the StreamToLogger stdout/stderr-redirect probes (isatty/writable/readable/fileno) in test_logger.py; QueueListener, SlackHandler, and the stderr-to-logger redirect are exercised by the integration smokes.
   • One line noting inference logic is unit-tested at the function level with end-to-end behavior in test_inference_smoke.py.

Every module/behavior cited was verified against the source on this branch. Relative links in the file still resolve (0 broken) and pre-commit run --all-files passes.