Consolidate the report bundle into one report with a table of contents

[johnjosephhorton]

The structural half (B) of the report cleanup we scoped.

Problem

The bundle shipped nine pages plus a digest index that re-summarized the executive summary — the same story told two or three times, the decisive comparison scattered, and constant "which page do I open?". And the row-level "by-twin" audit table explodes file size when inlined.

Change

The bundle entry point is now one scrollable report.html (= index.html) with a sticky table of contents:

• Decision & Evidence (executive summary), Technical Validation (twin), Survey Profile, One-Shot Marginals fold in as sections;
• the digest index is gone;
• row-level / reference material — twin run audit, twin comparison, per-twin microdata — is linked in a Downloads section, never inlined. The bundle never generates microdata inline, so the file is lean by construction.

consolidated_report.py composes the existing standalone pages by lifting each page's <main> body into a titled section and merging the page-specific CSS once into a shared head (shared base CSS deduped). No ready twin yet → falls back to the readiness/next-steps index so the bundle stays navigable.

This also makes the report more robust: previously every section was a separate page the index linked to; now the substance is inline and only the audit/comparison links point out.

Testing

• pytest — 258 passed (new test_consolidated_report.py for the composer; updated the incremental-bundle integration test to assert the TOC + folded-in sections + linked audit instead of the old digest markers).
• ruff clean.

Deliberate follow-ups

• De-dup the generated narrative that can appear in both Decision and Technical Validation.
• Give the twin-validate workflow report the same consolidated shape.
• Fix relative download links in the secondary report/ copy (pre-existing behavior; main entry is correct).

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

This PR replaces the old multi-page digest bundle with a single scrollable report.html / index.html that embeds all ready pages as sections under a sticky table of contents. Row-level audit material (twin run audit, comparison, microdata) is moved to a Downloads section instead of being inlined, keeping file size lean.

• zwill/consolidated_report.py is a new module that regex-extracts each page's <main> body and page-specific CSS, deduplicates the shared base CSS, and renders one valid HTML document with a TOC.
• zwill/report_bundle.py gains build_consolidated_report_html which gates on twin-validation readiness and, if ready, writes both report.html and index.html with the consolidated output; if not ready it falls back to the old readiness index for index.html only.
• Tests cover the composer unit-level and the incremental-bundle integration path; 258 tests pass.

Confidence Score: 4/5

Safe to merge; the structural consolidation is correct and well-tested, with two minor edge cases that do not affect normal operation.

The greedy _MAIN_RE regex is only a risk if source pages ever contain a second closing main tag, which they currently do not. The report.html staleness concern requires an unusual re-run sequence and only affects that file's disk copy, not the live index.html users typically land on.

zwill/report_bundle.py around the else-branch where report.html is not cleared on a non-ready re-run; zwill/consolidated_report.py line 19 for the greedy _MAIN_RE pattern.

Important Files Changed

Filename	Overview
zwill/consolidated_report.py	New module that lifts each page's <main> into a titled section, deduplicates page-specific CSS, and renders a single scrollable HTML report with a sticky TOC; regex-based HTML extraction is fragile for the greedy closing-tag edge case.
zwill/report_bundle.py	Wires the consolidated report into build_report_bundle; when twin is not ready, report.html is not written/cleared, leaving it potentially stale from a prior run while index.html shows the readiness fallback.
tests/test_consolidated_report.py	New unit tests for the composer: covers section folding, TOC link generation, CSS deduplication, and empty-section skipping.
tests/test_cli_workflows.py	Integration test updated to assert the consolidated structure (TOC class, folded sections, download link) instead of the old digest markers; assertions are accurate for the new layout.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[build_report_bundle] --> B{twin-validation ready?}
    B -- No --> C[render_report_bundle_index]
    C --> D[index.html = readiness fallback]
    D --> E[report.html untouched - may be stale]
    B -- Yes --> F[build_consolidated_report_html]
    F --> G[read executive_summary HTML]
    F --> H[read twin-validation / survey-profile / one-shot HTML]
    F --> I[build downloads links for audit, comparison, microdata]
    G & H & I --> J[render_consolidated_report]
    J --> K[report.html written]
    K --> L[index.html written = same content]
    L --> M[copy to report/index.html]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[build_report_bundle] --> B{twin-validation ready?}
    B -- No --> C[render_report_bundle_index]
    C --> D[index.html = readiness fallback]
    D --> E[report.html untouched - may be stale]
    B -- Yes --> F[build_consolidated_report_html]
    F --> G[read executive_summary HTML]
    F --> H[read twin-validation / survey-profile / one-shot HTML]
    F --> I[build downloads links for audit, comparison, microdata]
    G & H & I --> J[render_consolidated_report]
    J --> K[report.html written]
    K --> L[index.html written = same content]
    L --> M[copy to report/index.html]

Loading

_{Reviews (1): Last reviewed commit: "Consolidate the report bundle into one r..." | Re-trigger Greptile}

[greptile-apps]

The _MAIN_RE pattern uses a greedy (.*) with re.S, so it matches from the first <main…> opener to the last </main> in the entire document. For the current well-structured pages this is harmless, but if any source page ever gains a second </main> — e.g. inside a <template>, <script>, or a non-collapsed conditional block — the captured body would silently include everything between the two closing tags. Non-greedy is the semantically correct choice here.

Suggested change

	_MAIN_RE = re.compile(r"<main[^>]*>(.*)</main>", re.S)
	_MAIN_RE = re.compile(r"<main[^>]*>(.*?)</main>", re.S)

[greptile-apps]

Stale report.html after re-run in non-ready state

When consolidated_report is None (twin not yet ready), only index.html is refreshed; report.html is left untouched on disk. If a user ran report render once with a ready twin (writing report.html), then re-ran after the twin became un-ready (e.g. data reset, path moved), index.html correctly shows the readiness fallback but report.html silently serves the old, stale consolidated report. Clearing or not creating report.html in the else-branch would prevent this.

[greptile-apps]

_page_style uses _STYLE_RE.search() which stops at the first </style> tag. If a source page ever emits a second <style> block (e.g. dynamically injected component styles), those rules are silently dropped and won't appear in the consolidated report's <head>. A small comment here would flag the intentional limitation so future page authors know to keep all CSS in one block.

Suggested change

	def _page_style(html: str) -> str:
	match = _STYLE_RE.search(html)
	return match.group(1) if match else ""
	def _page_style(html: str) -> str:
	# Only the first <style> block is extracted; all page CSS must live there.
	match = _STYLE_RE.search(html)
	return match.group(1) if match else ""

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!