Flag stale narratives on model-set change; mark section pages as sections#76
Conversation
…ions Two revision-safety fixes for the consolidated report. Stale-narrative detection. The cached executive narrative (LLM prose) is matched to a rebuild on jobs + questions but NOT on the models inside those jobs, so adding a model to an existing job left the prose describing one model while the tables showed two — silently. The generation context now records the twin model set the narrative was written against (baseline excluded, twin-to-twin); on rebuild the report compares it to the current model set and, when they differ, renders a loud "this written summary is out of date — regenerate it" banner in the Decision section instead of showing stale prose. Narratives generated before fingerprinting record nothing and are never falsely flagged. Section-page banner. The intermediate page files (executive-summary, twin-validation, survey-profile, one-shot-marginals) still exist on disk as fold-in sources. Each now gets a "this is one section of a larger report — open the full report" banner, inserted outside <main> so the consolidated composer (which lifts only <main>) never pulls it into the report itself. Follow-up (noted): cache the deterministic executive-summary artifacts by the same fingerprint so unchanged rebuilds skip the permutation/bootstrap recompute. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Greptile SummaryThis PR adds two defences against a silent staleness problem in the consolidated report: a model-set fingerprint stored at narrative-generation time lets rebuilds detect when cached LLM prose describes a different set of twin models than the current tables, surfacing a loud banner instead of stale text; and a "this is a section" banner is stamped onto each intermediate page file so they are not mistaken for the full report when opened in isolation.
Confidence Score: 4/5Safe to merge; all changes are additive guardrails with no modifications to existing data paths or schemas. The staleness detection and section-banner logic are well-scoped and independently verified by tests. The notable gap is that the detection block in The staleness detection block in Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[build_report_bundle] --> B[find_imported_executive_summary_report]
B --> C{generated_executive found?}
C -- No --> D[generated_markdown = None]
C -- Yes --> E[narrative_models = generation.narrative_model_labels]
E --> F{narrative_models is not None?}
F -- No old narrative --> G[No stale flag back-compat]
F -- Yes --> H{sorted narrative_models != current_models?}
H -- Same --> I[No stale flag]
H -- Different --> J[generation.stale = True]
J --> K[build_executive_summary with stale generation]
G --> K
I --> K
D --> K
K --> L{generated_markdown truthful?}
L -- Yes --> M[executive_body = stale_banner + callout div]
L -- No --> N[Fallback body — stale_banner NOT shown]
M --> O[Write executive-summary.html]
N --> O
O --> P[build_consolidated_report_html lifts only main]
P --> Q[Write report.html / index.html]
Q --> R[mark_intermediate_page_html on section files]
R --> S[Banner outside main — excluded from consolidated report]
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A[build_report_bundle] --> B[find_imported_executive_summary_report]
B --> C{generated_executive found?}
C -- No --> D[generated_markdown = None]
C -- Yes --> E[narrative_models = generation.narrative_model_labels]
E --> F{narrative_models is not None?}
F -- No old narrative --> G[No stale flag back-compat]
F -- Yes --> H{sorted narrative_models != current_models?}
H -- Same --> I[No stale flag]
H -- Different --> J[generation.stale = True]
J --> K[build_executive_summary with stale generation]
G --> K
I --> K
D --> K
K --> L{generated_markdown truthful?}
L -- Yes --> M[executive_body = stale_banner + callout div]
L -- No --> N[Fallback body — stale_banner NOT shown]
M --> O[Write executive-summary.html]
N --> O
O --> P[build_consolidated_report_html lifts only main]
P --> Q[Write report.html / index.html]
Q --> R[mark_intermediate_page_html on section files]
R --> S[Banner outside main — excluded from consolidated report]
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| assert comp["share_twin_better"] == 1.0 | ||
|
|
||
|
|
||
| def test_stale_narrative_banner_when_model_set_changed(tmp_path) -> None: | ||
| twin = [_row("openai:gpt-5.5", "r1", "A", 0.20, 1), _row("openai:gpt-5.5", "r2", "B", 0.25, 1)] | ||
| build_executive_summary( | ||
| twin, | ||
| survey="demo", | ||
| path=tmp_path / "exec.html", | ||
| markdown_path=None, | ||
| simulations=25, | ||
| seed=1, | ||
| generated_markdown="The GPT-5.5 twin captures individual signal.", | ||
| generation={"report_id": "r", "stale": True, "stale_reason": "written for openai:gpt-5.5; current: anthropic:claude-opus-4-8, openai:gpt-5.5"}, | ||
| ) | ||
| html = (tmp_path / "exec.html").read_text() | ||
| assert "This written summary is out of date" in html | ||
| assert "anthropic:claude-opus-4-8" in html | ||
|
|
||
|
|
||
| def test_no_stale_banner_for_fresh_narrative(tmp_path) -> None: | ||
| twin = [_row("openai:gpt-5.5", "r1", "A", 0.20, 1), _row("openai:gpt-5.5", "r2", "B", 0.25, 1)] | ||
| build_executive_summary( | ||
| twin, | ||
| survey="demo", | ||
| path=tmp_path / "exec.html", | ||
| markdown_path=None, | ||
| simulations=25, | ||
| seed=1, | ||
| generated_markdown="Fresh narrative.", | ||
| generation={"report_id": "r"}, | ||
| ) | ||
| assert "This written summary is out of date" not in (tmp_path / "exec.html").read_text() | ||
|
|
||
|
|
||
| def test_executive_summary_unchanged_without_baseline(tmp_path) -> None: | ||
| twin = [_row("openai:gpt-5.5", "r1", "A", 0.20, 1), _row("openai:gpt-5.5", "r2", "B", 0.25, 1)] | ||
| result = build_executive_summary( |
There was a problem hiding this comment.
Both new tests call build_executive_summary with a hand-crafted generation={"stale": True, …} dict. They confirm the HTML banner appears when told to, but the code in build_report_bundle that computes whether staleness should be set (comparing narrative_model_labels vs the current model set) has no direct test. If, for example, the key name mismatched — narrative_model_labels vs model_labels — or the None-guard accidentally fired on an empty list, the detection would silently skip marking and neither test would catch it.
| return html[: match.end()] + "\n" + banner + html[match.end() :] | ||
|
|
||
| _CONSOLIDATED_CSS = """ |
There was a problem hiding this comment.
mark_intermediate_page_html and the module-level _CONSOLIDATED_CSS string — PEP 8 recommends two blank lines surrounding top-level definitions.
| return html[: match.end()] + "\n" + banner + html[match.end() :] | |
| _CONSOLIDATED_CSS = """ | |
| return html[: match.end()] + "\n" + banner + html[match.end() :] | |
| _CONSOLIDATED_CSS = """ |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
1 participant
Follow-through on the "is report.html vulnerable to revisions?" discussion.
1. Stale-narrative detection (the real fragility)
The cached executive narrative (LLM prose) is matched to a rebuild on jobs + questions but not on the models inside those jobs. So adding a model to an existing job left the prose describing one model while the tables showed two — silently.
Now:
2. "Not meant to be read separately" banner
The intermediate page files (
executive-summary,twin-validation,survey-profile,one-shot-marginals) still exist on disk as the fold-in sources for the consolidated report. Each now carries a "this is one section of a larger report — open the full report" banner. It's inserted outside<main>, so the composer (which lifts only<main>) never pulls it into the report itself — verified by test.Testing
pytest— 261 passed (adds: stale banner shows on model-set change / absent for a fresh narrative; section banner is added outside<main>, is idempotent, and is excluded from the composed report).ruffclean.Follow-up (noted, not done)
Cache the deterministic executive-summary artifacts (permutation/bootstrap) by the same fingerprint so an unchanged rebuild skips the recompute — the remaining "cheap rewrite" win. Right now the compose is already instant; this would remove the permutation cost on no-op rebuilds.
🤖 Generated with Claude Code