Feature the conditional baseline on the executive summary and index

[johnjosephhorton]

Part A of the report cleanup (per our discussion — B, the structural de-dup, is still open).

Problem

The executive summary and the bundle index — the two pages a decision-maker reads — compared the twin only against uniform guessing. The decisive comparison (twin vs the deployable XGBoost conditional baseline) lived only on the technical validation page (after #73). So the headline pages told a weaker, different story than the detail page, and "leaned on uniform" exactly as flagged.

Change

build_executive_summary now accepts the baseline rows and computes a twin-vs-conditional-baseline comparison — aggregate baseline metrics plus a paired win rate and p(actual) delta over the (respondent, question) rows both scored. Then:

• Executive summary → Main Evidence gains a "Conditional baseline (XGBoost)" column beside Twin/Uniform, and a lead sentence: "Versus the deployable bar: the twin beats/ties/trails the XGBoost conditional baseline … +X.X% p(actual), wins on Y% of predictions. This is the comparison that matters; uniform and the empirical oracle are context."
• Bundle index gains a "Beats conditional baseline / p(actual) vs conditional baseline" row right under the twin's p(actual).
• The comparison is returned as conditional_comparison for the index to consume.

All gated on a baseline being present — no-baseline reports are byte-for-byte unchanged. The bundle passes the baseline rows folded into scope by #73; the standalone twin-results executive-summary command is left unchanged for now (noted follow-up).

Testing

• pytest — 256 passed (adds exec-summary tests: conditional column + lead + conditional_comparison present with a correct paired win rate; unchanged when no baseline).
• ruff clean.

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

This PR adds a twin-vs-conditional-baseline (XGBoost) comparison to the executive summary and bundle index, so decision-maker pages now show the deployable bar alongside uniform guessing. The change is gated on a baseline being present and passes the baseline rows through from the bundle build path.

• executive_summary.py: adds conditional_baseline_comparison to compute paired win rate / mean p(actual) delta, and injects a new column plus a lead sentence into the Main Evidence table when a baseline is present.
• report_bundle.py: reads conditional_comparison from the executive summary result and renders a "Beats conditional baseline / p(actual) vs conditional baseline" row in the index tile.
• tests/test_report_conditional_baseline.py: adds two tests covering the executive summary path — conditional column present with a baseline, and unchanged without one.

Confidence Score: 4/5

Safe to merge for the common path; the zero-matched-rows edge case is the only current defect.

The new conditional_baseline_comparison function returns a non-None dict whenever baseline rows are present, even when no (respondent_id, heldout_question) pairs overlap between the twin and baseline datasets. Both render sites check if conditional: on that dict, so a zero-match situation renders a misleading ties verdict with +0.0% / 0% of predictions on exactly the pages this PR is intended to improve.

zwill/executive_summary.py — specifically the return value of conditional_baseline_comparison when matched_rows == 0, and the corresponding if conditional: guard in render_html; zwill/report_bundle.py carries the same guard pattern.

Important Files Changed

Filename	Overview
zwill/executive_summary.py	Adds conditional_baseline_comparison and threads its output through render_html and build_executive_summary; zero-match case when baseline keys don't align produces misleading rendered output
zwill/report_bundle.py	Reads conditional_comparison from executive summary payload and injects a conditional row into the bundle index tile; inherits the zero-match rendering issue from executive_summary.py
tests/test_report_conditional_baseline.py	Adds two new tests covering the executive summary path — conditional column present with a baseline, and unchanged without one; both use fully matched datasets

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[build_report_bundle] -->|baseline_rows| B[build_executive_summary]
    B --> C[conditional_baseline_comparison]
    C -->|baseline_rows empty| D[return None]
    C -->|baseline_rows present| E{matched_rows > 0?}
    E -->|yes| F[return dict with metrics + win rate + delta]
    E -->|no — keys don't align| G[return dict with matched_rows=0\nmean_p_delta=0.0\nshare_twin_better=0.0]
    F --> H[render_html\nconditional column + lead sentence]
    G -->|if conditional: is truthy| H
    H --> I[Executive Summary HTML]
    F --> J[render_report_bundle_index\nconditional_row]
    G -->|if conditional: is truthy| J
    J --> K[Bundle Index HTML]
    D -->|no conditional block rendered| I
    D -->|no conditional_row rendered| K

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[build_report_bundle] -->|baseline_rows| B[build_executive_summary]
    B --> C[conditional_baseline_comparison]
    C -->|baseline_rows empty| D[return None]
    C -->|baseline_rows present| E{matched_rows > 0?}
    E -->|yes| F[return dict with metrics + win rate + delta]
    E -->|no — keys don't align| G[return dict with matched_rows=0\nmean_p_delta=0.0\nshare_twin_better=0.0]
    F --> H[render_html\nconditional column + lead sentence]
    G -->|if conditional: is truthy| H
    H --> I[Executive Summary HTML]
    F --> J[render_report_bundle_index\nconditional_row]
    G -->|if conditional: is truthy| J
    J --> K[Bundle Index HTML]
    D -->|no conditional block rendered| I
    D -->|no conditional_row rendered| K

Loading

_{Reviews (1): Last reviewed commit: "Feature the conditional baseline on the ..." | Re-trigger Greptile}

[greptile-apps]

Zero-match baseline silently renders misleading stats

When baseline_rows is non-empty but no (respondent_id, heldout_question) key pairs match the twin rows (e.g. ID mismatch in the data pipeline), conditional_baseline_comparison returns a valid dict with matched_rows=0, mean_p_delta=0.0, share_twin_better=0.0. Both render_html and render_report_bundle_index then check if conditional: — which is truthy for any non-empty dict — and render the block on the decision-maker pages with a "ties" verdict, "+0.0% probability", and "wins on 0% of matched predictions", even though zero predictions were actually compared. The fix is to either return None when matched_rows == 0, or gate both render sites with conditional.get("matched_rows", 0) > 0.