From b667b5c41b4e93a78187748d737de8c22afe570f Mon Sep 17 00:00:00 2001 From: John Horton Date: Fri, 10 Jul 2026 07:53:00 -0400 Subject: [PATCH] Study-runner skill: force a delivered report; refresh stale baseline guidance MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A research-agent run went passive ("Standing by", "Here whenever you need me") when asked to validate and produce the report — it never ran the commands, so no report was created. The skill that drives the agent had no non-negotiable terminal step and was stale on the baseline/embedder. - Add step 6, "Build and deliver the full report — do not stop before this": run `zwill report build --survey --path report_out`, confirm report.html exists, and tell the user its path. Explicitly: never end a validation request with "standing by"; a "blocked" interpretation stage is NOT "no report" (the deterministic report is complete and viewable without the optional narrative). - Refresh the conditional-baseline description: it is XGBoost over embeddings + respondent covariates (not "embedding + logistic"), a strong deployable bar. - Refresh the embedder guidance: auto tries Expected Parrot first (health-probed, fails over in seconds), then OpenAI, then local sentence-transformers, then a built-in lexical embedder — so it runs with no key and never hangs. The old text said it "needs OPENAI_API_KEY," which could make the agent skip it. - Stop-condition updated: the baseline always runs; flag the weaker lexical fallback rather than claiming it can't run. Co-Authored-By: Claude Opus 4.8 (1M context) --- zwill/skills/digital-twin-study-runner/SKILL.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/zwill/skills/digital-twin-study-runner/SKILL.md b/zwill/skills/digital-twin-study-runner/SKILL.md index 44e5692..0f3d5bf 100644 --- a/zwill/skills/digital-twin-study-runner/SKILL.md +++ b/zwill/skills/digital-twin-study-runner/SKILL.md @@ -43,7 +43,7 @@ Core rule: always expand survey codebooks before import/export. Use human-readab `twin-validate` runs, in order: - **Leakage audit** over the held-out targets — flags any context question that near-deterministically predicts a target (bias-corrected Cramér's V). Leakage is the top validity threat; a "twin" that only works on leaky targets is copying, not modelling. - - **Conditional baseline** — a cheap embedding + logistic model fit on the *same respondents* the twins scored, using only what is available for a genuinely new question (question/option wording + the respondent's other answers, leave-one-question-out; it never sees the target's own marginal). This is the fair yardstick. Needs `OPENAI_API_KEY` for embeddings; pass `--skip-baseline` to omit it, or `--require-baseline` to fail rather than warn. + - **Conditional baseline** — an **XGBoost** model over question/option embeddings **plus the respondent's panel covariates**, fit on the *same respondents* the twins scored, leave-one-question-out (it never sees the target's own marginal). This is the fair, deployable yardstick, and with covariates it is a *strong* bar. The embedder auto-selects a working backend — Expected Parrot first (behind a short health probe, so an unavailable endpoint fails over in seconds instead of hanging), then a direct `OPENAI_API_KEY`, then a local sentence-transformers model, then a built-in lexical embedder — so it **runs even with no key and never hangs**. Install `zwill[conditional-baseline]` for the strongest (semantic) baseline. Pass `--skip-baseline` to omit it, or `--require-baseline` to fail rather than warn. - **Bootstrap confidence intervals** on each model's scores and on the paired twin-minus-baseline deltas, resampling respondents. This answers "is the gap real or sampling noise?" - **HTML report** embedding skill scores, the bootstrap panel, the probability-granularity check, the correlation-attenuation verdict, and the baseline appendix. @@ -57,6 +57,16 @@ Core rule: always expand survey codebooks before import/export. Use human-readab - **Probability granularity**: a model flagged "coarse" (mass piled on round numbers) has quantization-limited Brier/calibration — read those scores with that ceiling. - **Correlation attenuation**: if the twin's implied cross-question association is systematically below the empirical association, the twin is over-shrinking toward a common distribution — its marginals can look fine while its joint structure is washed out. +6. **Build and deliver the full report — do not stop before this.** + + The validation bundle above is the technical readout. The *full* report — survey profile, one-shot marginals, twin validation, and executive summary as **one scrollable page with a table of contents** — comes from: + + ```bash + zwill report build --survey --path report_out + ``` + + This writes `report_out/report.html` (also `report_out/index.html`). **The task is not complete until that file exists and you have told the user its path.** Never end a validation request with "standing by" or "here when you need me" — run the command, confirm the file was written, and report where it is. If the frontier-model executive narrative has not been generated, the report is still complete and viewable (the narrative is optional and can be added later); say so rather than treating a "blocked" interpretation stage as "no report." + ## Useful single-step commands Run or re-run one piece (all of these feed the same prediction store `twin-validate` reads): @@ -82,4 +92,4 @@ Do not proceed to a practitioner report or make a positive claim until: - calibration is inspected: a model with high accuracy but poor NLL/ECE (overconfident misses) is not reported as good without that caveat; - the held-out questions are documented well enough to interpret successes and failures. -If the conditional baseline cannot run (no embedding key) and the study still needs a readout, say so explicitly: without it you can only claim the twin beats uniform/marginal, not that it beats a cheap individual model. +The conditional baseline now **always runs** — with no embedding key it falls back to a built-in lexical embedder. If only that lexical fallback was available (a stderr warning says so), note the baseline is a *weaker* bar leaning mostly on covariates — install `zwill[conditional-baseline]` for the semantic model — but you can still make the twin-vs-baseline claim.