Tracking: zwill guide end-to-end dogfooding findings (smb.csv twin validation)

[johnjosephhorton]

Tracking issue for findings from dogfooding the full zwill guide workflow end-to-end: converting a real 33-respondent SMB survey (smb.csv, Qualtrics choice-text export) into a validated digital-twin report — survey → import → commit → one-shot marginals → twin-probability job → twin-validate --require-baseline → report index.html.

The workflow worked and produced a defensible gated result (gpt-5.5 twin beat the conditional baseline on p(actual)/NLL/Brier/top-1 with bootstrap CIs clearing zero; leakage audit clean). These issues are the friction, gaps, and defects encountered along the way. We'll work through them one by one.

🐞 Bugs

• twin-validate and the conditional baseline don't load the nearest .env #12 — twin-validate & baseline don't load the nearest .env (high)
• Plan-driven twin export hardcodes include_survey_context=False #13 — plan-driven twin export hardcodes include_survey_context=False
• init-plan produces a plan that export-plan rejects (Twin approach not found) #14 — init-plan produces a plan export-plan rejects ("Twin approach not found")
• twin-experiment approve overwrites a user-provided prediction_count_estimate #15 — twin-experiment approve overwrites a user-set prediction_count_estimate
• Inconsistent command output contract (stdout envelope vs job dump vs unwrapped file) #16 — inconsistent command output contract (stdout envelope vs job dump vs unwrapped file)

✨ Features / enhancements

• Conditional baseline requires a direct OpenAI key; route embeddings via Expected Parrot / make embedder pluggable #17 — baseline needs a direct OpenAI key; route embeddings via Expected Parrot / pluggable embedder (high)
• First-class multi-select / checkbox question support #18 — first-class multi-select / checkbox question support
• Make rank-battery detection explicit/robust instead of heuristic #19 — make rank-battery detection explicit/robust instead of heuristic
• Context-question selection is positional; make it relevance-aware or prioritizable #20 — context-question selection is positional; make it relevance-aware
• Support glob / rank-task / battery-level leakage exclusions #21 — glob / rank-task / battery-level leakage exclusions
• Prediction-count estimate should reflect actual fill / complete-cases; warn when sample > population #22 — prediction-count estimate should reflect actual fill / complete-cases; warn when sample > population
• One command to generate + render the required report interpretations #23 — one command to generate + render the required report interpretations
• Rank batteries have no place in the flagship twin-validate gate / report #24 — rank batteries have no place in the flagship twin-validate gate / report
• Optional auto-retry for malformed provider rows #25 — optional auto-retry for malformed provider rows

📄 Documentation

• Document the import JSONL schemas (questions / respondents / answers) #26 — document the import JSONL schemas (questions / respondents / answers) (high)
• Fix incorrect/incomplete command examples in the agent-workflow guide #27 — fix incorrect/incomplete command examples in the agent-workflow guide
• Document hidden behaviors and improve CLI help discoverability #28 — document hidden behaviors & improve CLI help discoverability

Suggested order

1. Document the import JSONL schemas (questions / respondents / answers) #26 (import schema) and twin-validate and the conditional baseline don't load the nearest .env #12 (.env) — remove the two things that most block a first clean run.
2. Conditional baseline requires a direct OpenAI key; route embeddings via Expected Parrot / make embedder pluggable #17 (EP-routed baseline) — unblocks the gated validation without a separate OpenAI key.
3. Fix incorrect/incomplete command examples in the agent-workflow guide #27 (guide example fixes) — cheap, stops the guide from mis-instructing.
4. Then the remaining enhancements as prioritized.

All issues carry the twin-workflow-review label.