Make the conditional baseline XGBoost over a full feature set

[johnjosephhorton]

What

Replaces the conditional-baseline model (logistic regression over two cosine-similarity features) with an XGBoost classifier over the full available feature set, and adds a local, API-key-free embedding backend so the baseline can run anywhere.

The feature set (per candidate option)

• raw question and option embeddings — so the model generalizes across questions
• one-hot respondent covariates from panel metadata (party, region, age) — the demographic signal cosine similarity alone can't recover
• embedding-similarity scalars vs the leave-one-question-out respondent profile
• the option index

Training stays leave-one-question-out: the model never sees the target question's labels, so the baseline is deployable, not an oracle.

Why

The old baseline ignored covariates entirely, so it understated how well a cheap model predicts individuals — making the LLM twin look better than it is. A good baseline should be the honest bar.

On Pew w158 the new baseline beats the GPT-5.5 twin on two of three metrics:

model	acc	p(actual)	NLL
GPT-5.5 twin	0.735	0.658	0.553
conditional baseline (XGBoost + covariates)	0.802	0.738	0.571
(old logistic baseline)	0.749	0.728	0.603

The twin now has to earn the claim that it adds individual-level signal.

Also in this PR

• Local sentence-transformers embedder (--embedder sentence-transformers) — the conditional baseline runs with no API key. auto falls back to it when no OpenAI/Expected Parrot key is present.
• Optional-dependency extras: zwill[local-embeddings] (embeddings) and zwill[conditional-baseline] (xgboost + embeddings). Both lazy-imported with actionable install hints.
• Docs updated (module docstring, report appendix HTML, interpreting-results and agent-workflow guides).

Testing

• pytest — 242 passed (adds test_baseline_uses_respondent_covariates, which proves a covariate-only signal lifts the baseline well above chance, and the ST-embedder resolver test).
• ruff check zwill/ tests/ clean.
• Live end-to-end on Pew w158 with local embeddings + covariates (numbers above).

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

This PR replaces the conditional-baseline logistic regression (two cosine-similarity features) with an XGBoost classifier trained over the full feature set — raw question/option embeddings, one-hot respondent covariates, and embedding-similarity scalars — while adding a local sentence-transformers embedder so the baseline can run without any API key.

• Model upgrade: build_conditional_baseline_predictions now uses XGBClassifier (lazy-imported) with leave-one-question-out training on a concatenated feature vector of question embedding + option embedding + covariate one-hots + two similarity scalars + option index; FEATURE_VERSION bumped to v2-xgboost, invalidating old cached job IDs.
• New local embedder: sentence_transformers_embedder with a module-level model cache is added; resolve_baseline_embedder auto-falls back to it when no API keys are present and the sentence-transformers package is installed.
• CLI / docs: --embedder gains a sentence-transformers choice; optional extras zwill[local-embeddings] and zwill[conditional-baseline] added to pyproject.toml; two new tests cover covariate-signal lift and embedder-resolver fallback.

Confidence Score: 4/5

Safe to merge. The core training and prediction logic is well-structured, leave-one-question-out isolation is maintained, and the new covariate/embedder paths are covered by tests.

The XGBoost replacement is substantive but the design is correct: feature construction is consistent across training and inference, binary labels flow correctly through predict_proba, and the feature version bump invalidates prior cached results. The stale --l2 help text and the missing covariate fingerprint in baseline_job_id are real but non-blocking.

zwill/twin_baseline.py (baseline_job_id hash, n_jobs value) and zwill/cli_parser.py (--l2 help text) deserve a second look before merging.

Important Files Changed

Filename	Overview
zwill/twin_baseline.py	Core change: replaces the hand-rolled logistic regression with XGBoost and adds covariate one-hot features plus raw question/option embeddings. Leave-one-question-out training structure is preserved correctly. Minor: baseline_job_id does not fingerprint covariate metadata; n_jobs=0 is XGBoost-specific.
zwill/twin_baseline_commands.py	Adds sentence-transformers backend resolution and passes metadata_by_respondent from respondents.jsonl into the baseline builder. Clean; auto-fallback logic is correct and tested.
tests/test_twin_baseline.py	Adds two well-scoped tests: one verifying covariate signal lifts the baseline above chance, and one verifying sentence-transformers backend resolution and auto-fallback error handling.
pyproject.toml	Adds optional extras local-embeddings and conditional-baseline with sensible minimum version pins (sentence-transformers>=2.2, xgboost>=2.0).
zwill/cli_parser.py	Adds sentence-transformers to --embedder choices. The --l2 help text on both baseline subcommands still says logistic model rather than XGBoost.
zwill/guides/agent-workflow.md	Docs updated to reflect new embedder fallback chain and XGBoost extra install hint. No issues.
zwill/guides/interpreting-results.md	Docs correctly updated to describe XGBoost + covariates baseline and the stronger evaluation bar it sets.

Comments Outside Diff (2)

1. zwill/cli_parser.py, line 121 (link)

   --l2 help text still references the old logistic model

   Both occurrences of --l2 in cli_parser.py (lines 121 and 130) still say "logistic model" in the help string, but the underlying model is now XGBoost and l2 is passed as reg_lambda. Users reading --help output will be confused about what model is being regularized.

2. zwill/twin_baseline.py, line 381-396 (link)

   baseline_job_id does not include covariate metadata in its hash

   The job ID is derived from survey, heldout questions, respondent IDs, embedding model, and FEATURE_VERSION, but not from the respondent metadata (covariates). If someone runs the baseline without metadata and then re-runs after populating respondents.jsonl with metadata (same survey/respondents/questions), both runs produce the same job ID. Without --replace, the second run raises "already exists"; with --replace, the user must remember to use it. Including a stable fingerprint of the metadata keys (e.g. a sorted hash of covariate_keys) in the payload would make the cache key self-validating.

_{Reviews (1): Last reviewed commit: "Make the conditional baseline XGBoost ov..." | Re-trigger Greptile}

[greptile-apps]

n_jobs=0 is XGBoost-specific: it maps to "use all available threads", which is correct, but most readers familiar with the scikit-learn API expect n_jobs=-1 for that behaviour (n_jobs=0 would raise a ValueError in sklearn). Using -1 is the idiomatic cross-library convention and avoids the ambiguity.

Suggested change

	n_jobs=0,
	n_jobs=-1,

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!