Make the baseline embedder auto-path reliable so --require-baseline never hangs#77
Conversation
…ever hangs A research-agent run stalled `twin-validate --require-baseline` for 600s and gave up (ran `--skip-baseline`), losing the whole point of the validation. Root cause: `--embedder auto` preferred the REMOTE Expected Parrot embeddings endpoint whenever EXPECTED_PARROT_API_KEY was set — which it always is for twin runs — and that endpoint is unavailable/404ing, so it hung. The XGBoost compute itself is fine (~22s at full Pew scale: 69 questions, 200 respondents, 8 held-out, 544k training rows). Fixes: - auto now prefers reliable, local backends and NEVER the remote endpoint: direct OPENAI_API_KEY, else local sentence-transformers if installed, else a new zero-dependency built-in lexical (hashing) embedder that always runs. The remote Expected Parrot embeddings stay reachable via explicit `--embedder edsl`. - add hashing_embedder: bag-of-words feature hashing, no model download, no network, instant. Weaker than a semantic embedder (leans on covariates) but the baseline always runs — so `--require-baseline` can't hang or be skipped. Emits a stderr warning when auto falls back to it. - `--embedder` gains a `hashing` choice; help + guide updated to describe the new auto order. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Greptile SummaryThis PR fixes a live hang in
Confidence Score: 3/5Safe to merge for the common case; one concrete crash path in the newly added hashing embedder needs attention before running on FIPS-compliant hosts. The hashing embedder is the entire reliability guarantee of this PR — it's what prevents
Important Files Changed
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| for text in texts: | ||
| vector = [0.0] * dim | ||
| for token in str(text).lower().split(): | ||
| bucket = int(hashlib.sha1(token.encode()).hexdigest(), 16) % dim |
There was a problem hiding this comment.
usedforsecurity=False breaks on FIPS systems
hashlib.sha1(token.encode()) raises ValueError on Python builds compiled against OpenSSL in FIPS mode (e.g., RHEL 8+, some AWS/GovCloud images). This directly contradicts the stated invariant — the hashing embedder is supposed to "always run" as the last-resort fallback — but on FIPS-constrained hosts it would crash with a ValueError instead. Adding usedforsecurity=False (Python ≥ 3.9) tells OpenSSL the hash is used for non-cryptographic purposes and is always accepted.
| bucket = int(hashlib.sha1(token.encode()).hexdigest(), 16) % dim | |
| bucket = int(hashlib.sha1(token.encode(), usedforsecurity=False).hexdigest(), 16) % dim |
1 participant
The bug (from a live research-agent run)
zwill twin-validate --survey w152 --require-baselinestalled for 600s and was abandoned — the agent fell back to--skip-baseline, losing the conditional-baseline comparison entirely (the whole point of the validation).Root cause:
--embedder autopreferred the remote Expected Parrot embeddings endpoint wheneverEXPECTED_PARROT_API_KEYwas set — which it always is for twin runs — and that endpoint is unavailable, so it hung.I profiled the baseline at full Pew scale (69 questions, 200 respondents, 8 held-out, 98 covariates, 544k training rows) with a fast embedder: 22s. So the compute was never the problem — the embedding backend selection was.
Fix
autoprefers reliable, local backends and never the remote endpoint: directOPENAI_API_KEY→ local sentence-transformers (if installed) → a new built-in lexical embedder that always runs. The remote endpoint is opt-in only via--embedder edsl.hashing_embedder— bag-of-words feature hashing, zero dependencies, no model download, no network, instant. Weaker than a semantic embedder (it leans on respondent covariates), but the baseline always runs, so--require-baselinecan't hang or be silently skipped. Auto emits a stderr warning when it falls back to it.--embeddergains ahashingchoice; help text and the agent-workflow guide updated for the new order.Testing
pytest— 261 passed (rewrote the resolver test: forced local/hashing resolve without keys;autostays local even with an Expected Parrot key set;autowith no keys/no ST returns the lexical embedder that actually embeds, plus a warning — never an error/hang).ruffclean.Note: the same run also didn't produce a final report (the agent went quiet after fighting the baseline). That looks like agent behavior downstream of this stall rather than a separate CLI error; fixing the stall should remove the thing it got stuck on. Happy to dig further if it recurs.
🤖 Generated with Claude Code