research: rewrite-efficacy Study 4 — stage 2 (en) - #720
Merged
Conversation
…k exclusion Owner asked to resume after stage 1 closed; the note is recorded before any stage 2 row. The bridge now takes BRIDGE_STAGE=en (Arm A1, English judge prompt, 84 passages) and the runner inherits Study 1's spok exclusion and reads the stage-specific admission verdict. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…iption-path gemini judge - judge-gemini-3.7-flash (Gemini API) on the 84 archived Arm-A1 passages: AUC 0.994, Spearman vs gpt 0.911 (grok reference 0.894) — admitted. - Adds judge-gemini-3.7-flash-cli: the same model through the gemini CLI's subscription login (no API key), bridged separately; the gemini binary is resolved once so detached runners work without nvm on PATH. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
…on EN; CLI transport not admitted (timeouts) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
…uthenticated on this machine Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
…ges closed - 42/42 documents, gpt + gemini-3.7-flash + det (binary disabled: fresh-corpus accuracy 0.762). - Primary paired d(S − P) on AI docs +3.7 [−3.5, +10.8]; det chief +12.1 [+4.5, +20.6]; AI-call rate unchanged 95.2%. - The 98% floor was met on 42/42 (plain rewrite compresses English to 0.81), so this is the clean test stage 1 could not deliver — and S reads more AI. - Guard rail 1 violated (38/42, four real dropped numbers); rails 2–3 held, rail 3 vacuous on English (detail-token regex matches every word; recorded as a dated note, not a criterion change). - Combined verdict: nothing ships; next candidate H-4a. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Study 4 stage 2 (English) complete: H-4b NOT supported, same direction as Korean. Both stages closed; nothing ships; next candidate H-4a. Results section added to
docs/research/2026-rewrite-efficacy-study4.md; dated notes in the prereg cover the resume, the judge seat, the API-key correction, and a post-hoc metric caveat.spokexcluded). Same rewriterclaude-sonnet-4-6; English production prompt + English constraint block, all fixed before the first row.judge-gpt+judge-gemini-3.7-flash(Gemini API; re-admitted on the 84 archived English passages: AUC 0.994, Spearman vs gpt 0.911 vs grok's 0.894) + det chief with the binary column disabled (fresh-corpus accuracy 0.762). The gemini CLI transport of the same model timed out and was not admitted; a correction note records that on this machine the CLI was API-key authenticated anyway.Results (42/42)
Test plan
node --check+ eslint on the changed scriptsMerge with a merge commit: rows reference commits on this branch.
🤖 Generated with Claude Code
https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN