research: rewrite-efficacy Study 4 — specificity-preservation constraint (H-4b), registration + harness - #719
Merged
Merged
Conversation
… constraint, H-4b) Registered before any Study 4 data: two fresh arms (P plain production prompt, S plain + a fixed specificity block with a 98% length floor and capped feedback retries) on the 54 Study 1 ko documents, stage 2 en; panel v2 chief det + gpt, with gemini replacing grok under a registered deviation (xAI credits exhausted 2026-09-02) gated by a bridge admission rule; primary paired d(S - P) <= -3 with CI upper < 0; guard rails on meaning, human controls, invented detail tokens and copy similarity. Survey H-4b marked activated; backlog records the owner's reorder; study4 artifacts ignored. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
- study4-common.mjs: S1-identical judge prompt/parse, S3-shape claude call, gemini/gpt/grok judge definitions, the pre-registered detail-token and trigram-copy measures, bootstrap/Spearman/AUC/McNemar helpers. - rewrite-efficacy-study4-bridge.mjs: gemini on the 108 archived S1-D passages; applies the registered admission rule and writes bridge-verdict. - rewrite-efficacy-study4.mjs: two fresh arms per document (P production minimal prompt, S + fixed specificity block with a 98% floor and two capped feedback retries), production body extraction, meaning gate, det chief with the run-time fresh-corpus gate, admitted LLM judges; fail-soft rows, resume, circuit breaker; smoke mode on a synthetic paragraph only. - loop + analyze scripts mirror the Study 3 pair and the prereg decision rules. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
…e; stage 1 runs single-perceptual-judge Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…the registered admission rule Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
…al judge (bridge AUC 1.000, rho 0.884) Adds the OpenAI-compatible HTTP judge CLI, the four API candidates, a candidate-parameterized bridge and the rejudge script; records every candidate's bridge verdict and the selection in the prereg. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
…r diagnosis Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
…ote, Study 3 precedent) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
…udget Sleep until claude's stated session-limit reset instead of 30-minute polls, and allow 240 passes; plumbing only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
…ships - 54/54 documents, both arms, gpt + gemini-3.7-flash + det chief. - Primary paired d(S − P) on AI docs +1.6 [−3.1, +6.5] (needed ≤ −3, CI upper < 0); det chief +8.5 [+0.8, +17.0]; AI-call rate unchanged 94.4%. - The 98% floor was met on 15/27 AI docs after up to three attempts; copy check fires (S median trigram Jaccard 0.667 vs P 0.517): S rewrote less. - Guard rail 1 violated (50/54 meaning gate; two known artifact docs plus two real dropped numbers); guard rails 2 and 3 held. - Committed artifacts are hash-and-score rows plus bridge rows/verdicts for all five judge candidates; raw texts stay ignored. - Stage 2 (en) not started; deferral is an owner decision (dated note). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Study 4 (H-4b, specificity-preservation constraint) — stage 1 (ko) complete: NOT supported, nothing ships. Registration, harness, judge-panel deviation notes, hash-only artifacts and the results doc
docs/research/2026-rewrite-efficacy-study4.md.claude-sonnet-4-6, back-to-back. Registered before any data.judge-gpt+judge-gemini-3.7-flash(Gemini API), admitted by the registered bridge rule on the 108 archived passages (AUC 1.000, Spearman vs gpt 0.884; grok reference 0.967 / 0.857). xAI credits were exhausted; gemini-2.5-pro CLI failed admission (AUC 0.556); deepseek incomplete; kimi billing. All bridge verdicts committed.Results (54/54 docs)
Test plan
node --check+ eslint on all new scripts; smoke run on a synthetic paragraph only🤖 Generated with Claude Code
https://claude.ai/code/session_018ePnwo4nBWCWvFvpy1YgCN