workflow-fix #777: flip dual-DV continuous-companion polarity (fixed +/- margin PREFERRED, logp_pos_mean OPT-IN)#579
Closed
superkaiba wants to merge 2 commits into
Closed
workflow-fix #777: flip dual-DV continuous-companion polarity (fixed +/- margin PREFERRED, logp_pos_mean OPT-IN)#579superkaiba wants to merge 2 commits into
superkaiba wants to merge 2 commits into
Conversation
…ed fixed +/- margin (preferred); logp_pos_mean is selection-confounded opt-in Per #722: the teacher-forced fixed positive-vs-negative completion margin (FIXED answer pools across all contexts => no selection-on-outcome bias) PASSES the rho(DV, rate) > 0 validation gate (refusal +0.56, sycophancy +0.40, broad_em +0.31; all-positive). The judged-positive-conditional-mean log-P (logp_pos_mean) is a conditional mean over an outcome-selected subset (selection bias) and FAILS the validation for 3 of 4 behaviors (rho ~= -0.3). Flip the PREFERRED/OPT-IN polarity in CLAUDE.md § Measurement validity + 5 enforcer agent files; add the recipe + failure-mode subsection to .claude/rules/llm-judging.md; add a pure additive cross-reference note to .claude/rules/marker-leakage-measurement.md (the marker three-space recipe is NOT touched). Invariants preserved: on-policy rate PRIMARY, rho(DV, rate) > 0 validation gate verbatim, #432->#456 teacher-forcing caution, graded-score-primary ranking/regression extension, marker three-space recipe. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…fter § E2 addition
Codex code-review CONCERN (minor): the new § E2 continuous-companion-DV
section in .claude/rules/llm-judging.md added numbered guidelines 19 and
20, but three cross-references still advertised "18 guidelines":
- .claude/rules/llm-judging.md:8 ("full 18-guideline recipe")
- CLAUDE.md:20 ("full 18-guideline set")
- CLAUDE.md:20 ("the 18 judging guidelines")
Update each to acknowledge both halves: "20-guideline (the 18 judging
guidelines plus the § E2 continuous-companion-DV recipe)". Preserves
the original 18-count provenance while making the addendum visible. No
recipe lines touched.
Generated with [Claude Code](https://claude.ai/code)
via [Happy](https://happy.engineering)
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Happy <yesreply@happy.engineering>
Owner
Author
|
Content already on main via concurrent-session rebase-merge (see main commits |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes task #777.
Summary
Corrects the dual-DV continuous-companion guidance in the project's
measurement-validity rules. The PREFERRED non-saturating continuous DV
is now the teacher-forced fixed positive-vs-negative completion margin
(fixed answer set across all contexts → no selection-on-outcome bias);
the length-normalized log-P of judged-positive on-policy completions
(
logp_pos_mean) is flagged as an OPT-IN alternative that carriesselection-on-outcome bias and FAILED task #722's ρ(DV, rate) > 0
validation gate for 3 of 4 behaviors (ρ ≈ −0.3).
Evidence
Task #722's
eval_results/issue_722/tf_margin/margin_chain.json:logp_pos_meanρ(DV, rate) ≈ −0.3 for 3 of 4 behaviors — FAILED the same gate.Diff
10 markdown files (0
.py, 0 configs), +65/-13 lines total across 2 commits:d30d1fd600— polarity flip inCLAUDE.md+.claude/rules/{marker-leakage-measurement,llm-judging}.md+ 5 enforcer agent files (planner.md§6,critic.mdStats lens,analyzer.mdstep 3,interpretation-critic.mdLens 1,clean-result-critic.mddual-DV lens).883352fcc9— folded-in Codex Minor CONCERN: bump 18→20 guideline count references to acknowledge the new§ E2continuous-companion-DV recipe alongside the 18 judging guidelines.5 MUST-NOT-CHANGE invariants preserved
Log and analyze ALL THREE spaces) untouched.Test plan
scripts/select_step9c_tests.py --base maintouched-file scope: 2318 pass / 5 fail / 6 skip.main(byte-identical evidence): 3× the#745scripts/issue744_dump_and_stream.py:87bare-dotenv lint (tracked separately), 2× HF-Hub network-state audit-claim tests.🤖 Generated with Claude Code