Add CoT baseline solver for H1 comparison#46
Open
Gun-Akash wants to merge 1 commit into
Open
Conversation
Member
|
@Gun-Akash, The branch is stale dont merge yet. rebase onto main first & CoT uses k=1 (pass@1) but the MARC main eval uses k=4 (pass@4). N and delta match, but MARC getting 4 candidates vs CoT's 1 isn't same H1 comparison either bump CoT to k=4 (4× gpt-5 token usage) or run MARC at k=1 for this table. Your call, but worth settling before the number goes in the paper. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
P2: token-level CoT baseline (H1 comparison)
Implements the CoT baseline for the H1 fairness comparison per the P2 ticket.
What's here
marc/eval/baselines/cot_baseline.py— CoTSolver (render_prompt → gpt-5 → parse_answer),plugs into the existing
run_split_evalharness. Reusesperturb_constantsand theChecker; no separate perturbation script needed.
marc/eval/baselines/README.md— model, prompt, run config, known limitations.Results (N=25, perturb_delta=0.1, k=1)
Needs input before this is "done":
results/is gitignored, socot_baseline.jsonisn't in this PR. How do youwant the results file — direct send, shared drive, or a
!results/p2_main/exceptionlike the existing p4_scale one? It's on disk and ready either way.
(I used the defaults).
does show a gap, but flagging that a harder suite (e.g. length_extrapolation, higher n_vars)
may be needed for solve_rate to be an informative H1 signal.
perturb_constants+ Checker on gpt-5 output;parse failures come through as NaN (Checker rejects them correctly).
Ticket: akash/p2-baseline