Skip to content

Add CoT baseline solver for H1 comparison#46

Open
Gun-Akash wants to merge 1 commit into
mainfrom
akash/p2-baseline
Open

Add CoT baseline solver for H1 comparison#46
Gun-Akash wants to merge 1 commit into
mainfrom
akash/p2-baseline

Conversation

@Gun-Akash

Copy link
Copy Markdown
Collaborator

P2: token-level CoT baseline (H1 comparison)

Implements the CoT baseline for the H1 fairness comparison per the P2 ticket.

What's here

  • marc/eval/baselines/cot_baseline.py — CoTSolver (render_prompt → gpt-5 → parse_answer),
    plugs into the existing run_split_eval harness. Reuses perturb_constants and the
    Checker; no separate perturbation script needed.
  • marc/eval/baselines/README.md — model, prompt, run config, known limitations.

Results (N=25, perturb_delta=0.1, k=1)

  • solve_rate: 1.0 on both splits → generalization_gap 0.0
  • perturbation_robustness: 0.0 (in-dist) vs 0.12 (held-out)

Needs input before this is "done":

  • @sparshresults/ is gitignored, so cot_baseline.json isn't in this PR. How do you
    want the results file — direct send, shared drive, or a !results/p2_main/ exception
    like the existing p4_scale one? It's on disk and ready either way.
  • @sparsh — confirm N and perturb_delta match what you need for cross-baseline comparability
    (I used the defaults).
  • solve_rate saturates at 1.0, so it doesn't separate the splits on this suite. Perturbation
    does show a gap, but flagging that a harder suite (e.g. length_extrapolation, higher n_vars)
    may be needed for solve_rate to be an informative H1 signal.
  • @davin — heads up the baseline exercises perturb_constants + Checker on gpt-5 output;
    parse failures come through as NaN (Checker rejects them correctly).

Ticket: akash/p2-baseline

@ImSpxrsh

ImSpxrsh commented Jul 9, 2026

Copy link
Copy Markdown
Member

@Gun-Akash, The branch is stale dont merge yet. rebase onto main first & CoT uses k=1 (pass@1) but the MARC main eval uses k=4 (pass@4). N and delta match, but MARC getting 4 candidates vs CoT's 1 isn't same H1 comparison either bump CoT to k=4 (4× gpt-5 token usage) or run MARC at k=1 for this table. Your call, but worth settling before the number goes in the paper.

@ImSpxrsh ImSpxrsh self-requested a review July 9, 2026 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants