Warn when a twin is validated on a superseded model

[johnjosephhorton]

Why

The research-agent run we've been dissecting validated its twin on openai:gpt-4.1 while the frontier was gpt-5.5. Its own plan said "gpt-5.5" — then it free-typed gpt-4.1 into --model and nothing flagged it. A weak model understates twin capability and makes the whole validation uninformative (it looked like "twins barely beat random" when the real story may be "we used a weak model"). The choice was invisible in the output.

What

zwill already defaults to gpt-5.5 when --model is omitted (parse_model_specs), so it never silently falls to EDSL's gpt-4o default. The only real gap is an explicitly-chosen weak model. For the validation-critical exports (twin-probability-job, rank-utility-twin-job, numeric-twin-job), edsl-export now:

• names the model(s) the twin will run on in its output envelope (data.models) — the choice is reviewable at the decision point, before spending anything;
• emits a loud superseded_twin_model warning when a chosen model matches a superseded family (gpt-4*, gpt-3.5, claude-3/2, gemini-1.x, legacy completions), pointing at a current frontier model and noting the gpt-5.5 default.

Advisory, never blocking — ignore the warning to benchmark an older model deliberately.

Behavior (live-verified)

--model openai:gpt-4.1   -> data.models=[openai:gpt-4.1] + superseded_twin_model warning
--model openai:gpt-5.5   -> data.models=[openai:gpt-5.5], no warning
(no --model)             -> data.models=[gpt-5.5],        no warning

Testing

• pytest — 252 passed (adds tests/test_twin_model_guard.py: superseded detection, warning fires only for weak models, the default model isn't itself flagged, target set).
• ruff clean.
• Documented the rule in the agent-workflow guide.

Note: the SUPERSEDED_TWIN_MODEL_MARKERS list is a deny-list of clearly-older families and should be kept current as models evolve.

🤖 Generated with Claude Code

[greptile-apps]

Greptile Summary

This PR adds a guardrail to edsl-export for the three twin-validation targets (twin-probability-job, rank-utility-twin-job, numeric-twin-job): it surfaces the chosen model(s) in the output envelope and emits a superseded_twin_model warning when the model matches a known-older family. The motivation is a real incident where a research-agent run validated on gpt-4.1 while the frontier was gpt-5.5, silently understating twin capability.

• Extracts DEFAULT_EXPORT_MODEL = \"gpt-5.5\" as a named constant, used in parse_model_specs and cited in the warning text.
• Adds SUPERSEDED_TWIN_MODEL_MARKERS (deny-list), superseded_twin_model_labels, and superseded_twin_model_warning — all advisory and never blocking.
• New test file covers: superseded-model detection, warning firing only for weak models, the default model is not itself flagged, and the target set is stable.

Confidence Score: 4/5

Safe to merge; the warning logic is advisory and never blocking, so the worst case for any false positive or future marker drift is an unwanted console warning, not broken exports or incorrect results.

The core detection and warning path is correct and thoroughly tested. The only nit is that model_count and the new models list both land in the same output envelope for twin-validation targets, computed from different sources; a future builder that populates its EDSL model list differently could cause them to disagree silently. Everything else is clean.

zwill/edsl_integration.py — specifically the relationship between model_count (from the serialised EDSL job dict) and the new models list (from a second parse_model_specs call) in export_data.

Important Files Changed

Filename	Overview
zwill/edsl_integration.py	Adds DEFAULT_EXPORT_MODEL constant, SUPERSEDED_TWIN_MODEL_MARKERS deny-list, two helper functions, and emits model labels + optional warning for twin-validation targets in cmd_edsl_export; minor redundancy between model_count and the new models list in the output envelope.
tests/test_twin_model_guard.py	New test file covering the four key behaviors of the guard; deterministic and self-contained, no issues.
zwill/guides/agent-workflow.md	Adds documentation guidance to validate twins on a current frontier model and explains the superseded_twin_model warning; no issues.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["cmd_edsl_export(args)"] --> B{args.target in\nTWIN_VALIDATION_TARGETS?}
    B -- No --> C[Build export_dict\nno model labeling]
    B -- Yes --> D["parse_model_specs(args)\n→ specs"]
    D --> E["model_labels = [model_label(...)\n for each spec]"]
    E --> F["superseded_twin_model_warning(specs)"]
    F --> G{Any spec matches\nSUPERSEDED_TWIN_MODEL_MARKERS?}
    G -- No --> H[No warning]
    G -- Yes --> I["append superseded_twin_model\nwarning to warnings[]"]
    H --> J["export_data includes\nmodels: model_labels"]
    I --> J
    C --> K["emit_raw_export(…, export_data, warnings)"]
    J --> K

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A["cmd_edsl_export(args)"] --> B{args.target in\nTWIN_VALIDATION_TARGETS?}
    B -- No --> C[Build export_dict\nno model labeling]
    B -- Yes --> D["parse_model_specs(args)\n→ specs"]
    D --> E["model_labels = [model_label(...)\n for each spec]"]
    E --> F["superseded_twin_model_warning(specs)"]
    F --> G{Any spec matches\nSUPERSEDED_TWIN_MODEL_MARKERS?}
    G -- No --> H[No warning]
    G -- Yes --> I["append superseded_twin_model\nwarning to warnings[]"]
    H --> J["export_data includes\nmodels: model_labels"]
    I --> J
    C --> K["emit_raw_export(…, export_data, warnings)"]
    J --> K

Loading

_{Reviews (1): Last reviewed commit: "Warn when a twin is validated on a super..." | Re-trigger Greptile}

[greptile-apps]

model_count and models can coexist with potentially different counts

For twin-validation targets, export_data gets both model_count (derived from export_dict.get("models", []), i.e. the serialised EDSL Jobs dict) and models (derived from a fresh parse_model_specs(args) call). In practice these agree because every builder calls parse_model_specs(args) from the same args, but if a future builder sources its model list differently the two fields would diverge silently. Consider populating model_count from len(model_labels) when model_labels is available, so the envelope is internally consistent by construction.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

Three blank lines between superseded_twin_model_warning and build_edsl_agent_list_dict; PEP 8 uses two.

Suggested change

	}
	def build_edsl_agent_list_dict(survey_name: str, args: argparse.Namespace) -> dict[str, Any]:
	}
	def build_edsl_agent_list_dict(survey_name: str, args: argparse.Namespace) -> dict[str, Any]:

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!