repeated-run stability analysis, token tracking, resumable experiments, and auditor model separation

[finnschwall]

Overview

This branch contains the code used to generate the experimental data for the "When No Benchmark Exists" paper.

---

Major Contributions

🪙 Token Usage Tracking

• _call_async now returns (content, input_tokens, output_tokens); token counts are propagated through probe generation, judging, and target calls in run_scenario
• AuditResult gains six token count fields (auditor/judge/target × input/output)
• AuditResults gains a token_usage property (per-role and total) and prints a Token Usage table at the bottom of .summary()
• Token usage is included in the serialized JSON output

🔁 Repeated Experiments & Stability Analysis

• AuditExperiment gains n_repetitions (default 1) to run each model N times and measure judge non-determinism
• New RepeatedExperimentResults class wraps all runs with a backward-compatible dict interface (old code iterating over results continues to work)
• .stability(model_name) returns a ModelStabilityReport with mean score, std, CV%, min/max, and per-scenario pass rates and agreement rates
• .summary() prints stability reports for all models
• Full JSON serialization via .save() / .load()
• save_dir parameter: each run is saved to disk immediately after completion ({save_dir}/{label}/run_{i}.json). On re-run, completed runs are loaded from disk and skipped — crash recovery with no repeated work
• ModelStabilityReport and RepeatedExperimentResults exported from the top-level package

🤖 Separate Auditor Model

• ModelAuditor and AuditExperiment now accept auditor_model, auditor_provider, auditor_api_key, auditor_base_url — a third LLM role for probe/attack generation, separate from the judge
• Falls back to judge config when not specified; reuses the same client object when configs are identical

---

Minor Changes

• AuditExperiment validates for duplicate model labels at construction time and raises a clear ValueError
• Mode description now correctly says "Sequential" when max_workers == 1
• examples/fake_audit.py added: a self-contained runnable demo using fake LLM clients, no API keys required. Supports --repeat N to exercise the stability analysis path
• README updated with documentation for all new parameters and a new Stability Analysis section with code examples

---

Bug Fixes

• mock_server.py was broken on main: the /v1/chat/completions endpoint returned no usage field (required by the token tracking update), and had no way to simulate judge responses. Fixed by making the endpoint async, adding a usage field, and adding a separate judge response path when model == "judge". Also adds --min-delay / --max-delay CLI flags to simulate realistic latency
• quickstart.ipynb was broken on main: it contained an inline copy of the mock server that was out of date and used a deprecated ModelAuditor instantiation style. Fixed by removing the duplicated server code, updating to the current ModelAuditor API, and switching demo cells from auditor.run() to await auditor.run_async()

---

Tests

• Seven new test files added: test_fakes.py, test_token_counting.py, test_repeated_experiments.py, test_resumable_experiments.py, test_json_format.py, test_judge_registry.py, and a shared fakes.py test harness
• Existing test files ported to use fake LLM clients
• Two obsolete test files removed: test_expected_behavior.py, test_local_providers.py

✅ All tests pass. Changes have been validated end-to-end using the mock server and the fake client harness.
⚠️ No validation has been done against live LLM API runs.

[Copilot]

Pull request overview

This PR expands SimpleAudit’s experiment and auditing pipeline to support (1) per-role token usage tracking, (2) repeated/resumable experiments with stability analysis across runs, and (3) separating the “auditor” (probe generator) model from the judge model, with accompanying documentation and test infrastructure.

Changes:

• Add token usage propagation through ModelAuditor into AuditResult / AuditResults, including summary display and JSON serialization.
• Introduce n_repetitions, save_dir caching/resume, and RepeatedExperimentResults + ModelStabilityReport for repeated-run stability statistics and serialization.
• Add a comprehensive fake-client test harness and expand/modernize the test suite and examples (mock server + notebook + offline demo).

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
simpleaudit/model_auditor.py	Adds auditor-model separation, token-returning _call_async, and token accounting through scenario runs.
simpleaudit/results.py	Extends result objects with token fields and adds aggregated token usage reporting/serialization.
simpleaudit/experiment.py	Adds repeated runs, disk caching via save_dir, and returns RepeatedExperimentResults.
simpleaudit/repeated_results.py	New repeated-run container + stability stats + JSON save/load.
simpleaudit/__init__.py	Exports RepeatedExperimentResults and ModelStabilityReport.
tests/fakes.py	New fake AnyLLM-compatible client utilities and make_auditor() helper for tests.
tests/test_fakes.py	Sanity tests for the fake client harness.
tests/test_token_counting.py	Tests for per-role token tracking and aggregation + end-to-end fake-client path.
tests/test_json_format.py	Tests that json_format is stored and forwarded into judge calls.
tests/test_repeated_experiments.py	Tests repeated-run behavior, stability calculations, and save/load round-trips.
tests/test_resumable_experiments.py	Tests save_dir run caching and resume/skip behavior.
tests/test_judge_registry.py	Tests judge registry/listing and named-judge prompt resolution.
tests/test_strip_thinking.py	Updates for _call_async returning (content, in_tokens, out_tokens) and adds history semantics tests.
tests/test_custom_prompts.py	Updates mocks to new _call_async return type and expands expected-behavior edge case testing.
tests/test_audit_flow.py	Migrates to fake clients and adds coverage for system prompt forwarding, language, max_workers, etc.
tests/test_basic.py	Adds severity distribution test and makes API-key error expectation explicit.
tests/test_model_auditor.py	Adds tests for separate auditor client vs reusing judge client.
tests/test_scenario_data.py	Adds assertions for BullshitBench test_prompt presence/shape.
tests/test_local_providers.py	Removed obsolete/skip-only local provider tests.
tests/test_expected_behavior.py	Removed obsolete minimal unittest-based coverage.
README.md	Documents new auditor-model params, repeated runs/stability analysis, and updated experiment usage.
examples/mock_server.py	Fixes mock server to return usage, support judge behavior, and simulate latency.
examples/quickstart.ipynb	Removes duplicated server code and updates to async API + new mock server behavior.
examples/fake_audit.py	New offline demo script using fake clients; supports repeated runs for stability reporting.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[finnschwall]

dont see this as a real issue. no overhead or unintended behavior is introduced

[finnschwall]

dont see as relevant as this feature is only for SA contributors that clone the repo anyways

[SushantGautam]

@copilot resolve the merge conflicts in this pull request