Make Responses API fallback / web_search degradation observable and controllable

[tangym]

Summary

PR #146 added two safety nets for regions / API versions where the Azure Responses API is unavailable (e.g. West Europe today):

1. Auto-fallback from /responses to /chat/completions.
2. Graceful degradation of the web_search tool (which is Responses-API-only) when fallback is active — the call is rewritten without web_search so the run keeps going instead of crashing.

That keeps interactive runs working out of the box, but leaves two gaps:

• Invisible to the user after the fact. The only signal that web_search was dropped is a single WARN log line, gated by a module-level _web_search_drop_warned flag so it appears at most once per process. There is nothing in metrics.json, nothing in the viewer, and nothing in the run summary. Re-opening a run a week later, or sharing artifacts with a teammate, you can't tell whether web_search was actually used.
• No opt-in fail-fast. CI / reproducibility pipelines may explicitly need web grounding. Today they have no way to say "if the framework would silently drop web_search, fail loudly instead." A green CI run can hide a degraded eval.

This issue tracks both gaps as one workstream. It supersedes #147 (observability) and #148 (strict mode), which were filed separately and are being consolidated.

Current behavior (as of PR #146)

Relevant code in assert_eval/core/model_client.py:

• _supports_web_search_preview(model) — gates whether web_search_preview tool is added.
• _drop_web_search_for_fallback(options, model, *, reason) — strips web_search/web_search_preview from request options, logs a one-time WARN via _web_search_drop_warned, returns the cleaned options.
• generate() and generate_structured() call _drop_web_search_for_fallback in two places each:
  • Proactive: at entry when _force_chat_completions is already set (because a prior call in the same process already discovered the Responses API is unavailable, or because ASSERT_PREFER_CHAT_COMPLETIONS=1).
  • Reactive: in the _with_retries catch for _ResponsesApiNotAvailableError (first call in the process that discovers the endpoint doesn't support Responses).

Both stages that hardcode web_search=True rely on this:

• assert_eval/stages/systematize.py:196 (taxonomy research)
• assert_eval/stages/stratification.py:339 (policy stratification)

Proposed behavior

A. Persist degradation events (observability)

When _drop_web_search_for_fallback fires for the first time in a run, record it in the run's metrics.json:

{
  …,
  "degradations": [
    {
      "kind": "web_search_disabled",
      "reason": "responses_api_unavailable",
      "stages_affected": ["systematize", "stratification"],
      "first_seen_at": "2026-05-30T12:34:56Z"
    },
    {
      "kind": "responses_api_fallback",
      "reason": "api_version_not_supported",
      "first_seen_at": "2026-05-30T12:34:56Z"
    }
  ]
}

stages_affected can be populated by checking which stages have already run (or by accumulating stage names from the caller context — TBD during implementation).

The viewer (viewer/) shows a banner on the run summary page:

⚠️ This run degraded: web_search was disabled because the Responses API is unavailable on this endpoint. Web grounding results in the systematize and stratification stages were not used.

The banner should be dismissible per-run but persistent on reload.

B. Strict mode (controllability)

Add an opt-in toggle that converts graceful degradation into a hard failure:

• CLI flag: assert-eval run --strict-web-search (preferred surface for CI)
• Env var: ASSERT_STRICT_WEB_SEARCH=1 (for CI systems that don't easily inject flags)

When strict mode is enabled and _drop_web_search_for_fallback would otherwise fire, raise an error with an actionable message:

ERROR  web_search requested but the Responses API is unavailable on this endpoint.
       Suggestions:
         - Use an Azure region that supports the Responses API
           (e.g. East US 2, Sweden Central)
         - Set web_search: false in eval_config.yaml for stages that don't
           need web grounding
         - Set ASSERT_PREFER_CHAT_COMPLETIONS=1 to skip Responses API probing
           (note: this still disables web_search; only use if you've already
           accepted that)

The exception type should be something distinct (e.g. StrictWebSearchUnavailableError) so users can catch it explicitly if they wrap assert-eval in their own scripts.

Strict mode is off by default — graceful degradation remains the right default for interactive / exploratory use.

Implementation notes

• Both A and B hook into the same code path (_drop_web_search_for_fallback in model_client.py), which is why they're now one issue.
• A clean order is A first (so even strict-mode failures are auditable in metrics.json), then B (which gates whether A's "degradation happened" record blocks the run).
• Tests live in tests/test_model_client.py (WebSearchFallbackDegradationTest). Both A and B should extend this class:
  • A: assert metrics.json contains a degradations entry after a run that triggered degradation.
  • B: assert that with ASSERT_STRICT_WEB_SEARCH=1 set, both the proactive and reactive drop paths raise instead of degrading.
• Viewer changes can ship in a separate PR since viewer/ has its own dev loop.

Why this matters

• Reproducibility audits: someone re-running an eval in a different region needs to know whether the previous run had web grounding or not.
• Comparability: comparing two runs without knowing one had web_search and the other didn't is misleading.
• CI signal integrity: a green CI run should mean the eval actually ran the way the config says it would.
• Customer trust: silent degradation is OK for interactive exploration; it's not OK when you're shipping evals on someone else's behalf.

Out of scope

• Per-test-case accounting (which specific test cases ran without web_search). Could be a later enhancement once stage-level tracking lands.
• Auto-discovery of which Azure region supports Responses API (currently hardcoded suggestions in the error message).
• Persisting degradations from other sources (rate limiting, content filter, etc.). The schema in (A) is designed to be extensible, but only web_search_disabled and responses_api_fallback need to be populated initially.

Related

• PR Auto-fallback from Azure Responses API to Chat Completions for regions without Responses support #146 — adds the underlying fallback + graceful degradation that this issue makes observable and controllable.
• Surface fallback/degradation events in metrics.json and viewer #147 (superseded by this issue) — original observability proposal.
• Strict mode for hard-fail on web_search/Responses API unavailability #148 (superseded by this issue) — original strict-mode proposal.

[github-actions]

This issue has had no activity for 90 days and is being marked as stale. It will be closed in 14 days unless updated. Reopen anytime if still relevant.