fix: count a job-reported error as a failed row

CHANGELOG.md

@@ -11,6 +11,8 @@ All notable changes to `evaluatorq` are documented here.
 - **`sim_model=` is removed from every simulation entry point; `llm_config=` carries the model.** `simulate()`, `generate_and_simulate()`, `generate()`, `generate_personas()` / `generate_scenarios()` (and their singular forms) and `extend_from_experiment()` no longer accept the keyword — pass `llm_config=LLMCallConfig(model=...)`, which says the same thing and can carry `temperature`, `reasoning_effort`, `timeout_ms`, `extra_body` and a client beside it. Two spellings of one setting could not report an explicitly-passed default as a contradiction: `simulate(sim_model=DEFAULT_MODEL, llm_config=LLMCallConfig(model='other'))` ran on `other` and warned about nothing. **Update any call passing `sim_model=`** — there is no deprecation shim. Unaffected: the CLI's `--sim-model` flag, which now builds that config for you, and the `model=` argument on `SimulationRunner`, the generators and the trace helpers, which still folds into a config beside it.
 - **`JudgeAgent` pins itself to the Responses API and logs when a config says otherwise.** `llm_config.api` is one knob for both simulation agents, and only one of them can honour every value: the judge sends function tools and `reasoning_effort` in one request, which chat completions answers with a 400 on models like `gpt-5.4-mini`, while the user simulator's plain completion works on either endpoint. Same model, two roles, two viable protocols. `LLMCallConfig(api='chat_completions')` now applies to the user simulator and is overridden on the judge with a `WARNING` naming it, where it previously reached the judge and broke the run. The pin is `JudgeAgent.REQUIRED_API`; the general default is still `BaseAgent.DEFAULT_API`.
 - **Every simulation agent defaults to the Responses API when its `LLMCallConfig` leaves `api` unset**, overriding that class's own `chat_completions` default. `JudgeAgentConfig` and `UserSimulatorAgentConfig` supplied this before; a caller handing an agent a bare `LLMCallConfig` used to get chat completions instead, which returns a 400 on models like `gpt-5.4-mini` the moment the judge sends function tools and `reasoning_effort` together. The default is `BaseAgent.DEFAULT_API` — a class attribute, not an environment variable, because it is a per-call setting. Set `LLMCallConfig(api='chat_completions')` to opt out.
+- **A job that returns a top-level `error` key now marks the row as failed, instead of counting as a clean success.** `process_job` read only `name` and `output` from a job's return value, so `JobResult.error` was set only when the job *raised*. A job that caught its own failure and reported it — which is what the simulation jobs do, because one dead row must not kill the batch — came back with `error=None`, and a run whose conversation never happened printed `Failed Jobs 0`, `Success Rate 100%` and exited 0. Both simulation jobs — `simulate()`'s own and `wrap_simulation_agent`'s — now emit `error` unconditionally (`None` on success, the runner's reason when the run ended in `error` or `timeout`), and `process_job` honours the key while **keeping** the output, so the transcript survives for diagnosis — its evaluators are skipped, though, because scoring a conversation already known to be dead buys nothing and costs an LLM judge call per row. `wrap_langchain_agent` now catches a failing `invoke`/`ainvoke` and reports it the same way rather than letting it raise; a caller mistake (no prompt and no usable `messages` column) still raises. The job's OTel span is now marked `ERROR` on every failed row, including the two raise paths, which previously closed as `OK`. Presence is the failure signal, not the message's truthiness: a payload that flattens to nothing is reported as `job reported a failure with no readable message` and logged, rather than silently becoming a clean row. `check_pass_failures(results, treat_errors_as_failure=True)` catches these rows as a result — **that flag still defaults to `False`, so a caller who does not pass it gets the same exit code as before**; the change is to the reported counts, not to any exit status. Any other job already returning a top-level `error` key for a non-failure reason will now be counted as failed.
+- **`simulate(exit_on_failure=True)` now raises when a run ended in `error` or `timeout`, not only when a row was dropped.** The check counted rows missing from the result cache, and a run the runner ended in `error` *is* in the cache — so `eq sim` and `simulate()` returned normally over a target that was 401 throughout, which is what the caller asked to exit on. `SimulationDroppedError`'s message now names both counts. `exit_on_failure=False` is unchanged: the same rows are reported as a `WARNING`. Scorer verdicts are still reporting only — an agent that answered badly does not raise.
 - **`LLMCallConfig.temperature` has no default — unset means the parameter is not sent, and the provider applies its own.** It previously defaulted to `1.0`, and evaluatorq's own call sites layered literals of their own on top (`0.8` for persona and first-message generation, `0.9` for edge-case scenarios, `0.7` for the executive summary and chat-completions agent calls, `0.3` for trace analysis, `0.0` for the judge). Reasoning-class models reject the parameter outright rather than clamping it — `gpt-5.6-luna` answers `400 Unsupported parameter: 'temperature' is not supported with this model` — so a hardcoded temperature on the default model turned every persona x scenario pair into a failure and `simulate()` into a `RuntimeError`. No call site sends a temperature now; a caller who wants one sets `LLMCallConfig(temperature=...)`, and a per-call `temperature=None` means unset rather than an explicit null. `BaseAgent._resolved_temperature` lost its `fallback` argument accordingly and now gates on `model_fields_set` like its sibling resolvers, so an explicit `LLMCallConfig(temperature=None)` opts an agent out rather than deferring to a call site. **The judge is the one behaviour change worth planning around: its scoring calls are no longer pinned to `temperature=0.0`, so judgments are no longer reproducible run-to-run by default.** That affects every `simulate()` caller, not only those on a reasoning model. Pass `JudgeAgentConfig(temperature=0.0)` to restore it — on a model that accepts the parameter.
 - **A set-but-empty `ORQ_OTEL_*` tuning variable now logs a `WARNING` and falls back to the default, instead of falling back silently.** `_env_int` treated an empty string like an unset variable, so an unresolved workflow variable in a CI `env:` block expands to the empty string and disabled the knob with no signal. Whitespace-only values are treated the same way. Related: `ORQ_OTEL_MAX_BATCH_SIZE` larger than `ORQ_OTEL_MAX_QUEUE_SIZE` is still clamped down to the queue size, but the clamp now announces itself with a `WARNING` rather than happening silently.
 - **`EVALUATORQ_REASONING_EFFORT` has no default — unset means the parameter is not sent, and the model applies its own.** It previously fell back to `"medium"` for the simulator's own calls (user simulator, judge). A global effort is the wrong default in both directions: on a model that does not accept the parameter it costs a rejected request plus a retry per `(model, tool shape)` — memoised per process, so a short run or CI job never amortises it — and on a model that does, it silently overrides the provider's own tuned value. Set the env var, or `LLMCallConfig.reasoning_effort` on the agent's config, when you actually want a specific effort. **Simulation only**; red teaming's `target_reasoning_effort` was already opt-in.

CLAUDE.md

@@ -190,6 +190,8 @@ Do not add a directory tree, a file inventory, or anything else the filesystem a
 
 The reverse failure is quieter and worse: a job that returns **no** `error` key at all makes `output_error_text` return `None`, so the judge scores the literal `[ERROR: ...]` marker as a genuine agent reply and a dead target comes back RESISTANT. Both static legs now emit the key unconditionally (`None` on success). A new job that calls a target must do the same.
 
+**Two different `error` keys, one word.** The one above is *inside the output payload* of a target or LLM invocation, and it decides what a judge reads. `JobReturn['error']` is at the **top level of a job's return value**, and it decides whether the row counts as failed in `Failed Jobs` and under `check_pass_failures(treat_errors_as_failure=True)`. An invocation lives inside a job, so a job can carry both, one, or neither. The rule above is about the nested one; a job that swallows a failure to keep the batch alive — `simulate()`'s and `wrap_simulation_agent()`'s do — also emits the top-level one, `None` on success. `@job()` nests everything it is handed under `output`, so a decorated job reports failures by raising; that is the contract, not a gap.
+
 ### Adding New Features
 
 - New vulnerabilities: see `docs/custom-evaluators-and-frameworks.md`

docs/evaluation-reference.md

@@ -81,6 +81,24 @@ await evaluatorq(
 )
 ```
 
+### Reporting a failure the job handled
+
+A job that lets a failure raise needs nothing: `process_job` records it and the row counts as failed. A job that *catches* its failure to keep the rest of the batch alive — a target that answered `401`, a simulation the runner ended in `error` — must say so, because a returned dict looks like a clean run:
+
+```python
+async def resilient_job(data: DataPoint, row: int) -> dict:
+    try:
+        answer = await call_my_agent(data.inputs["text"])
+    except MyAgentError as exc:
+        # Keeps the partial output for diagnosis, and still fails the row.
+        return {"name": "my-agent", "output": None, "error": str(exc)}
+    return {"name": "my-agent", "output": answer, "error": None}
+```
+
+Emit `error` on every path, `None` on success: an omitted key and a clean run are indistinguishable, so a job that forgets the key on one branch reports a dead target as a passing one. A row with a non-empty `error` is counted in the summary table's `Failed Jobs` and keeps its output for diagnosis, but its evaluators are **skipped** — scoring a transcript you already know is dead buys nothing and costs an LLM judge call per row. `check_pass_failures(results, treat_errors_as_failure=True)` is what turns it into a CI failure; the default (`False`) gates on evaluator `pass_` alone.
+
+This is the raw-dict job contract. `@job()` wraps a function's return value into `{"name", "output"}`, so an `error` key returned from a decorated function lands *inside* `output`, where a judge reads it as the target's failure rather than the runner reading it as the row's. A decorated job reports a row failure by raising.
+
 ## Data sources
 
 `data` accepts inline `DataPoint`s, an Orq dataset, or awaitables that resolve to `DataPoint`s — the last of which lets you stream rows in from a slow source without blocking the run:
@@ -151,10 +169,12 @@ When any evaluator returns `pass_: False`, `evaluatorq()` returns the results; t
 from evaluatorq.evaluatorq import check_pass_failures
 
 results = await evaluatorq(...)
-if check_pass_failures(results):
+if check_pass_failures(results, treat_errors_as_failure=True):
     raise SystemExit(1)
 ```
 
+`treat_errors_as_failure=True` also gates on rows that errored — a job that raised, a job that reported its own failure, and an evaluator whose every call failed. It defaults to `False`, which gates on evaluator `pass_` alone, so a run whose target was dead throughout can pass a gate that leaves it off.
+
 The results table gains a pass rate row — `Pass Rate | 75% (3/4)`.
 
 ## Controlling the run

docs/guides/agent-simulation.md

@@ -145,9 +145,9 @@ The fastest start: `generate_and_simulate()` synthesizes the personas, scenarios
 `agent_description` drives generation; `num_personas × num_scenarios` is how many conversations run. The simulation-side LLMs resolve their provider by precedence: an explicitly passed `generation_client` wins, then `llm_config.client`, then `ORQ_API_KEY` (the Orq AI Router), then `OPENAI_API_KEY`. See [Configuration](../configuration.md).
 
 !!! note "CI and local runs"
-    Dropped simulations raise by default; ordinary failed goals remain in the returned results. Set `exit_on_failure=False` for exploratory runs. When `ORQ_API_KEY` is available, results upload to Orq by default; pass `upload_results=False` to suppress the Experiment upload. That is not an offline mode — see [What gets uploaded](simulation-in-evaluatorq.md#what-gets-uploaded).
+    A simulation that produced no conversation — dropped, or ended in `error`/`timeout` — raises by default; ordinary failed goals remain in the returned results. Set `exit_on_failure=False` for exploratory runs. When `ORQ_API_KEY` is available, results upload to Orq by default; pass `upload_results=False` to suppress the Experiment upload. That is not an offline mode — see [What gets uploaded](simulation-in-evaluatorq.md#what-gets-uploaded).
 
-    `exit_on_failure` gates on dropped datapoints, not on scores, so it will not fail a build for an agent that simply answered badly. For a gate on the scores themselves — turning evaluator results into an exit code, with the env vars and workflow step to go with it — see [In an evaluatorq Run › In CI](simulation-in-evaluatorq.md#in-ci).
+    `exit_on_failure` gates on datapoints that never produced a conversation, not on scores, so it will not fail a build for an agent that simply answered badly. For a gate on the scores themselves — turning evaluator results into an exit code, with the env vars and workflow step to go with it — see [In an evaluatorq Run › In CI](simulation-in-evaluatorq.md#in-ci).
 
 ## Seed by archetype
 

docs/guides/simulation-in-evaluatorq.md

@@ -129,12 +129,11 @@ async def main() -> None:
             metadata = job_result.output.get("metadata", {})
             print("metadata keys:", sorted(metadata))
             print("criteria:", metadata.get("criteria_results"))
-            # evaluatorq reports a 100% success rate for this run even when the
-            # conversation itself never happened: the job returned a result dict,
-            # so nothing errored from its point of view. Check terminated_by, or
-            # a dead target reads as a passing one.
-            if metadata.get("terminated_by") in {"error", "timeout"}:
-                raise SystemExit(f"simulation did not run: {metadata.get('reason')}")
+            # A run the runner ended in error or timeout lands in job_result.error
+            # and in the table's "Failed Jobs"; this exits on it instead of reading
+            # a partial transcript as a result.
+            if job_result.error:
+                raise SystemExit(f"simulation did not run: {job_result.error}")
 
 
 asyncio.run(main())
@@ -256,7 +255,7 @@ Three things change for a non-interactive run:
 - `EVALUATORQ_DIR` pointed at a scratch directory, so the run store does not persist between jobs and one workflow cannot resolve another's runs.
 - `ORQ_DISABLE_TRACING=1` if you do not want CI spans in your traces.
 
-If you are arriving from `simulate()`, note what does and does not carry over. `simulate()` takes `exit_on_failure`, which defaults to `True` and raises `SimulationDroppedError` when a datapoint is *dropped* — a job raised and no result was cached. That is an infrastructure gate, not a quality gate: scorer verdicts are reporting only there too, so an agent that answered every question badly still exits 0. Moving to `wrap_simulation_agent()` costs you that infrastructure gate, because the parameter belongs to `simulate()` and there is no equivalent on `evaluatorq()`.
+If you are arriving from `simulate()`, note what does and does not carry over. `simulate()` takes `exit_on_failure`, which defaults to `True` and raises `SimulationDroppedError` when a datapoint produced no conversation — *dropped* (a job raised and no result was cached) or ended in `error`/`timeout`. That is an infrastructure gate, not a quality gate: scorer verdicts are reporting only there too, so an agent that answered every question badly still exits 0. Moving to `wrap_simulation_agent()` costs you that infrastructure gate, because the parameter belongs to `simulate()` and there is no equivalent on `evaluatorq()` — gate on `job_result.error` yourself, as the example above does.
 
 `evaluatorq()` does not fail the process for you either — it returns results and exits 0 whatever the scores are. So on this path the build verdict is yours to write, and it is one line:
 

src/evaluatorq/evaluatorq.py

@@ -53,7 +53,10 @@ def check_pass_failures(results: EvaluatorqResult, *, treat_errors_as_failure: b
             (e.g. every judge call raised), also counts as a failure. Without this, an
             errored job has no evaluator scores and an errored evaluator leaves ``pass_``
             unset, so both would be invisible here, letting a run with no usable
-            responses or no usable scores exit successfully.
+            responses or no usable scores exit successfully. Neither errored path is scored:
+            a job that raised has no output to score, and one that reported its own
+            failure through a top-level ``error`` key keeps its output but skips its
+            evaluators — so this flag is the only thing that fails such a row.
 
     Returns:
         True if any evaluator failed (pass_=False), False otherwise

src/evaluatorq/integrations/langchain_integration/wrap_agent.py

@@ -6,6 +6,7 @@
 from typing import TYPE_CHECKING, Any
 
 from langchain_core.tools import BaseTool
+from loguru import logger
 
 from .convert import convert_to_open_responses
 
@@ -131,10 +132,26 @@ async def job(data: DataPoint, _row: int) -> dict[str, Any]:
 
         # Invoke the LangChain agent off the event loop; prefer the native
         # async entry point when the agent exposes one.
-        if hasattr(agent, 'ainvoke'):
-            result = await agent.ainvoke({'messages': messages})
-        else:
-            result = await asyncio.to_thread(agent.invoke, {'messages': messages})
+        #
+        # The agent call is the one step here that talks to a live target, so its
+        # failure is reported through the top-level 'error' key rather than raised:
+        # a raise is per-row invisible to anything that reads JobResult.error only
+        # when the job caught nothing. Everything above this — a missing prompt, an
+        # unusable messages column — is a caller mistake and still raises.
+        try:
+            if hasattr(agent, 'ainvoke'):
+                result = await agent.ainvoke({'messages': messages})
+            else:
+                result = await asyncio.to_thread(agent.invoke, {'messages': messages})
+        except Exception as agent_error:  # noqa: BLE001 - a target's failure is reported, not raised
+            logger.warning('langchain agent invocation failed: {}: {}', type(agent_error).__name__, agent_error)
+            # No transcript to keep — the call never returned one. The row carries the
+            # reason and no output, which is what an evaluator scoring it will see.
+            return {
+                'name': name,
+                'output': None,
+                'error': f'{type(agent_error).__name__}: {agent_error}',
+            }
 
         # Extract messages from result
         result_messages: list[BaseMessage] = result.get('messages', [])
@@ -148,6 +165,10 @@ async def job(data: DataPoint, _row: int) -> dict[str, Any]:
         return {
             'name': name,
             'output': open_responses_output,
+            # Emitted unconditionally, None on success: a job that omits the key on a
+            # good row and sets it on a bad one is indistinguishable from one that
+            # never reports failures at all.
+            'error': None,
         }
 
     return job