fix(tracing): enforce the OTEL flush timeout, and document ORQ_OTEL_* batching

.claude/skills/docs-autofill/ledger.md

@@ -7,3 +7,4 @@ Outcomes: `prepared` (work committed, PR not yet created) · `opened` · `blocke
 | date | matrix cell | branch | outcome |
 |---|---|---|---|
 | 2026-08-23 | `entry point (red_team()) × data source (replay)` | docs/autofill-redteam-replay | opened |
+| 2026-08-24 | `tier 1: env var (ORQ_OTEL_MAX_QUEUE_SIZE, ORQ_OTEL_MAX_BATCH_SIZE, ORQ_OTEL_SCHEDULE_DELAY_MS, ORQ_OTEL_FLUSH_TIMEOUT_MS)` | docs/autofill-otel-batching | opened |

.claude/skills/docs-coverage/axes.md

@@ -79,4 +79,6 @@ Marked `N/A` in the matrix, never reported as a gap.
 
 **Tier 1 — needs prose.** Top-level `evaluatorq.__all__` entry points, every CLI command and subcommand, every env var. The generated API reference does **not** count: a docstring is not discovery.
 
+Tier-1 items are **not** matrix cells — there is no `env var` axis, and `surface` has exactly the three values above. Record a Tier-1 gap in the `docs-autofill` ledger as `tier 1: <kind> (<name>, …)` with no second axis, e.g. `tier 1: env var (ORQ_OTEL_MAX_QUEUE_SIZE, ORQ_OTEL_MAX_BATCH_SIZE)`. The ledger dedupe compares that cell as free text, so naming the items is what stops the same gap being re-derived next week; an invented axis pair never matches.
+
 **Tier 2 — API reference suffices.** Supporting types, contracts, backends, and subpackage `__all__` members. Flag only when there is no docstring at all.

CHANGELOG.md

@@ -9,6 +9,7 @@ All notable changes to `evaluatorq` are documented here.
 ### Notable defaults
 
 - **`LLMCallConfig.temperature` has no default — unset means the parameter is not sent, and the provider applies its own.** It previously defaulted to `1.0`, and evaluatorq's own call sites layered literals of their own on top (`0.8` for persona and first-message generation, `0.9` for edge-case scenarios, `0.7` for the executive summary and chat-completions agent calls, `0.3` for trace analysis, `0.0` for the judge). Reasoning-class models reject the parameter outright rather than clamping it — `gpt-5.6-luna` answers `400 Unsupported parameter: 'temperature' is not supported with this model` — so a hardcoded temperature on the default model turned every persona x scenario pair into a failure and `simulate()` into a `RuntimeError`. No call site sends a temperature now; a caller who wants one sets `LLMCallConfig(temperature=...)`, and a per-call `temperature=None` means unset rather than an explicit null. `BaseAgent._resolved_temperature` lost its `fallback` argument accordingly and now gates on `model_fields_set` like its sibling resolvers, so an explicit `LLMCallConfig(temperature=None)` opts an agent out rather than deferring to a call site. **The judge is the one behaviour change worth planning around: its scoring calls are no longer pinned to `temperature=0.0`, so judgments are no longer reproducible run-to-run by default.** That affects every `simulate()` caller, not only those on a reasoning model. Pass `JudgeAgentConfig(temperature=0.0)` to restore it — on a model that accepts the parameter.
+- **A set-but-empty `ORQ_OTEL_*` tuning variable now logs a `WARNING` and falls back to the default, instead of falling back silently.** `_env_int` treated an empty string like an unset variable, so an unresolved workflow variable in a CI `env:` block expands to the empty string and disabled the knob with no signal. Whitespace-only values are treated the same way. Related: `ORQ_OTEL_MAX_BATCH_SIZE` larger than `ORQ_OTEL_MAX_QUEUE_SIZE` is still clamped down to the queue size, but the clamp now announces itself with a `WARNING` rather than happening silently.
 - **`EVALUATORQ_REASONING_EFFORT` has no default — unset means the parameter is not sent, and the model applies its own.** It previously fell back to `"medium"` for the simulator's own calls (user simulator, judge). A global effort is the wrong default in both directions: on a model that does not accept the parameter it costs a rejected request plus a retry per `(model, tool shape)` — memoised per process, so a short run or CI job never amortises it — and on a model that does, it silently overrides the provider's own tuned value. Set the env var, or `LLMCallConfig.reasoning_effort` on the agent's config, when you actually want a specific effort. **Simulation only**; red teaming's `target_reasoning_effort` was already opt-in.
 - **`OrqResponsesTarget.retry_attempts` now defaults to `1` — a single attempt, no
   retry — down from falling through to `with_retry`'s default of `5`.**

docs/configuration.md

@@ -22,6 +22,10 @@ All configuration is via environment variables. No config file is required.
 | `OTEL_EXPORTER_OTLP_HEADERS` | No | — | Comma-separated `key=value` pairs added to every OTLP export request. Format: `key1=value1,key2=value2`. |
 | `OTEL_SERVICE_NAME` | No | `evaluatorq` | Service name recorded on every span's `service.name` resource attribute. |
 | `OTEL_SERVICE_VERSION` | No | `1.0.0` | Service version recorded on every span's `service.version` resource attribute. |
+| `ORQ_OTEL_MAX_QUEUE_SIZE` | No | `4096` | Maximum spans buffered by the `BatchSpanProcessor`. When it is full the SDK evicts the oldest buffered span and logs `Queue full, dropping Span.` on the stdlib `opentelemetry.sdk._shared_internal` logger. See [Tracing › Batching and flush](tracing.md#batching-and-flush). |
+| `ORQ_OTEL_MAX_BATCH_SIZE` | No | `512` | Maximum spans per OTLP export request. Reaching it wakes the exporter immediately. A value larger than the queue size is clamped to the queue size, with a warning. |
+| `ORQ_OTEL_SCHEDULE_DELAY_MS` | No | `5000` | Milliseconds a partial batch waits before it is exported. It does not throttle a full batch. |
+| `ORQ_OTEL_FLUSH_TIMEOUT_MS` | No | `5000` | Milliseconds the end-of-run force-flush waits before giving up and logging a warning. Read per run; enforced by evaluatorq rather than the SDK. Bounds the final flush only — the per-request export timeout is a fixed 5s. |
 | `EVALUATORQ_CAPTURE_MESSAGE_CONTENT` | No | `true` | Set to `false` or `0` to strip LLM message content (prompts and responses) from spans. Token counts, model name, and latency are still recorded. Useful when exporting to third-party backends or to avoid capturing PII. |
 | `EVALUATORQ_SPAN_MAX_TEXT_CHARS` | No | unset (no limit) | Maximum characters per span text attribute. Set a positive integer (e.g. `8192`) to truncate long strings. Unset or `0` / `-1` means capture all. |
 | `EVALUATORQ_LLM_TIMEOUT_S` | No | `60.0` | Per-LLM-call timeout in seconds. **Simulation only** — has no effect on red teaming or core evaluation. A fallback default: `LLMCallConfig.timeout_ms` on the agent's config wins when set. Read at call time, so setting it after import takes effect. Increase for slow self-hosted endpoints; for the *target's* timeout rather than the simulator's, pass `target_agent_timeout_ms` to `simulate()`. |
@@ -33,15 +37,9 @@ All configuration is via environment variables. No config file is required.
 
 ## Model catalogue overrides
 
-Prices, provider ids, Responses support and accepted reasoning-effort values come
-from Orq's `GET /v2/models`, fetched once per process. A model that catalogue does
-not list — a self-hosted deployment, or one newer than your workspace's catalogue —
-degrades silently in three ways: the call stays unpriced, `qualified_model()` sends
-it to Chat Completions instead of Responses, and its reasoning effort cannot be
-pre-validated.
+Prices, provider ids, Responses support and accepted reasoning-effort values come from Orq's `GET /v2/models`, fetched once per process. A model that catalogue does not list — a self-hosted deployment, or one newer than your workspace's catalogue — degrades silently in three ways: the call stays unpriced, `qualified_model()` sends it to Chat Completions instead of Responses, and its reasoning effort cannot be pre-validated.
 
-Register an entry to fix that. Registered entries take priority over the fetched
-catalogue, so this also corrects an entry that is wrong:
+Register an entry to fix that. Registered entries take priority over the fetched catalogue, so this also corrects an entry that is wrong:
 
 ```python
 from evaluatorq.common.model_catalogue import ModelInfo, get_model_info, register_model
@@ -62,12 +60,7 @@ info = await get_model_info('my-self-hosted-llama')
 
 Costs are USD **per 1000 tokens**, matching what `/v2/models` publishes.
 
-The id is stored unprefixed, so `'openai/gpt-x'` and `'gpt-x'` register and resolve
-the same entry — register either spelling and both lookups find it. Registering
-both replaces rather than duplicates: there is one model. `reasoning_efforts=None`
-means "unknown, cannot pre-validate"; an empty set means the same thing and is
-normalized to `None`, because a literally-empty accepted-values list would reject
-every effort including the defaults.
+The id is stored unprefixed, so `'openai/gpt-x'` and `'gpt-x'` register and resolve the same entry — register either spelling and both lookups find it. Registering both replaces rather than duplicates: there is one model. `reasoning_efforts=None` means "unknown, cannot pre-validate"; an empty set means the same thing and is normalized to `None`, because a literally-empty accepted-values list would reject every effort including the defaults.
 
 ## `.env` file
 

docs/faq.md

@@ -72,12 +72,16 @@ Simulator/attacker/judge LLM calls go to OpenAI or the Orq router. Results uploa
 
 ### How much does a run cost, and how do I keep it cheap?
 
-Cost and wall-clock scale with cases × turns × LLM calls. The levers are how many cases you run (`max_dynamic_datapoints` / `max_static_datapoints` for red teaming, `num_personas` × `num_scenarios` for simulation), `max_turns`, and `datapoint_parallelism` (default 10 everywhere). To size against a provider concurrency limit, set `llm_parallelism=` (on `evaluatorq()`, `red_team()` or `simulate()`) rather than lowering `datapoint_parallelism` — it counts requests instead of tasks, so the number means the same thing however the fan-out nests. Red teaming's report tracks spend in `report.summary.token_usage_total`.
+Cost and wall-clock scale with cases × turns × LLM calls. The levers are how many cases you run (`max_dynamic_datapoints` / `max_static_datapoints` for red teaming, `num_personas` × `num_scenarios` for simulation), `max_turns`, and `datapoint_parallelism` (default 10 on `evaluatorq()`, `red_team()` and `simulate()`; 5 in the adaptive red-team pipeline). To size against a provider concurrency limit, set `llm_parallelism=` (on `evaluatorq()`, `red_team()` or `simulate()`) rather than lowering `datapoint_parallelism` — it counts requests instead of tasks, so the number means the same thing however the fan-out nests. Red teaming's report tracks spend in `report.summary.token_usage_total`.
 
 ### Where do results go, and how do I view a past run?
 
 Runs auto-save locally (red-team runs to `.evaluatorq/runs/`; simulation runs to `.evaluatorq/sim-runs/`). Browse them in the multi-run FastHTML dashboard with `eq dashboard` (no path browses both stores; `eq dashboard .evaluatorq/sim-runs` scopes to simulation), or list runs with `eq redteam runs` / `eq sim runs`. See [Dashboard](dashboard.md).
 
+### Some spans are missing from my traces
+
+The span exporter batches in the background, so spans can be lost two ways, neither of which fails the run. Either the in-memory queue overflowed — spans produced faster than the exporter drained them — or the process exited before the final flush finished. Raise `ORQ_OTEL_MAX_QUEUE_SIZE` for the first and `ORQ_OTEL_FLUSH_TIMEOUT_MS` for the second. Both log a warning; a hard `SIGKILL` drops whatever was still buffered without one. See [Tracing › Batching and flush](tracing.md#batching-and-flush).
+
 ### How do I run a plain evaluation?
 
 Decorate a function with `@job`, hand `evaluatorq()` your data and evaluators, and it runs the jobs in parallel and scores each row: