From 11864cf08d9aa599b5a767f5dc4e8173b9854af7 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 24 Aug 2026 06:24:08 +0000 Subject: [PATCH 1/9] docs: document ORQ_OTEL_* span export batching env vars Add a 'Batching and flush' section to the Tracing page covering the four BatchSpanProcessor tuning env vars (ORQ_OTEL_MAX_QUEUE_SIZE, ORQ_OTEL_MAX_BATCH_SIZE, ORQ_OTEL_SCHEDULE_DELAY_MS, ORQ_OTEL_FLUSH_TIMEOUT_MS), previously documented only in the tracing/setup.py docstring. Explains queue-overflow and exit-before-flush span loss and gives a runnable CI configuration. --- docs/tracing.md | 81 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) diff --git a/docs/tracing.md b/docs/tracing.md index 3f1e26b5..40460611 100644 --- a/docs/tracing.md +++ b/docs/tracing.md @@ -93,6 +93,87 @@ errors to stdout. - **Custom headers**: parsed from `OTEL_EXPORTER_OTLP_HEADERS` as `key1=value1,key2=value2`. +### Batching and flush + +Spans are exported asynchronously by a `BatchSpanProcessor`: each finished span +lands in an in-memory queue, and a background thread drains the queue in batches +on a fixed schedule. The `TracerProvider` lives for the whole process — a +long-lived host (the dashboard, a worker that runs many red-team or simulation +runs back to back) never tears it down between runs, so the same queue absorbs +every run's spans. + +Two failure modes follow from that, and both drop spans **silently** unless you +tune the processor: + +- **Queue overflow.** If spans are produced faster than the exporter drains them — + a burst of parallel jobs, or a slow/unreachable OTLP endpoint — the queue fills + and the SDK discards the overflow. Nothing is raised; the trace just arrives + incomplete. +- **Exit before flush.** At the end of each run evaluatorq force-flushes the + processor, and the SDK's `atexit` hook flushes again on a clean shutdown. But a + force-flush that exceeds its timeout, or a hard `SIGKILL` (an OOM kill, a CI job + cancelled mid-run), leaves whatever is still buffered unexported. A flush that + times out logs a `WARNING` — *"OTEL span flush timed out … some spans may not + have been exported"* — rather than failing the run. + +Four environment variables tune this. All are read once, when tracing +initializes: + +| Variable | Default | What it controls | +|---|---|---| +| `ORQ_OTEL_MAX_QUEUE_SIZE` | `4096` | Maximum spans buffered before the processor drops the overflow. Raise it for a long-lived host that batches many runs. | +| `ORQ_OTEL_MAX_BATCH_SIZE` | `512` | Maximum spans per export request. Capped to the queue size, so a batch size larger than the queue is silently clamped down. | +| `ORQ_OTEL_SCHEDULE_DELAY_MS` | `5000` | Delay between scheduled batch exports. Lower it to export more eagerly (fewer spans sitting in the queue at any moment); raise it to send larger, less frequent batches. | +| `ORQ_OTEL_FLUSH_TIMEOUT_MS` | `5000` | How long the per-run force-flush blocks waiting for the exporter before giving up and logging the timeout warning. | + +Each value must be a positive integer. A set-but-invalid value (non-numeric, zero +or negative) is ignored with a `WARNING` and the default is used, so a typo never +takes tracing down — but it also never silently takes effect. + +A CI or batch run that must not lose spans typically drains the queue more +eagerly and allows a longer final flush: + +```bash +export ORQ_API_KEY="your_orq_api_key" # enables tracing +export ORQ_OTEL_MAX_QUEUE_SIZE=16384 # absorb a burst of parallel jobs +export ORQ_OTEL_SCHEDULE_DELAY_MS=1000 # export every second, not every five +export ORQ_OTEL_FLUSH_TIMEOUT_MS=30000 # give the final flush up to 30s +``` + +With those set, a normal run needs no code changes — evaluatorq flushes at the +end of each run, and the knobs above decide how much headroom that flush has: + +```python +import asyncio + +from evaluatorq import DataPoint, evaluatorq, job, string_contains_evaluator + + +@job("echo") +async def echo(data: DataPoint, _row: int) -> str: + return str(data.inputs["question"]) + + +async def main() -> None: + data = [DataPoint(inputs={"question": "30 days"}, expected_output="30 days")] + # Spans for this run are queued, batched on the schedule above, and + # force-flushed when evaluatorq() returns — within ORQ_OTEL_FLUSH_TIMEOUT_MS. + await evaluatorq( + "tracing-batching-demo", + data=data, + jobs=[echo], + evaluators=[string_contains_evaluator()], + ) + + +if __name__ == "__main__": + asyncio.run(main()) +``` + +Set `ORQ_DEBUG=1` to print the resolved endpoint and initialization diagnostics, +which is the quickest way to confirm the exporter is configured the way you +expect before a long run. + ## Span hierarchy ### Evaluation runner spans From feb073f283220031ee4141d1ac3b0b5d00c5a277 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 24 Aug 2026 06:34:37 +0000 Subject: [PATCH 2/9] docs: refine ORQ_OTEL_* batching section and add canonical table rows Apply review feedback: correct the per-run vs init read timing of ORQ_OTEL_FLUSH_TIMEOUT_MS, quote the flush warning verbatim, drop duplicated prose (BatchSpanProcessor restatement, ORQ_DEBUG tip, second example), add silent-overflow detection guidance, and list the four vars in configuration.md's canonical env-var table with cross-references. Records the docs-autofill ledger row for this attempt. --- .claude/skills/docs-autofill/ledger.md | 1 + docs/configuration.md | 4 + docs/tracing.md | 105 ++++++++++--------------- 3 files changed, 46 insertions(+), 64 deletions(-) diff --git a/.claude/skills/docs-autofill/ledger.md b/.claude/skills/docs-autofill/ledger.md index 9c45742e..586eaa5f 100644 --- a/.claude/skills/docs-autofill/ledger.md +++ b/.claude/skills/docs-autofill/ledger.md @@ -8,3 +8,4 @@ Outcomes: `opened` · `blocked` (3 rounds, persona still stuck) · `skipped-no-k | date | matrix cell | branch | outcome | |---|---|---|---| +| 2026-08-24 | `env var (ORQ_OTEL_* export batching) × surface (tracing)` | docs/autofill-otel-batching | opened | diff --git a/docs/configuration.md b/docs/configuration.md index 6501ea6c..9d8aa3c5 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -22,6 +22,10 @@ All configuration is via environment variables. No config file is required. | `OTEL_EXPORTER_OTLP_HEADERS` | No | — | Comma-separated `key=value` pairs added to every OTLP export request. Format: `key1=value1,key2=value2`. | | `OTEL_SERVICE_NAME` | No | `evaluatorq` | Service name recorded on every span's `service.name` resource attribute. | | `OTEL_SERVICE_VERSION` | No | `1.0.0` | Service version recorded on every span's `service.version` resource attribute. | +| `ORQ_OTEL_MAX_QUEUE_SIZE` | No | `4096` | Maximum spans buffered by the `BatchSpanProcessor` before overflow is silently dropped. Raise it for a long-lived host that batches many runs. See [Tracing → Batching and flush](tracing.md#batching-and-flush). | +| `ORQ_OTEL_MAX_BATCH_SIZE` | No | `512` | Maximum spans per OTLP export request, clamped down to the queue size. See [Tracing → Batching and flush](tracing.md#batching-and-flush). | +| `ORQ_OTEL_SCHEDULE_DELAY_MS` | No | `5000` | Milliseconds between scheduled batch exports. Lower it to export more eagerly. See [Tracing → Batching and flush](tracing.md#batching-and-flush). | +| `ORQ_OTEL_FLUSH_TIMEOUT_MS` | No | `5000` | Milliseconds the end-of-run force-flush blocks before giving up and logging a warning. Read per run, so a long-lived host can raise it before a big run. See [Tracing → Batching and flush](tracing.md#batching-and-flush). | | `EVALUATORQ_CAPTURE_MESSAGE_CONTENT` | No | `true` | Set to `false` or `0` to strip LLM message content (prompts and responses) from spans. Token counts, model name, and latency are still recorded. Useful when exporting to third-party backends or to avoid capturing PII. | | `EVALUATORQ_SPAN_MAX_TEXT_CHARS` | No | unset (no limit) | Maximum characters per span text attribute. Set a positive integer (e.g. `8192`) to truncate long strings. Unset or `0` / `-1` means capture all. | | `EVALUATORQ_LLM_TIMEOUT_S` | No | `60.0` | Per-LLM-call timeout in seconds. **Simulation only** — has no effect on red teaming or core evaluation. A fallback default: `LLMCallConfig.timeout_ms` on the agent's config wins when set. Read at call time, so setting it after import takes effect. Increase for slow self-hosted endpoints; for the *target's* timeout rather than the simulator's, pass `target_agent_timeout_ms` to `simulate()`. | diff --git a/docs/tracing.md b/docs/tracing.md index 40460611..edbe9366 100644 --- a/docs/tracing.md +++ b/docs/tracing.md @@ -95,43 +95,50 @@ errors to stdout. ### Batching and flush -Spans are exported asynchronously by a `BatchSpanProcessor`: each finished span -lands in an in-memory queue, and a background thread drains the queue in batches -on a fixed schedule. The `TracerProvider` lives for the whole process — a -long-lived host (the dashboard, a worker that runs many red-team or simulation -runs back to back) never tears it down between runs, so the same queue absorbs -every run's spans. +The `BatchSpanProcessor` above buffers each finished span in an in-memory queue +and drains it in batches from a background thread. Because the `TracerProvider` +lives for the whole process, a long-lived host — the dashboard, or a worker that +runs many red-team or simulation runs back to back — never tears the queue down +between runs; the same queue absorbs every run's spans. -Two failure modes follow from that, and both drop spans **silently** unless you -tune the processor: +Two failure modes follow, and both drop spans **silently** unless you tune the +processor: - **Queue overflow.** If spans are produced faster than the exporter drains them — - a burst of parallel jobs, or a slow/unreachable OTLP endpoint — the queue fills - and the SDK discards the overflow. Nothing is raised; the trace just arrives - incomplete. + a burst of parallel jobs (the default `datapoint_parallelism` is 10), or a + slow or unreachable OTLP endpoint — the queue fills and the SDK discards the + overflow. Nothing is raised and nothing is logged. There is no first-party + signal for this: to confirm it, compare the span count in Orq's trace UI against + what a run should produce (see [Span hierarchy](#span-hierarchy)) and raise + `ORQ_OTEL_MAX_QUEUE_SIZE` if they disagree. - **Exit before flush.** At the end of each run evaluatorq force-flushes the - processor, and the SDK's `atexit` hook flushes again on a clean shutdown. But a - force-flush that exceeds its timeout, or a hard `SIGKILL` (an OOM kill, a CI job - cancelled mid-run), leaves whatever is still buffered unexported. A flush that - times out logs a `WARNING` — *"OTEL span flush timed out … some spans may not - have been exported"* — rather than failing the run. - -Four environment variables tune this. All are read once, when tracing -initializes: - -| Variable | Default | What it controls | -|---|---|---| -| `ORQ_OTEL_MAX_QUEUE_SIZE` | `4096` | Maximum spans buffered before the processor drops the overflow. Raise it for a long-lived host that batches many runs. | -| `ORQ_OTEL_MAX_BATCH_SIZE` | `512` | Maximum spans per export request. Capped to the queue size, so a batch size larger than the queue is silently clamped down. | -| `ORQ_OTEL_SCHEDULE_DELAY_MS` | `5000` | Delay between scheduled batch exports. Lower it to export more eagerly (fewer spans sitting in the queue at any moment); raise it to send larger, less frequent batches. | -| `ORQ_OTEL_FLUSH_TIMEOUT_MS` | `5000` | How long the per-run force-flush blocks waiting for the exporter before giving up and logging the timeout warning. | - -Each value must be a positive integer. A set-but-invalid value (non-numeric, zero -or negative) is ignored with a `WARNING` and the default is used, so a typo never -takes tracing down — but it also never silently takes effect. + processor — via the tracing session that wraps every `evaluatorq()`, + `red_team()` and `simulate()` call — and the SDK's `atexit` hook flushes again on + a clean shutdown. But a force-flush that exceeds its timeout, or a hard + `SIGKILL` (an OOM kill, a CI job cancelled mid-run), leaves whatever is still + buffered unexported. A flush that times out logs a `WARNING` rather than failing + the run: `OTEL span flush timed out after ms; some spans may not have been + exported.` + +Four environment variables tune this: + +| Variable | Default | Read | What it controls | +|---|---|---|---| +| `ORQ_OTEL_MAX_QUEUE_SIZE` | `4096` | at init | Maximum spans buffered before the processor drops the overflow. Raise it for a long-lived host that batches many runs. | +| `ORQ_OTEL_MAX_BATCH_SIZE` | `512` | at init | Maximum spans per export request. Capped to the queue size, so a batch size larger than the queue is silently clamped down. | +| `ORQ_OTEL_SCHEDULE_DELAY_MS` | `5000` | at init | Delay in milliseconds between scheduled batch exports. Lower it to export more eagerly (fewer spans sitting in the queue at any moment); raise it to send larger, less frequent batches. | +| `ORQ_OTEL_FLUSH_TIMEOUT_MS` | `5000` | per run | Milliseconds the end-of-run force-flush blocks waiting for the exporter before giving up and logging the timeout warning. | + +The first three are baked into the processor when tracing initializes and cannot +change for the life of the process; `ORQ_OTEL_FLUSH_TIMEOUT_MS` is read on every +flush, so a long-lived host can raise it before a big run. Each value must be a +positive integer — a set-but-invalid value (non-numeric, zero or negative) is +ignored with a `WARNING` and the default is used, so a typo never takes tracing +down, but it never silently takes effect either. A CI or batch run that must not lose spans typically drains the queue more -eagerly and allows a longer final flush: +eagerly and allows a longer final flush. These are read from the environment, so +in a GitHub Action they go straight in the workflow's `env:` block: ```bash export ORQ_API_KEY="your_orq_api_key" # enables tracing @@ -140,39 +147,9 @@ export ORQ_OTEL_SCHEDULE_DELAY_MS=1000 # export every second, not every five export ORQ_OTEL_FLUSH_TIMEOUT_MS=30000 # give the final flush up to 30s ``` -With those set, a normal run needs no code changes — evaluatorq flushes at the -end of each run, and the knobs above decide how much headroom that flush has: - -```python -import asyncio - -from evaluatorq import DataPoint, evaluatorq, job, string_contains_evaluator - - -@job("echo") -async def echo(data: DataPoint, _row: int) -> str: - return str(data.inputs["question"]) - - -async def main() -> None: - data = [DataPoint(inputs={"question": "30 days"}, expected_output="30 days")] - # Spans for this run are queued, batched on the schedule above, and - # force-flushed when evaluatorq() returns — within ORQ_OTEL_FLUSH_TIMEOUT_MS. - await evaluatorq( - "tracing-batching-demo", - data=data, - jobs=[echo], - evaluators=[string_contains_evaluator()], - ) - - -if __name__ == "__main__": - asyncio.run(main()) -``` - -Set `ORQ_DEBUG=1` to print the resolved endpoint and initialization diagnostics, -which is the quickest way to confirm the exporter is configured the way you -expect before a long run. +No code changes are needed: run the [minimal enable example](#minimal-enable-example) +above with these variables set, and evaluatorq force-flushes the run's spans +within `ORQ_OTEL_FLUSH_TIMEOUT_MS` when it returns. ## Span hierarchy From 3a009d9d1618dd0e92b715d199a257defb2f2b6e Mon Sep 17 00:00:00 2001 From: Bauke Brenninkmeijer Date: Fri, 28 Aug 2026 11:02:21 +0200 Subject: [PATCH 3/9] fix(tracing): warn on empty ORQ_OTEL_* values and on batch-size clamping MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `_env_int` used a falsy check on the raw env value, so a set-but-empty variable took the same silent path as an unset one. That is the exact shape a CI `env:` block produces when `${{ vars.X }}` does not resolve, so the tuning knob the tracing docs recommend for CI could be disabled with no signal at all. Treat empty (and whitespace-only) as set-but-invalid: warn and fall back, like every other rejected value. `max_export_batch_size` was clamped to the queue size with a bare `min()` next to a helper that warns on every rejected value — two adjacent branches differing in whether they log, which is what the "a degraded path announces itself" rule exists to prevent. It now warns. Tests: pin the three documented batching defaults (4096/512/5000) where they are actually consumed, so a change to the literals fails instead of silently contradicting docs/tracing.md; add the empty and whitespace cases to the `_env_int` parametrizations; assert the clamp warning. The processor-faking scaffolding moves into `_fake_tracing_sdk` rather than being copied for the second case. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 1 + src/evaluatorq/tracing/setup.py | 25 +++++++++++-- tests/common/test_tracing_lifecycle.py | 52 ++++++++++++++++++++++---- 3 files changed, 66 insertions(+), 12 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b3d92b9e..23cf416c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ All notable changes to `evaluatorq` are documented here. ### Notable defaults +- **A set-but-empty `ORQ_OTEL_*` tuning variable now logs a `WARNING` and falls back to the default, instead of falling back silently.** `_env_int` treated an empty string like an unset variable, so an unresolved `${{ vars.X }}` in a CI `env:` block — the exact shape the tracing docs recommend — disabled the knob with no signal. Whitespace-only values are treated the same way. Related: `ORQ_OTEL_MAX_BATCH_SIZE` larger than `ORQ_OTEL_MAX_QUEUE_SIZE` is still clamped down to the queue size, but the clamp now announces itself with a `WARNING` rather than happening silently. - **`EVALUATORQ_REASONING_EFFORT` has no default — unset means the parameter is not sent, and the model applies its own.** It previously fell back to `"medium"` for the simulator's own calls (user simulator, judge). A global effort is the wrong default in both directions: on a model that does not accept the parameter it costs a rejected request plus a retry per `(model, tool shape)` — memoised per process, so a short run or CI job never amortises it — and on a model that does, it silently overrides the provider's own tuned value. Set the env var, or `LLMCallConfig.reasoning_effort` on the agent's config, when you actually want a specific effort. **Simulation only**; red teaming's `target_reasoning_effort` was already opt-in. - **`OrqResponsesTarget.retry_attempts` now defaults to `1` — a single attempt, no retry — down from falling through to `with_retry`'s default of `5`.** diff --git a/src/evaluatorq/tracing/setup.py b/src/evaluatorq/tracing/setup.py index d8648e3a..658f187a 100644 --- a/src/evaluatorq/tracing/setup.py +++ b/src/evaluatorq/tracing/setup.py @@ -23,7 +23,8 @@ tracing to initialize again in that interpreter. - Tune the long-lived batch processor with ``ORQ_OTEL_MAX_QUEUE_SIZE`` (default 4096), ``ORQ_OTEL_SCHEDULE_DELAY_MS`` (default 5000), and ``ORQ_OTEL_MAX_BATCH_SIZE`` - (default 512). + (default 512, clamped to the queue size with a warning). These are passed to the + processor explicitly, so the SDK's own ``OTEL_BSP_*`` variables have no effect. - ``ORQ_OTEL_FLUSH_TIMEOUT_MS`` controls per-run force-flush (default 5000). A timeout logs a warning because some spans may not have been exported. - Exporter headers are bound at initialization: changing ``ORQ_API_KEY`` requires a @@ -51,11 +52,18 @@ def _env_int(name: str, default: int) -> int: """Read a positive int from the environment, falling back to *default*. - A set-but-invalid value (non-integer or non-positive) logs a WARNING so a - misconfigured tuning knob is actionable instead of silently ignored. + A set-but-invalid value (empty, non-integer or non-positive) logs a WARNING + so a misconfigured tuning knob is actionable instead of silently ignored. + An empty value counts as set-but-invalid: in a CI ``env:`` block an + unresolved ``${{ vars.X }}`` expands to the empty string, which would + otherwise fall back to the default with no signal. """ raw = os.environ.get(name) + if raw is None: + return default + raw = raw.strip() if not raw: + logger.warning('{} is set but empty; using default {}.', name, default) return default try: value = int(raw) @@ -213,11 +221,20 @@ async def init_tracing_if_needed() -> bool: # noqa: RUF029 # drop spans. Larger defaults + env overrides reduce that risk. max_queue_size = _env_int('ORQ_OTEL_MAX_QUEUE_SIZE', 4096) requested_batch_size = _env_int('ORQ_OTEL_MAX_BATCH_SIZE', 512) + batch_size = min(requested_batch_size, max_queue_size) + if batch_size != requested_batch_size: + logger.warning( + 'ORQ_OTEL_MAX_BATCH_SIZE ({}) exceeds ORQ_OTEL_MAX_QUEUE_SIZE ({}); ' + 'clamping the export batch size to {}.', + requested_batch_size, + max_queue_size, + batch_size, + ) span_processor = BatchSpanProcessor( exporter, max_queue_size=max_queue_size, schedule_delay_millis=_env_int('ORQ_OTEL_SCHEDULE_DELAY_MS', 5000), - max_export_batch_size=min(requested_batch_size, max_queue_size), + max_export_batch_size=batch_size, ) # Rely on the SDK default shutdown_on_exit=True for atexit teardown: diff --git a/tests/common/test_tracing_lifecycle.py b/tests/common/test_tracing_lifecycle.py index 0026b539..69bac0bf 100644 --- a/tests/common/test_tracing_lifecycle.py +++ b/tests/common/test_tracing_lifecycle.py @@ -131,7 +131,7 @@ async def capture_processing(*args: object) -> list[object]: # noqa: RUF029 ] -@pytest.mark.parametrize('raw', [None, 'invalid', '0', '-1']) +@pytest.mark.parametrize('raw', [None, '', ' ', 'invalid', '0', '-1']) def test_env_int_uses_default_for_unset_or_non_positive_values( monkeypatch: pytest.MonkeyPatch, raw: str | None ) -> None: @@ -149,7 +149,7 @@ def test_env_int_returns_positive_parsed_value(monkeypatch: pytest.MonkeyPatch) assert tracing_setup._env_int('X', 4096) == 8192 -@pytest.mark.parametrize('raw', ['invalid', '0', '-1']) +@pytest.mark.parametrize('raw', ['', ' ', 'invalid', '0', '-1']) def test_env_int_warns_on_set_but_invalid_value(monkeypatch: pytest.MonkeyPatch, raw: str) -> None: warning = Mock() monkeypatch.setenv('X', raw) @@ -159,10 +159,12 @@ def test_env_int_warns_on_set_but_invalid_value(monkeypatch: pytest.MonkeyPatch, warning.assert_called_once() -@pytest.mark.asyncio -async def test_initialization_caps_export_batch_size_to_queue_size( - monkeypatch: pytest.MonkeyPatch, -) -> None: +def _fake_tracing_sdk(monkeypatch: pytest.MonkeyPatch) -> dict[str, int]: + """Fake out the exporter and provider, returning the captured processor kwargs. + + Drives real ``init_tracing_if_needed`` so the values the documented + ``ORQ_OTEL_*`` knobs resolve to are observed where they are actually used. + """ from opentelemetry import trace from opentelemetry.exporter.otlp.proto.http import trace_exporter from opentelemetry.sdk import trace as sdk_trace @@ -191,10 +193,41 @@ def add_span_processor(self, processor: object) -> None: monkeypatch.setattr(tracing_setup, '_tracer', None) monkeypatch.setattr(tracing_setup, '_is_initialized', False) monkeypatch.setattr(tracing_setup, '_initialization_attempted', False) - # Opt out of the suite-wide export guard: this test drives real setup with - # the exporter and provider faked out above. + # Opt out of the suite-wide export guard: this drives real setup with the + # exporter and provider faked out above. monkeypatch.delenv('ORQ_DISABLE_TRACING', raising=False) monkeypatch.setenv('OTEL_EXPORTER_OTLP_ENDPOINT', 'https://example.test') + return processor_options + + +@pytest.mark.asyncio +async def test_initialization_uses_documented_batching_defaults( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Pin the defaults published in docs/tracing.md and docs/configuration.md.""" + processor_options = _fake_tracing_sdk(monkeypatch) + for name in ( + 'ORQ_OTEL_MAX_QUEUE_SIZE', + 'ORQ_OTEL_MAX_BATCH_SIZE', + 'ORQ_OTEL_SCHEDULE_DELAY_MS', + ): + monkeypatch.delenv(name, raising=False) + + assert await tracing_setup.init_tracing_if_needed() is True + assert processor_options == { + 'max_queue_size': 4096, + 'max_export_batch_size': 512, + 'schedule_delay_millis': 5000, + } + + +@pytest.mark.asyncio +async def test_initialization_caps_export_batch_size_to_queue_size( + monkeypatch: pytest.MonkeyPatch, +) -> None: + processor_options = _fake_tracing_sdk(monkeypatch) + warning = Mock() + monkeypatch.setattr(tracing_setup.logger, 'warning', warning) monkeypatch.setenv('ORQ_OTEL_MAX_QUEUE_SIZE', '100') monkeypatch.setenv('ORQ_OTEL_MAX_BATCH_SIZE', '200') monkeypatch.setenv('ORQ_OTEL_SCHEDULE_DELAY_MS', '300') @@ -205,6 +238,9 @@ def add_span_processor(self, processor: object) -> None: 'max_export_batch_size': 100, 'schedule_delay_millis': 300, } + # The clamp is a degraded path, so it announces itself. + warning.assert_called_once() + assert 'clamping' in warning.call_args.args[0] @pytest.mark.asyncio From 3868a5f9dcd48b396b74df61ac4b49d2756a92b3 Mon Sep 17 00:00:00 2001 From: Bauke Brenninkmeijer Date: Fri, 28 Aug 2026 11:02:37 +0200 Subject: [PATCH 4/9] docs: correct the span-loss claims in the batching section MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The section asserted that queue overflow is completely silent and has "no first-party signal", so the only remedy it offered was hand-counting spans in the Orq trace UI. The SDK logs `Queue full, dropping Span.` on the stdlib `opentelemetry` logger (verified against the pinned 1.42.1 with a blocking exporter), deduplicated per process. It also evicts the *oldest* buffered span, not the newest, so a truncated trace is missing its early spans — the opposite of what the page implied. Other corrections and additions: - Both failure modes log; neither is silent. Reframed accordingly, and the second flush warning (`OTEL span flush failed (...)`) is documented alongside the timeout one it shares a code path with. - State the flush guarantee rather than the mechanism: `simulate()` flushes in its own teardown, not via `tracing_session`, so naming the session was drift bait even though the outcome is the same. - Sizing: the queue drains continuously, so raising it buys stall tolerance, not capacity. Gives the practitioner an actual formula instead of "raise it if the counts disagree". - The per-request export timeout is a fixed 5s and is not the flush timeout; three different "5 seconds" sat within 45 lines undisambiguated. - The SDK's own `OTEL_BSP_*` variables are inert here. - Adds the YAML `env:` block the prose promised, and a note that the flush timeout bounds wall-clock added to the job. - Drops the four-row table: `configuration.md` is the canonical env-var reference, and the page's existing `OTEL_EXPORTER_OTLP_*` bullets already follow that one-directional precedent. Also drops the opening and closing paragraphs, which restated bullets six lines away. - A scale boundary for readers on `deployment:` targets, and a discovery path from the symptom: an FAQ entry and a link from `dashboard.md`, the long-lived host the section is about. `axes.md` gains the ledger convention for Tier-1 gaps — they are not matrix cells, and the row recorded a `surface` value that does not exist, which next week's free-text dedupe would never match. Co-Authored-By: Claude Opus 5 (1M context) --- .claude/skills/docs-autofill/ledger.md | 2 +- .claude/skills/docs-coverage/axes.md | 7 ++ docs/configuration.md | 8 +- docs/dashboard.md | 5 ++ docs/faq.md | 9 ++ docs/tracing.md | 114 +++++++++++++++---------- 6 files changed, 95 insertions(+), 50 deletions(-) diff --git a/.claude/skills/docs-autofill/ledger.md b/.claude/skills/docs-autofill/ledger.md index 586eaa5f..d2ae4485 100644 --- a/.claude/skills/docs-autofill/ledger.md +++ b/.claude/skills/docs-autofill/ledger.md @@ -8,4 +8,4 @@ Outcomes: `opened` · `blocked` (3 rounds, persona still stuck) · `skipped-no-k | date | matrix cell | branch | outcome | |---|---|---|---| -| 2026-08-24 | `env var (ORQ_OTEL_* export batching) × surface (tracing)` | docs/autofill-otel-batching | opened | +| 2026-08-24 | `tier 1: env var (ORQ_OTEL_MAX_QUEUE_SIZE, ORQ_OTEL_MAX_BATCH_SIZE, ORQ_OTEL_SCHEDULE_DELAY_MS, ORQ_OTEL_FLUSH_TIMEOUT_MS)` | docs/autofill-otel-batching | opened | diff --git a/.claude/skills/docs-coverage/axes.md b/.claude/skills/docs-coverage/axes.md index b2b1864b..46acf45c 100644 --- a/.claude/skills/docs-coverage/axes.md +++ b/.claude/skills/docs-coverage/axes.md @@ -106,5 +106,12 @@ Marked `N/A` in the matrix, never reported as a gap. command and subcommand, every env var. The generated API reference does **not** count: a docstring is not discovery. +Tier-1 items are **not** matrix cells — there is no `env var` axis, and `surface` +has exactly the three values above. Record a Tier-1 gap in the `docs-autofill` +ledger as `tier 1: (, …)` with no second axis, e.g. +`tier 1: env var (ORQ_OTEL_MAX_QUEUE_SIZE, ORQ_OTEL_MAX_BATCH_SIZE)`. The ledger +dedupe compares that cell as free text, so naming the items is what stops the +same gap being re-derived next week; an invented axis pair never matches. + **Tier 2 — API reference suffices.** Supporting types, contracts, backends, and subpackage `__all__` members. Flag only when there is no docstring at all. diff --git a/docs/configuration.md b/docs/configuration.md index 9d8aa3c5..2d116b4b 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -22,10 +22,10 @@ All configuration is via environment variables. No config file is required. | `OTEL_EXPORTER_OTLP_HEADERS` | No | — | Comma-separated `key=value` pairs added to every OTLP export request. Format: `key1=value1,key2=value2`. | | `OTEL_SERVICE_NAME` | No | `evaluatorq` | Service name recorded on every span's `service.name` resource attribute. | | `OTEL_SERVICE_VERSION` | No | `1.0.0` | Service version recorded on every span's `service.version` resource attribute. | -| `ORQ_OTEL_MAX_QUEUE_SIZE` | No | `4096` | Maximum spans buffered by the `BatchSpanProcessor` before overflow is silently dropped. Raise it for a long-lived host that batches many runs. See [Tracing → Batching and flush](tracing.md#batching-and-flush). | -| `ORQ_OTEL_MAX_BATCH_SIZE` | No | `512` | Maximum spans per OTLP export request, clamped down to the queue size. See [Tracing → Batching and flush](tracing.md#batching-and-flush). | -| `ORQ_OTEL_SCHEDULE_DELAY_MS` | No | `5000` | Milliseconds between scheduled batch exports. Lower it to export more eagerly. See [Tracing → Batching and flush](tracing.md#batching-and-flush). | -| `ORQ_OTEL_FLUSH_TIMEOUT_MS` | No | `5000` | Milliseconds the end-of-run force-flush blocks before giving up and logging a warning. Read per run, so a long-lived host can raise it before a big run. See [Tracing → Batching and flush](tracing.md#batching-and-flush). | +| `ORQ_OTEL_MAX_QUEUE_SIZE` | No | `4096` | Maximum spans buffered by the `BatchSpanProcessor`. When it is full the SDK evicts the oldest buffered span and logs a warning on the stdlib `opentelemetry` logger. See [Tracing → Batching and flush](tracing.md#batching-and-flush). | +| `ORQ_OTEL_MAX_BATCH_SIZE` | No | `512` | Maximum spans per OTLP export request. A value larger than the queue size is clamped down to it with a `WARNING`. | +| `ORQ_OTEL_SCHEDULE_DELAY_MS` | No | `5000` | Milliseconds between scheduled batch exports. Lower it to drain the queue more eagerly. | +| `ORQ_OTEL_FLUSH_TIMEOUT_MS` | No | `5000` | Milliseconds the end-of-run force-flush blocks before giving up and logging a warning. Read per run, so a long-lived host can raise it before a big run. Bounds the final flush only — the per-request export timeout is a fixed 5s. | | `EVALUATORQ_CAPTURE_MESSAGE_CONTENT` | No | `true` | Set to `false` or `0` to strip LLM message content (prompts and responses) from spans. Token counts, model name, and latency are still recorded. Useful when exporting to third-party backends or to avoid capturing PII. | | `EVALUATORQ_SPAN_MAX_TEXT_CHARS` | No | unset (no limit) | Maximum characters per span text attribute. Set a positive integer (e.g. `8192`) to truncate long strings. Unset or `0` / `-1` means capture all. | | `EVALUATORQ_LLM_TIMEOUT_S` | No | `60.0` | Per-LLM-call timeout in seconds. **Simulation only** — has no effect on red teaming or core evaluation. A fallback default: `LLMCallConfig.timeout_ms` on the agent's config wins when set. Read at call time, so setting it after import takes effect. Increase for slow self-hosted endpoints; for the *target's* timeout rather than the simulator's, pass `target_agent_timeout_ms` to `simulate()`. | diff --git a/docs/dashboard.md b/docs/dashboard.md index 74c32d07..bf50a8d3 100644 --- a/docs/dashboard.md +++ b/docs/dashboard.md @@ -55,6 +55,11 @@ eq dashboard --host 0.0.0.0 --port 8888 ORQ_WORKSPACE=orq-research eq dashboard ``` +The dashboard is a long-lived process, so its tracer provider — and the span +queue behind it — is shared by every run it serves. If spans go missing from +traces produced under it, see +[Tracing → Batching and flush](tracing.md#batching-and-flush). + | Invocation | What it scans | |---|---| | `eq dashboard` | Both default stores: `.evaluatorq/runs` (red team) and `.evaluatorq/sim-runs` (simulation) | diff --git a/docs/faq.md b/docs/faq.md index 4a13dc19..a0105d5a 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -78,6 +78,15 @@ Cost and wall-clock scale with cases × turns × LLM calls. The levers are how m Runs auto-save locally (red-team runs to `.evaluatorq/runs/`; simulation runs to `.evaluatorq/sim-runs/`). Browse them in the multi-run FastHTML dashboard with `eq dashboard` (no path browses both stores; `eq dashboard .evaluatorq/sim-runs` scopes to simulation), or list runs with `eq redteam runs` / `eq sim runs`. See [Dashboard](dashboard.md). +### Some spans are missing from my traces + +The span exporter batches in the background, so spans can be lost two ways, both of +which log a warning rather than failing the run: the in-memory queue overflowed +(spans produced faster than the exporter drained them), or the process exited before +the final flush finished. Raise `ORQ_OTEL_MAX_QUEUE_SIZE` for the first and +`ORQ_OTEL_FLUSH_TIMEOUT_MS` for the second. See +[Tracing → Batching and flush](tracing.md#batching-and-flush). + ### How do I run a plain evaluation? Decorate a function with `@job`, hand `evaluatorq()` your data and evaluators, and it runs the jobs in parallel and scores each row: diff --git a/docs/tracing.md b/docs/tracing.md index edbe9366..b8832a23 100644 --- a/docs/tracing.md +++ b/docs/tracing.md @@ -85,7 +85,8 @@ errors to stdout. - **Protocol**: HTTP/protobuf (`OTLPSpanExporter` from `opentelemetry-exporter-otlp-proto-http`) - **Export mode**: `BatchSpanProcessor` (asynchronous batching) -- **Timeout**: 5 seconds per export request +- **Timeout**: 5 seconds per export request (fixed; distinct from the tunable + end-of-run flush timeout below) - **Auth**: `Authorization: Bearer ` is added automatically when the resolved endpoint's hostname ends in `.orq.ai` or is exactly `orq.ai`. For any other endpoint the header is not added; use `OTEL_EXPORTER_OTLP_HEADERS` @@ -95,50 +96,63 @@ errors to stdout. ### Batching and flush -The `BatchSpanProcessor` above buffers each finished span in an in-memory queue -and drains it in batches from a background thread. Because the `TracerProvider` -lives for the whole process, a long-lived host — the dashboard, or a worker that -runs many red-team or simulation runs back to back — never tears the queue down -between runs; the same queue absorbs every run's spans. +Defaults are fine for one-off runs — read this if you run a long-lived host or a +CI job that must not lose spans. The four `ORQ_OTEL_*` variables named below are +listed with their defaults in +[Configuration](configuration.md#environment-variables). -Two failure modes follow, and both drop spans **silently** unless you tune the -processor: +Because the `TracerProvider` lives for the whole process, a long-lived host — the +dashboard, or a worker that runs many red-team or simulation runs back to back — +never tears the span queue down between runs; the same queue absorbs every run's +spans. The knobs behave identically whether you export to Orq or to a third-party +collector; only endpoint latency differs. + +Two failure modes follow. Each drops spans without failing the run, so span loss +shows up in your logs rather than your exit code: - **Queue overflow.** If spans are produced faster than the exporter drains them — - a burst of parallel jobs (the default `datapoint_parallelism` is 10), or a - slow or unreachable OTLP endpoint — the queue fills and the SDK discards the - overflow. Nothing is raised and nothing is logged. There is no first-party - signal for this: to confirm it, compare the span count in Orq's trace UI against - what a run should produce (see [Span hierarchy](#span-hierarchy)) and raise - `ORQ_OTEL_MAX_QUEUE_SIZE` if they disagree. -- **Exit before flush.** At the end of each run evaluatorq force-flushes the - processor — via the tracing session that wraps every `evaluatorq()`, - `red_team()` and `simulate()` call — and the SDK's `atexit` hook flushes again on - a clean shutdown. But a force-flush that exceeds its timeout, or a hard - `SIGKILL` (an OOM kill, a CI job cancelled mid-run), leaves whatever is still - buffered unexported. A flush that times out logs a `WARNING` rather than failing - the run: `OTEL span flush timed out after ms; some spans may not have been - exported.` - -Four environment variables tune this: - -| Variable | Default | Read | What it controls | -|---|---|---|---| -| `ORQ_OTEL_MAX_QUEUE_SIZE` | `4096` | at init | Maximum spans buffered before the processor drops the overflow. Raise it for a long-lived host that batches many runs. | -| `ORQ_OTEL_MAX_BATCH_SIZE` | `512` | at init | Maximum spans per export request. Capped to the queue size, so a batch size larger than the queue is silently clamped down. | -| `ORQ_OTEL_SCHEDULE_DELAY_MS` | `5000` | at init | Delay in milliseconds between scheduled batch exports. Lower it to export more eagerly (fewer spans sitting in the queue at any moment); raise it to send larger, less frequent batches. | -| `ORQ_OTEL_FLUSH_TIMEOUT_MS` | `5000` | per run | Milliseconds the end-of-run force-flush blocks waiting for the exporter before giving up and logging the timeout warning. | - -The first three are baked into the processor when tracing initializes and cannot -change for the life of the process; `ORQ_OTEL_FLUSH_TIMEOUT_MS` is read on every -flush, so a long-lived host can raise it before a big run. Each value must be a -positive integer — a set-but-invalid value (non-numeric, zero or negative) is -ignored with a `WARNING` and the default is used, so a typo never takes tracing -down, but it never silently takes effect either. - -A CI or batch run that must not lose spans typically drains the queue more -eagerly and allows a longer final flush. These are read from the environment, so -in a GitHub Action they go straight in the workflow's `env:` block: + a burst of parallel jobs (the `datapoint_parallelism` argument), or a slow or + unreachable OTLP endpoint — the queue is full and the SDK evicts the *oldest* + buffered span to make room, so a trace comes back missing its early spans rather + than its last ones. The SDK logs a warning naming the full queue on the stdlib + `opentelemetry` logger (`Queue full, dropping Span.` on current releases, + worded differently on older ones); the message is deduplicated, so one + occurrence means loss started, not that exactly one span was lost. evaluatorq + does not route that logger through loguru, so if you configure logging yourself, + intercept stdlib logging or you will not see it. Raise + `ORQ_OTEL_MAX_QUEUE_SIZE` when it appears. +- **Exit before flush.** Every `evaluatorq()`, `red_team()` and `simulate()` call + force-flushes the processor when it returns, including on failure, and the SDK's + `atexit` hook flushes again on a clean shutdown. But a force-flush that exceeds + its timeout, or a hard `SIGKILL` (an OOM kill, a CI job cancelled mid-run), + leaves whatever is still buffered unexported. A flush that times out logs + `OTEL span flush timed out after ms; some spans may not have been exported.` + and one that fails outright — unreachable endpoint, rejected auth — logs + `OTEL span flush failed (); some spans may not have been exported.` + +The queue drains continuously, not at the end of a run: the processor exports up +to `ORQ_OTEL_MAX_BATCH_SIZE` spans every `ORQ_OTEL_SCHEDULE_DELAY_MS`, which on +the defaults sustains roughly 512 spans per 5 seconds. Back-to-back runs do not +accumulate. Raising `ORQ_OTEL_MAX_QUEUE_SIZE` therefore buys *stall tolerance* — +queue size divided by that sustained rate is how many seconds of exporter stall +you survive — rather than extra capacity. When spans go missing and the endpoint +is healthy, the lever is a lower `ORQ_OTEL_SCHEDULE_DELAY_MS`. + +`ORQ_OTEL_MAX_QUEUE_SIZE`, `ORQ_OTEL_MAX_BATCH_SIZE` and +`ORQ_OTEL_SCHEDULE_DELAY_MS` are baked into the processor when tracing +initializes and cannot change for the life of the process; a batch size larger +than the queue size is clamped down to it with a `WARNING`. +`ORQ_OTEL_FLUSH_TIMEOUT_MS` is read on every flush, so a long-lived host can raise +it before a big run — it bounds the end-of-run flush only, while the per-request +export timeout above is a fixed 5 seconds and is not tunable. Each value must be a +positive integer: a set-but-invalid value (empty, non-numeric, zero or negative) +is ignored with a `WARNING` and the default is used, so a typo never takes tracing +down but never silently takes effect either. The SDK's own `OTEL_BSP_*` variables +have no effect — evaluatorq passes these values to the processor explicitly, which +takes precedence over the SDK's env-var defaults. + +A CI run that must not lose spans drains the queue more eagerly and allows a +longer final flush: ```bash export ORQ_API_KEY="your_orq_api_key" # enables tracing @@ -147,9 +161,19 @@ export ORQ_OTEL_SCHEDULE_DELAY_MS=1000 # export every second, not every five export ORQ_OTEL_FLUSH_TIMEOUT_MS=30000 # give the final flush up to 30s ``` -No code changes are needed: run the [minimal enable example](#minimal-enable-example) -above with these variables set, and evaluatorq force-flushes the run's spans -within `ORQ_OTEL_FLUSH_TIMEOUT_MS` when it returns. +In a GitHub Action the same values go straight in the workflow's `env:` block: + +```yaml +env: + ORQ_API_KEY: ${{ secrets.ORQ_API_KEY }} + ORQ_OTEL_MAX_QUEUE_SIZE: 16384 + ORQ_OTEL_SCHEDULE_DELAY_MS: 1000 + ORQ_OTEL_FLUSH_TIMEOUT_MS: 30000 +``` + +The flush timeout is an upper bound on wall-clock added to the job: a run whose +exporter has become unreachable waits the full 30 seconds before warning and +moving on. ## Span hierarchy From 1c9e07fe6971542043ebf85137b2199665c9d19a Mon Sep 17 00:00:00 2001 From: Bauke Brenninkmeijer Date: Fri, 28 Aug 2026 11:35:21 +0200 Subject: [PATCH 5/9] fix(tracing): enforce ORQ_OTEL_FLUSH_TIMEOUT_MS ourselves; correct the batching docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A Codex fact-check pass over every claim in the batching section against the installed SDK found the variable did nothing and three of the surrounding statements were wrong. `BatchProcessor.force_flush` ignores its `timeout_millis` argument and unconditionally returns `True` (upstream issue #4568), so passing `ORQ_OTEL_FLUSH_TIMEOUT_MS` down bounded nothing and the `ok is False` warning branch was unreachable. A run whose collector had stopped responding waited for the export to finish however long that took — the opposite of what the docs promised. `flush_tracing` now wraps the call in `asyncio.wait_for`, so the timeout is real and the warning fires. The exporter thread is a daemon and keeps draining behind it, which the docstring says. Doc corrections, each verified against source: - The processor does not export on a fixed cadence. `emit` wakes the worker the moment the queue reaches `max_export_batch_size`, which then drains while the queue stays above the threshold; the schedule delay is the idle timer for a *partial* batch. So the old "512 spans per 5 seconds" arithmetic, and the advice to lower the schedule delay under a burst, were both wrong. Rewritten around what the worker actually does. - The overflow warning is on `opentelemetry.sdk._shared_internal`, not a bare `opentelemetry` logger, and its `DuplicateFilter` dedupes in 20-second buckets rather than once per process. - The flush guarantee is narrower than "every call, including on failure": validation that fails before the tracing scope opens does not flush, and the standalone pairwise entry points never do. A hard `SIGKILL` drops the buffer with no warning, which the FAQ entry claimed otherwise. - The dashboard paragraph is removed outright: `eq dashboard` reads saved reports and manifests and never initializes tracing, so it has no span queue to share. The same wrong example is dropped from `setup.py`'s comment. - `docs/faq.md` said `datapoint_parallelism` defaults to 10 "everywhere"; the adaptive red-team pipeline defaults to 5. Style, from a corpus check rather than taste: `›` for section breadcrumbs in link labels (8 existing uses, no `→`), `**Label** —` list labels (31 existing uses, no `**Label.**`), and "warning" as prose rather than `WARNING` as an identifier. Long sentences split; the announce-the-next-sentence lines cut. Tests: assert an all-default initialization does not trip the clamp warning (it was conditional and unpinned), and assert a hanging force-flush is bounded and warns. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 2 +- docs/configuration.md | 8 +- docs/dashboard.md | 5 - docs/faq.md | 15 +-- docs/tracing.md | 124 ++++++++++++------------- src/evaluatorq/tracing/setup.py | 48 +++++++--- tests/common/test_tracing_lifecycle.py | 25 +++++ 7 files changed, 132 insertions(+), 95 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 23cf416c..d8c641db 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,7 @@ All notable changes to `evaluatorq` are documented here. ### Notable defaults -- **A set-but-empty `ORQ_OTEL_*` tuning variable now logs a `WARNING` and falls back to the default, instead of falling back silently.** `_env_int` treated an empty string like an unset variable, so an unresolved `${{ vars.X }}` in a CI `env:` block — the exact shape the tracing docs recommend — disabled the knob with no signal. Whitespace-only values are treated the same way. Related: `ORQ_OTEL_MAX_BATCH_SIZE` larger than `ORQ_OTEL_MAX_QUEUE_SIZE` is still clamped down to the queue size, but the clamp now announces itself with a `WARNING` rather than happening silently. +- **A set-but-empty `ORQ_OTEL_*` tuning variable now logs a `WARNING` and falls back to the default, instead of falling back silently.** `_env_int` treated an empty string like an unset variable, so an unresolved workflow variable in a CI `env:` block expands to the empty string and disabled the knob with no signal. Whitespace-only values are treated the same way. Related: `ORQ_OTEL_MAX_BATCH_SIZE` larger than `ORQ_OTEL_MAX_QUEUE_SIZE` is still clamped down to the queue size, but the clamp now announces itself with a `WARNING` rather than happening silently. - **`EVALUATORQ_REASONING_EFFORT` has no default — unset means the parameter is not sent, and the model applies its own.** It previously fell back to `"medium"` for the simulator's own calls (user simulator, judge). A global effort is the wrong default in both directions: on a model that does not accept the parameter it costs a rejected request plus a retry per `(model, tool shape)` — memoised per process, so a short run or CI job never amortises it — and on a model that does, it silently overrides the provider's own tuned value. Set the env var, or `LLMCallConfig.reasoning_effort` on the agent's config, when you actually want a specific effort. **Simulation only**; red teaming's `target_reasoning_effort` was already opt-in. - **`OrqResponsesTarget.retry_attempts` now defaults to `1` — a single attempt, no retry — down from falling through to `with_retry`'s default of `5`.** diff --git a/docs/configuration.md b/docs/configuration.md index 2d116b4b..af2eae74 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -22,10 +22,10 @@ All configuration is via environment variables. No config file is required. | `OTEL_EXPORTER_OTLP_HEADERS` | No | — | Comma-separated `key=value` pairs added to every OTLP export request. Format: `key1=value1,key2=value2`. | | `OTEL_SERVICE_NAME` | No | `evaluatorq` | Service name recorded on every span's `service.name` resource attribute. | | `OTEL_SERVICE_VERSION` | No | `1.0.0` | Service version recorded on every span's `service.version` resource attribute. | -| `ORQ_OTEL_MAX_QUEUE_SIZE` | No | `4096` | Maximum spans buffered by the `BatchSpanProcessor`. When it is full the SDK evicts the oldest buffered span and logs a warning on the stdlib `opentelemetry` logger. See [Tracing → Batching and flush](tracing.md#batching-and-flush). | -| `ORQ_OTEL_MAX_BATCH_SIZE` | No | `512` | Maximum spans per OTLP export request. A value larger than the queue size is clamped down to it with a `WARNING`. | -| `ORQ_OTEL_SCHEDULE_DELAY_MS` | No | `5000` | Milliseconds between scheduled batch exports. Lower it to drain the queue more eagerly. | -| `ORQ_OTEL_FLUSH_TIMEOUT_MS` | No | `5000` | Milliseconds the end-of-run force-flush blocks before giving up and logging a warning. Read per run, so a long-lived host can raise it before a big run. Bounds the final flush only — the per-request export timeout is a fixed 5s. | +| `ORQ_OTEL_MAX_QUEUE_SIZE` | No | `4096` | Maximum spans buffered by the `BatchSpanProcessor`. When it is full the SDK evicts the oldest buffered span and logs `Queue full, dropping Span.` on the stdlib `opentelemetry.sdk._shared_internal` logger. See [Tracing › Batching and flush](tracing.md#batching-and-flush). | +| `ORQ_OTEL_MAX_BATCH_SIZE` | No | `512` | Maximum spans per OTLP export request. Reaching it wakes the exporter immediately. A value larger than the queue size is clamped to the queue size, with a warning. | +| `ORQ_OTEL_SCHEDULE_DELAY_MS` | No | `5000` | Milliseconds a partial batch waits before it is exported. It does not throttle a full batch. | +| `ORQ_OTEL_FLUSH_TIMEOUT_MS` | No | `5000` | Milliseconds the end-of-run force-flush waits before giving up and logging a warning. Read per run; enforced by evaluatorq rather than the SDK. Bounds the final flush only — the per-request export timeout is a fixed 5s. | | `EVALUATORQ_CAPTURE_MESSAGE_CONTENT` | No | `true` | Set to `false` or `0` to strip LLM message content (prompts and responses) from spans. Token counts, model name, and latency are still recorded. Useful when exporting to third-party backends or to avoid capturing PII. | | `EVALUATORQ_SPAN_MAX_TEXT_CHARS` | No | unset (no limit) | Maximum characters per span text attribute. Set a positive integer (e.g. `8192`) to truncate long strings. Unset or `0` / `-1` means capture all. | | `EVALUATORQ_LLM_TIMEOUT_S` | No | `60.0` | Per-LLM-call timeout in seconds. **Simulation only** — has no effect on red teaming or core evaluation. A fallback default: `LLMCallConfig.timeout_ms` on the agent's config wins when set. Read at call time, so setting it after import takes effect. Increase for slow self-hosted endpoints; for the *target's* timeout rather than the simulator's, pass `target_agent_timeout_ms` to `simulate()`. | diff --git a/docs/dashboard.md b/docs/dashboard.md index bf50a8d3..74c32d07 100644 --- a/docs/dashboard.md +++ b/docs/dashboard.md @@ -55,11 +55,6 @@ eq dashboard --host 0.0.0.0 --port 8888 ORQ_WORKSPACE=orq-research eq dashboard ``` -The dashboard is a long-lived process, so its tracer provider — and the span -queue behind it — is shared by every run it serves. If spans go missing from -traces produced under it, see -[Tracing → Batching and flush](tracing.md#batching-and-flush). - | Invocation | What it scans | |---|---| | `eq dashboard` | Both default stores: `.evaluatorq/runs` (red team) and `.evaluatorq/sim-runs` (simulation) | diff --git a/docs/faq.md b/docs/faq.md index a0105d5a..5da1cf36 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -72,7 +72,7 @@ Simulator/attacker/judge LLM calls go to OpenAI or the Orq router. Results uploa ### How much does a run cost, and how do I keep it cheap? -Cost and wall-clock scale with cases × turns × LLM calls. The levers are how many cases you run (`max_dynamic_datapoints` / `max_static_datapoints` for red teaming, `num_personas` × `num_scenarios` for simulation), `max_turns`, and `datapoint_parallelism` (default 10 everywhere). To size against a provider concurrency limit, set `llm_parallelism=` (on `evaluatorq()`, `red_team()` or `simulate()`) rather than lowering `datapoint_parallelism` — it counts requests instead of tasks, so the number means the same thing however the fan-out nests. Red teaming's report tracks spend in `report.summary.token_usage_total`. +Cost and wall-clock scale with cases × turns × LLM calls. The levers are how many cases you run (`max_dynamic_datapoints` / `max_static_datapoints` for red teaming, `num_personas` × `num_scenarios` for simulation), `max_turns`, and `datapoint_parallelism` (default 10 on `evaluatorq()`, `red_team()` and `simulate()`; 5 in the adaptive red-team pipeline). To size against a provider concurrency limit, set `llm_parallelism=` (on `evaluatorq()`, `red_team()` or `simulate()`) rather than lowering `datapoint_parallelism` — it counts requests instead of tasks, so the number means the same thing however the fan-out nests. Red teaming's report tracks spend in `report.summary.token_usage_total`. ### Where do results go, and how do I view a past run? @@ -80,12 +80,13 @@ Runs auto-save locally (red-team runs to `.evaluatorq/runs/`; simulation runs to ### Some spans are missing from my traces -The span exporter batches in the background, so spans can be lost two ways, both of -which log a warning rather than failing the run: the in-memory queue overflowed -(spans produced faster than the exporter drained them), or the process exited before -the final flush finished. Raise `ORQ_OTEL_MAX_QUEUE_SIZE` for the first and -`ORQ_OTEL_FLUSH_TIMEOUT_MS` for the second. See -[Tracing → Batching and flush](tracing.md#batching-and-flush). +The span exporter batches in the background, so spans can be lost two ways, neither +of which fails the run. Either the in-memory queue overflowed — spans produced faster +than the exporter drained them — or the process exited before the final flush +finished. Raise `ORQ_OTEL_MAX_QUEUE_SIZE` for the first and +`ORQ_OTEL_FLUSH_TIMEOUT_MS` for the second. Both log a warning; a hard `SIGKILL` +drops whatever was still buffered without one. See +[Tracing › Batching and flush](tracing.md#batching-and-flush). ### How do I run a plain evaluation? diff --git a/docs/tracing.md b/docs/tracing.md index b8832a23..91d010be 100644 --- a/docs/tracing.md +++ b/docs/tracing.md @@ -85,8 +85,8 @@ errors to stdout. - **Protocol**: HTTP/protobuf (`OTLPSpanExporter` from `opentelemetry-exporter-otlp-proto-http`) - **Export mode**: `BatchSpanProcessor` (asynchronous batching) -- **Timeout**: 5 seconds per export request (fixed; distinct from the tunable - end-of-run flush timeout below) +- **Timeout**: 5 seconds per export request. The separate end-of-run flush timeout + is set with `ORQ_OTEL_FLUSH_TIMEOUT_MS` (see below). - **Auth**: `Authorization: Bearer ` is added automatically when the resolved endpoint's hostname ends in `.orq.ai` or is exactly `orq.ai`. For any other endpoint the header is not added; use `OTEL_EXPORTER_OTLP_HEADERS` @@ -96,84 +96,82 @@ errors to stdout. ### Batching and flush -Defaults are fine for one-off runs — read this if you run a long-lived host or a -CI job that must not lose spans. The four `ORQ_OTEL_*` variables named below are -listed with their defaults in -[Configuration](configuration.md#environment-variables). - -Because the `TracerProvider` lives for the whole process, a long-lived host — the -dashboard, or a worker that runs many red-team or simulation runs back to back — -never tears the span queue down between runs; the same queue absorbs every run's -spans. The knobs behave identically whether you export to Orq or to a third-party -collector; only endpoint latency differs. - -Two failure modes follow. Each drops spans without failing the run, so span loss -shows up in your logs rather than your exit code: - -- **Queue overflow.** If spans are produced faster than the exporter drains them — - a burst of parallel jobs (the `datapoint_parallelism` argument), or a slow or - unreachable OTLP endpoint — the queue is full and the SDK evicts the *oldest* - buffered span to make room, so a trace comes back missing its early spans rather - than its last ones. The SDK logs a warning naming the full queue on the stdlib - `opentelemetry` logger (`Queue full, dropping Span.` on current releases, - worded differently on older ones); the message is deduplicated, so one - occurrence means loss started, not that exactly one span was lost. evaluatorq - does not route that logger through loguru, so if you configure logging yourself, - intercept stdlib logging or you will not see it. Raise - `ORQ_OTEL_MAX_QUEUE_SIZE` when it appears. -- **Exit before flush.** Every `evaluatorq()`, `red_team()` and `simulate()` call - force-flushes the processor when it returns, including on failure, and the SDK's - `atexit` hook flushes again on a clean shutdown. But a force-flush that exceeds - its timeout, or a hard `SIGKILL` (an OOM kill, a CI job cancelled mid-run), - leaves whatever is still buffered unexported. A flush that times out logs - `OTEL span flush timed out after ms; some spans may not have been exported.` - and one that fails outright — unreachable endpoint, rejected auth — logs - `OTEL span flush failed (); some spans may not have been exported.` - -The queue drains continuously, not at the end of a run: the processor exports up -to `ORQ_OTEL_MAX_BATCH_SIZE` spans every `ORQ_OTEL_SCHEDULE_DELAY_MS`, which on -the defaults sustains roughly 512 spans per 5 seconds. Back-to-back runs do not -accumulate. Raising `ORQ_OTEL_MAX_QUEUE_SIZE` therefore buys *stall tolerance* — -queue size divided by that sustained rate is how many seconds of exporter stall -you survive — rather than extra capacity. When spans go missing and the endpoint -is healthy, the lever is a lower `ORQ_OTEL_SCHEDULE_DELAY_MS`. +The defaults suit one-off runs. Tune them for a long-lived worker process, or for a +CI job where losing spans is unacceptable. The four `ORQ_OTEL_*` variables named +below are listed with their defaults in +[Configuration › Environment variables](configuration.md#environment-variables). + +The `TracerProvider` lives for the whole process. In a long-lived process that runs +many evaluations, red-team or simulation runs back to back, the same span queue +serves every run. These settings behave the same whether you export to Orq or to a +third-party collector; only endpoint latency differs. + +Spans are dropped two ways. Neither fails the run, so the loss shows up in your logs +rather than your exit code: + +- **Queue overflow** — spans arrive faster than the exporter drains them. A burst of + parallel jobs (the `datapoint_parallelism` argument) or a slow endpoint will do it. + Once the queue is full the SDK evicts the *oldest* buffered span to make room, so a + trace comes back missing its early spans rather than its last ones. It logs + `Queue full, dropping Span.` through the stdlib logger + `opentelemetry.sdk._shared_internal`. Identical warnings are suppressed within + 20-second buckets, so one line means loss started, not that exactly one span was + lost. Raise `ORQ_OTEL_MAX_QUEUE_SIZE` when it appears. +- **Exit before flush** — `evaluatorq()`, `red_team()`, `simulate()` and + `generate_and_simulate()` each force-flush in a `finally`, so a run that raises + still flushes. Argument validation that fails before the run's tracing scope opens + does not, and the standalone pairwise entry points do not flush at all. The SDK's + `atexit` hook flushes again on a clean shutdown. What survives none of that is a + flush that hits its timeout, or a hard `SIGKILL` — an OOM kill, a CI job cancelled + mid-run — which drops the buffer with no warning at all. A flush that times out + logs `OTEL span flush timed out after ms; some spans may not have been + exported.` and one that fails outright logs `OTEL span flush failed (); + some spans may not have been exported.` + +The queue drains continuously, not at the end of a run, and it does not drain on a +fixed cadence. `ORQ_OTEL_SCHEDULE_DELAY_MS` is the idle timer: it decides how long a +*partial* batch waits before going out. As soon as the queue reaches +`ORQ_OTEL_MAX_BATCH_SIZE` the exporter wakes immediately and keeps exporting while +the queue stays above that threshold. Two consequences follow. Under a burst, +lowering the schedule delay changes nothing — throughput is bounded by how fast the +collector accepts batches. And raising `ORQ_OTEL_MAX_QUEUE_SIZE` buys tolerance for +bursts and for a stalled exporter; it does not raise throughput. Lower +`ORQ_OTEL_SCHEDULE_DELAY_MS` when a low-volume run leaves spans sitting in a partial +batch for too long. `ORQ_OTEL_MAX_QUEUE_SIZE`, `ORQ_OTEL_MAX_BATCH_SIZE` and -`ORQ_OTEL_SCHEDULE_DELAY_MS` are baked into the processor when tracing -initializes and cannot change for the life of the process; a batch size larger -than the queue size is clamped down to it with a `WARNING`. -`ORQ_OTEL_FLUSH_TIMEOUT_MS` is read on every flush, so a long-lived host can raise -it before a big run — it bounds the end-of-run flush only, while the per-request -export timeout above is a fixed 5 seconds and is not tunable. Each value must be a -positive integer: a set-but-invalid value (empty, non-numeric, zero or negative) -is ignored with a `WARNING` and the default is used, so a typo never takes tracing -down but never silently takes effect either. The SDK's own `OTEL_BSP_*` variables -have no effect — evaluatorq passes these values to the processor explicitly, which -takes precedence over the SDK's env-var defaults. - -A CI run that must not lose spans drains the queue more eagerly and allows a -longer final flush: +`ORQ_OTEL_SCHEDULE_DELAY_MS` are fixed into the processor when tracing initializes +and cannot change for the life of the process. A batch size larger than the queue +size is clamped down to it, with a warning. `ORQ_OTEL_FLUSH_TIMEOUT_MS` is read on +every flush, so a long-running process can raise it before a big run; it bounds the +end-of-run flush only, and the per-request export timeout stays fixed at 5 seconds. +evaluatorq enforces that bound itself, because the SDK currently ignores the timeout +handed to `force_flush`. + +Each value must be a positive integer. An empty, non-numeric, zero or negative value +is ignored: evaluatorq logs a warning and uses the default, so a typo neither +disables tracing nor takes effect unnoticed. + +For CI, absorb the burst and allow a longer final flush: ```bash export ORQ_API_KEY="your_orq_api_key" # enables tracing export ORQ_OTEL_MAX_QUEUE_SIZE=16384 # absorb a burst of parallel jobs -export ORQ_OTEL_SCHEDULE_DELAY_MS=1000 # export every second, not every five export ORQ_OTEL_FLUSH_TIMEOUT_MS=30000 # give the final flush up to 30s ``` -In a GitHub Action the same values go straight in the workflow's `env:` block: +In a GitHub Actions workflow, set the same values in the job's `env:` block: ```yaml env: ORQ_API_KEY: ${{ secrets.ORQ_API_KEY }} ORQ_OTEL_MAX_QUEUE_SIZE: 16384 - ORQ_OTEL_SCHEDULE_DELAY_MS: 1000 ORQ_OTEL_FLUSH_TIMEOUT_MS: 30000 ``` -The flush timeout is an upper bound on wall-clock added to the job: a run whose -exporter has become unreachable waits the full 30 seconds before warning and -moving on. +The flush timeout is an upper bound on the wall-clock time added to the job. If the +collector stops responding, the run waits the full 30 seconds, logs a warning, and +continues. ## Span hierarchy diff --git a/src/evaluatorq/tracing/setup.py b/src/evaluatorq/tracing/setup.py index 658f187a..e79de1e4 100644 --- a/src/evaluatorq/tracing/setup.py +++ b/src/evaluatorq/tracing/setup.py @@ -25,8 +25,10 @@ ``ORQ_OTEL_SCHEDULE_DELAY_MS`` (default 5000), and ``ORQ_OTEL_MAX_BATCH_SIZE`` (default 512, clamped to the queue size with a warning). These are passed to the processor explicitly, so the SDK's own ``OTEL_BSP_*`` variables have no effect. -- ``ORQ_OTEL_FLUSH_TIMEOUT_MS`` controls per-run force-flush (default 5000). A timeout - logs a warning because some spans may not have been exported. +- ``ORQ_OTEL_FLUSH_TIMEOUT_MS`` bounds the per-run force-flush (default 5000). The SDK + ignores the timeout it is handed, so ``flush_tracing`` enforces it with + ``asyncio.wait_for``; a timeout logs a warning because some spans may not have been + exported. - Exporter headers are bound at initialization: changing ``ORQ_API_KEY`` requires a process restart. A hard ``SIGKILL`` can lose spans still buffered by the batch exporter. """ @@ -215,10 +217,10 @@ async def init_tracing_if_needed() -> bool: # noqa: RUF029 ) # Use BatchSpanProcessor to export spans asynchronously in batches. - # Queue/scheduling are env-tunable: in a long-lived process (e.g. the - # dashboard) that runs many red-team/simulation runs without ever tearing - # the provider down, the default 2048-span queue can overflow and silently - # drop spans. Larger defaults + env overrides reduce that risk. + # Queue/scheduling are env-tunable: in a long-lived worker process that + # runs many red-team/simulation runs without ever tearing the provider + # down, the default 2048-span queue can overflow and drop spans. Larger + # defaults + env overrides reduce that risk. max_queue_size = _env_int('ORQ_OTEL_MAX_QUEUE_SIZE', 4096) requested_batch_size = _env_int('ORQ_OTEL_MAX_BATCH_SIZE', 512) batch_size = min(requested_batch_size, max_queue_size) @@ -265,22 +267,38 @@ async def flush_tracing() -> None: Force flush all pending spans, blocking until export completes or times out. ``force_flush`` is a synchronous, blocking SDK call, so it runs on a worker - thread to avoid stalling the event loop. A ``False`` return means the flush - timed out with spans still unexported; a raised exception means the export - failed outright — both leave spans unexported and are surfaced as warnings - rather than silently dropped. + thread to avoid stalling the event loop. + + ``ORQ_OTEL_FLUSH_TIMEOUT_MS`` is enforced here with ``asyncio.wait_for``, not + by the SDK: ``BatchProcessor.force_flush`` currently ignores its + ``timeout_millis`` argument and unconditionally returns ``True`` + (https://github.com/open-telemetry/opentelemetry-python/issues/4568), so + passing the timeout down bounds nothing. Without the wrapper a run whose + collector has stopped responding waits for the export to finish, however long + that takes. The timeout only stops this coroutine *waiting*; the exporter + thread is a daemon and keeps draining behind it, so a late flush may still + land. The ``ok is False`` branch stays for an SDK that does honour the + argument. + + A timeout leaves spans unexported and a raised exception means the export + failed outright — both are surfaced as warnings rather than silently dropped. + Exporter errors raised inside ``_export`` are caught and logged by the SDK, + so they reach neither branch. """ if _sdk is None: return provider = _sdk # TracerProvider timeout_ms = _env_int('ORQ_OTEL_FLUSH_TIMEOUT_MS', 5000) + timed_out = 'OTEL span flush timed out after {}ms; some spans may not have been exported.' try: - ok = await asyncio.to_thread(provider.force_flush, timeout_ms) + ok = await asyncio.wait_for( + asyncio.to_thread(provider.force_flush, timeout_ms), + timeout=timeout_ms / 1000, + ) if ok is False: - logger.warning( - 'OTEL span flush timed out after {}ms; some spans may not have been exported.', - timeout_ms, - ) + logger.warning(timed_out, timeout_ms) + except asyncio.TimeoutError: + logger.warning(timed_out, timeout_ms) except Exception as e: logger.warning('OTEL span flush failed ({}); some spans may not have been exported.', e) diff --git a/tests/common/test_tracing_lifecycle.py b/tests/common/test_tracing_lifecycle.py index 69bac0bf..10801687 100644 --- a/tests/common/test_tracing_lifecycle.py +++ b/tests/common/test_tracing_lifecycle.py @@ -206,6 +206,8 @@ async def test_initialization_uses_documented_batching_defaults( ) -> None: """Pin the defaults published in docs/tracing.md and docs/configuration.md.""" processor_options = _fake_tracing_sdk(monkeypatch) + warning = Mock() + monkeypatch.setattr(tracing_setup.logger, 'warning', warning) for name in ( 'ORQ_OTEL_MAX_QUEUE_SIZE', 'ORQ_OTEL_MAX_BATCH_SIZE', @@ -219,6 +221,8 @@ async def test_initialization_uses_documented_batching_defaults( 'max_export_batch_size': 512, 'schedule_delay_millis': 5000, } + # The clamp warning is conditional: defaults must not trip it. + warning.assert_not_called() @pytest.mark.asyncio @@ -275,6 +279,27 @@ async def test_flush_tracing_does_not_warn_on_success(monkeypatch: pytest.Monkey warning.assert_not_called() +@pytest.mark.asyncio +async def test_flush_tracing_bounds_a_hanging_force_flush(monkeypatch: pytest.MonkeyPatch) -> None: + """The SDK ignores the timeout it is handed, so ``asyncio.wait_for`` enforces it.""" + warning = Mock() + + async def never_returns(*_args: object, **_kwargs: object) -> bool: + await asyncio.sleep(30) + return True + + monkeypatch.setattr(tracing_setup, '_sdk', Mock()) + monkeypatch.setattr(tracing_setup.asyncio, 'to_thread', never_returns) + monkeypatch.setattr(tracing_setup.logger, 'warning', warning) + monkeypatch.setenv('ORQ_OTEL_FLUSH_TIMEOUT_MS', '10') + + await tracing_setup.flush_tracing() + + warning.assert_called_once_with( + 'OTEL span flush timed out after {}ms; some spans may not have been exported.', 10 + ) + + @pytest.mark.asyncio async def test_flush_tracing_warns_when_force_flush_raises(monkeypatch: pytest.MonkeyPatch) -> None: warning = Mock() From e18c87cf780729e95053156bed5cea4040e65e2d Mon Sep 17 00:00:00 2001 From: Bauke Brenninkmeijer Date: Fri, 28 Aug 2026 11:59:29 +0200 Subject: [PATCH 6/9] docs: correct the red-team span tree and judge span naming Both defects are contradicted by `redteam/tracing.py`'s own module docstring, which the page was written against but drifted from. - The tree drew `orq.redteam.security_evaluation` between `orq.evaluation` and the judge's LLM span. That span does not exist: the OWASP scorer calls `annotate_current_span`, tagging the framework's `orq.evaluation` span in place, precisely to avoid that layer. - `attack_turn`, `adversarial_generation`, `context_retrieval` and `datapoint_generation` were shown as the general red-team shape. Static mode is single-shot and emits none of them. The page now shows both trees and says which mode each belongs to. - Judge LLM spans were documented as `chat {model}` in the red-team tree, the jury tree and the trace-context seam paragraph. `EvaluatorConfig.api` defaults to `'responses'` (the endpoint the Orq router prices), so the span is normally `responses {provider}/{model}`; `chat` is the fallback. `gen_ai.operation.name` documented both values. `annotate_current_span`'s docstring said "the judge `chat` span" for the same reason and is updated with it. Co-Authored-By: Claude Opus 5 (1M context) --- docs/tracing.md | 42 +++++++++++++++++++++++++------ src/evaluatorq/redteam/tracing.py | 4 +-- 2 files changed, 37 insertions(+), 9 deletions(-) diff --git a/docs/tracing.md b/docs/tracing.md index 91d010be..1e1ed273 100644 --- a/docs/tracing.md +++ b/docs/tracing.md @@ -238,6 +238,8 @@ Span attributes on `orq.evaluation`: ### Red teaming spans +This is the **dynamic** and **hybrid** shape: + ``` Evaluatorq - Red Teaming # root — one per red_team() call ├── orq.redteam.context_retrieval @@ -254,17 +256,40 @@ Evaluatorq - Red Teaming # root — one per red_team() call │ ├── orq.redteam.adversarial_generation │ │ └── chat (llm_purpose=adversarial) │ └── orq.redteam.target_call - ├── orq.evaluation # security evaluator result - │ └── orq.redteam.security_evaluation - │ └── chat (llm_purpose=evaluation) + ├── orq.evaluation # security evaluator result, annotated in place + │ └── responses (llm_purpose=evaluation) └── orq.redteam.memory_cleanup # post-run agent memory entity cleanup (only when cleanup is enabled, entities exist, and the target has configured memory stores) ``` -LLM spans (`chat ...`) carry standard GenAI attributes: +**Static mode is single-shot**, and its tree is correspondingly shorter: no +`context_retrieval` or `datapoint_generation` work, and no `attack_turn` or +`adversarial_generation` spans. One `target_call` per attack, then the evaluation: + +``` +Evaluatorq - Red Teaming + ├── orq.job + │ └── orq.redteam.attack + │ └── orq.redteam.target_call + ├── orq.evaluation + │ └── responses (llm_purpose=evaluation) + └── orq.redteam.memory_cleanup +``` + +There is no `orq.redteam.security_evaluation` span. The OWASP scorer annotates the +framework's own `orq.evaluation` span in place rather than nesting a redundant layer +between it and the judge's LLM span. + +The judge's span is named for the endpoint that served it. Evaluator configs default +to `api='responses'`, so it is usually `responses {provider}/{model}`; it is +`chat {model}` when the call falls back to Chat Completions — a non-router client, +`structured_output=False`, a model the catalogue cannot qualify as +Responses-capable, or `api='chat_completions'` set explicitly. + +LLM spans (`chat ...` / `responses ...`) carry standard GenAI attributes: | Attribute | Value | |---|---| -| `gen_ai.operation.name` | Operation name (e.g. `"chat"`) | +| `gen_ai.operation.name` | Operation name (`"chat"` or `"responses"`) | | `gen_ai.system` | Provider name | | `gen_ai.request.model` | Model identifier | | `gen_ai.usage.input_tokens` | Prompt token count | @@ -403,9 +428,12 @@ hierarchy: orq.evaluation {evaluator} # from the core runner, when a jury backs an evaluator └── orq.jury # one per deliberation (orq.pairwise_jury in comparative mode) └── orq.judge # one per judge (x2 in comparative mode — see below) - └── chat {model} # the judge's own LLM call(s), tagged orq.llm.purpose="judge" + └── responses {model} # the judge's own LLM call(s), tagged orq.llm.purpose="judge" ``` +The leaf span is `responses {provider}/{model}` on the default `api='responses'`, and +`chat {model}` when the call falls back to Chat Completions. + The panel opens no span of its own outside `orq.jury` — it can equally be called standalone (not nested under `orq.evaluation`), in which case `orq.jury` is the root. All jury/judge spans are opened via `evaluatorq.common.tracing`'s @@ -520,7 +548,7 @@ judges with no reconciled vote *and* no flip (i.e. one or both orderings raised an error). A judge can be flipped, failed, or a normal decisive vote, but never counted under more than one of those buckets. -`orq.evaluation`, `orq.jury` / `orq.pairwise_jury`, `orq.judge`, and the nested `chat {model}` LLM +`orq.evaluation`, `orq.jury` / `orq.pairwise_jury`, `orq.judge`, and the nested LLM spans follow the ambient OTel context — nothing threads an explicit parent across the `orq.evaluation` → `orq.jury` seam, so a jury backing a custom evaluator's scorer nests correctly without extra plumbing. diff --git a/src/evaluatorq/redteam/tracing.py b/src/evaluatorq/redteam/tracing.py index 0f071043..c1cefd8c 100644 --- a/src/evaluatorq/redteam/tracing.py +++ b/src/evaluatorq/redteam/tracing.py @@ -131,8 +131,8 @@ async def annotate_current_span( # noqa: RUF029 Used by evaluator scorers: the evaluatorq framework already runs them inside the ``orq.evaluation`` evaluator span (via ``start_as_current_span``), which carries the verdict/score/explanation. Annotating that span directly avoids a - redundant ``orq.redteam.security_evaluation`` layer between it and the judge - ``chat`` span. + redundant ``orq.redteam.security_evaluation`` layer between it and the judge's + own LLM span (``responses`` by default, ``chat`` on fallback). Yields: The current span when tracing is enabled, None otherwise. From 0a0f8952336176be82da06d7b5d13a17ea1e4536 Mon Sep 17 00:00:00 2001 From: Bauke Brenninkmeijer Date: Fri, 28 Aug 2026 12:06:03 +0200 Subject: [PATCH 7/9] docs: restore the batching table and unwrap the new prose Puts the four-variable reference back on the tracing page, where a reader tuning span export actually is. It earns its place against the canonical table in configuration.md by carrying two columns that one cannot: when each value is read (at init versus per run), and which direction to move it and on what signal. The prose that previously carried those facts in narrative form is shorter for it. Paragraphs are one line each, matching the reflow main applied to docs-coverage/axes.md, and leaving diffs on a sentence edit instead of a reflowed block. Co-Authored-By: Claude Opus 5 (1M context) --- docs/faq.md | 8 +--- docs/tracing.md | 98 +++++++++++++------------------------------------ 2 files changed, 26 insertions(+), 80 deletions(-) diff --git a/docs/faq.md b/docs/faq.md index 5da1cf36..0985fc58 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -80,13 +80,7 @@ Runs auto-save locally (red-team runs to `.evaluatorq/runs/`; simulation runs to ### Some spans are missing from my traces -The span exporter batches in the background, so spans can be lost two ways, neither -of which fails the run. Either the in-memory queue overflowed — spans produced faster -than the exporter drained them — or the process exited before the final flush -finished. Raise `ORQ_OTEL_MAX_QUEUE_SIZE` for the first and -`ORQ_OTEL_FLUSH_TIMEOUT_MS` for the second. Both log a warning; a hard `SIGKILL` -drops whatever was still buffered without one. See -[Tracing › Batching and flush](tracing.md#batching-and-flush). +The span exporter batches in the background, so spans can be lost two ways, neither of which fails the run. Either the in-memory queue overflowed — spans produced faster than the exporter drained them — or the process exited before the final flush finished. Raise `ORQ_OTEL_MAX_QUEUE_SIZE` for the first and `ORQ_OTEL_FLUSH_TIMEOUT_MS` for the second. Both log a warning; a hard `SIGKILL` drops whatever was still buffered without one. See [Tracing › Batching and flush](tracing.md#batching-and-flush). ### How do I run a plain evaluation? diff --git a/docs/tracing.md b/docs/tracing.md index 1e1ed273..8899337a 100644 --- a/docs/tracing.md +++ b/docs/tracing.md @@ -85,8 +85,7 @@ errors to stdout. - **Protocol**: HTTP/protobuf (`OTLPSpanExporter` from `opentelemetry-exporter-otlp-proto-http`) - **Export mode**: `BatchSpanProcessor` (asynchronous batching) -- **Timeout**: 5 seconds per export request. The separate end-of-run flush timeout - is set with `ORQ_OTEL_FLUSH_TIMEOUT_MS` (see below). +- **Timeout**: 5 seconds per export request. The separate end-of-run flush timeout is set with `ORQ_OTEL_FLUSH_TIMEOUT_MS` (see below). - **Auth**: `Authorization: Bearer ` is added automatically when the resolved endpoint's hostname ends in `.orq.ai` or is exactly `orq.ai`. For any other endpoint the header is not added; use `OTEL_EXPORTER_OTLP_HEADERS` @@ -96,61 +95,25 @@ errors to stdout. ### Batching and flush -The defaults suit one-off runs. Tune them for a long-lived worker process, or for a -CI job where losing spans is unacceptable. The four `ORQ_OTEL_*` variables named -below are listed with their defaults in -[Configuration › Environment variables](configuration.md#environment-variables). - -The `TracerProvider` lives for the whole process. In a long-lived process that runs -many evaluations, red-team or simulation runs back to back, the same span queue -serves every run. These settings behave the same whether you export to Orq or to a -third-party collector; only endpoint latency differs. - -Spans are dropped two ways. Neither fails the run, so the loss shows up in your logs -rather than your exit code: - -- **Queue overflow** — spans arrive faster than the exporter drains them. A burst of - parallel jobs (the `datapoint_parallelism` argument) or a slow endpoint will do it. - Once the queue is full the SDK evicts the *oldest* buffered span to make room, so a - trace comes back missing its early spans rather than its last ones. It logs - `Queue full, dropping Span.` through the stdlib logger - `opentelemetry.sdk._shared_internal`. Identical warnings are suppressed within - 20-second buckets, so one line means loss started, not that exactly one span was - lost. Raise `ORQ_OTEL_MAX_QUEUE_SIZE` when it appears. -- **Exit before flush** — `evaluatorq()`, `red_team()`, `simulate()` and - `generate_and_simulate()` each force-flush in a `finally`, so a run that raises - still flushes. Argument validation that fails before the run's tracing scope opens - does not, and the standalone pairwise entry points do not flush at all. The SDK's - `atexit` hook flushes again on a clean shutdown. What survives none of that is a - flush that hits its timeout, or a hard `SIGKILL` — an OOM kill, a CI job cancelled - mid-run — which drops the buffer with no warning at all. A flush that times out - logs `OTEL span flush timed out after ms; some spans may not have been - exported.` and one that fails outright logs `OTEL span flush failed (); - some spans may not have been exported.` - -The queue drains continuously, not at the end of a run, and it does not drain on a -fixed cadence. `ORQ_OTEL_SCHEDULE_DELAY_MS` is the idle timer: it decides how long a -*partial* batch waits before going out. As soon as the queue reaches -`ORQ_OTEL_MAX_BATCH_SIZE` the exporter wakes immediately and keeps exporting while -the queue stays above that threshold. Two consequences follow. Under a burst, -lowering the schedule delay changes nothing — throughput is bounded by how fast the -collector accepts batches. And raising `ORQ_OTEL_MAX_QUEUE_SIZE` buys tolerance for -bursts and for a stalled exporter; it does not raise throughput. Lower -`ORQ_OTEL_SCHEDULE_DELAY_MS` when a low-volume run leaves spans sitting in a partial -batch for too long. - -`ORQ_OTEL_MAX_QUEUE_SIZE`, `ORQ_OTEL_MAX_BATCH_SIZE` and -`ORQ_OTEL_SCHEDULE_DELAY_MS` are fixed into the processor when tracing initializes -and cannot change for the life of the process. A batch size larger than the queue -size is clamped down to it, with a warning. `ORQ_OTEL_FLUSH_TIMEOUT_MS` is read on -every flush, so a long-running process can raise it before a big run; it bounds the -end-of-run flush only, and the per-request export timeout stays fixed at 5 seconds. -evaluatorq enforces that bound itself, because the SDK currently ignores the timeout -handed to `force_flush`. - -Each value must be a positive integer. An empty, non-numeric, zero or negative value -is ignored: evaluatorq logs a warning and uses the default, so a typo neither -disables tracing nor takes effect unnoticed. +The defaults suit one-off runs. Tune them for a long-lived worker process, or for a CI job where losing spans is unacceptable. + +| Variable | Default | Read | Raise or lower it when | +|---|---|---|---| +| `ORQ_OTEL_MAX_QUEUE_SIZE` | `4096` | at init | **Raise** after a `Queue full, dropping Span.` warning. Buys tolerance for bursts and for a stalled exporter; it does not raise throughput. | +| `ORQ_OTEL_MAX_BATCH_SIZE` | `512` | at init | **Raise** to send fewer, larger export requests. Reaching it wakes the exporter immediately. Clamped down to the queue size, with a warning. | +| `ORQ_OTEL_SCHEDULE_DELAY_MS` | `5000` | at init | **Lower** when a low-volume run leaves spans sitting in a partial batch. It does not throttle a full batch, so it does not help under a burst. | +| `ORQ_OTEL_FLUSH_TIMEOUT_MS` | `5000` | per run | **Raise** when the end-of-run flush warns before finishing. Bounds the final flush only; the per-request export timeout stays fixed at 5 seconds. | + +The first three are fixed into the processor when tracing initializes and cannot change for the life of the process. `ORQ_OTEL_FLUSH_TIMEOUT_MS` is read on every flush, so a long-running process can raise it before a big run — and evaluatorq enforces that bound itself, because the SDK currently ignores the timeout handed to `force_flush`. Each value must be a positive integer: an empty, non-numeric, zero or negative value is ignored, and evaluatorq logs a warning and uses the default, so a typo neither disables tracing nor takes effect unnoticed. + +The `TracerProvider` lives for the whole process. In a long-lived process that runs many evaluations, red-team or simulation runs back to back, the same span queue serves every run. These settings behave the same whether you export to Orq or to a third-party collector; only endpoint latency differs. + +Spans are dropped two ways. Neither fails the run, so the loss shows up in your logs rather than your exit code: + +- **Queue overflow** — spans arrive faster than the exporter drains them. A burst of parallel jobs (the `datapoint_parallelism` argument) or a slow endpoint will do it. Once the queue is full the SDK evicts the *oldest* buffered span to make room, so a trace comes back missing its early spans rather than its last ones. It logs `Queue full, dropping Span.` through the stdlib logger `opentelemetry.sdk._shared_internal`. Identical warnings are suppressed within 20-second buckets, so one line means loss started, not that exactly one span was lost. +- **Exit before flush** — `evaluatorq()`, `red_team()`, `simulate()` and `generate_and_simulate()` each force-flush in a `finally`, so a run that raises still flushes. Argument validation that fails before the run's tracing scope opens does not, and the standalone pairwise entry points do not flush at all. The SDK's `atexit` hook flushes again on a clean shutdown. What survives none of that is a flush that hits its timeout, or a hard `SIGKILL` — an OOM kill, a CI job cancelled mid-run — which drops the buffer with no warning at all. A flush that times out logs `OTEL span flush timed out after ms; some spans may not have been exported.` and one that fails outright logs `OTEL span flush failed (); some spans may not have been exported.` + +The queue drains continuously, not at the end of a run, and it does not drain on a fixed cadence. `ORQ_OTEL_SCHEDULE_DELAY_MS` is the idle timer: it decides how long a *partial* batch waits before going out. As soon as the queue reaches `ORQ_OTEL_MAX_BATCH_SIZE` the exporter wakes immediately and keeps exporting while the queue stays above that threshold. Under a burst, then, lowering the schedule delay changes nothing — throughput is bounded by how fast the collector accepts batches. For CI, absorb the burst and allow a longer final flush: @@ -169,9 +132,7 @@ env: ORQ_OTEL_FLUSH_TIMEOUT_MS: 30000 ``` -The flush timeout is an upper bound on the wall-clock time added to the job. If the -collector stops responding, the run waits the full 30 seconds, logs a warning, and -continues. +The flush timeout is an upper bound on the wall-clock time added to the job. If the collector stops responding, the run waits the full 30 seconds, logs a warning, and continues. ## Span hierarchy @@ -261,9 +222,7 @@ Evaluatorq - Red Teaming # root — one per red_team() call └── orq.redteam.memory_cleanup # post-run agent memory entity cleanup (only when cleanup is enabled, entities exist, and the target has configured memory stores) ``` -**Static mode is single-shot**, and its tree is correspondingly shorter: no -`context_retrieval` or `datapoint_generation` work, and no `attack_turn` or -`adversarial_generation` spans. One `target_call` per attack, then the evaluation: +**Static mode is single-shot**, and its tree is correspondingly shorter: no `context_retrieval` or `datapoint_generation` work, and no `attack_turn` or `adversarial_generation` spans. One `target_call` per attack, then the evaluation: ``` Evaluatorq - Red Teaming @@ -275,15 +234,9 @@ Evaluatorq - Red Teaming └── orq.redteam.memory_cleanup ``` -There is no `orq.redteam.security_evaluation` span. The OWASP scorer annotates the -framework's own `orq.evaluation` span in place rather than nesting a redundant layer -between it and the judge's LLM span. +There is no `orq.redteam.security_evaluation` span. The OWASP scorer annotates the framework's own `orq.evaluation` span in place rather than nesting a redundant layer between it and the judge's LLM span. -The judge's span is named for the endpoint that served it. Evaluator configs default -to `api='responses'`, so it is usually `responses {provider}/{model}`; it is -`chat {model}` when the call falls back to Chat Completions — a non-router client, -`structured_output=False`, a model the catalogue cannot qualify as -Responses-capable, or `api='chat_completions'` set explicitly. +The judge's span is named for the endpoint that served it. Evaluator configs default to `api='responses'`, so it is usually `responses {provider}/{model}`; it is `chat {model}` when the call falls back to Chat Completions — a non-router client, `structured_output=False`, a model the catalogue cannot qualify as Responses-capable, or `api='chat_completions'` set explicitly. LLM spans (`chat ...` / `responses ...`) carry standard GenAI attributes: @@ -431,8 +384,7 @@ orq.evaluation {evaluator} # from the core runner, when a jury backs an └── responses {model} # the judge's own LLM call(s), tagged orq.llm.purpose="judge" ``` -The leaf span is `responses {provider}/{model}` on the default `api='responses'`, and -`chat {model}` when the call falls back to Chat Completions. +The leaf span is `responses {provider}/{model}` on the default `api='responses'`, and `chat {model}` when the call falls back to Chat Completions. The panel opens no span of its own outside `orq.jury` — it can equally be called standalone (not nested under `orq.evaluation`), in which case `orq.jury` From 7085f25390c273660c90224b854b7d02ddcb2e5f Mon Sep 17 00:00:00 2001 From: Bauke Brenninkmeijer Date: Fri, 28 Aug 2026 12:24:33 +0200 Subject: [PATCH 8/9] docs: trim added comments and smooth the batching prose comment-scrub over the added comments: the batch-processor comment drops from four lines to two, and the test's export-guard note to one. Two one-line test comments stay; each says why an assertion exists, which the assertion cannot. humanizer over the added prose: "a typo neither disables tracing nor takes effect unnoticed" was a negative parallelism and now says what does happen; "What survives none of that is..." was written as a closing line rather than a statement; and two sentences that announced their own inference say it plainly. Em dashes are kept deliberately. The skill treats them as a hard cut, but its voice-calibration rule puts a writing sample above that, and the corpus is the sample: 777 across docs/, and this section sits below the density of tuning.md or guides/red-teaming.md. Stripping them here alone would make the section read as though a different person wrote it. Co-Authored-By: Claude Opus 5 (1M context) --- docs/tracing.md | 8 ++++---- src/evaluatorq/tracing/setup.py | 6 ++---- tests/common/test_tracing_lifecycle.py | 3 +-- 3 files changed, 7 insertions(+), 10 deletions(-) diff --git a/docs/tracing.md b/docs/tracing.md index 8899337a..cc563a4c 100644 --- a/docs/tracing.md +++ b/docs/tracing.md @@ -104,16 +104,16 @@ The defaults suit one-off runs. Tune them for a long-lived worker process, or fo | `ORQ_OTEL_SCHEDULE_DELAY_MS` | `5000` | at init | **Lower** when a low-volume run leaves spans sitting in a partial batch. It does not throttle a full batch, so it does not help under a burst. | | `ORQ_OTEL_FLUSH_TIMEOUT_MS` | `5000` | per run | **Raise** when the end-of-run flush warns before finishing. Bounds the final flush only; the per-request export timeout stays fixed at 5 seconds. | -The first three are fixed into the processor when tracing initializes and cannot change for the life of the process. `ORQ_OTEL_FLUSH_TIMEOUT_MS` is read on every flush, so a long-running process can raise it before a big run — and evaluatorq enforces that bound itself, because the SDK currently ignores the timeout handed to `force_flush`. Each value must be a positive integer: an empty, non-numeric, zero or negative value is ignored, and evaluatorq logs a warning and uses the default, so a typo neither disables tracing nor takes effect unnoticed. +The first three are fixed into the processor when tracing initializes and cannot change for the life of the process. `ORQ_OTEL_FLUSH_TIMEOUT_MS` is read on every flush, so a long-running process can raise it before a big run — and evaluatorq enforces that bound itself, because the SDK currently ignores the timeout handed to `force_flush`. Each value must be a positive integer. An empty, non-numeric, zero or negative value is ignored: evaluatorq logs a warning and falls back to the default, so a typo leaves tracing running on a value you can see in the logs. The `TracerProvider` lives for the whole process. In a long-lived process that runs many evaluations, red-team or simulation runs back to back, the same span queue serves every run. These settings behave the same whether you export to Orq or to a third-party collector; only endpoint latency differs. -Spans are dropped two ways. Neither fails the run, so the loss shows up in your logs rather than your exit code: +Spans are dropped two ways, and neither one fails the run, so the loss shows up in your logs instead of your exit code: - **Queue overflow** — spans arrive faster than the exporter drains them. A burst of parallel jobs (the `datapoint_parallelism` argument) or a slow endpoint will do it. Once the queue is full the SDK evicts the *oldest* buffered span to make room, so a trace comes back missing its early spans rather than its last ones. It logs `Queue full, dropping Span.` through the stdlib logger `opentelemetry.sdk._shared_internal`. Identical warnings are suppressed within 20-second buckets, so one line means loss started, not that exactly one span was lost. -- **Exit before flush** — `evaluatorq()`, `red_team()`, `simulate()` and `generate_and_simulate()` each force-flush in a `finally`, so a run that raises still flushes. Argument validation that fails before the run's tracing scope opens does not, and the standalone pairwise entry points do not flush at all. The SDK's `atexit` hook flushes again on a clean shutdown. What survives none of that is a flush that hits its timeout, or a hard `SIGKILL` — an OOM kill, a CI job cancelled mid-run — which drops the buffer with no warning at all. A flush that times out logs `OTEL span flush timed out after ms; some spans may not have been exported.` and one that fails outright logs `OTEL span flush failed (); some spans may not have been exported.` +- **Exit before flush** — `evaluatorq()`, `red_team()`, `simulate()` and `generate_and_simulate()` each force-flush in a `finally`, so a run that raises still flushes. Argument validation that fails before the run's tracing scope opens does not, and the standalone pairwise entry points do not flush at all. The SDK's `atexit` hook flushes again on a clean shutdown. Neither helps against a flush that hits its timeout, or against a hard `SIGKILL` such as an OOM kill or a CI job cancelled mid-run, which drops the buffer without a warning. A flush that times out logs `OTEL span flush timed out after ms; some spans may not have been exported.` and one that fails outright logs `OTEL span flush failed (); some spans may not have been exported.` -The queue drains continuously, not at the end of a run, and it does not drain on a fixed cadence. `ORQ_OTEL_SCHEDULE_DELAY_MS` is the idle timer: it decides how long a *partial* batch waits before going out. As soon as the queue reaches `ORQ_OTEL_MAX_BATCH_SIZE` the exporter wakes immediately and keeps exporting while the queue stays above that threshold. Under a burst, then, lowering the schedule delay changes nothing — throughput is bounded by how fast the collector accepts batches. +The queue drains continuously, not at the end of a run, and it does not drain on a fixed cadence. `ORQ_OTEL_SCHEDULE_DELAY_MS` is the idle timer: it decides how long a *partial* batch waits before going out. As soon as the queue reaches `ORQ_OTEL_MAX_BATCH_SIZE` the exporter wakes immediately and keeps exporting while the queue stays above that threshold. Under a burst, lowering the schedule delay changes nothing, because throughput is bounded by how fast the collector accepts batches. For CI, absorb the burst and allow a longer final flush: diff --git a/src/evaluatorq/tracing/setup.py b/src/evaluatorq/tracing/setup.py index e79de1e4..d2a5866e 100644 --- a/src/evaluatorq/tracing/setup.py +++ b/src/evaluatorq/tracing/setup.py @@ -217,10 +217,8 @@ async def init_tracing_if_needed() -> bool: # noqa: RUF029 ) # Use BatchSpanProcessor to export spans asynchronously in batches. - # Queue/scheduling are env-tunable: in a long-lived worker process that - # runs many red-team/simulation runs without ever tearing the provider - # down, the default 2048-span queue can overflow and drop spans. Larger - # defaults + env overrides reduce that risk. + # Env-tunable because a long-lived process never tears the provider down, + # so one queue absorbs every run and the SDK's 2048 default overflows. max_queue_size = _env_int('ORQ_OTEL_MAX_QUEUE_SIZE', 4096) requested_batch_size = _env_int('ORQ_OTEL_MAX_BATCH_SIZE', 512) batch_size = min(requested_batch_size, max_queue_size) diff --git a/tests/common/test_tracing_lifecycle.py b/tests/common/test_tracing_lifecycle.py index 10801687..ce542ab6 100644 --- a/tests/common/test_tracing_lifecycle.py +++ b/tests/common/test_tracing_lifecycle.py @@ -193,8 +193,7 @@ def add_span_processor(self, processor: object) -> None: monkeypatch.setattr(tracing_setup, '_tracer', None) monkeypatch.setattr(tracing_setup, '_is_initialized', False) monkeypatch.setattr(tracing_setup, '_initialization_attempted', False) - # Opt out of the suite-wide export guard: this drives real setup with the - # exporter and provider faked out above. + # Opt out of the suite-wide export guard: exporter and provider are faked above. monkeypatch.delenv('ORQ_DISABLE_TRACING', raising=False) monkeypatch.setenv('OTEL_EXPORTER_OTLP_ENDPOINT', 'https://example.test') return processor_options From 35cfd102af89e17fa6807e9be23d592e3b2e026d Mon Sep 17 00:00:00 2001 From: Bauke Brenninkmeijer Date: Fri, 28 Aug 2026 14:15:25 +0200 Subject: [PATCH 9/9] docs: unwrap hard-wrapped prose in tracing, configuration and faq pages One line per paragraph, list item and admonition body. Whitespace only: the word-level diff is empty. Code fences and tables are untouched. --- docs/configuration.md | 17 +-- docs/tracing.md | 259 +++++++++--------------------------------- 2 files changed, 57 insertions(+), 219 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index af2eae74..e082cc36 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -37,15 +37,9 @@ All configuration is via environment variables. No config file is required. ## Model catalogue overrides -Prices, provider ids, Responses support and accepted reasoning-effort values come -from Orq's `GET /v2/models`, fetched once per process. A model that catalogue does -not list — a self-hosted deployment, or one newer than your workspace's catalogue — -degrades silently in three ways: the call stays unpriced, `qualified_model()` sends -it to Chat Completions instead of Responses, and its reasoning effort cannot be -pre-validated. +Prices, provider ids, Responses support and accepted reasoning-effort values come from Orq's `GET /v2/models`, fetched once per process. A model that catalogue does not list — a self-hosted deployment, or one newer than your workspace's catalogue — degrades silently in three ways: the call stays unpriced, `qualified_model()` sends it to Chat Completions instead of Responses, and its reasoning effort cannot be pre-validated. -Register an entry to fix that. Registered entries take priority over the fetched -catalogue, so this also corrects an entry that is wrong: +Register an entry to fix that. Registered entries take priority over the fetched catalogue, so this also corrects an entry that is wrong: ```python from evaluatorq.common.model_catalogue import ModelInfo, get_model_info, register_model @@ -66,12 +60,7 @@ info = await get_model_info('my-self-hosted-llama') Costs are USD **per 1000 tokens**, matching what `/v2/models` publishes. -The id is stored unprefixed, so `'openai/gpt-x'` and `'gpt-x'` register and resolve -the same entry — register either spelling and both lookups find it. Registering -both replaces rather than duplicates: there is one model. `reasoning_efforts=None` -means "unknown, cannot pre-validate"; an empty set means the same thing and is -normalized to `None`, because a literally-empty accepted-values list would reject -every effort including the defaults. +The id is stored unprefixed, so `'openai/gpt-x'` and `'gpt-x'` register and resolve the same entry — register either spelling and both lookups find it. Registering both replaces rather than duplicates: there is one model. `reasoning_efforts=None` means "unknown, cannot pre-validate"; an empty set means the same thing and is normalized to `None`, because a literally-empty accepted-values list would reject every effort including the defaults. ## `.env` file diff --git a/docs/tracing.md b/docs/tracing.md index cc563a4c..3b4ae2bf 100644 --- a/docs/tracing.md +++ b/docs/tracing.md @@ -1,24 +1,17 @@ # Tracing -evaluatorq ships optional OpenTelemetry tracing. When enabled, every evaluation -run, job, evaluator, and LLM call becomes a span you can view in the Orq -dashboard or any OTLP-compatible backend. +evaluatorq ships optional OpenTelemetry tracing. When enabled, every evaluation run, job, evaluator, and LLM call becomes a span you can view in the Orq dashboard or any OTLP-compatible backend. ## How tracing is enabled -Tracing initialises lazily on the first evaluation run. It turns on automatically -when either condition is true: +Tracing initialises lazily on the first evaluation run. It turns on automatically when either condition is true: -- `ORQ_API_KEY` is set — the OTLP base endpoint is `https://my.orq.ai/v2/otel` - (or `/v2/otel` if `ORQ_BASE_URL` is set); the exporter appends - `/v1/traces`, so spans POST to `…/v2/otel/v1/traces`. +- `ORQ_API_KEY` is set — the OTLP base endpoint is `https://my.orq.ai/v2/otel` (or `/v2/otel` if `ORQ_BASE_URL` is set); the exporter appends `/v1/traces`, so spans POST to `…/v2/otel/v1/traces`. - `OTEL_EXPORTER_OTLP_ENDPOINT` is set — that endpoint is used as the OTLP base. -If neither variable is set, no tracer is created and all span context managers -are no-ops. +If neither variable is set, no tracer is created and all span context managers are no-ops. -Set `ORQ_DISABLE_TRACING=1` or `ORQ_DISABLE_TRACING=true` to suppress tracing -even when the above variables are present. +Set `ORQ_DISABLE_TRACING=1` or `ORQ_DISABLE_TRACING=true` to suppress tracing even when the above variables are present. ## Install the OTEL packages @@ -32,12 +25,9 @@ uv add opentelemetry-api opentelemetry-sdk \ uv add "evaluatorq[otel]" ``` -Prefer pip? Use `python -m pip install "evaluatorq[otel]"`, which installs into -the interpreter you just named rather than whichever `pip` happens to be first -on your `PATH`. +Prefer pip? Use `python -m pip install "evaluatorq[otel]"`, which installs into the interpreter you just named rather than whichever `pip` happens to be first on your `PATH`. -If these packages are absent the SDK silently skips initialisation — no error is -raised. +If these packages are absent the SDK silently skips initialisation — no error is raised. ## Minimal enable example @@ -77,21 +67,15 @@ To debug tracing setup: ORQ_DEBUG=1 uv run my_eval.py ``` -This prints the resolved endpoint, auth header presence, and any initialisation -errors to stdout. +This prints the resolved endpoint, auth header presence, and any initialisation errors to stdout. ## OTLP exporter details -- **Protocol**: HTTP/protobuf (`OTLPSpanExporter` from - `opentelemetry-exporter-otlp-proto-http`) +- **Protocol**: HTTP/protobuf (`OTLPSpanExporter` from `opentelemetry-exporter-otlp-proto-http`) - **Export mode**: `BatchSpanProcessor` (asynchronous batching) - **Timeout**: 5 seconds per export request. The separate end-of-run flush timeout is set with `ORQ_OTEL_FLUSH_TIMEOUT_MS` (see below). -- **Auth**: `Authorization: Bearer ` is added automatically when - the resolved endpoint's hostname ends in `.orq.ai` or is exactly `orq.ai`. - For any other endpoint the header is not added; use `OTEL_EXPORTER_OTLP_HEADERS` - to supply auth manually. -- **Custom headers**: parsed from `OTEL_EXPORTER_OTLP_HEADERS` as - `key1=value1,key2=value2`. +- **Auth**: `Authorization: Bearer ` is added automatically when the resolved endpoint's hostname ends in `.orq.ai` or is exactly `orq.ai`. For any other endpoint the header is not added; use `OTEL_EXPORTER_OTLP_HEADERS` to supply auth manually. +- **Custom headers**: parsed from `OTEL_EXPORTER_OTLP_HEADERS` as `key1=value1,key2=value2`. ### Batching and flush @@ -144,15 +128,11 @@ orq.job # one per DataPoint — root when no ambient tr └── orq.evaluation # one per evaluator applied to this job ``` -All `orq.job` spans from a single `evaluatorq()` call share the same `orq.run_id` -attribute, which ties them together as a logical run without requiring a common -parent span. Because there is no common parent, though, an N-row run arrives as -**N separate traces** — one rooted at each `orq.job`. +All `orq.job` spans from a single `evaluatorq()` call share the same `orq.run_id` attribute, which ties them together as a logical run without requiring a common parent span. Because there is no common parent, though, an N-row run arrives as **N separate traces** — one rooted at each `orq.job`. #### One trace per run: `single_trace=True` -Pass `single_trace=True` to bracket the whole run in one `evaluatorq.run` span, -so every row lands in a single trace: +Pass `single_trace=True` to bracket the whole run in one `evaluatorq.run` span, so every row lands in a single trace: ```python await evaluatorq("my-eval", data=rows, jobs=[my_job], single_trace=True) @@ -164,10 +144,7 @@ evaluatorq.run # one per evaluatorq() call — the root └── orq.evaluation ``` -It defaults to `False` so existing traces keep their shape. Red teaming and -simulation do not need the flag — they already open their own root spans -(`Evaluatorq - Red Teaming` / `Evaluatorq - Agent Simulation`), and `orq.job` -nests under those. +It defaults to `False` so existing traces keep their shape. Red teaming and simulation do not need the flag — they already open their own root spans (`Evaluatorq - Red Teaming` / `Evaluatorq - Agent Simulation`), and `orq.job` nests under those. Span attributes on `evaluatorq.run`: @@ -258,22 +235,7 @@ LLM spans (`chat ...` / `responses ...`) carry standard GenAI attributes: | `orq.llm.purpose` | Cross-domain purpose tag (e.g. `"adversarial"`, `"evaluation"`, `"target"`) | !!! note "Attribute aliases removed (August 2026, RES-985)" - Earlier releases emitted every token count under up to three names: the - canonical `gen_ai.usage.*` key above, a legacy alias - (`gen_ai.usage.prompt_tokens`, `gen_ai.usage.completion_tokens`, - `gen_ai.usage.prompt_tokens_details.cached_tokens`), and a bare - un-namespaced key (`prompt_tokens`, `completion_tokens`, `input_tokens`, - `output_tokens`, `total_tokens`, `calls`). The aliases and bare keys are no - longer emitted. This was verified against the Orq platform's OTel ingest - (`extractCommonUsage` in `orquesta-web` `apps/traces-api`): its attribute - pattern lists try the canonical `gen_ai.usage.*` spellings first, cache - counts are read from the `cache_read.input_tokens` / - `cache_creation.input_tokens` keys kept here, and the bare keys and `calls` - are read nowhere. Reasoning tokens moved from - `gen_ai.usage.completion_tokens_details.reasoning_tokens` (a spelling the - platform never read) to `gen_ai.usage.reasoning.output_tokens`, the one it - does. Third-party OTLP consumers that matched the removed aliases must - switch to the canonical keys. + Earlier releases emitted every token count under up to three names: the canonical `gen_ai.usage.*` key above, a legacy alias (`gen_ai.usage.prompt_tokens`, `gen_ai.usage.completion_tokens`, `gen_ai.usage.prompt_tokens_details.cached_tokens`), and a bare un-namespaced key (`prompt_tokens`, `completion_tokens`, `input_tokens`, `output_tokens`, `total_tokens`, `calls`). The aliases and bare keys are no longer emitted. This was verified against the Orq platform's OTel ingest (`extractCommonUsage` in `orquesta-web` `apps/traces-api`): its attribute pattern lists try the canonical `gen_ai.usage.*` spellings first, cache counts are read from the `cache_read.input_tokens` / `cache_creation.input_tokens` keys kept here, and the bare keys and `calls` are read nowhere. Reasoning tokens moved from `gen_ai.usage.completion_tokens_details.reasoning_tokens` (a spelling the platform never read) to `gen_ai.usage.reasoning.output_tokens`, the one it does. Third-party OTLP consumers that matched the removed aliases must switch to the canonical keys. The root `Evaluatorq - Red Teaming` span additionally carries: @@ -302,11 +264,7 @@ orq.simulation.generate # root — one per standalone generate() call └── chat/responses {model} # persona/scenario/first-message generation calls ``` -`generate_personas()` and `generate_scenarios()` don't open a synthetic root span -when invoked standalone. They do create `orq.simulation.persona_generation` / -`orq.simulation.scenario_generation` spans around their LLM calls. Those generation -spans carry the active run metadata when called inside an outer simulation or -red-team scope; standalone helpers intentionally have no synthetic run id to stamp. +`generate_personas()` and `generate_scenarios()` don't open a synthetic root span when invoked standalone. They do create `orq.simulation.persona_generation` / `orq.simulation.scenario_generation` spans around their LLM calls. Those generation spans carry the active run metadata when called inside an outer simulation or red-team scope; standalone helpers intentionally have no synthetic run id to stamp. Span attributes on `Evaluatorq - Agent Simulation` / `orq.simulation.generate`: @@ -364,18 +322,11 @@ Span attributes on `orq.simulation.turn`: | `orq.simulation.goal_completion_score` | Judge's goal-completion score | | `orq.simulation.should_terminate` | Whether the judge signalled the conversation should end | -`orq.simulation.target_call`, `orq.simulation.judge_evaluation`, and -`orq.simulation.user_simulator_call` carry no span attributes of their own — they -exist purely to scope the nested LLM call (and, for `target_call`, the target's own -input/output recording). LLM spans nested under `judge_evaluation` and -`user_simulator_call` carry the same GenAI attributes as the red teaming LLM spans -above, tagged via `orq.llm.purpose`. +`orq.simulation.target_call`, `orq.simulation.judge_evaluation`, and `orq.simulation.user_simulator_call` carry no span attributes of their own — they exist purely to scope the nested LLM call (and, for `target_call`, the target's own input/output recording). LLM spans nested under `judge_evaluation` and `user_simulator_call` carry the same GenAI attributes as the red teaming LLM spans above, tagged via `orq.llm.purpose`. ### Judge-panel spans -`llm_jury()`, `run_jury()`, and `run_pairwise()` (`src/evaluatorq/common/jury.py`, -`src/evaluatorq/pairwise.py`) run a panel of judges under a shared span -hierarchy: +`llm_jury()`, `run_jury()`, and `run_pairwise()` (`src/evaluatorq/common/jury.py`, `src/evaluatorq/pairwise.py`) run a panel of judges under a shared span hierarchy: ``` orq.evaluation {evaluator} # from the core runner, when a jury backs an evaluator @@ -386,16 +337,9 @@ orq.evaluation {evaluator} # from the core runner, when a jury backs an The leaf span is `responses {provider}/{model}` on the default `api='responses'`, and `chat {model}` when the call falls back to Chat Completions. -The panel opens no span of its own outside `orq.jury` — it can equally be -called standalone (not nested under `orq.evaluation`), in which case `orq.jury` -is the root. All jury/judge spans are opened via `evaluatorq.common.tracing`'s -`with_span()`, so — like every other span in this document — they are a no-op -when tracing is disabled; verdicts and aggregation are unaffected either way. +The panel opens no span of its own outside `orq.jury` — it can equally be called standalone (not nested under `orq.evaluation`), in which case `orq.jury` is the root. All jury/judge spans are opened via `evaluatorq.common.tracing`'s `with_span()`, so — like every other span in this document — they are a no-op when tracing is disabled; verdicts and aggregation are unaffected either way. -A judge whose call failed leaves its `orq.judge` span with OTel status `ERROR` -(via `set_span_error`), but the failure is swallowed at the panel level — the -jury carries on with whatever judges succeeded (or promotes a replacement) and -the parent `orq.jury` span stays OK. +A judge whose call failed leaves its `orq.judge` span with OTel status `ERROR` (via `set_span_error`), but the failure is swallowed at the panel level — the jury carries on with whatever judges succeeded (or promotes a replacement) and the parent `orq.jury` span stays OK. Span attributes on `orq.judge`: @@ -412,22 +356,11 @@ Span attributes on `orq.judge`: | `judge.error` | Error string when the judge failed (truncated per `EVALUATORQ_SPAN_MAX_TEXT_CHARS`) | | `judge.repetitions_failed` | Count of repetitions that failed to produce a usable verdict (an error, or a non-decisive non-abstained pass; a clean abstention is not counted), out of the configured repetition count | -No token usage or cost here: those are recorded once, on the `chat` spans -underneath, and rolled up by the consumer. Stamping them on every ancestor as -well made the same tokens appear three times in one trace. +No token usage or cost here: those are recorded once, on the `chat` spans underneath, and rolled up by the consumer. Stamping them on every ancestor as well made the same tokens appear three times in one trace. -`judge.label_swapped` is only ever set (`True`/`False`) in comparative mode — -in plain `run_jury()` deliberations it is absent, since each judge votes once. +`judge.label_swapped` is only ever set (`True`/`False`) in comparative mode — in plain `run_jury()` deliberations it is absent, since each judge votes once. -One judge attribute lives a level *down*, on the `chat` / `responses` span that -made the call rather than on `orq.judge`: `judge.verdict_coerced`. It is set to -`abstain_with_value` when the model returned `abstain=true` together with a -non-null `value` — a self-contradictory verdict, kept as an abstention with the -value dropped. It is absent on a well-formed verdict, so counting it per judge -model in the trace store answers "can this model follow the verdict schema". -Nothing aggregates it today — it does not reach `JuryVote`, the run manifest or -any report, so a coerced verdict is indistinguishable from a clean abstention -once it leaves the judge call. Query the spans, not the run artifact. +One judge attribute lives a level *down*, on the `chat` / `responses` span that made the call rather than on `orq.judge`: `judge.verdict_coerced`. It is set to `abstain_with_value` when the model returned `abstain=true` together with a non-null `value` — a self-contradictory verdict, kept as an abstention with the value dropped. It is absent on a well-formed verdict, so counting it per judge model in the trace store answers "can this model follow the verdict schema". Nothing aggregates it today — it does not reach `JuryVote`, the run manifest or any report, so a coerced verdict is indistinguishable from a clean abstention once it leaves the judge call. Query the spans, not the run artifact. Span attributes on `orq.jury`: @@ -444,31 +377,15 @@ Span attributes on `orq.jury`: | `jury.tie` | Whether the verdict came from a tie-break | | `jury.inconclusive` | Whether the panel failed to reach quorum | -`pairwise_plurality` is a **reported value, not an accepted argument** — you -cannot pass it to `aggregator=`, and `validate_aggregator()` rejects it. It -names the rule `run_pairwise()` applies internally: `pairwise_consensus()`, -a strict plurality over *reconciled pair* votes, run after judges that flipped -across the two orderings have already been dropped to abstentions. The six -`aggregator=` keywords reduce raw per-judge votes instead, so labelling this -one `mode` would name it after a function it does not call. +`pairwise_plurality` is a **reported value, not an accepted argument** — you cannot pass it to `aggregator=`, and `validate_aggregator()` rejects it. It names the rule `run_pairwise()` applies internally: `pairwise_consensus()`, a strict plurality over *reconciled pair* votes, run after judges that flipped across the two orderings have already been dropped to abstentions. The six `aggregator=` keywords reduce raw per-judge votes instead, so labelling this one `mode` would name it after a function it does not call. #### Comparative (pairwise) mode -`run_pairwise()` compares two responses (A vs. B) and, to control for position -bias, runs every judge in **both** label orderings. This changes the span -shape from the plain jury case: - -- **One `orq.pairwise_jury` span covers the whole comparison** — both orderings - drive the same span rather than each minting its own; `run_pairwise` calls the - internal `_run_jury_core` directly (not `run_jury`) so it doesn't open a - second jury span per ordering. -- **Each judge appears twice** under that one `orq.pairwise_jury` span — one - `orq.judge` span per ordering, distinguished by `judge.label_swapped` - (`False` for the A/B ordering, `True` for the swapped B/A ordering). -- **The span is named `orq.pairwise_jury`, not `orq.jury`** — it aggregates - reconciled *pair* votes rather than raw per-judge votes, so it gets its own - name rather than masquerading as a plain jury. Its attributes stay in the - `jury.*` namespace, plus these comparative-only extras: +`run_pairwise()` compares two responses (A vs. B) and, to control for position bias, runs every judge in **both** label orderings. This changes the span shape from the plain jury case: + +- **One `orq.pairwise_jury` span covers the whole comparison** — both orderings drive the same span rather than each minting its own; `run_pairwise` calls the internal `_run_jury_core` directly (not `run_jury`) so it doesn't open a second jury span per ordering. +- **Each judge appears twice** under that one `orq.pairwise_jury` span — one `orq.judge` span per ordering, distinguished by `judge.label_swapped` (`False` for the A/B ordering, `True` for the swapped B/A ordering). +- **The span is named `orq.pairwise_jury`, not `orq.jury`** — it aggregates reconciled *pair* votes rather than raw per-judge votes, so it gets its own name rather than masquerading as a plain jury. Its attributes stay in the `jury.*` namespace, plus these comparative-only extras: | Attribute | Value | |---|---| @@ -476,111 +393,52 @@ shape from the plain jury case: | `jury.flipped_judges` | Comma-separated model names of the flipped judges | | `jury.swap` | Whether the comparison ran both orderings (`swap=True`, the default) or only one | -**`judge.verdict` is already un-swapped in comparative mode.** The labels a judge -returns there name a *position*, not a response: a judge that picks the same -response both times says `A` in one ordering and `B` in the other, which reads as -a self-contradiction and is in fact the opposite, a perfectly consistent judge. -`label_swapped=True` spans are mapped back to the canonical frame before the -attribute is written, so "how often did this judge pick response A" is answerable -from `judge.verdict` alone, with no join against `judge.label_swapped`. - -There is deliberately no raw-frame twin. The text the verdict was parsed from is -one level down, on the `chat` child's `gen_ai.output.messages`, so a second -attribute here would only restate what the trace already holds — the same -reasoning as the alias removal noted above. - -Un-swapping is per-ordering and needs nothing but `label_swapped`. *Flip -detection* is what needs both orderings, and it stays on the parent — -`jury.flipped_judges` names the judges that really did follow slot order. - -`jury.flipped` counts judges that answered in both orderings but disagreed -with themselves — that is position bias, not a failure, so a flipped judge is -deliberately excluded from `jury.judges_failed`: `judges_failed` counts only -judges with no reconciled vote *and* no flip (i.e. one or both orderings -raised an error). A judge can be flipped, failed, or a normal decisive vote, -but never counted under more than one of those buckets. - -`orq.evaluation`, `orq.jury` / `orq.pairwise_jury`, `orq.judge`, and the nested LLM -spans follow the ambient OTel context — nothing threads an explicit parent -across the `orq.evaluation` → `orq.jury` seam, so a jury backing a custom -evaluator's scorer nests correctly without extra plumbing. +**`judge.verdict` is already un-swapped in comparative mode.** The labels a judge returns there name a *position*, not a response: a judge that picks the same response both times says `A` in one ordering and `B` in the other, which reads as a self-contradiction and is in fact the opposite, a perfectly consistent judge. `label_swapped=True` spans are mapped back to the canonical frame before the attribute is written, so "how often did this judge pick response A" is answerable from `judge.verdict` alone, with no join against `judge.label_swapped`. + +There is deliberately no raw-frame twin. The text the verdict was parsed from is one level down, on the `chat` child's `gen_ai.output.messages`, so a second attribute here would only restate what the trace already holds — the same reasoning as the alias removal noted above. + +Un-swapping is per-ordering and needs nothing but `label_swapped`. *Flip detection* is what needs both orderings, and it stays on the parent — `jury.flipped_judges` names the judges that really did follow slot order. + +`jury.flipped` counts judges that answered in both orderings but disagreed with themselves — that is position bias, not a failure, so a flipped judge is deliberately excluded from `jury.judges_failed`: `judges_failed` counts only judges with no reconciled vote *and* no flip (i.e. one or both orderings raised an error). A judge can be flipped, failed, or a normal decisive vote, but never counted under more than one of those buckets. + +`orq.evaluation`, `orq.jury` / `orq.pairwise_jury`, `orq.judge`, and the nested LLM spans follow the ambient OTel context — nothing threads an explicit parent across the `orq.evaluation` → `orq.jury` seam, so a jury backing a custom evaluator's scorer nests correctly without extra plumbing. ## Run correlation -Every LLM invocation issued during a `red_team()` or simulation run (`simulate()`, -`generate_and_simulate()`, or `generate()`) is tagged so an operator can filter -Orq's trace UI down to exactly the model calls belonging to one run. The same -metadata is inherited by `generate_personas()` and `generate_scenarios()` when -they are called inside an outer simulation or red-team scope; standalone calls -have no synthetic root run id. +Every LLM invocation issued during a `red_team()` or simulation run (`simulate()`, `generate_and_simulate()`, or `generate()`) is tagged so an operator can filter Orq's trace UI down to exactly the model calls belonging to one run. The same metadata is inherited by `generate_personas()` and `generate_scenarios()` when they are called inside an outer simulation or red-team scope; standalone calls have no synthetic root run id. | Surface | Key | Where | |---|---|---| | Request `metadata` on every LLM invocation | `evaluatorq_run_id` | red-team + simulation runs, including inherited nested work | | Root span attribute | `orq.evaluatorq_run_id` | `Evaluatorq - Red Teaming` root span; `Evaluatorq - Agent Simulation` / `orq.simulation.generate` root spans | -A companion key rides the same rail: `evaluatorq_pipeline`, whose value is -`"red_teaming"` or `"agent_simulation"`. It identifies which surface issued the call -and is sent as request metadata alongside `evaluatorq_run_id` — filter on it to -separate red-team traffic from simulation traffic regardless of run. Both -`evaluatorq_run_id` and `evaluatorq_pipeline` are native request `metadata` fields -on Chat Completions and Responses calls. They are sent to direct -OpenAI-compatible endpoints as well as through the Orq router. +A companion key rides the same rail: `evaluatorq_pipeline`, whose value is `"red_teaming"` or `"agent_simulation"`. It identifies which surface issued the call and is sent as request metadata alongside `evaluatorq_run_id` — filter on it to separate red-team traffic from simulation traffic regardless of run. Both `evaluatorq_run_id` and `evaluatorq_pipeline` are native request `metadata` fields on Chat Completions and Responses calls. They are sent to direct OpenAI-compatible endpoints as well as through the Orq router. ### How it reaches every call -Both red-team and simulation route their datapoints through a nested `evaluatorq()` -call. The run id isn't threaded through function arguments — it's bound to a -`contextvars.ContextVar` (`src/evaluatorq/common/thread_context.py`) at the run's -entrypoint and read back at the call site. Because a `ContextVar` set in an ancestor -scope is visible to nested calls (and copied into child `asyncio` tasks), every LLM -call issued from inside the nested `evaluatorq()` run automatically carries the SAME -`evaluatorq_run_id` as the outer red-team/sim run — no explicit plumbing required. - -Call sites read it back one of two ways, and the difference matters when you are -tracking down a missing tag: - -- **Chat Completions** (`create` / `.parse`) and **Responses** calls read the same - context and send it as native request `metadata`. -- The router-specific `thread` body parameter is separate and remains endpoint- - gated: it is included only when the client routes through Orq and a conversation - thread is active. It is never required for run correlation. - -Separate root invocations receive separate ids: two calls to `simulate()`, -`generate_and_simulate()`, or `generate()` each get a distinct -`evaluatorq_run_id`, even if called back-to-back in the same process. Nested -`evaluatorq()` work within one red-team or simulation root receives that root's id, -and nested generation helpers inherit it. Standalone `generate_personas()` and -`generate_scenarios()` do not mint ids of their own. The evaluatorq-core -`orq.run_id` attributes continue to describe evaluatorq evaluation runs and are -unchanged by this correlation mechanism. +Both red-team and simulation route their datapoints through a nested `evaluatorq()` call. The run id isn't threaded through function arguments — it's bound to a `contextvars.ContextVar` (`src/evaluatorq/common/thread_context.py`) at the run's entrypoint and read back at the call site. Because a `ContextVar` set in an ancestor scope is visible to nested calls (and copied into child `asyncio` tasks), every LLM call issued from inside the nested `evaluatorq()` run automatically carries the SAME `evaluatorq_run_id` as the outer red-team/sim run — no explicit plumbing required. + +Call sites read it back one of two ways, and the difference matters when you are tracking down a missing tag: + +- **Chat Completions** (`create` / `.parse`) and **Responses** calls read the same context and send it as native request `metadata`. +- The router-specific `thread` body parameter is separate and remains endpoint- gated: it is included only when the client routes through Orq and a conversation thread is active. It is never required for run correlation. + +Separate root invocations receive separate ids: two calls to `simulate()`, `generate_and_simulate()`, or `generate()` each get a distinct `evaluatorq_run_id`, even if called back-to-back in the same process. Nested `evaluatorq()` work within one red-team or simulation root receives that root's id, and nested generation helpers inherit it. Standalone `generate_personas()` and `generate_scenarios()` do not mint ids of their own. The evaluatorq-core `orq.run_id` attributes continue to describe evaluatorq evaluation runs and are unchanged by this correlation mechanism. ### Using it -In Orq's trace UI, filter spans/traces on the `evaluatorq_run_id` request-metadata -value (copy it from the `orq.evaluatorq_run_id` attribute on the run's root span, or -from your own logs/hooks that captured the run id) to see every model call — target, -judge, user-simulator, attacker, evaluator, generation — that belongs to one -`red_team()` or `simulate()`/`generate_and_simulate()`/`generate()` invocation, including calls -made through the nested `evaluatorq()` run. Add `evaluatorq_pipeline` to the filter to -scope further to just red-team or just simulation traffic. +In Orq's trace UI, filter spans/traces on the `evaluatorq_run_id` request-metadata value (copy it from the `orq.evaluatorq_run_id` attribute on the run's root span, or from your own logs/hooks that captured the run id) to see every model call — target, judge, user-simulator, attacker, evaluator, generation — that belongs to one `red_team()` or `simulate()`/`generate_and_simulate()`/`generate()` invocation, including calls made through the nested `evaluatorq()` run. Add `evaluatorq_pipeline` to the filter to scope further to just red-team or just simulation traffic. ## Content capture and truncation Two env vars control how much text is stored on spans: -- **`EVALUATORQ_CAPTURE_MESSAGE_CONTENT`** (default `true`): set to `false` or - `0` to keep LLM message content out of traces entirely. Token counts and - model name are still recorded. -- **`EVALUATORQ_SPAN_MAX_TEXT_CHARS`** (default: no limit): set to a positive - integer to truncate span text attributes. Truncated strings end with - `... [truncated]`. +- **`EVALUATORQ_CAPTURE_MESSAGE_CONTENT`** (default `true`): set to `false` or `0` to keep LLM message content out of traces entirely. Token counts and model name are still recorded. +- **`EVALUATORQ_SPAN_MAX_TEXT_CHARS`** (default: no limit): set to a positive integer to truncate span text attributes. Truncated strings end with `... [truncated]`. ## W3C trace context propagation -To propagate trace context across service boundaries, inject the active span's -W3C `traceparent`/`tracestate` headers into your outgoing HTTP requests. Use the -OpenTelemetry SDK's public `inject()` helper — a stable, supported API: +To propagate trace context across service boundaries, inject the active span's W3C `traceparent`/`tracestate` headers into your outgoing HTTP requests. Use the OpenTelemetry SDK's public `inject()` helper — a stable, supported API: ```python from opentelemetry.propagate import inject @@ -590,19 +448,10 @@ inject(headers) # writes `traceparent` (+ `tracestate`) for the active # pass `headers` into your outgoing request, e.g. httpx.get(url, headers=headers) ``` -`inject()` is a no-op when no span is active, so it is safe to call whenever -OpenTelemetry is installed. (The `from opentelemetry.propagate import inject` -import itself requires OTel; if you need code that also runs without it -installed, use the internal helper below, which degrades to an empty dict.) +`inject()` is a no-op when no span is active, so it is safe to call whenever OpenTelemetry is installed. (The `from opentelemetry.propagate import inject` import itself requires OTel; if you need code that also runs without it installed, use the internal helper below, which degrades to an empty dict.) !!! note "Internal convenience helper" - evaluatorq also ships `get_trace_context_headers()` in - `evaluatorq.common.tracing`, an `async` helper you `await` for the same - headers as a dict (empty when OTel is unavailable). It is an internal - utility — **not** re-exported - from the public `evaluatorq.tracing` namespace, and its import path may - change without a deprecation cycle. Prefer the OpenTelemetry `inject()` path - above for anything stable. + evaluatorq also ships `get_trace_context_headers()` in `evaluatorq.common.tracing`, an `async` helper you `await` for the same headers as a dict (empty when OTel is unavailable). It is an internal utility — **not** re-exported from the public `evaluatorq.tracing` namespace, and its import path may change without a deprecation cycle. Prefer the OpenTelemetry `inject()` path above for anything stable. ## Where to next