[PENDING: #744, #712] fix: record a clean but short completion under ignore_eos as a failure

[Bslabe123]

Stacked on #744, which is stacked on #712: the first two commits are theirs, the last one is this change. Will rebase to a single commit once they merge.

Closes #655. Part of #606, Success classification.

What

A stream the server closes cleanly after a handful of tokens is a well-formed 200: no stream break (#530), no token-count mismatch (#565), no error. It landed in the success bucket and dragged the output-length distribution down silently.

• ResponseMetrics.finish_reason: the server's reason verbatim (OpenAI finish_reason, Anthropic stop_reason), captured on every streaming and unary path (completion, chat, Anthropic, trace replay). ResponseMetrics.delivered_output_tokens(): server usage.completion_tokens when reported, else the client count.
• RequestLifecycleMetric.max_tokens: what the request body asked for, also on each per-request report entry.
• Under ignore_eos, a completion that delivered fewer tokens than max_tokens is recorded as a TruncatedResponseError failure (label truncatedresponseerror; the message names delivered-of-requested and the finish_reason). Its metrics and body stay on the record. The rule reads ignore_eos from the request body, not the client setting: Anthropic bodies never carry it, and the trace replay turns it off per request for tool calls.
• Without ignore_eos, a short response stays a success and the summary reports successes.finish_reasons (reason to count) and successes.output_shortfalls.
• The OTel finish_reason attribute read a never-populated extra_info key; it now reads the persisted field.

Two calls to check

1. length with fewer tokens is still a truncation. A server that caps below max_tokens (a max_model_len cap) reports length and fewer tokens; under ignore_eos the requested budget was not delivered, so it fails, with the finish_reason in the message for diagnosis.
2. The rule is on by default, since ignore_eos defaults to true. A server that ignores the ignore_eos field will now report every natural stop as truncated. That is the config misdescribing the run, and the fix is ignore_eos: false; the field description and docs/reports.md say so. Fallback if wanted: a report-config toggle.

Tests

• tests/required/integration/test_truncated_response.py against the test: integration coverage for partial response body on a broken SSE stream #712 fake: short-with-stop under ignore_eos on (failure) and off (success plus shortfall), full budget, length-capped, server count over client count, unary chat, the report bucket, and a no-choices body that is not judged.
• Parser (finish_reason on the last content frame, on a content-free frame, Anthropic message_delta), API layer, and reportgen fields.
• e2e/tests/test_output_length_sim.py: sim v0.6.1 honours ignore_eos; on gives finish_reasons {"length": 10} and 0 shortfalls; off gives shortfalls equal to the per-request entries whose completion_tokens is below max_tokens. The fidelity test asserts the same on its ground truth.
• The golden fixture serves fixed n_tokens regardless of max_tokens, so it now runs with ignore_eos: false (comment explains); the backend suite's 5-token canned replies likewise.

pdm run validate clean, 935 unit tests pass, sim e2e run locally.

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Bslabe123

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [Bslabe123]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment