Add OSL / E2EL x-axis metric as the agentic default / 新增 OSL / E2EL X 轴指标并设为智能体默认 - #638
Open
cquil11 wants to merge 8 commits into
Open
Add OSL / E2EL x-axis metric as the agentic default / 新增 OSL / E2EL X 轴指标并设为智能体默认#638cquil11 wants to merge 8 commits into
cquil11 wants to merge 8 commits into
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
cquil11
changed the title
…xis modes Retire the three experimental agentic x-axis modes and everything that existed solely to serve them, leaving Interactivity / E2E Latency / TTFT. - chart: drop the three mode buttons, DERIVED_X_MODE_SPECS and the derived-metric fetch/remap plumbing in ChartDisplay; the remaining modes apply to both scenario kinds, so the agentic-only button filter (and its `mounted` SSR guard) goes too - API/hook/db: delete /api/v1/derived-agentic-metrics, the useDerivedAgenticMetrics hook, and queries/derived-agentic-metrics.ts - aggregate_stats v6: drop normalizedSessionTimeS, p90PrefillTpsPerUser and normalizedE2e400; mergeProfileStatsUpgrade no longer carries them forward, and the backfill's profile-only fast path now covers every v3+ bundle instead of only v3 - drop the NORMALIZED_E2E_OUTPUT_TOKENS constant and the overlay suppression helper that existed only for Normalized E2E 中文：移除三个实验性智能体 X 轴模式（Normalized E2E、会话时长、 Prefill TPS / user）及其专属实现，仅保留交互性 / 端到端延迟 / TTFT。 同时删除对应的 API 路由、React Query hook 与数据库查询模块； aggregate_stats 升级至 v6，去掉三个已废弃字段，backfill 的 profile-only 快速路径扩展至所有 v3 及以上版本。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Introduces "OSL / E2EL" ("e2e interactivity") for agentic scenarios: the
per-request output-token rate INCLUDING the prefill wait,
OSL / (TTFT + generation time) ≈ 1 / (ITL + TTFT/OSL). Unlike plain
interactivity it cannot be inflated by delaying prefill.
- aggregate_stats v7: store `e2elPerOsl`, percentiles of the per-request
E2EL/OSL ratio (seconds per output token). The read path inverts, so
pXX OSL/E2EL = 1 / pXX(E2EL/OSL) — the slow-tail convention the ETL
already enforces for `*_intvty`
- API: /api/v1/derived-agentic-metrics returns p75/p90_osl_per_e2el
- chart: new mode, agentic default, listed first; fixed-seq is unchanged
and never shows the button
- overlays: suppressed in this mode (unofficial rows carry no persisted
per-request trace) with a caption disclaimer
- shared cypress helper interceptDerivedAgenticMetrics stubs the fetch
the default mode fires on mount; overlay specs switch to Interactivity
explicitly before asserting overlay points
中文：为智能体场景新增 "OSL / E2EL"（e2e interactivity）指标：每请求
输出 token 速率（含 prefill 等待），即 OSL /（TTFT + 生成时间）。
aggregate_stats 升级至 v7，存储每请求 E2EL/OSL 比值的分位数，读取时取
倒数以沿用 `*_intvty` 的慢尾约定；新增 API 字段与图表模式，并设为智能体
默认模式且置于最左侧；该模式下隐藏非官方运行覆盖层并显示提示文案。
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cquil11
force-pushed
the
feat/agentic-osl-e2el-metric
branch
from
July 29, 2026 21:58
318b573 to
a65a098
Compare
cquil11
changed the base branch from
master
to
chore/remove-agentic-derived-x-axis-modes
July 29, 2026 21:58
cquil11
changed the title
The "switches back to Interactivity" test asserted a P75 axis label, but nothing in it selected p75 — it inherited the selector state from the Normalized E2E test that ran before it. Removing that test left the selector on p90 and the assertion failed. Assert P90 there (the default) and cover the p75 case in its own test that selects the percentile itself. 中文：「switches back to Interactivity」用例断言 P75 轴标签，但其自身 并未切换分位数，而是依赖此前 Normalized E2E 用例遗留的选择器状态。 该用例被移除后选择器停留在 p90，断言随之失败。现改为断言默认的 P90， 并将 p75 场景拆分为独立用例，由其自行切换分位数。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…entic-osl-e2el-metric
The percentile selector is page state shared across the whole suite, so leaving it on p75 made the following Interactivity test's P90 assertion depend on test order. Restore the default at the end of the test that changes it. 中文：分位数选择器是整个套件共享的页面状态，停留在 p75 会使后续 Interactivity 用例的 P90 断言依赖执行顺序。现在由改动它的用例在结束时 恢复默认值。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
…red mode The reconcile effect ran before availability resolved the sequence. On a cold load it recorded the fixed-seq placeholder kind, then treated the switch to agentic as a user-driven kind change and snapped to the agentic default — so /inference?i_seq=agentic-traces&i_xmode=interactivity landed on OSL / E2EL instead. Gate the effect on `sequenceResolved`, matching the label effect above it, so the first run sees the real kind and the URL-restored mode survives. Found by Cursor Bugbot; reproduced against the live DB before and after. Pre-existing on master (it clobbered a URL-restored TTFT/E2E with interactivity there) — visible now because the agentic default differs. 中文：X 轴模式协调 effect 在 availability 解析出场景之前就已运行：冷启动 时它记录了固定序列占位场景，随后把切换到智能体场景误判为用户主动切换， 从而覆盖 URL 中的 `i_xmode`。现与其上方的 label effect 一致，改为在 `sequenceResolved` 之后才执行。该问题在 master 上已存在，因智能体默认模式 变更后才变得可见。新增回归用例（去掉修复即失败）。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…2EL chart The mode hardcoded `upper_left` for every y-metric. That is right for throughput but inverts the frontier for cost and joules metrics, whose good direction is a LOWER corner — and Optimal Only filters off the same flag, so the wrong points were hidden on what is now the agentic default view. Derived modes render on the e2e chart definition (lower-x-is-better) while OSL / E2EL is higher-is-better, so the correct transform is a horizontal mirror of each configured corner, not a constant: upper_right → upper_left for throughput, lower_left → lower_right for cost/joules. Extracted as `derivedModeRoofline` and unit-tested against the real chart config — the mirrored e2e corner must equal the interactivity chart's corner for every y-metric, since both axes are higher-is-better. Found by Cursor Bugbot. Confirmed the tests fail with the hardcoded corner restored and pass with the mirror. 中文：该模式此前对所有 Y 轴指标硬编码 `upper_left`，对吞吐量正确，但会让 成本与能耗类指标的帕累托前沿方向反转（Optimal Only 也依赖同一标记， 导致隐藏了错误的点）。派生模式渲染在 e2e 图表定义之上（x 越小越好）， 而 OSL / E2EL 是越大越好，因此正确做法是对配置中的角位做水平镜像而非 固定取值。已抽出 `derivedModeRoofline` 并针对真实图表配置编写单元测试。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 263ed05. Configure here.
The derived-metrics read path only sees the profile blob, so it cannot recompute kvCacheUtil / prefixCacheHitRate — it carries them forward from the stale row. When the stale row had none (null stats, or a pre-v3 bundle), it still wrote nulls stamped at the current STATS_VERSION. That looks complete to everyone downstream: the backfill's candidate query matches on version and skips the row, and agentic-aggregates takes the fast path, so those server-derived fields stay null permanently. It also broke writeBackTraceReplayJsonb's documented contract that callers only persist COMPLETE payloads. Self-heal now only stamps the bundle when there are server-derived fields to preserve. Otherwise the row stays stale, costing one repeat profile parse and letting a reader that CAN see the server blob heal it fully. Pre-existing behavior carried over from before the metric split; found by Cursor Bugbot. Test asserts no UPDATE is issued for a null-stats row and fails if the guard is removed. 中文：派生指标读取路径只能看到 profile blob，无法重算 kvCacheUtil / prefixCacheHitRate，只能从旧行继承。当旧行没有这些字段时， 此前仍会写入 null 并打上当前 STATS_VERSION，使 backfill 跳过该行、 agentic-aggregates 走快速路径，导致这些字段永久为空。现仅在确实有服务端 字段可保留时才写回，否则保持 stale，交由能读取 server blob 的读取方修复。 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Base automatically changed from
chore/remove-agentic-derived-x-axis-modes
to
master
July 30, 2026 03:48
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds OSL / E2EL ("e2e interactivity") for agentic scenarios: per-request
OSL / E2E latencyin tok/s/user — the rate the user receives output tokens including the prefill wait,≈ 1/(ITL + TTFT/OSL).What changed
aggregate_statsv6 → v7 addse2elPerOsl, percentiles of the per-request E2EL/OSL ratio (seconds per output token)pXX OSL/E2EL = 1 / pXX(E2EL/OSL)— the slow-tail convention the ETL already enforces for*_intvty. The existing p75/p90 selector drives it/api/v1/derived-agentic-metricsreturnsp75/p90_osl_per_e2elinterceptDerivedAgenticMetricscypress helper stubs the fetch the default mode fires on mount; overlay specs switch to Interactivity explicitly before asserting overlay pointsWhy
Backfill
v7 needs a backfill (
db:backfill-aggregate-stats, profile-only fast path, ~20 min for the current row count). Rows also self-heal on read. I ran the equivalent backfill against the target DB while validating; it will be re-run after merge to stamp v7.中文说明
为智能体场景新增 OSL / E2EL（e2e interactivity）：每请求
输出 token 数 / 端到端延迟（tok/s/user），衡量含 prefill 等待在内的输出速率，≈ 1/(ITL + TTFT/OSL)。改动：
aggregate_statsv6 → v7，新增e2elPerOsl（每请求 E2EL/OSL 比值的分位数）*_intvty的慢尾约定：读取时取倒数，pXX OSL/E2EL = 1 / pXX(E2EL/OSL)p75/p90_osl_per_e2el原因： 1/TPOT 可通过延迟 prefill 刷分，本指标对等待计入惩罚；端到端延迟分位数受 OSL 主导，除以 OSL 后既计入 prefill 又不受输出长度支配。
回填： v7 需要执行
db:backfill-aggregate-stats（profile-only 快速路径，约 20 分钟），行也会在读取时自愈。🤖 Generated with Claude Code
Note
Medium Risk
Changes default agentic chart behavior and introduces a versioned
aggregate_statspayload requiring backfill; mitigated by read-path self-heal, versioned blob cache, and broad unit/e2e coverage.Overview
Introduces OSL / E2EL (slow-tail per-request output rate including prefill wait, tok/s/user) as a new agentic-only x-axis mode and makes it the default for Agentic Traces—listed first in the mode tabs ahead of Interactivity.
Backend: Bumps
STATS_VERSIONto 7 and storese2elPerOsl(percentiles of E2EL/OSL in s/tok; chart uses1 / pXXlike*_intvty). NewGET /api/v1/derived-agentic-metricsreads fromaggregate_statswith profile-blob fallback and optional self-heal write-back; blob cache key is tied toSTATS_VERSION. ETL/backfill and aggregate paths compute the same ratio bundle fromprofile_export.jsonl.Chart UI:
ChartDisplayloads derived metrics via React Query (chunked past 200 ids), remaps pointxvalues, appliesderivedModeRooflineso higher-is-better x mirrors e2e roofline corners per y-metric (fixes cost/joules frontier inversion). Unofficial-run overlays are hidden in this mode with a caption disclaimer.InferenceContextwaits until the sequence is resolved before reconciling x-mode (fixesi_xmode=interactivitybeing overwritten) and defaults agentic toosl-e2elinstead of interactivity.Tests: Shared Cypress
interceptDerivedAgenticMetrics; agentic/overlay specs stub the default-mode fetch and explicitly switch to Interactivity where overlays are asserted.Reviewed by Cursor Bugbot for commit 02554dd. Bugbot is set up for automated code reviews on this repo. Configure here.