CollectiveX: add a per-component bandwidth view to the sweep summary / CollectiveX：为扫描摘要新增按组件的带宽视图 - #2434
CollectiveX: add a per-component bandwidth view to the sweep summary / CollectiveX：为扫描摘要新增按组件的带宽视图#2434Oseltamivir wants to merge 2 commits into
Conversation
Every case-attempt artifact already carries per-component bytes and latency, but nothing turns the pair into a rate, so the sweep reports how long a dispatch took without saying how close it came to moving that payload at link speed. Add a consumer that reads only committed artifacts -- no harness, runtime, or GPU change, and it gates nothing. Two views. Effective bandwidth divides byte_provenance[c].total_logical_bytes by the component's latency at each ladder point, reported per GPU. The alpha/beta fit regresses per-component p50 latency on bytes across the ladder, latency(bytes) ~= alpha + bytes/beta, which separates the bandwidth term from the fixed per-call launch/sync/rendezvous floor that dominates small token counts. The regression is against emitted bytes rather than T, because routed_copies is sub-linear in T at high fan-out, and rungs that failed the correctness gate are excluded -- a rung whose payload came back corrupt is not a measurement of how fast correct data moves. A straight line through a ladder is a MODEL, and on real data it is sometimes a bad one, so the fit reports its own trustworthiness rather than leaving that to the reader. Two rungs of noise once yielded beta = 1018 GB/s per GPU on a B200 -- above the link's per-direction peak, i.e. physically impossible -- from a fit with R^2 = 0.29, printed indistinguishably from a neighbouring R^2 = 0.99 fit. R^2 alone is not enough either: a dispatch ladder that stayed latency-bound throughout came back at 3763 GB/s with R^2 = 0.92, because alpha explained nearly everything and the slope was not identifiable. So Fit carries r2, point count and bandwidth_share as results rather than diagnostics, beta is withheld unless both R^2 and the transfer term's share of the top rung clear their thresholds, and alpha is marked as an extrapolation when the ladder never approaches zero bytes (prefill ladders start at T=1024). Deliberately NOT added: a per-SKU peak-bandwidth table to check beta against. That would put hardware knowledge belonging to platform_config.json into a consumer, where it would age badly and mislead in its own right; the fit-quality gate is what actually caught the pathology above. The interpretation the numbers need is documented in the module docstring: logical bytes include the copies a rank routes to itself, which never cross the interconnect, so wire traffic is (1 - routing.locality.local_rank_fraction) x the reported figures, and beta is a marginal rate that legitimately sits above every measured point. Tests are math-only (unit conversion, the fit, and each gate) and need no GPU; the existing unittest discovery picks them up. 中文：新增按组件统计的带宽消费者(consumer)。 每个 case-attempt 产物本就带有各组件的字节数与延迟，但没有任何代码把这两者换算成速率， 于是扫描只报告 dispatch 耗时多少，却不说明它离以链路速率搬运该载荷还有多远。本提交新增一个 只读取已落盘产物的消费者 —— 不改动 harness、runtime 或 GPU 路径，也不作为任何门禁。 两种视图。有效带宽在每个 ladder 测点上用 byte_provenance[c].total_logical_bytes 除以该组件的 延迟，按每 GPU 报告。alpha/beta 拟合则在整个 ladder 上对各组件 p50 延迟与字节数做回归， latency(bytes) ≈ alpha + bytes/beta，从而把带宽项与小 token 数下占主导的固定单次调用开销 (launch/sync/rendezvous)分离开。回归对的是产物中的字节数而非 T，因为在高 fanout 下 routed_copies relative T 是次线性的；未通过正确性门禁的档位会被排除 —— 载荷已损坏的档位， 不构成「正确数据搬运速度」的测量。 穿过 ladder 的直线是一个模型，而在真实数据上有时是个糟糕的模型，所以拟合结果自己携带可信度， 而不是把判断留给读者。曾经有两档噪声在 B200 上拟合出 beta = 1018 GB/s 每 GPU —— 高于链路单向 峰值，物理上不可能 —— 而其 R² 仅为 0.29，却与相邻 R² = 0.99 的拟合打印得一模一样。单看 R² 也不够：一条全程受延迟主导的 dispatch ladder 拟合出 3763 GB/s 而 R² = 0.92，因为 alpha 解释了 几乎全部变化、斜率不可辨识。因此 Fit 把 r2、点数与 bandwidth_share 作为结果而非诊断信息返回， beta 仅在 R² 与顶档传输项占比双双达标时才输出，而当 ladder 从未接近零字节时(预填充 ladder 从 T=1024 起步)alpha 会被标记为外推值。 有意不做的事：不引入按 SKU 的峰值带宽表来校验 beta。那会把本属于 platform_config.json 的硬件 知识塞进消费者，既容易过时又会自行产生误导；真正抓住上述异常的是拟合质量门禁。 数值解读所需的前提写在模块 docstring 中：逻辑字节包含 rank 路由给自己、根本不过互联的那部分 拷贝，因此实际上链流量为 (1 - routing.locality.local_rank_fraction) × 报告值；而 beta 是边际 速率，理应高于每一个实测点。 测试只覆盖数学部分(单位换算、拟合、各道门禁)，无需 GPU；现有的 unittest discovery 会自动收集。
The consumer added in the previous commit was never invoked, so the second axis existed only for whoever ran the script by hand. The sweep's summary step renders the latency table and nothing else, and since the workflow's only other output is the neutral per-case JSON, an unwired renderer in this repo cannot reach a reader at all. Invoke it next to summarize.py, over the same staged results directory and into the same step summary, so every leg reports how fast it moved its payload alongside how long the point took. Both are pure renderers over already-written documents and neither gates the leg, which is why both keep `|| true`: a formatting failure must not turn a green sweep red or mask a real one. 中文：在扫描摘要中渲染带宽视图。 上一个提交加入的消费者从未被调用，因此这条「第二坐标轴」只对手动执行脚本的人存在。扫描的摘要 步骤只渲染延迟表格；而该工作流唯一的其他输出是中立的按 case JSON 产物，所以一个没有接线的 渲染器在本仓库里根本无法触达任何读者。 把它接在 summarize.py 旁边调用，读取同一个已落盘的结果目录、写入同一份 step summary，这样每条 leg 在报告测点耗时的同时，也报告它搬运载荷的速率。两者都是对既有产物的纯渲染器，都不作为门禁， 这也是二者都保留 `|| true` 的原因：格式化失败既不能把绿色扫描变红，也不能掩盖真正的失败。
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
No bugs found, but this adds a fairly substantial new statistical-fitting consumer (OLS regression, R²/bandwidth-share gating with several hand-picked thresholds, MAX-vs-MIN latency choice interacting with #2422) — worth a human sanity check on the modeling choices even though the code itself is well-isolated and tested.
What was reviewed: the OLS fit and R²/bandwidth-share gating math, the correctness-gate row exclusion, the alpha-extrapolation flag, and the workflow wiring (|| true, non-gating). Checked and ruled out: the bandwidth_share denominator using ladder-wide max latency rather than the top rung's own latency — this matches the documented intent (share of the largest measured latency, not necessarily the max-bytes rung) rather than being a bug; the correctness-gate rung-exclusion note undercounting when a component's fit fails entirely; and the new script not being listed in the README's core-path list.
Extended reasoning...
Overview
This PR adds experimental/CollectiveX/bandwidth.py, a new pure-renderer consumer that computes per-component effective bandwidth (bytes/latency) and an alpha/beta OLS fit (latency ~= alpha + bytes/beta) from existing case-attempt JSON documents, plus wiring it into .github/workflows/collectivex-sweep.yml next to the existing summarize.py step, and 10 new unit tests covering the math.
Security risks
None. This is a read-only renderer over already-written JSON artifacts in an experimental CI sweep workflow — no new inputs, no auth/crypto/permissions surface, no network or credential handling. The workflow step is appended with || true so it cannot affect job pass/fail status.
Level of scrutiny
Low production risk (non-gating, experimental tooling), but the file itself carries real quantitative logic: an OLS regression, two hand-tuned gating thresholds (FIT_MIN_R2=0.9, FIT_MIN_BANDWIDTH_SHARE=0.25), an extrapolation heuristic, and a documented design choice to derive bandwidth from cross-rank MAX latency (consistent with the adjacent latency table) rather than the MIN latency introduced by #2422. None of this is mechanical — it's the kind of statistical-modeling choice where a domain expert's sign-off adds value beyond what a correctness-focused bug hunt can verify, even though no bugs were found in the implementation.
Other factors
Test coverage is strong: 10 new tests directly target the two real failure modes described in the PR body (a noisy fit with low R² that must withhold beta, and a latency-bound ladder with high R² but non-identifiable slope that must also withhold beta via the bandwidth-share gate), plus the correctness-gate exclusion and alpha-extrapolation flag. The bug-hunting system's candidate findings around the bandwidth_share denominator and dead-code rung exclusion were examined and found to match documented intent rather than being bugs. Given the design tradeoffs called out explicitly in the PR description (MAX vs MIN latency, no per-SKU peak table), this reads like a PR the author already expects a human to weigh in on.
1 participant
Why
Every case-attempt artifact already carries per-component bytes and latency, but nothing turns the pair into a rate. The sweep summary reports how long a dispatch took and never says how close it came to moving that payload at link speed — so
## CollectiveX EP resultshas one axis, latency, and no second one.What changed
A bandwidth consumer (
bandwidth.py), deliberately built as a structural twin ofsummarize.py: same--results-dir/--runner/--tsflags, samerender() -> markdown, sameprint(render(...)), reading the same case documents. It is a pure consumer — no harness, runtime, or GPU change — and it gates nothing.Two views. Effective bandwidth divides
byte_provenance[c].total_logical_bytesby the component's latency at each ladder point, per GPU. The alpha/beta fit regresses per-component p50 latency on bytes across the ladder,latency(bytes) ~= alpha + bytes/beta, separating the bandwidth term from the fixed per-call launch/sync/rendezvous floor that dominates small token counts. The regression is against emitted bytes rather than T, becauserouted_copiesis sub-linear in T at high fan-out, and rungs that failed the correctness gate are excluded — a rung whose payload came back corrupt is not a measurement of how fast correct data moves.Wired into the sweep summary (
.github/workflows/collectivex-sweep.yml). Called next tosummarize.py, over the same staged results directory and into the same$GITHUB_STEP_SUMMARY, so every leg reports how fast it moved its payload alongside how long the point took. Both are pure renderers over already-written documents and neither gates the leg, which is why both keep|| true: a formatting failure must not turn a green sweep red or mask a real one. Without this the consumer only exists for whoever runs the script by hand — the workflow's only other output is the neutral per-case JSON, so an unwired renderer in this repo cannot reach a reader at all.Fit quality is part of the result, not a footnote
A straight line through a ladder is a MODEL, and on real data it is sometimes a bad one. Two rungs of noise once yielded
beta = 1018 GB/sper GPU on a B200 — above the link's per-direction peak, i.e. physically impossible — from a fit with R² = 0.29, printed indistinguishably from a neighbouring R² = 0.99 fit. R² alone is not enough either: a dispatch ladder that stayed latency-bound throughout came back at 3763 GB/s with R² = 0.92, because alpha explained nearly everything and the slope was not identifiable. SoFitcarriesr2, point count andbandwidth_shareas results rather than diagnostics, beta is withheld unless both R² and the transfer term's share of the top rung clear their thresholds, and alpha is marked as an extrapolation when the ladder never approaches zero bytes (prefill ladders start at T=1024).Deliberately NOT added: a per-SKU peak-bandwidth table to check beta against. That would put hardware knowledge belonging to
platform_config.jsoninto a consumer, where it would age badly and mislead in its own right; the fit-quality gate is what actually caught the pathology above.Reading the numbers
Documented in the module docstring: logical bytes include the copies a rank routes to itself, which never cross the interconnect, so wire traffic is
(1 - routing.locality.local_rank_fraction)× the reported figures; and beta is a marginal rate that legitimately sits above every measured point.One deliberate choice worth flagging now that #2422 has landed: GB/s is derived from
components[c]— the cross-rank MAX latency — so it stays consistent with the latency table it sits beside. #2422'scross_rank_min_usis the skew-excluded companion, and a skew-inflated point therefore reads as lower bandwidth here. Switching the rate to MIN, or emitting both, is a follow-up rather than something this PR assumes.Verification
unittest discoverpicks them up).jobs.sweep.steps.summarize.pyrenders its table andbandwidth.pyrenders per-point GB/s per component plus the fit line, confirming the render path, percentile keying, locality suffix and fit gate all execute against the real document shape.main(includes CollectiveX: stop charging rank entry skew to component latency / CollectiveX：不再把 rank 进入时间差计入组件延迟 #2422), so the additions-onlyperf-changelog.yamlgate is satisfied.中文说明
为什么
每个 case-attempt 产物本就同时带有各组件的字节数与延迟，但没有任何代码把这两者换算成速率。扫描摘要只报告 dispatch 耗时多少，从不说明它离以链路速率搬运该载荷还有多远 —— 也就是说
## CollectiveX EP results只有延迟这一个坐标轴，没有第二个。改了什么
带宽消费者(
bandwidth.py)，有意做成summarize.py的结构孪生：相同的--results-dir/--runner/--ts参数、相同的render() -> markdown、相同的print(render(...))，读取同一批 case 文档。它是纯消费者 —— 不改动 harness、runtime 或 GPU 路径 —— 也不作为任何门禁。两种视图。有效带宽在每个 ladder 测点上用
byte_provenance[c].total_logical_bytes除以该组件的延迟，按每 GPU 报告。alpha/beta 拟合在整个 ladder 上对各组件 p50 延迟与字节数做回归，latency(bytes) ≈ alpha + bytes/beta，从而把带宽项与小 token 数下占主导的固定单次调用开销(launch/sync/rendezvous)分离开。回归对的是产物中的字节数而非 T，因为在高 fanout 下routed_copies相对 T 是次线性的；未通过正确性门禁的档位会被排除 —— 载荷已损坏的档位，不构成「正确数据搬运速度」的测量。接入扫描摘要(
.github/workflows/collectivex-sweep.yml)。接在summarize.py旁边调用，读取同一个已落盘的结果目录、写入同一份$GITHUB_STEP_SUMMARY，这样每条 leg 在报告测点耗时的同时，也报告它搬运载荷的速率。两者都是对既有产物的纯渲染器，都不作为门禁，这也是二者都保留|| true的原因：格式化失败既不能把绿色扫描变红，也不能掩盖真正的失败。若不接入，这个消费者只对手动执行脚本的人存在 —— 该工作流唯一的其他输出是中立的按 case JSON 产物，因此一个没有接线的渲染器在本仓库里根本无法触达任何读者。拟合质量属于结果本身，而非脚注
穿过 ladder 的直线是一个模型，而在真实数据上有时是个糟糕的模型。曾经有两档噪声在 B200 上拟合出
beta = 1018 GB/s每 GPU —— 高于链路单向峰值，物理上不可能 —— 而其 R² 仅为 0.29，却与相邻 R² = 0.99 的拟合打印得一模一样。单看 R² 也不够：一条全程受延迟主导的 dispatch ladder 拟合出 3763 GB/s 而 R² = 0.92，因为 alpha 解释了几乎全部变化、斜率不可辨识。因此Fit把r2、点数与bandwidth_share作为结果而非诊断信息返回；beta 仅在 R² 与顶档传输项占比双双达标时才输出；而当 ladder 从未接近零字节时(预填充 ladder 从 T=1024 起步)alpha 会被标记为外推值。有意不做的事：不引入按 SKU 的峰值带宽表来校验 beta。那会把本属于
platform_config.json的硬件知识塞进消费者，既容易过时又会自行产生误导；真正抓住上述异常的是拟合质量门禁。数值该怎么读
写在模块 docstring 中：逻辑字节包含 rank 路由给自己、根本不过互联的那部分拷贝，因此实际上链流量为
(1 - routing.locality.local_rank_fraction)× 报告值；而 beta 是边际速率，理应高于每一个实测点。#2422 已合入，因此有一处取舍值得在此说明：GB/s 由
components[c](即跨 rank MAX 延迟)推导，以便与紧邻的延迟表格保持一致。#2422 的cross_rank_min_us是排除 skew 后的对照量，所以一个被 skew 抬高的测点在这里会表现为更低的带宽。把速率改用 MIN、或同时输出两者，属于后续工作，本 PR 不预设该结论。验证
unittest discover会自动收集)。jobs.sweep.steps中。summarize.py正常渲染表格，bandwidth.py正常渲染各组件的逐点 GB/s 与拟合行，确认渲染路径、百分位键、locality 后缀与拟合门禁都能在真实文档结构上执行。main(含 CollectiveX: stop charging rank entry skew to component latency / CollectiveX：不再把 rank 进入时间差计入组件延迟 #2422)，因此满足perf-changelog.yaml仅允许新增的校验。