Skip to content

Merge the 2026-09-13 hardening commits with the October updates - #3

Merged
LUOaini1213 merged 8 commits into
mainfrom
hardening-2026-09-13
Nov 1, 2026
Merged

Merge the 2026-09-13 hardening commits with the October updates#3
LUOaini1213 merged 8 commits into
mainfrom
hardening-2026-09-13

Conversation

@LUOaini1213

@LUOaini1213 LUOaini1213 commented Nov 1, 2026

Copy link
Copy Markdown
Owner

Brings the seven hardening commits from 2026-09-13, which were never pushed, together with the October work already on main (PR #1 durable OTLP export, PR #2 recoverable Agent deletion).

The seven local commits

  • 3feda83 Rank spans that failed above spans the Run's collapse merely closed
  • 3dd0246 Make all three replay Runs reachable, and stop the no-key demo warning about a key it does not need
  • d2133d7 Put the Trace Plane screenshots in the README, re-shot from the current build
  • b2d5bda Remove the organizer-side challenge specs from docs/
  • 6e114fb Update the test counts in the CI comment to match the suite
  • ae88ac2 Drop the hard-coded commit count from the authorship line
  • 51253f3 Show the pre-run policy deny in the README, and pin its prompt in the smoke run

Conflict resolution (merge commit c14f1a4)

  • apps/web/src/App.tsx (only textual conflict): both sides fixed the runtime-configuration banner that showed in replay. RuntimeConfigBanner from main is kept, since runtime-config.test.tsx imports it, and it now renders the text from runtimeConfigWarning in apps/web/src/system.ts. The helper also takes main's runtimeProvider === "replay" check. Both system.test.ts and runtime-config.test.tsx now test one implementation. The replay starter prompts and the OTLP delivery and deletion-cleanup UI from main are all kept.
  • README.md (merged without a textual conflict): keeps the hardening edits (replay prompt table, re-shot Trace Plane screenshots, the pre-run deny section, no link to the expired Devpost docs/SUBMISSION.md, no hard-coded commit count) and keeps main's OTLP demo, deletion-recovery demo, durable-export feature line, new OTEL_* variables and the npm run check OTLP paragraph.
  • .github/workflows/ci.yml (merged without a textual conflict): has every step from both sides, including smoke:otlp. The comment's test count was out of date on both sides (22/160 on the hardening branch, 21/150 on main). I counted the merged suite again: 28 test files and 5 + 186 + 80 = 271 tests.

Local checks (Windows 11, Node 24.19.0; CI also runs Node 22)

  • npm ci, npm run typecheck, npm run build, npm run smoke, npm run smoke:replay and npm run smoke:otlp all passed.
  • npm test: contract 5/5 and web 80/80 passed. The server suite had 185/186 passing. On this machine, one or two timing-sensitive tests from the October PRs fail intermittently: otlp-outbox.test.ts, deletion-recovery.test.ts, otlp-delivery-http.test.ts and the 20 ms cancel test in replay-runner.test.ts. A different test fails on each run. They exceed vitest's 1 s expect.poll window or its 5 s test timeout. With --expect.poll.timeout=10000, otlp-outbox.test.ts passed 34/34 in three runs out of three. The OTLP and deletion code is the same as on main, whose Linux CI is green. This PR does not change those tests.
  • I checked the README screenshot captions against the merged build by running apps/server/dist/replay-dev.js at REPLAY_SPEED=1 over HTTP. All of them still hold: the successful Run has 12 spans over a 9.3 s timeline with 21518 in / 1093 out tokens. The failed Run has 6 problem spans, and the failing step is execute_tool shell with exitCode 1. The in-flight deny has 5 problem spans, and the failing step is policy.live with protected-env-file. The pre-run deny is 2 spans and lands on policy.check.

Needs a human look

  • The Trace Plane screenshots were taken before the October UI change. The current build shows an "OTLP delivery · Export off" block in the trace panel, and the PNGs do not.

🤖 Generated with Claude Code

LUOaini1213 and others added 8 commits September 13, 2026 18:31
@LUOaini1213 @claude
Rank spans that failed above spans the Run's collapse merely closed
3feda83 
"Open failing step" landed on the wrong span in both failure fixtures, and had
done so for a long time.

当一个 Run 中途死掉，`endOpenSpans` 会把所有还开着的 span 一律关成
`errorText: "Run ended before this step completed"`（deny 路径是
"Terminated by the policy gate ..."）。这句占位话在 `problemSpans` 的排序里
和真实的 `exitCode: 1` 一样算「有诊断属性」，第 2 条规则分不出来；两者又是
`runtime.spawn` 下的兄弟节点，深度打平；于是第 3 条「最早优先」让 40ms 就
开始的 `chat` 赢过 1600ms 才失败的命令。UI 里那颗按钮取的正是 `[0]`。

所以在原有规则前面加一条：自己失败的 span 排在「只是还没关」的 span 之前。
判据用 `attributes.unterminated`（`TraceCollector.endOpenSpans` 早就在打这个
标记，这里只是第一次去读它）。写成布尔比较而不是
`Number(left.attributes.unterminated) - Number(...)`：两边都没有该属性时那
个写法是 `NaN`，而 `if (x !== 0) return x` 对 NaN 为真，比较器会直接返回
NaN，后面的深度与顺序规则从此永远不执行——排序会退化成不一致比较器。
第二条新增用例就是钉这件事的。

本地用编译后的 `apps/server/dist/replay-dev.js` 走 HTTP 跑两个失败夹具复核：
- "Fix the failing test in this workspace." 现在落在
  `execute_tool shell` / `exitCode: 1`（原先是 `chat`）
- "Summarize the config files in this workspace." 现在落在
  `policy.live` / `ruleId: protected-env-file`（原先也是 `chat`）
后者更要紧：README 的 3 分钟演示脚本第 5 步写的就是「the denied policy span
is highlighted」。

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@LUOaini1213 @claude
Make all three replay Runs reachable, and stop the no-key demo warnin…
3dd0246 
…g about a key it does not need

Two defects that only the browser could see, in the path the README leads with.

一、三个夹具有两个点不到。夹具靠提示词关键字选择，而 Playground 上那三个
启动提示是官方启动套件自带的，一个关键字都不命中：第一条撞上 `cli` 走成功
夹具，另外两条落到 default——也是同一个成功夹具。评审照 README 跑
`npm run demo`，能复现的只有成功那一种；失败步定位和运行中拒绝这两样
——正是 Trace Plane 存在的理由——只有翻 `fixtures/replay/*.json` 的 `match`
数组才找得到。

把三条提示词、各自必须命中的夹具、以及各自应该结束在什么状态，写成
`apps/contract/src/replay-demo.ts` 里的一份清单，四个地方共用：Playground 的
按钮、`replay-runner.test.ts`（断言每条提示词确实选中它声明的夹具，并用这三条
提示词驱动三个控制面 Run 去断言状态）、CI 的 `smoke:replay`，以及 README。
smoke 里还加了一条：README 必须逐字包含这三句话，否则报错——文件和清单
脱节正是这个缺陷的成因。

二、无 key 演示首屏挂着「Runtime configuration needed — Set ARK_API_KEY and
ARK_MODEL in .env」。replay 模式本来就没有 key 也不需要（`agent-service.ts`
明确放行，无 key 也能在 Playground 里跑完整 Run），所以这条提示在 replay 下
是错的，不只是吓人。另外 `system` 初值是 null，`!system?.arkConfigured` 对
null 也成立，于是**任何模式**在 `/api/system` 返回之前都会先闪一下这条横幅。

把判断抽成 `apps/web/src/system.ts` 的 `runtimeConfigWarning`：未加载和
replay 都返回 null，其余分支照旧。`apps/web` 此前只有两个纯函数测试、零处
渲染断言，所以顺带补了 `system.test.ts` 把首屏这两条行为钉住。

本机用 headless Chrome 载入构建产物复核：banner 从首帧到稳态都没出现过
（MutationObserver 全程监听），侧栏 Runtime 卡片显示 "Replay of recorded
Codex events (no model, no key)"，三个启动按钮就是上面那三句提示词。

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@LUOaini1213 @claude
Put the Trace Plane screenshots in the README, re-shot from the curre…
d2133d7 
…nt build

四张 Trace Plane 截图早就提交在 `docs/assets/` 里，全仓引用为零；README 里
唯一露出的两张 jpg，恰恰是正文自己声明「come from the official starter」的
Agent 列表和创建表单。读者看不到这个仓真正做的那一块。

但那四张不能直接挂上去——它们拍摄之后代码变了：图里有一个已经不存在的
"Export OTLP" 按钮，有一条现在不再产生的 `runtime.event` span，成功 Run 是
13 个 span（现在是 12 个），`failing-step.png` 里「Open failing step」选中的
还是修好排序之前的落点。挂过期截图等于挂一份过期证据。

所以用当前构建重拍：起 `apps/server/dist/replay-dev.js`（就是 `npm run demo`
的服务端，REPLAY_SPEED 默认 1，所以时间轴是真实节奏），用 headless Chrome
经 CDP 载入 `apps/web/dist`，走 HTTP 触发三个 Run，点开 Trace Plane、按下
「Open failing step」后截 `section.trace-panel`。四张全部重拍，包括没有被
README 引用的 `policy-denied.png`。

README 侧同时做了三处：
- 无 key 上手段补上三条提示词各演示什么，并把成功 Run 的瀑布图放在紧接着的
  位置，让「这是什么」和「它长什么样」挨在一起；
- Screenshots 段改成先放失败步、再放运行中拒绝，官方启动套件那两张降到最后
  并写明来历；
- 删掉第一屏那条指向 `docs/SUBMISSION.md` 的链接。那是一份 2026-09-01 已过期
  的 Devpost 投稿操作说明，通篇 "we"、"Every teammate"，和「个人开源项目」的
  口径当面撞车；把已有的分工陈述补完整更准确：`git shortlog -sne` 显示 32 个
  commit 里 31 个是他本人的，剩下 1 个是官方启动套件导入。

图注里的数字都来自这次重拍时打印的实测值：成功 Run 12 span / 9.3s /
21518 in / 1093 out tokens，失败 Run 问题计数 1/6，拒绝 Run 1/5。

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@LUOaini1213 @claude
Remove the organizer-side challenge specs from docs/
b2d5bda 
`docs/` 里躺着 10 个 `hackathon-v2-*.xml`，共 210 行，全仓零引用，README 的
Documentation 段也没列。内容不是这个项目，是赛题说明书的组织方视角：
`hackathon-v2-section-9.xml` 开头是「Organizers should complete these checks
before distributing the challenge」，下面 8 条 checkbox 包括「Brief judges on
track-specific success tests and the 40/25/20/15 rubric」；skeleton 里写的是
「Audience: second- and third-year undergraduates · Duration: 3 days」。

赛道编号还和本仓自相矛盾：这些文件里是 Track #5 与 Track A/B/C，README 写的
是 Track 1 / Glass Box。一个会翻 `docs/` 的读者先要分辨这个仓到底是参赛作品
还是赛事组织材料，再要分辨他做的是哪一块。

文件本身留档在本机 `04_作品与代码/_归档/track1-docs-hackathon-v2-xml/`，
仓库历史里也仍然完整。

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@LUOaini1213 @claude
Update the test counts in the CI comment to match the suite
6e114fb 
这一轮补了 7 条用例、多了一个测试文件（`apps/web/src/system.test.ts`），
workflow 注释里「21 个测试文件 / 150 个用例」就不再成立了。

重新数过：`git ls-files | grep -c '\.test\.ts$'` = 22，全部 `it(` 合计 160，
与 `npm test` 三个 workspace 的汇总 5 + 131 + 24 = 160 逐位吻合。

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@LUOaini1213 @claude
Drop the hard-coded commit count from the authorship line
ae88ac2 
「of the 32 commits here … the other 31 are mine」里的 32 是写这句话时的线上值。
本地这五个提交推上去之后仓库就是 37 个，这句话当场变假——一个用来自证署名的句子
自己先对不上数，比不写更糟。

改成不带计数的说法：起点是一个 starter-kit 导入提交，其余都是自己的。
`git shortlog -sne` 仍然打得出这个划分，验证路径没丢，只是不再依赖一个会过期的数。

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@LUOaini1213 @claude
Show the pre-run policy deny in the README, and pin its prompt in the…
51253f3 
… smoke run

docs/assets/policy-denied.png 一直是孤儿：仓里没有任何一处引用它，
git grep policy-denied 在全部历史上都无命中。它拍到的又不是重复内容——
README 原有的两张 Trace Plane 截图讲的是「跑起来之后失败」和「跑到一半被拦」，
这张讲的是第三种、也是策略面最先生效的那一种：提示词在 Run 起来之前就被
policy.check 拒掉，整条 trace 只有两个 span，没有 runtime.spawn，runner
根本没被调用。所以选择用上它，而不是删掉。

截图里的每一项都是对着编译产物回算出来的，不是照着图写的：起
apps/server/dist/replay-dev.js（无 ARK_API_KEY / 无 ARK_MODEL），
对一个名为 Leaker 的 agent 发这条提示词，走 HTTP 读 /api/runs/:id/trace，
得到 status=failed、2 个 span（invoke_agent Leaker denied、policy.check
denied）、pickFailingSpan 落在 policy.check、ruleId=protected-env-file、
reason="Attempt to read the protected secret fixture"——与图中计数器 1/2、
选中行和 JSON 面板逐字一致。时长不写进正文：实测是 1ms，UI 显示 0ms，
这种会漂的数不该写死。

smoke:replay 里补了同一条路径的断言。这条提示词不是 Playground 的
starter 按钮，是手打的，原来的三条 starter 断言覆盖不到它；如果哪天策略
规则改了不再命中，README 这一段和这张图会一起变成假话而 CI 全绿。
断言里连 agent 名字都取自截图，保证图里的 span 名和断言里的是同一个字符串。
已验证这个守卫会咬：把 README 里的提示词改一个字，smoke 立即红。

本机按 .github/workflows/ci.yml 五步全跑过：typecheck、test（22 文件 /
160 例全绿）、build、smoke、smoke:replay，全部 exit 0。

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@LUOaini1213 @claude
Merge origin/main into the 2026-09-13 hardening commits
c14f1a4 
App.tsx was the only textual conflict: both sides had fixed the runtime banner
that warned about a missing key in replay. The exported RuntimeConfigBanner from
origin/main stays (runtime-config.test.tsx imports it) but now renders the text
from runtimeConfigWarning in system.ts, which also takes origin/main's
runtimeProvider === "replay" guard, so both test files pin one implementation.
README.md and ci.yml merged cleanly: the hardening edits (screenshots, replay
prompt table, no SUBMISSION.md link, no hard-coded commit count) are kept with
the October OTLP and deletion-recovery sections, CI keeps every step from both
sides, and the CI comment's test count is recounted for the merged suite
(28 test files, 5 + 186 + 80 = 271 tests).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@LUOaini1213
LUOaini1213 merged commit 09b4cdb into main Nov 1, 2026
4 checks passed
@LUOaini1213 LUOaini1213 mentioned this pull request Nov 3, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@LUOaini1213