Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 70 additions & 27 deletions docs/agents/cli-e2e-tests.md
Original file line number Diff line number Diff line change
@@ -1,55 +1,86 @@
# CLI E2E 测试规范

## 架构分层

| 层级 | 路径 | 测什么 |
| --------------- | ----------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| **共享基建** | `packages/e2e` | gating、子进程 runner、output、globalSetup(`private`,不发布) |
| **命令 E2E** | `packages/commands/tests/e2e` | help、缺参、dry-run、live(gated);每用例最小路由 |
| **bl smoke** | `packages/cli/tests/e2e/registry.smoke.e2e.test.ts` | 产品 map 全部 path `--help`、分组 help、根 help |
| **kscli smoke** | `packages/kscli/tests/e2e/registry.smoke.e2e.test.ts` | 从 `kscli/src/commands.ts` 推导 path/分组;identity(`--version`、`search --help` path) |
| **runtime** | `packages/runtime/tests` | `proxy.e2e`、console 跨域 flag 拒绝 |

**依赖边界**:`e2e` → `core`;`commands/tests` → `e2e` + `commands/src`;产品 tests → `e2e` + 各自 `src`。**禁止**产品 import `commands/tests/**`(子进程 spawn harness 路径除外)。

## 触发条件

- 新增/修改 `packages/commands/src/commands` 下的 command 实现
- 新增/修改 `packages/cli/src/commands.ts` 的 `bl` 命令路径 map
- 新建或扩展 `packages/cli/tests/e2e/*.e2e.test.ts` 用例
- 为命令补 help / 缺参 / dry-run / 真实集成测试
- 新建或扩展 `packages/commands/tests/e2e/<topic>.e2e.test.ts`
- 新增 bl 产品 path → `registry.smoke` 自动覆盖 leaf path;commands topic 测试在 `topic-routes.ts` 补最小路由

跑测与环境变量见 `.cursor/skills/bailian-cli-e2e/SKILL.md`。

以上情况必须同步维护 `packages/cli/tests/e2e/<topic>.e2e.test.ts`。跑测与环境变量见 `.cursor/skills/bailian-cli-e2e/SKILL.md`。
> **规则**:共享 command 行为在 `commands/tests/e2e`;产品 map、identity、CLI-only 命令留在对应产品 `tests/e2e`。

## 文件与工具

- 路径:`packages/cli/tests/e2e/<kebab-topic>.e2e.test.ts`
- 框架:`vite-plus/test`;子进程跑 CLI:`runCli` from `./helpers.ts`
### commands E2E

- 路径:`packages/commands/tests/e2e/<kebab-topic>.e2e.test.ts`
- 子进程:`runCommandE2e(routes, args)` from `./helpers.ts`(spawn `harness/main.ts`,`routes` 为本 topic 最小 path → export 映射)
- fixtures:`packages/commands/tests/e2e/fixtures/`
- 路由常量:`topic-routes.ts`(按 topic 维护,**非**全量产品 map)

### 产品 smoke

- bl:`runCli` from `packages/cli/tests/e2e/helpers.ts`
- kscli:`runKscli` from `packages/kscli/tests/e2e/helpers.ts`

### 共享

- gating / output / runner:`e2e/gating`、`e2e/output`、`e2e/runner`
- globalSetup:根 `vite.config.ts` → `packages/e2e/src/global-setup.ts`
- 解析 JSON stdout:`parseStdoutJson`;输出目录:`makeE2eOutputDir(e2eLabelFromMetaUrl(import.meta.url))`
- 长任务:`cliTimeoutPrefix()`;视频用例加 `test(..., 3_600_000)` 等显式超时

## 双层 describe(固定结构)

```ts
// 1) 不 skip:分组 + --help,无密钥、无真实 API
// 1) 不 skip:--help,无密钥、无真实 API(分组 help 由 bl registry.smoke 覆盖)
describe("e2e: <topic>", () => {
test("<group> 分组展示子命令帮助且成功退出", ...);
test("<subcommand> --help 正常退出", ...);
});

// 2) skipIf:缺参 / dry-run / 真实集成;原有集成用例放最后、勿改逻辑
// 2) skipIf:缺参 / dry-run / 真实集成
describe.skipIf(<ready>)("e2e: <topic>(DashScope …)", () => {
test("缺少 --<flag> 时退出为用法错误 (2)", ...);
test("<cmd> --dry-run ...", ...); // 若适用
test("【model】真实流程", ..., LONG_TIMEOUT);
});
```

## skip 条件(helpers.ts
## skip 条件(`e2e/gating`,commands helpers re-export

| 场景 | 条件 |
| ------------------- | ----------------------------------------------------- |
| 文本/搜索/记忆/配置 | `isDashScopeE2EReady()` |
| 图像/语音 | `isBailianE2EMediaEnabled() && isDashScopeE2EReady()` |
| 视频 | `isBailianE2EVideoEnabled() && isDashScopeE2EReady()` |
| 知识库 | `isKnowledgeE2EReady()` |
| 视频 download/task | 另需 `BAILIAN_E2E_VIDEO_TASK_ID` |
| 场景 | 条件 |
| ----------------------- | ---------------------------------------------------------------------------------------------------------- |
| 文本/搜索/记忆/配置 | `isDashScopeE2EReady()` |
| 图像/语音 | `isBailianE2EMediaEnabled() && isDashScopeE2EReady()` |
| 视频 | `isBailianE2EVideoEnabled() && isDashScopeE2EReady()` |
| 视频 download/task | 另需 `BAILIAN_E2E_VIDEO_TASK_ID` |
| 知识库 chat/search live | `isChatE2EReady()` / `isSearchE2EReady()`(`knowledge chat/search`,需 `BAILIAN_WORKSPACE_ID` + agent ID) |

## 用例类型

1. **分组 help**:`runCli(["image"])` → `exitCode === 0`,stdout+stderr 含子命令名
2. **--help**:`runCli([..., "--help"])` → stderr 含主要 flags
3. **缺参**:带一个无害全局 flag(如 `--quiet`)且不传 required flag → `exitCode === 2`,stderr 匹配 `--flag|Missing required argument`
4. **--dry-run**:仅当实现在联网/上传/写盘**之前**返回;断言 stdout JSON/文本,不入网
5. **真实集成**:保留既有用例名称与断言;放在 skip 块**末尾**
1. **--help**:`runCommandE2e(ROUTES, [..., "--help"])` → stderr 含主要 flags
2. **缺参**:带无害全局 flag(如 `--quiet`)且不传 required flag → `exitCode === 2`
3. **--dry-run**:实现在联网/上传/写盘**之前**返回;断言 stdout JSON/文本
4. **真实集成**:放在 skip 块**末尾**

## 增删命令同步

- **commands export** + **topic 路由**(`topic-routes.ts` 或测试文件内 `ROUTES`)+ **产品 map**(`cli/commands.ts` / `kscli/commands.ts`)
- 分组 help 由产品 `registry.smoke` 负责,无需在 commands 重复

## 安全与例外

Expand All @@ -60,24 +91,36 @@ describe.skipIf(<ready>)("e2e: <topic>(DashScope …)", () => {

## 新增 command 检查清单

- [ ] `packages/commands/src/index.ts` 导出 + `packages/cli/src/commands.ts` 暴露路径 + `tests/e2e/<topic>.e2e.test.ts`(新建或扩展)
- [ ] `packages/commands/src/index.ts` 导出 + `packages/cli/src/commands.ts` 暴露路径 + `topic-routes.ts` 补最小路由
- [ ] `packages/commands/tests/e2e/<topic>.e2e.test.ts`(新建或扩展)
- [ ] 若改了 `usageArgs` / `flags` / `exampleArgs`,跑 `pnpm --filter bailian-cli run generate:reference` 更新 `skills/bailian-cli/reference/` 并提交
- [ ] 顶层:分组 help + 子命令 `--help`(多子命令则各一条 help)
- [ ] 子命令 `--help`(分组 help 由 bl `registry.smoke` 覆盖
- [ ] skip 块:每个 required flag 缺参;可 dry-run 则加一条
- [ ] 至少一条真实集成(或说明为何仅 smoke);不破坏已有集成用例顺序
- [ ] `pnpm test packages/cli/tests/e2e/<file>` 通过
- [ ] `vp test packages/commands/tests/e2e/<file>` 通过

## 调试命令

```sh
pnpm --filter bailian-cli-commands exec vp test packages/commands/tests/e2e/text-chat.e2e.test.ts
pnpm --filter bailian-cli exec vp test packages/cli/tests/e2e/registry.smoke.e2e.test.ts
pnpm --filter knowledge-studio-cli exec vp test packages/kscli/tests/e2e/registry.smoke.e2e.test.ts
pnpm --filter bailian-cli-runtime exec vp test packages/runtime/tests/proxy.e2e.test.ts
```

## 示例片段

```ts
import { FOO_ROUTES } from "./topic-routes.ts";

test("foo bar 缺少 --prompt 时退出为用法错误 (2)", async () => {
const { stderr, exitCode } = await runCli(["foo", "bar", "--quiet"]);
const { stderr, exitCode } = await runCommandE2e(FOO_ROUTES, ["foo", "bar", "--quiet"]);
expect(exitCode).toBe(2);
expect(stderr).toMatch(/--prompt|Missing required argument/i);
});

test("foo bar --dry-run 仅输出计划", async () => {
const { stdout, stderr, exitCode } = await runCli([
const { stdout, stderr, exitCode } = await runCommandE2e(FOO_ROUTES, [
"foo",
"bar",
"--dry-run",
Expand All @@ -97,4 +140,4 @@ test("foo bar --dry-run 仅输出计划", async () => {
- **E2E**:单条/少量调用、断言固定、可进 `vp test`(见上文 skip 条件)
- **批量压测**:`packages/cli/tests/stress/run.mjs` + `targets/*.mjs`,并发 + 报告,**仅手动** `pnpm run test:stress -- <target>`

勿把压测并入 E2E 或默认 CI。详见 [stress-batch-tests.md](stress-batch-tests.md)
勿把压测并入 E2E 或默认 CI。详见 [stress-batch-tests.md](stress-batch-tests.md).
12 changes: 7 additions & 5 deletions docs/agents/command-add-remove.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,15 +92,17 @@ packages/commands/src/index.ts

### D. 测试层

- [ ] 按 [cli-e2e-tests.md](cli-e2e-tests.md) 新建或更新 `packages/cli/tests/e2e/<topic>.e2e.test.ts`
- [ ] 删除命令时一并删对应 e2e / README 示例 / reference 生成结果
- [ ] 如果 shared command 在不同入口路径下复用,至少确保 `bl` 入口 e2e 覆盖;`kscli` 入口改动需补对应入口测试或手工 smoke
- [ ] 按 [cli-e2e-tests.md](cli-e2e-tests.md) 新建或更新 `packages/commands/tests/e2e/<topic>.e2e.test.ts`
- [ ] 同步 `packages/commands/tests/e2e/topic-routes.ts`(该 topic 的最小 path → export 映射)
- [ ] bl 产品 path 变更由 `registry.smoke` 自动覆盖;kscli 变更同步 `kscli/src/commands.ts` 与 `registry.smoke`
- [ ] 删除命令时一并删对应 commands e2e / README 示例 / reference / topic 路由条目
- [ ] 如果 shared command 在不同入口路径下复用,至少确保 commands e2e 覆盖 `bl` path;`kscli` 入口改动需补对应 smoke 或说明不测 flat path live

### E. 重命名特殊处理

- [ ] 全仓 grep **旧命令名字符串**,确保以下位置全部更新:
- `packages/cli/src/commands.ts` map key
- `packages/kscli/src/main.ts` map key(如适用)
- `packages/kscli/src/commands.ts` map key(如适用)
- 用户可见 hint / README / tests
- `skills/bailian-cli/reference/`(重建后检查并提交)
- [ ] 检查 `usageArgs` / `exampleArgs` 没有硬编码旧的 `bl <path>` 前缀
Expand All @@ -111,7 +113,7 @@ packages/commands/src/index.ts
pnpm run sync:skill-assets
pnpm -F bailian-cli exec tsx src/main.ts <new-command> --help
pnpm -F bailian-cli exec tsx src/main.ts
vp test packages/cli/tests/e2e/<topic>.e2e.test.ts
vp test packages/commands/tests/e2e/<topic>.e2e.test.ts
```

如改了 `kscli` 入口:
Expand Down
1 change: 1 addition & 0 deletions packages/cli/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@
"ajv": "catalog:",
"boxen": "catalog:",
"chalk": "catalog:",
"e2e": "workspace:*",
"typescript": "^6.0.2",
"undici": "catalog:",
"vite-plus": "0.1.22",
Expand Down
Loading