Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/quality.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ jobs:
--build-arg "DEBIAN_SECURITY_MIRROR=${DEBIAN_SECURITY_MIRROR}"
--tag luowang:quality .

- name: Phase 9 acceptance and quality matrix
run: docker run --rm --init --ipc=host luowang:quality npm run test:acceptance
- name: Local acceptance and quality matrix (live is not run in CI)
run: docker run --rm --init --ipc=host luowang:quality npm run test:acceptance:local

- name: Build production image
run: >-
Expand Down
41 changes: 24 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,20 @@
# LuoWang

罗网(LuoWang)是一个独立部署的 AI 场景测试 Harness。当前仓库已发布 v0.1.0,并已实现 Phase 0–9 的主要模块:安全配置控制台、唯一 GitHub 目标仓库索引、Main → Runner → Reviewer → Main Run、受控 Playwright MCP UI 执行、S3-compatible OSS 证据 Gateway、幂等归档和持久 FIFO 自动化队列,以及长期场景生命周期、三种场景维护模式、陌生项目初始化和运维控制台。

v0.1.0 的 Phase 9 验收已使用隔离的本地样例仓库、样例 Web 应用和 test doubles 完成 34 个 AC 的自动化回归;这证明的是 local 基线,不是生产闭环的 live 联合验收。当前尚未完成真实 Provider + Pi Agent + Playwright MCP + 私有 OSS + GitHub + 非生产应用的联合证明,因此不能把 v0.1.0 或当前 `develop` 视为下一版本的发布验收已经通过。
罗网(LuoWang)是一个独立部署的 AI 场景测试 Harness。当前仓库已完成 Phase 0–9 的主要模块,并已发布 v0.1.0:除了安全配置控制台、唯一 GitHub 目标仓库索引、Main → Runner → Reviewer → Main 的本地 Run、受控 Playwright MCP UI 执行、S3-compatible OSS 证据 Gateway、幂等归档和持久 FIFO 自动化队列,还支持长期场景生命周期、三种场景维护模式、陌生项目初始化,以及完整运维控制台。34 个原 MVP AC 已有本地 fixture/自动化回归;真实 GitHub、Provider、Pi、Playwright MCP、私有 OSS 和非生产应用的联合验收尚未完成,不能据此声明可发布。

## 本地启动

优先使用 Docker Compose 复现固定的 Node、原生依赖和 Chromium 环境:
优先使用下文的 Docker Compose 或固定 quality/runtime 镜像复现。原生运行需要 Node.js 24 和 npm;当 `better-sqlite3` 没有匹配的预编译包时,还需要 `python3`、`make` 和 `g++`。生产模式:

```bash
npm ci
npm run build
npm start
```

默认只监听 `127.0.0.1:3000`。打开 <http://127.0.0.1:3000/> 可查看控制台壳,健康检查地址为 <http://127.0.0.1:3000/health>。数据默认保存在 `/data`;本地开发可以设置 `LUOWANG_DATA_DIR` 到可写目录。执行 `npm run dev` 可以同时启动 Vite 和开发服务器。

也可以使用 Docker Compose:

```bash
# 首次启动空数据卷前必须设置这两个值;不要把真实值提交到 Git。
Expand All @@ -19,25 +27,24 @@ docker compose down

Compose 将数据保存到 `luowang-data` 卷,并把宿主机端口绑定到 `127.0.0.1`。管理员密码只在空数据库首次启动时读取;主密钥只用于进程内派生 Secret Store 密钥,二者都不会写入 SQLite。

如需直接在宿主机运行,需要 Node.js 24 和 npm 10 或更高版本。`better-sqlite3` 等依赖没有适用的预编译包时,`npm ci` 还需要系统已安装 `python3`、`make` 和 `g++`;缺少这些工具时请优先使用 Docker,不要跳过依赖安装或质量检查。

```bash
npm ci
npm run build
npm start
```

默认只监听 `127.0.0.1:3000`。打开 <http://127.0.0.1:3000/> 可查看控制台壳,健康检查地址为 <http://127.0.0.1:3000/health>。数据默认保存在 `/data`;本地开发可以设置 `LUOWANG_DATA_DIR` 到可写目录。执行 `npm run dev` 可以同时启动 Vite 和开发服务器。

## 验收状态

当前命令
验收命令按证明范围严格分层

```bash
npm run test:acceptance
# 无外部凭据;真实经过 Pi SDK Session 和本地模型协议服务。兼容别名 npm run test:acceptance 也只指向 local。
npm run test:acceptance:local

# 真实外部联合验收;缺少任一必需输入时列出 missing 名称并非零退出。
npm run test:acceptance:live

# 先执行公共质量与 local,再执行 live;live blocked/failed 时非零退出。
npm run test:acceptance:release
```

该命令当前只代表 **local acceptance**。它会运行公共质量命令,并使用临时 Git bare 仓库、Cynos 官网登录/注册样例应用、SQLite、队列、归档和 headless Chromium 回归 34 个 AC;报告保存在 `.cynos/acceptance/<timestamp>/`。其中 Agent Run 证明注入 `FixtureSessionFactory`,浏览器证明直接使用 Playwright;它们不能代替真实 `createAgentSession()`、模型 custom tool 循环和 Playwright MCP 联合路径。正式的 local/live/release 分层命令将在生产闭环 Closure Phase 6 提供。
`local` 使用临时 Git bare 仓库、样例应用、SQLite、队列、归档、headless Chromium 和本地可控模型协议服务;Agent 流程真实调用生产 `createAgentSession()`,但本地 test double 只能证明 `local.status=passed`。报告保存在 `.cynos/acceptance/<timestamp>-<mode>/`,分别记录 `local.status`、`live.status`、`release.status`、资源检查、逐 AC 证据和命令。CI 只运行 local,并明确不读取 live Secret。

Closure 6 只提供 live 输入预检和诚实的 blocked 结果;真实 GitHub、Provider、Playwright MCP、私有 OSS、非生产应用和测试账号的联合流程在 Closure 7 执行。在它通过前,`release.status` 必须保持 blocked,不能用现有 GitHub 只读 smoke 或本地服务冒充 live passed。

当前可选的 live 路径只包含有限的 GitHub smoke,不覆盖真实 Provider、Pi Agent、Playwright MCP、私有 OSS 和非生产应用联合验收。它不会默认执行;如需运行,必须显式提供 `LUOWANG_ACCEPTANCE_LIVE=1`、`LUOWANG_SMOKE_REPOSITORY=https://github.com/cynos-ai/cynos-website` 和临时 `LUOWANG_SMOKE_GITHUB_TOKEN`。即使该有限 smoke 通过,也不能视为 live 或 release 证明完成。

Expand Down
5 changes: 4 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,10 @@
"lint": "eslint .",
"start": "node dist/server/main.js",
"test": "vitest run",
"test:acceptance": "tsx tests/acceptance/phase9.ts",
"test:acceptance": "npm run test:acceptance:local",
"test:acceptance:local": "tsx tests/acceptance/closure.ts local",
"test:acceptance:live": "tsx tests/acceptance/closure.ts live",
"test:acceptance:release": "tsx tests/acceptance/closure.ts release",
"test:e2e": "npm run build && tsx tests/e2e/smoke.ts && tsx tests/e2e/phase8-ui-smoke.ts",
"test:e2e:github": "npm run build && tsx tests/e2e/phase2-github-smoke.ts",
"typecheck": "tsc --noEmit -p tsconfig.json && tsc --noEmit -p tsconfig.web.json",
Expand Down
54 changes: 44 additions & 10 deletions src/server/automation/service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ class DefaultAutomationService implements AutomationService {
private activeQueueId: number | null = null;
private activeRunId: string | null = null;
private dispatching = false;
private dispatchScheduled = false;
private recovering = false;

constructor(
Expand All @@ -110,12 +111,8 @@ class DefaultAutomationService implements AutomationService {

async submitTestRequest(input: TestRequestInput): Promise<AutomationSubmission> {
const queued = this.options.queue.enqueue(input);
const dispatched = await this.dispatchNext();
const current = this.options.queue.get(queued.queueId) ?? queued;
return {
queue: current,
run: dispatched?.queueId === queued.queueId ? dispatched.run : null,
};
this.kickDispatch();
return { queue: queued, run: null };
}

listQueue(): TestRequestRecord[] {
Expand Down Expand Up @@ -164,10 +161,15 @@ class DefaultAutomationService implements AutomationService {
if (!this.options.runStore) return result;

const cutoff = this.now().getTime() - retentionDays * 24 * 60 * 60 * 1_000;
const current = await this.options.runs.current();
for (const run of this.options.runStore.list()) {
if (run.archiveStatus !== 'completed') continue;
const finishedAt = Date.parse(run.finishedAt);
if (Number.isNaN(finishedAt) || finishedAt > cutoff) continue;
if (current?.runId === run.runId || this.hasPendingRunOperations(run.runId)) {
result.skippedRunIds.push(run.runId);
continue;
}
try {
await this.workspaceStore.remove(run.runId, 'completed');
result.removedRunIds.push(run.runId);
Expand Down Expand Up @@ -201,6 +203,19 @@ class DefaultAutomationService implements AutomationService {
throw new AutomationServiceError('AUTOMATION_RUN_ACTIVE', '当前 Run 正在执行,不能清理');
}
const knownRun = await this.options.runs.get(runId);
const storedRun = this.options.runStore?.get(runId);
if (!storedRun || storedRun.archiveStatus !== 'completed') {
throw new AutomationServiceError(
'AUTOMATION_CLEANUP_FAILED',
'Run 归档尚未完成,必须保留 report、场景 patch 和 Issue 重试工件',
);
}
if (this.hasPendingRunOperations(runId)) {
throw new AutomationServiceError(
'AUTOMATION_CLEANUP_FAILED',
'Run 仍有关联的队列、归档或发布重试操作,不能清理',
);
}
let removed = false;
for (const placement of ['running', 'completed'] as const) {
if ((await this.workspaceStore.list(placement)).includes(runId)) {
Expand All @@ -218,6 +233,13 @@ class DefaultAutomationService implements AutomationService {
return this.options.state;
}

private hasPendingRunOperations(runId: string): boolean {
return this.options.queue
.list()
.filter((item) => item.runId === runId || queueRunId(item) === runId)
.some((item) => item.status !== 'completed' || item.archiveStatus !== 'completed');
}

private async dispatchNext(): Promise<{ queueId: number; run: RunSummary } | null> {
if (this.activeQueueId !== null || this.dispatching || this.recovering) return null;
this.dispatching = true;
Expand Down Expand Up @@ -246,9 +268,7 @@ class DefaultAutomationService implements AutomationService {
await this.failQueueItem(item.queueId, safeMessage(error));
this.activeQueueId = null;
this.activeRunId = null;
Promise.resolve()
.then(() => this.dispatchNext())
.catch(() => undefined);
this.kickDispatch();
return null;
}
} finally {
Expand Down Expand Up @@ -290,10 +310,24 @@ class DefaultAutomationService implements AutomationService {
this.activeQueueId = null;
this.activeRunId = null;
}
await this.dispatchNext();
this.kickDispatch();
}
}

private kickDispatch(): void {
if (this.dispatchScheduled) return;
this.dispatchScheduled = true;
setImmediate(() => {
this.dispatchScheduled = false;
this.dispatchNext().catch((error: unknown) => {
this.options.logger?.error(
{ errorName: error instanceof Error ? error.name : 'UnknownError' },
'background automation dispatch failed',
);
});
});
}

private async recoverQueueItem(item: TestRequestRecord): Promise<void> {
if (item.status === 'waiting_archive') {
if (item.runId) {
Expand Down
6 changes: 5 additions & 1 deletion src/server/runs/agent-session.ts
Original file line number Diff line number Diff line change
Expand Up @@ -84,16 +84,20 @@ class PiAgentSessionFactory implements AgentSessionFactory {

class ManagedAgentSession implements AgentSession {
private disposed = false;
readonly sessionId: string;

constructor(
private readonly session: {
sessionId: string;
prompt(message: string): Promise<void>;
dispose(): void;
extensionRunner: {
emit(event: SessionShutdownEvent): Promise<unknown>;
};
},
) {}
) {
this.sessionId = session.sessionId;
}

prompt(message: string): Promise<void> {
if (this.disposed) return Promise.reject(new Error('Agent session 已释放'));
Expand Down
21 changes: 21 additions & 0 deletions src/server/runs/evidence.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,19 @@ export interface EvidenceUploadFailure {
message: string;
}

export interface EvidenceCleanupResult {
deleted: string[];
failures: EvidenceUploadFailure[];
}

export interface RunEvidenceStore {
list(): Promise<RunEvidenceFile[]>;
upload(filename: string): Promise<EvidenceReference>;
uploadAll(): Promise<EvidenceUploadResult>;
read(filename: string): Promise<EvidenceReadResult>;
readUploaded?(filename: string): Promise<EvidenceReadResult>;
cleanupLocal(): Promise<void>;
cleanupUploaded(): Promise<EvidenceCleanupResult>;
readFailureCount?: () => number;
recordReadFailure?: () => void;
reviewReadCount?: () => number;
Expand Down Expand Up @@ -156,6 +162,21 @@ class DefaultRunEvidenceStore implements RunEvidenceStore {
return this.workspace.removeEvidence();
}

async cleanupUploaded(): Promise<EvidenceCleanupResult> {
const deleted: string[] = [];
const failures: EvidenceUploadFailure[] = [];
for (const [filename, reference] of this.references) {
try {
await this.oss.deleteObject(reference.objectKey);
this.references.delete(filename);
deleted.push(filename);
} catch (error) {
failures.push({ filename, message: safeMessage(error) });
}
}
return { deleted, failures };
}

readFailureCount(): number {
return this.readFailures;
}
Expand Down
75 changes: 72 additions & 3 deletions src/server/runs/orchestrator.ts
Original file line number Diff line number Diff line change
Expand Up @@ -359,7 +359,13 @@ class DefaultRunOrchestrator implements RunOrchestrator {
}
scenarioDecision = await this.prepareScenarioPatch(workspace, prepared.repository, context);
if (scenarioDecision === 'review') {
await this.finishScenarioReviewRun(state, workspace, context);
const closure = await this.finishScenarioReviewRunner(
state,
workspace,
context,
evidenceStore,
);
await this.finishScenarioReviewRun(state, workspace, context, closure);
return;
}
let runnerCleanup: { uploaded: boolean; uploadFailed: boolean } = {
Expand Down Expand Up @@ -680,14 +686,60 @@ class DefaultRunOrchestrator implements RunOrchestrator {
return 'applied';
}

private async finishScenarioReviewRunner(
state: RunState,
workspace: RunWorkspace,
context: RunContext,
evidenceStore: RunEvidenceStore | undefined,
): Promise<ScenarioReviewClosure> {
try {
await this.finishRunner(state, workspace, context, evidenceStore);
} catch (error) {
this.addBlockingReason(context, `Runner 收尾失败:${safeMessage(error)}`);
await this.appendExecutionNotes(workspace, [
`Runner 收尾失败,特殊场景审核 Run 已阻塞:${safeMessage(error)}`,
]).catch(() => undefined);
}
const testData = await this.finalizeTestData(workspace, context);
if (!evidenceStore) {
return { testDataMessage: testData.message, evidenceDeleted: 0, evidenceDeleteFailures: 0 };
}

const cleanup = await evidenceStore.cleanupUploaded();
if (cleanup.deleted.length > 0) {
await this.appendExecutionNotes(workspace, [
`特殊场景审核不保留执行 evidence;已删除 ${cleanup.deleted.length} 个已上传 OSS 对象。`,
]);
}
for (const failure of cleanup.failures) {
this.addBlockingReason(context, `特殊场景审核 evidence 删除失败:${failure.filename}`);
}
if (cleanup.failures.length > 0) {
await this.appendExecutionNotes(
workspace,
cleanup.failures.map(
(failure) => `特殊场景审核 evidence 删除失败:${failure.filename}(${failure.message})`,
),
);
}
context.evidence = [];
state.evidence = [];
return {
testDataMessage: testData.message,
evidenceDeleted: cleanup.deleted.length,
evidenceDeleteFailures: cleanup.failures.length,
};
}

private async finishScenarioReviewRun(
state: RunState,
workspace: RunWorkspace,
context: RunContext,
closure: ScenarioReviewClosure,
): Promise<void> {
this.addBlockingReason(context, 'Run 等待场景变更人工审核,不等待 PR 合并');
const finishedAt = this.now().toISOString();
const reportContent = buildScenarioReviewReport(state, context, finishedAt);
const reportContent = buildScenarioReviewReport(state, context, finishedAt, closure);
await workspace.writer('main-b').writeReport(reportContent);
const report = parseReportMarkdown(
reportContent,
Expand Down Expand Up @@ -976,7 +1028,10 @@ class DefaultRunOrchestrator implements RunOrchestrator {
return { uploaded, uploadFailed };
}

private async finalizeTestData(workspace: RunWorkspace, context: RunContext): Promise<void> {
private async finalizeTestData(
workspace: RunWorkspace,
context: RunContext,
): Promise<ReturnType<TestDataManager['finalize']>> {
const result = (this.options.testData ?? createTestDataManager()).finalize(context.runId);
const notes = [`测试数据最终核验:${result.message}`];
if (!result.ok) {
Expand All @@ -992,6 +1047,7 @@ class DefaultRunOrchestrator implements RunOrchestrator {
notes.push(...context.blockingReasons.map((reason) => `Harness 阻塞:${reason}`));
}
await this.appendExecutionNotes(workspace, notes);
return result;
}

private async cleanupRetainedEvidence(
Expand Down Expand Up @@ -1512,10 +1568,17 @@ async function readOptionalScenarioPatch(workspace: RunWorkspace): Promise<strin
return workspace.read('scenario-changes.patch');
}

interface ScenarioReviewClosure {
testDataMessage: string;
evidenceDeleted: number;
evidenceDeleteFailures: number;
}

function buildScenarioReviewReport(
state: RunState,
context: RunContext,
finishedAt: string,
closure: ScenarioReviewClosure,
): string {
const changes = context.scenarioChanges
? `变更文件:${context.scenarioChanges.changedPaths.join(', ')}`
Expand All @@ -1541,6 +1604,12 @@ confirmed_bugs: []

${changes}

## Harness 收尾

- 测试数据:${closure.testDataMessage}
- 执行 evidence:已删除 ${closure.evidenceDeleted} 个已上传 OSS 对象;删除失败 ${closure.evidenceDeleteFailures} 个。
- 特殊归档仅保留 scenario-changes.patch 和 report.md。

## 阻塞原因

${reasons || '- 场景变更需要人工审核'}
Expand Down
1 change: 1 addition & 0 deletions src/server/runs/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ export interface AgentSessionInput {
}

export interface AgentSession {
readonly sessionId?: string;
prompt(message: string): Promise<void>;
dispose(): void | Promise<void>;
}
Expand Down
Loading
Loading