You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/Architecture.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -433,7 +433,7 @@ bun run ep:smoke-test
433
433
### Scripts (scaffold and lifecycle harness)
434
434
435
435
-**Scaffold** (`bun run scaffold`) creates new projects under `$HOME/Projects/TestRepos` (see [scripts/scaffold-test-project.ts](../scripts/scaffold-test-project.ts) and [docs/development/SCAFFOLD_USER_GUIDE.md](development/SCAFFOLD_USER_GUIDE.md)).
436
-
-**Lifecycle harness** (`bun run lifecycle-harness --seed <n>`) runs seedable E2E over real repos and the `ep` CLI. It depends on the scaffold’s output path: it uses the same root (`defaultScaffoldRootDir()` in `scripts/lifecycle-harness/src/paths.ts`). The harness discovers the monorepo root by walking up from `scripts/lifecycle-harness/src/` and looking for a root `package.json` containing `effect-patterns-hub` or a `"scaffold"` script.
436
+
-**Lifecycle harness** (`bun run lifecycle-harness --seed <n>`) runs seedable E2E over real repos and the `ep` CLI (no mocks; real network/API). It uses the same repo root as the scaffold (`defaultScaffoldRootDir()` in `scripts/lifecycle-harness/src/paths.ts`), discovers the monorepo root by walking up from its script dir, and writes JSON reports under `scripts/lifecycle-harness/reports/`. Templates are assigned in round-robin by scenario index. Full documentation: **[scripts/lifecycle-harness/README.md](../scripts/lifecycle-harness/README.md)**.
Copy file name to clipboardExpand all lines: scripts/lifecycle-harness/README.md
+13-1Lines changed: 13 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -64,6 +64,16 @@ This runs up to 10 scenarios (deterministic for seed `123`), creates repos under
64
64
-**Print summary for an existing report:**
65
65
`bun run lifecycle-harness --analyze scripts/lifecycle-harness/reports/run-20260226-172113-seed-20260226.json`
66
66
67
+
## Unit tests
68
+
69
+
The harness has a companion unit test suite under `scripts/__tests__/lifecycle-harness/` (e.g. `code-broken.test.ts`, `report.test.ts`, `scaffold-validate.test.ts`, `output-checks.test.ts`, `list-parse.test.ts`). Run from the monorepo root:
70
+
71
+
```bash
72
+
bunx vitest run scripts/__tests__/lifecycle-harness/ --config vitest.config.ts
73
+
```
74
+
75
+
These tests cover outcome classification, output validation, list parsing, scaffold validation, code-broken detection, args, disk helpers, and report summary. They do **not** run the full harness or hit the network.
76
+
67
77
## Report summary
68
78
69
79
At the end of each run the harness prints a **report summary**: coverage matrix (attempted/succeeded per ep surface), soft-fail counts by command type, and template distribution. The same summary can be printed for any saved report with `--analyze <path>`.
@@ -104,6 +114,8 @@ Per-command fields allow copy/paste or reconstructing the exact invocation.
104
114
105
115
If **expectedToFail** is true, a non-zero exit is treated as **outcome: "success"** for that command. A command with **expectedToFail: true** and **outcome: "success"** thus means the intentional negative test passed (e.g. bogus `ep show` correctly exited non-zero).
106
116
117
+
**Source layout:** The harness lives under `scripts/lifecycle-harness/`. Entry point is `src/index.ts`. Supporting modules: `args.ts` (CLI flags), `command.ts` (subprocess run with timeout/capture), `report.ts` (outcome classification, JSON shape), `report-summary.ts` (printed summary), `mutations.ts` (random lifecycle steps), `code-broken.ts` (detect broken project state), `output-checks.ts` (stdout validation), `list-parse.ts` (parse pattern IDs from `ep list`), `scaffold-validate.ts` (required files per template), `paths.ts` (monorepo root, scaffold root, reports dir), `disk.ts` (size helpers), `skills.ts` (break/fix SKILL.md), `types.ts` (TEMPLATES, TOOLS, mutation union). See **§ Unit tests** for the test suite.
118
+
107
119
**Repo root:** The harness discovers the monorepo root by walking up from its script location (`scripts/lifecycle-harness/src/`). It looks for a directory containing `package.json` with either `effect-patterns-hub` or a `"scaffold"` script. If not found, it throws a clear error (where it searched, what it expected). This contract is in `paths.ts`; if the root `package.json` is restructured, that check may need updating.
108
120
109
121
**Scenario directory names:** Each scenario repo is named `ep-life-YYYYMMDD-<seed>-s<index>-<template>-<shortRand>`. The date prefix (today’s date) is included so runs on different days get distinct directories and do not collide. The same seed still produces the same sequence of actions and the same logical scenario; only the directory path differs across days.
@@ -141,4 +153,4 @@ bun run lifecycle-harness --seed <seed> --scenarios <scenarioIndex + 1>
141
153
142
154
**Coverage gate:** At the end of a run, if any ep surface was attempted but never succeeded (e.g. `installAdd` always 401), the harness prints "Coverage gap: ..." and exits with code 1. When you use `--only-scenario N`, the coverage gate is **skipped** (and a one-line note is printed), since a single scenario cannot exercise all surfaces.
143
155
144
-
All randomness (templates after index 3, tools, mutation order, short random suffix) is driven by the seed, so the same seed produces the same sequence of actions. Repo directory names also include the current date (see above), so the exact path varies by run date.
156
+
All randomness (tools, mutation order, short random suffix) is driven by the seed, so the same seed produces the same sequence of actions. Only template assignment is deterministic (round-robin by scenario index). Repo directory names include the current date (see above), so the exact path varies by run date.
0 commit comments