Skip to content

Commit a8b7c05

Browse files
committed
docs: document lifecycle harness (source layout, unit tests, Architecture link)
- Architecture.md: expand lifecycle harness bullet, link to harness README - lifecycle-harness/README.md: add Source layout, Unit tests section; fix randomness note Made-with: Cursor
1 parent bfa74ba commit a8b7c05

2 files changed

Lines changed: 14 additions & 2 deletions

File tree

docs/Architecture.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -433,7 +433,7 @@ bun run ep:smoke-test
433433
### Scripts (scaffold and lifecycle harness)
434434

435435
- **Scaffold** (`bun run scaffold`) creates new projects under `$HOME/Projects/TestRepos` (see [scripts/scaffold-test-project.ts](../scripts/scaffold-test-project.ts) and [docs/development/SCAFFOLD_USER_GUIDE.md](development/SCAFFOLD_USER_GUIDE.md)).
436-
- **Lifecycle harness** (`bun run lifecycle-harness --seed <n>`) runs seedable E2E over real repos and the `ep` CLI. It depends on the scaffold’s output path: it uses the same root (`defaultScaffoldRootDir()` in `scripts/lifecycle-harness/src/paths.ts`). The harness discovers the monorepo root by walking up from `scripts/lifecycle-harness/src/` and looking for a root `package.json` containing `effect-patterns-hub` or a `"scaffold"` script.
436+
- **Lifecycle harness** (`bun run lifecycle-harness --seed <n>`) runs seedable E2E over real repos and the `ep` CLI (no mocks; real network/API). It uses the same repo root as the scaffold (`defaultScaffoldRootDir()` in `scripts/lifecycle-harness/src/paths.ts`), discovers the monorepo root by walking up from its script dir, and writes JSON reports under `scripts/lifecycle-harness/reports/`. Templates are assigned in round-robin by scenario index. Full documentation: **[scripts/lifecycle-harness/README.md](../scripts/lifecycle-harness/README.md)**.
437437

438438
### Testing
439439

scripts/lifecycle-harness/README.md

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,16 @@ This runs up to 10 scenarios (deterministic for seed `123`), creates repos under
6464
- **Print summary for an existing report:**
6565
`bun run lifecycle-harness --analyze scripts/lifecycle-harness/reports/run-20260226-172113-seed-20260226.json`
6666

67+
## Unit tests
68+
69+
The harness has a companion unit test suite under `scripts/__tests__/lifecycle-harness/` (e.g. `code-broken.test.ts`, `report.test.ts`, `scaffold-validate.test.ts`, `output-checks.test.ts`, `list-parse.test.ts`). Run from the monorepo root:
70+
71+
```bash
72+
bunx vitest run scripts/__tests__/lifecycle-harness/ --config vitest.config.ts
73+
```
74+
75+
These tests cover outcome classification, output validation, list parsing, scaffold validation, code-broken detection, args, disk helpers, and report summary. They do **not** run the full harness or hit the network.
76+
6777
## Report summary
6878

6979
At the end of each run the harness prints a **report summary**: coverage matrix (attempted/succeeded per ep surface), soft-fail counts by command type, and template distribution. The same summary can be printed for any saved report with `--analyze <path>`.
@@ -104,6 +114,8 @@ Per-command fields allow copy/paste or reconstructing the exact invocation.
104114

105115
If **expectedToFail** is true, a non-zero exit is treated as **outcome: "success"** for that command. A command with **expectedToFail: true** and **outcome: "success"** thus means the intentional negative test passed (e.g. bogus `ep show` correctly exited non-zero).
106116

117+
**Source layout:** The harness lives under `scripts/lifecycle-harness/`. Entry point is `src/index.ts`. Supporting modules: `args.ts` (CLI flags), `command.ts` (subprocess run with timeout/capture), `report.ts` (outcome classification, JSON shape), `report-summary.ts` (printed summary), `mutations.ts` (random lifecycle steps), `code-broken.ts` (detect broken project state), `output-checks.ts` (stdout validation), `list-parse.ts` (parse pattern IDs from `ep list`), `scaffold-validate.ts` (required files per template), `paths.ts` (monorepo root, scaffold root, reports dir), `disk.ts` (size helpers), `skills.ts` (break/fix SKILL.md), `types.ts` (TEMPLATES, TOOLS, mutation union). See **§ Unit tests** for the test suite.
118+
107119
**Repo root:** The harness discovers the monorepo root by walking up from its script location (`scripts/lifecycle-harness/src/`). It looks for a directory containing `package.json` with either `effect-patterns-hub` or a `"scaffold"` script. If not found, it throws a clear error (where it searched, what it expected). This contract is in `paths.ts`; if the root `package.json` is restructured, that check may need updating.
108120

109121
**Scenario directory names:** Each scenario repo is named `ep-life-YYYYMMDD-<seed>-s<index>-<template>-<shortRand>`. The date prefix (today’s date) is included so runs on different days get distinct directories and do not collide. The same seed still produces the same sequence of actions and the same logical scenario; only the directory path differs across days.
@@ -141,4 +153,4 @@ bun run lifecycle-harness --seed <seed> --scenarios <scenarioIndex + 1>
141153

142154
**Coverage gate:** At the end of a run, if any ep surface was attempted but never succeeded (e.g. `installAdd` always 401), the harness prints "Coverage gap: ..." and exits with code 1. When you use `--only-scenario N`, the coverage gate is **skipped** (and a one-line note is printed), since a single scenario cannot exercise all surfaces.
143155

144-
All randomness (templates after index 3, tools, mutation order, short random suffix) is driven by the seed, so the same seed produces the same sequence of actions. Repo directory names also include the current date (see above), so the exact path varies by run date.
156+
All randomness (tools, mutation order, short random suffix) is driven by the seed, so the same seed produces the same sequence of actions. Only template assignment is deterministic (round-robin by scenario index). Repo directory names include the current date (see above), so the exact path varies by run date.

0 commit comments

Comments
 (0)