diff --git a/forge/rules/examples/integration-embedded-testing/run_conformance_tests_java.sh b/forge/rules/examples/integration-embedded-testing/run_conformance_tests_java.sh index d2b5d8b..4c26ae6 100755 --- a/forge/rules/examples/integration-embedded-testing/run_conformance_tests_java.sh +++ b/forge/rules/examples/integration-embedded-testing/run_conformance_tests_java.sh @@ -21,7 +21,18 @@ fi current_dir=$(pwd) -CONFORMANCE_TESTS_FOLDER=".tmp/java_conformance" +# Resolve the conformance tests folder to an absolute path ($2 may be relative +# on older renderers; newer renderers pass it as an absolute path). +case "$2" in + /*) CONFORMANCE_TESTS_SOURCE="$2" ;; + *) CONFORMANCE_TESTS_SOURCE="$current_dir/$2" ;; +esac + +# Scratch folder lives in the system temp directory so no build debris is left +# inside the project. It is removed again when the script exits. +CONFORMANCE_TESTS_FOLDER="/tmp/java_conformance" + +trap 'rm -rf "$CONFORMANCE_TESTS_FOLDER"' EXIT cd "$current_dir" 2>/dev/null @@ -43,8 +54,8 @@ else mkdir -p $CONFORMANCE_TESTS_FOLDER fi -printf "Copying all files and folders from $2 to $CONFORMANCE_TESTS_FOLDER...\n" -cp -R $2/* $CONFORMANCE_TESTS_FOLDER +printf "Copying all files and folders from $CONFORMANCE_TESTS_SOURCE to $CONFORMANCE_TESTS_FOLDER...\n" +cp -R "$CONFORMANCE_TESTS_SOURCE"/* $CONFORMANCE_TESTS_FOLDER # Move to the subfolder printf "Moving to $CONFORMANCE_TESTS_FOLDER...\n" diff --git a/forge/rules/integration-embedded-testing.md b/forge/rules/integration-embedded-testing.md index 0241830..59c8381 100644 --- a/forge/rules/integration-embedded-testing.md +++ b/forge/rules/integration-embedded-testing.md @@ -9,13 +9,13 @@ When an embedded integration ships its three `test_scripts/` (prepare-environmen ## Staging model (read this first) -The three scripts do **not** all stage into the same place. Embedded integrations need a runnable host project, so the prepare and unit-test scripts operate **inside the host codebase itself**; only the conformance script uses a `.tmp/` scratch folder because the conformance suite lives in a separate project. +The three scripts do **not** all stage into the same place. Embedded integrations need a runnable host project, so the prepare and unit-test scripts operate **inside the host codebase itself**; only the conformance script uses a scratch folder in the system temp directory (`/tmp/_conformance/`) because the conformance suite lives in a separate project. | Script | Where it operates | What it does to the host source tree | |--------|-------------------|---------------------------------------| | `prepare_environment_` | Inside the host codebase | Copies `$1` into the module's package path under the host's source tree, then compiles / installs the host project | | `run_unittests_` | Inside the host codebase | Same copy as above (self-contained), then runs the module's unit tests + lint inside the host | -| `run_conformance_tests_` | `.tmp/_conformance/` | Copies `$2` (the conformance-tests folder) into the scratch directory and runs the conformance suite there, which depends on the host build that `prepare_environment` already installed | +| `run_conformance_tests_` | `/tmp/_conformance/` (system temp) | Copies `$2` (the conformance-tests folder) into the scratch directory and runs the conformance suite there, which depends on the host build that `prepare_environment` already installed | This deliberately writes into the host's `src/main/...` and `src/test/...` (or the language equivalent). Two consequences flow from that: @@ -126,11 +126,13 @@ All three scripts are invoked by the `codeplain` renderer with positional argume - `2` — missing or inaccessible input folder - `69` — unrecoverable environment failure (missing toolchain, cannot enter working folder, install failed) - Any other non-zero code — propagated verbatim from the underlying build / test tool + + The renderer passes `$1` (and `$2` for the conformance runner) as **absolute paths** (older renderer versions passed relative ones). Never build a derived path by concatenating a raw argument (`.tmp/$1`, `_$1`, `$current_dir/$2`) — with an absolute argument that produces a doubled path (`/project//abs/path/...`). Use the argument as-is where a path is needed, take its `basename` when naming a scratch folder, and resolve `$2` to absolute (if it isn't already) before any `cd` 3. **Echo the toolchain home at the top** — the first thing a developer checks when CI breaks. For Java: `JAVA_HOME`. For Python: the active interpreter (`python --version` / `which python`). For Node: `node --version`. For Go: `go env GOROOT`. Pick the variable whose value most often explains "why does it work locally but not on the CI runner?" 4. **Respect `VERBOSE=1`.** Gate chatty diagnostic prints behind `if [ "${VERBOSE:-}" = "1" ]` (and the PowerShell equivalent). Errors and key step markers print unconditionally 5. **Resolve paths relative to the script, not `$PWD`.** Use `"$(cd "$(dirname "$0")/" && pwd)"`. Hard-coded `../../` chains break the moment the renderer's `cwd` changes 6. **Create destination directories before copying.** `mkdir -p` before any `cp -R` / `rsync` / `robocopy` — most copy commands do not create intermediate directories and fail silently in some shells -7. **Scope destructive operations narrowly.** Any `rm -rf` (or `Remove-Item -Recurse -Force`) targets only the module's own package path inside the working folder — never the host's `src/`, `target/`, `node_modules/`, `build/`, or `dist/` at the project root. Only `prepare_environment` owns the build-output directory's lifecycle for its `.tmp/` working folder +7. **Scope destructive operations narrowly.** Any `rm -rf` (or `Remove-Item -Recurse -Force`) targets only the module's own package path inside the working folder — never the host's `src/`, `target/`, `node_modules/`, `build/`, or `dist/` at the project root. Only `prepare_environment` owns the build-output directory's lifecycle for its system-temp working folder 8. **Print where you are before you `cd`.** `echo "Moving to: $DIR"` saves an hour of debugging when paths are wrong. PowerShell: `Write-Host "Moving to: $DIR"` ## 1. `prepare_environment_` — copy into the host, then compile @@ -184,7 +186,7 @@ Hard rules: - **Never use a "clean and test" shortcut** (`mvn clean test`, `npm run rebuild`, …) — the cleanup belongs in `prepare_environment` and would erase whatever it installed - **Never run the full test suite** when scoping is possible. A per-module script that runs the entire host's tests wastes minutes on every render iteration -## 3. `run_conformance_tests_` — copy into `.tmp/`, then run the external suite +## 3. `run_conformance_tests_` — copy into the system temp directory, then run the external suite Receives two positional arguments: the renderer's build output folder (`$1`) and the conformance-tests folder (`$2`). @@ -192,8 +194,8 @@ Purpose: run the conformance suite in `$2` against the build that `prepare_envir Required steps: -1. Validate `$1` and `$2`. **Only `$2` is actually used by the script body today** — but keep `$1` in the signature; the renderer passes both positionally -2. Stage `$2/*` into a scratch directory under `.tmp/_conformance/`. Wipe it first if it exists. Use a deletion form that handles hidden files and odd shells safely: +1. Validate `$1` and `$2`. **Only `$2` is actually used by the script body today** — but keep `$1` in the signature; the renderer passes both positionally. Resolve `$2` to an absolute path if it isn't one already (newer renderers pass it absolute; older ones passed it relative to the invocation directory) +2. Stage `$2/*` into a scratch directory in the system temp directory (`/tmp/_conformance/` in Bash, the `GetTempPath()` equivalent in PowerShell). Wipe it first if it exists, and remove it again on exit (`trap 'rm -rf "$DIR"' EXIT` / `finally`). Use a deletion form that handles hidden files and odd shells safely: - Bash: `find "$DIR" -mindepth 1 -exec rm -rf {} +` (safer than `rm -rf "$DIR"/*`) - PowerShell: `Remove-Item -Recurse -Force "$DIR\*"` after confirming the path exists 3. `cd` into the scratch directory @@ -270,7 +272,7 @@ A working three-script set for a Java / Maven embedded integration lives under [ - [`prepare_environment_java.sh`](examples/integration-embedded-testing/prepare_environment_java.sh) — copies `$1` into the host's source tree, cleans `target/`, runs `mvn clean install -DskipTests` - [`run_unittests_java.sh`](examples/integration-embedded-testing/run_unittests_java.sh) — re-stages into the host and runs `mvn test -Dtest='.**.*Test' checkstyle:check` -- [`run_conformance_tests_java.sh`](examples/integration-embedded-testing/run_conformance_tests_java.sh) — copies `$2` into `.tmp/java_conformance/`, builds the conformance project, runs the suite, and parses the Surefire summary line per the strict pass criteria above +- [`run_conformance_tests_java.sh`](examples/integration-embedded-testing/run_conformance_tests_java.sh) — copies `$2` into `/tmp/java_conformance/` (system temp, removed again on exit), builds the conformance project, runs the suite, and parses the Surefire summary line per the strict pass criteria above Use them as a template when adding a new embedded integration in Java / Maven — copy, search-and-replace only the package segment and the `-Dtest` filter, then wire into `config.yaml`. For other languages (Python, Node.js, Go, Rust, …), the `implement-*-testing-script` skills generate the equivalent three-script set following the same contract. diff --git a/forge/rules/integration-embedded.md b/forge/rules/integration-embedded.md index 5657512..c60859c 100644 --- a/forge/rules/integration-embedded.md +++ b/forge/rules/integration-embedded.md @@ -9,7 +9,7 @@ When an integration `.plain` module is **embedded** — meaning the generated co Embedded means: the host codebase already exists, has its own language / framework / dependency manager / packaging layout, and the integration must conform to all of that without negotiation. -> **For test-script authoring**, also follow [`integration-embedded-testing.md`](integration-embedded-testing.md). It defines the per-script contract (`prepare_environment_`, `run_unittests_`, `run_conformance_tests_`) — staging into the host vs `.tmp/`, arg validation, exit codes, output parsing, the three `***implementation reqs***` entries the spec must declare so the scripts can be generated. This file (`integration-embedded.md`) only summarizes the test-script wiring; the testing rule is the source of truth. +> **For test-script authoring**, also follow [`integration-embedded-testing.md`](integration-embedded-testing.md). It defines the per-script contract (`prepare_environment_`, `run_unittests_`, `run_conformance_tests_`) — staging into the host vs the system temp directory, arg validation, exit codes, output parsing, the three `***implementation reqs***` entries the spec must declare so the scripts can be generated. This file (`integration-embedded.md`) only summarizes the test-script wiring; the testing rule is the source of truth. ## The host codebase dictates the tech stack (hard rule) @@ -118,7 +118,7 @@ The renderer reads the directives from the spec and the shapes from the linked s ## Test-script wiring — copy into the host, run tests there -Embedded integrations are tested **inside the host codebase itself**. The prepare and unit-test scripts copy the renderer's output (`$1`, i.e. `plain_modules//`) into the host's source tree at the module's package path, then compile / test the host project in place. Only the conformance script uses a `.tmp/` scratch folder, because the conformance suite is a separate project that consumes the host build as a dependency. +Embedded integrations are tested **inside the host codebase itself**. The prepare and unit-test scripts copy the renderer's output (`$1`, i.e. `plain_modules//`) into the host's source tree at the module's package path, then compile / test the host project in place. Only the conformance script uses a scratch folder in the system temp directory (`/tmp/_conformance/`), because the conformance suite is a separate project that consumes the host build as a dependency. This matters because the integration's generated code references host symbols by their full import path (e.g. `from host_project.integrations.base import IntegrationContract`). Those imports only resolve cleanly when the test process is rooted in the host's package layout — anything else creates path edge cases that bite later in conformance failures. @@ -126,7 +126,7 @@ See [`integration-embedded-testing.md`](integration-embedded-testing.md) for the - **`prepare_environment_`** copies `$1` into the host's source tree at the module's package path, cleans the host's build-output directory, then runs the host's install / build (e.g. `mvn clean install -DskipTests`). The conformance suite later resolves the integration from the host's local dependency cache - **`run_unittests_`** repeats the same copy into the host (self-contained — must work without `prepare_environment` having run first), then runs the module's unit tests + lint scoped to the module's package -- **`run_conformance_tests_`** copies `$2` (the conformance-tests folder) into `.tmp/_conformance/`, `cd`s in, builds the conformance project, and runs it against the build that `prepare_environment` already installed into the host +- **`run_conformance_tests_`** copies `$2` (the conformance-tests folder) into `/tmp/_conformance/` (system temp), `cd`s in, builds the conformance project, and runs it against the build that `prepare_environment` already installed into the host ### Invariants the scripts must enforce @@ -149,7 +149,7 @@ Before declaring an embedded integration done, in addition to the shared checkli - [ ] Host-package version pins are copied into `***implementation reqs***` - [ ] `prepare_environment` copies `$1` into the host's source tree at the module's package path, cleans the host's build-output directory, and runs the host's install / build so the conformance suite can resolve the integration from the local dependency cache - [ ] `run_unittests` runs the same copy-into-host sequence (self-contained — does not depend on `prepare_environment` having run) and invokes the host's test runner scoped to the module's package -- [ ] `run_conformance_tests` copies `$2` into `.tmp/_conformance/`, `cd`s in, builds the conformance project, and runs it against the host build that `prepare_environment` already installed +- [ ] `run_conformance_tests` copies `$2` into `/tmp/_conformance/` (system temp), `cd`s in, builds the conformance project, and runs it against the host build that `prepare_environment` already installed - [ ] Host codebase root is read from a named env var (default value documented in each script's usage) — never hardcoded; the env var name is captured in the integration's configuration concept - [ ] `***implementation reqs***` declares **everything about `:UnitTests:`** — integration source path, `:UnitTests:` source path, `:UnitTests:` package, framework + conventions, lint / static-analysis gate — per [`integration-embedded-testing.md`](integration-embedded-testing.md) - [ ] `***test reqs***` declares **everything about `:ConformanceTests:`** — source location, framework + execution command, package, mocking / network policy, pass criteria, build / install needs — per [`integration-embedded-testing.md`](integration-embedded-testing.md) diff --git a/forge/skills/implement-conformance-testing-script/SKILL.md b/forge/skills/implement-conformance-testing-script/SKILL.md index 1e518ad..fa944ac 100644 --- a/forge/skills/implement-conformance-testing-script/SKILL.md +++ b/forge/skills/implement-conformance-testing-script/SKILL.md @@ -27,8 +27,8 @@ Read both before writing anything — every script you produce must be a faithfu A conformance script is structurally very close to a unit-test script (see the sibling skill [`implement-unit-testing-script`](../implement-unit-testing-script/SKILL.md)) but with two important differences: -1. **Two positional arguments instead of one.** A conformance script takes both the **build folder** (source under test) and a **separate conformance tests folder** (the tests to execute against that build). -2. **Tests are loaded from outside the working folder.** The build is staged into `.tmp/_` and the script `cd`s into it, but the test command is pointed at the *original* `$current_dir/`. Tests are never copied into the staging area. +1. **Two positional arguments instead of one.** A conformance script takes both the **build folder** (source under test) and a **separate conformance tests folder** (the tests to execute against that build). Both arrive as **absolute paths** from current renderers (older renderer versions passed relative ones) — the script must work with both. +2. **Tests are loaded from outside the working folder.** The build is staged into the working folder `/_` (system temp directory + language id + basename of `$1` — see [Conventions](#conventions)) and the script `cd`s into it, but the test command is pointed at the conformance tests folder, resolved to an **absolute path** up front (`$CONFORMANCE_TESTS_FOLDER` below). Tests are never copied into the staging area. Everything else — toolchain check, build staging, dependency isolation, exit codes — is the same. @@ -50,9 +50,11 @@ That per-spec invocation pattern is what makes the install step expensive. A nai The two variants are a direct response to this: - **Install-inline** is correct only when N is small (a few specs) or dependencies are cheap. It is self-contained: stage, install, run, repeat from scratch every invocation. -- **Activate-only** is the production answer. [`prepare_environment_`](../implement-prepare-environment-script/SKILL.md) runs **once per render** and pays the install cost a single time, populating `.tmp/_/` with the warmed environment. Each of the N conformance invocations then just **attaches** to that working folder and runs the tests — no install, no compile, just activate-and-go. +- **Activate-only** is the production answer. [`prepare_environment_`](../implement-prepare-environment-script/SKILL.md) runs **once per render** and pays the install cost a single time, populating `/_/` with the warmed environment. Each of the N conformance invocations then just **attaches** to that working folder and runs the tests — no install, no compile, just activate-and-go. -**Why picking the right variant matters:** if you emit the install-inline variant alongside an existing prepare script, prepare's work is wiped (by the script's `rm -rf .tmp/_$1`) or duplicated (by re-running install) on every run — defeating prepare's whole purpose. Conversely, emitting activate-only without a prepare script means the "verify prepared environment" check fails on every run because nothing has populated the working folder. See [Anti-Patterns](#anti-patterns). +**The cleanup contract follows from this split:** the **install-inline** variant owns its working folder and removes it on exit (`trap 'rm -rf "$WORKING_FOLDER"' EXIT` / `Remove-Item` in `finally`). The **activate-only** variant **never deletes the working folder** — not on success, not on failure. Prepare owns that folder's lifecycle; if activate-only deleted it, the second of the N invocations would fail the "prepared environment missing" guard and the amortization would be destroyed. + +**Why picking the right variant matters:** if you emit the install-inline variant alongside an existing prepare script, prepare's work is wiped (by the script's staging `rm -rf` and its exit cleanup) or duplicated (by re-running install) on every run — defeating prepare's whole purpose. Conversely, emitting activate-only without a prepare script means the "verify prepared environment" check fails on every run because nothing has populated the working folder. See [Anti-Patterns](#anti-patterns). ## Pick the Shell First @@ -73,36 +75,42 @@ Steps **1–3** and **step 8** are identical in both variants. Steps **4–7** d 1. **Toolchain check.** Verify that the required language runtime / build tool (and the required version, if any) is installed. If not, print an error and exit with code `69`. 2. **Argument validation.** Require **two** positional arguments: `` and ``. If either is missing, print usage and exit with code `69`. -3. **Capture original cwd.** Store `pwd` in a variable (`current_dir` / `$PWD`) **before** changing directories — the test command in step 8 needs it to resolve the conformance tests folder. +3. **Resolve the conformance tests folder to an absolute path.** Current renderers pass `$2` as an absolute path; older ones passed it relative to the invocation directory. Resolve it once, up front, into a variable the test command in step 8 uses: + - Bash: `case "$2" in /*) CONFORMANCE_TESTS_FOLDER="$2" ;; *) CONFORMANCE_TESTS_FOLDER="$(pwd)/$2" ;; esac` + - PowerShell: `if (-not [System.IO.Path]::IsPathRooted($ConformanceTestsFolder)) { $ConformanceTestsFolder = Join-Path (Get-Location) $ConformanceTestsFolder }` + + Never concatenate `$current_dir/$2` at the point of use — with an absolute `$2` that produces a doubled path (`/project//abs/path/...`), the classic symptom being `ImportError: Start directory is not importable` pointing at a path that contains the project root twice. ### Steps 4–7 — install-inline variant (no prepare script) -4. **Working directory setup.** Define a working folder at `.tmp/_`. Wipe it (`rm -rf` / `Remove-Item -Recurse -Force`) and recreate it. This folder — and **only** this folder — is where every subsequent write must land. +4. **Working directory setup.** Define a working folder at `/_` (`/tmp/_$(basename "$1")` in Bash, `Join-Path ([System.IO.Path]::GetTempPath()) "_$(Split-Path $BuildFolder -Leaf)"` in PowerShell). Wipe it (`rm -rf` / `Remove-Item -Recurse -Force`) and recreate it. Register cleanup so the folder is removed when the script exits — `trap 'rm -rf "$WORKING_FOLDER"' EXIT` in Bash, `Remove-Item -Recurse -Force` in the `finally` block in PowerShell. This folder — and **only** this folder — is where every subsequent write must land. 5. **Copy the build.** Recursively copy everything from `` (`$1`) into the working folder. **Do not** copy the conformance tests — they stay where they are. After this step both `$1` (build folder) and `$2` (conformance tests folder) are treated as **read-only** for the rest of the script. -6. **Enter the working directory.** `cd` / `Set-Location` into `.tmp/_`. If that fails, exit with code `69`. All remaining steps run from inside the working folder; they must never write back to `$1` or `$2`. -7. **Install dependencies into an isolated environment inside `.tmp/_`.** Set up a per-working-folder dependency location (a Python venv at `./.venv`, a local `./node_modules`, a project-scoped Maven repo at `./.m2`, etc.) and install/resolve all dependencies into it. **Never** install into the source build folder (`$1`), the conformance tests folder (`$2`), the user's global cache (`~/.m2`, system-wide `pip`, `~/.cargo`, `~/.npm`, ...), or anywhere outside `.tmp/_`. If the install command fails, propagate its exit code immediately and **do not** proceed to step 8. See [Dependency isolation (install-inline)](#dependency-isolation-install-inline). +6. **Enter the working directory.** `cd` / `Set-Location` into `/_`. If that fails, exit with code `69`. All remaining steps run from inside the working folder; they must never write back to `$1` or `$2`. +7. **Install dependencies into an isolated environment inside `/_`.** Set up a per-working-folder dependency location (a Python venv at `./.venv`, a local `./node_modules`, a project-scoped Maven repo at `./.m2`, etc.) and install/resolve all dependencies into it. **Never** install into the source build folder (`$1`), the conformance tests folder (`$2`), the user's global cache (`~/.m2`, system-wide `pip`, `~/.cargo`, `~/.npm`, ...), or anywhere outside `/_`. If the install command fails, propagate its exit code immediately and **do not** proceed to step 8. See [Dependency isolation (install-inline)](#dependency-isolation-install-inline). ### Steps 4–7 — activate-only variant (prepare script exists) 4. **Verify the prepared environment.** Both: - - Check that the working folder `.tmp/_` exists. + - Check that the working folder `/_` exists — derived **exactly** the way prepare derives it (system temp dir + `_` + basename of `$1`). - Check that the language's isolation location inside it exists (e.g. `.venv/bin/activate` for Python, `.m2/` for Java, `node_modules/` for Node, `.gocache/` for Go, `.cargo/` for Rust). If either check fails, print a helpful error (`"Error: prepared environment missing — did you run prepare_environment_. first?"`) and exit `69`. **Do not silently fall back to creating it inline** — that would mask a real misconfiguration and turn this script into the install-inline variant in disguise. After this step both `$1` and `$2` are treated as **read-only** for the rest of the script. -5. **Enter the working directory.** `cd` / `Set-Location` into `.tmp/_`. If that fails, exit `69`. All remaining steps run from inside the working folder; they must never write back to `$1` or `$2`. + + **No cleanup registration in this variant.** The working folder belongs to prepare; this script must never delete it — not via a `trap`, not in a `finally` block, not on failure. See [the cleanup contract](#variant-decision-install-inline-vs-activate-only). +5. **Enter the working directory.** `cd` / `Set-Location` into `/_`. If that fails, exit `69`. All remaining steps run from inside the working folder; they must never write back to `$1` or `$2`. 6. **Activate the prepared dependency environment.** Per-language: - Python: `source .venv/bin/activate` (must succeed; exit `69` on failure). - Java: set `MAVEN_LOCAL_REPO="$(pwd)/.m2"` so it can be passed as `-Dmaven.repo.local="$MAVEN_LOCAL_REPO"` to `mvn` in step 8. - Node.js / Go / Rust: nothing to activate explicitly — the test command in step 8 just needs to receive the same isolation flag/env var that prepare used (`./node_modules` is found by default; pass `GOMODCACHE` / `CARGO_HOME`). - Activation is **always relative to the working folder**, never to `$1` or `$2` — prepare populated `.tmp/_/...`, and that is the only place to attach to. + Activation is **always relative to the working folder**, never to `$1` or `$2` — prepare populated `/_/...`, and that is the only place to attach to. 7. *(There is no step 7 in this variant — install was prepare's job. Skip straight to step 8.)* ### Common step 8 (both variants) -8. **Run the conformance tests.** Invoke the language's standard test command, **pointed at `$current_dir/`** (the original cwd from step 3 + the second arg). The script's final exit code is whatever the test command returns — except for the "no tests discovered" case below. +8. **Run the conformance tests.** Invoke the language's standard test command, **pointed at `$CONFORMANCE_TESTS_FOLDER`** (the absolute path resolved in step 3). The script's final exit code is whatever the test command returns — except for the "no tests discovered" case below. - The test command is **read-only** with respect to `$current_dir/$2`. It loads test files from there, but any artifacts the runner produces (caches, JUnit XML, coverage reports, compiled test classes, etc.) must land inside `.tmp/_`, not next to the test files. If your chosen runner defaults to writing output beside the tests, pass an explicit output-directory flag pointing inside the working folder (e.g. `pytest --basetemp=./.pytest_tmp`, `jest --cacheDirectory=./.jest_cache`, Maven `target/` under `.tmp` via `mvn -f "$current_dir/$2/pom.xml" -Dproject.build.directory="$(pwd)/target" test`). + The test command is **read-only** with respect to `$CONFORMANCE_TESTS_FOLDER`. It loads test files from there, but any artifacts the runner produces (caches, JUnit XML, coverage reports, compiled test classes, etc.) must land inside `/_`, not next to the test files. If your chosen runner defaults to writing output beside the tests, pass an explicit output-directory flag pointing inside the working folder (e.g. `pytest --basetemp=./.pytest_tmp`, `jest --cacheDirectory=./.jest_cache`, Maven `target/` under the working folder via `mvn -f "$CONFORMANCE_TESTS_FOLDER/pom.xml" -Dproject.build.directory="$(pwd)/target" test`). ### Read-only inputs — hard rule @@ -110,15 +118,15 @@ A conformance script has **two** read-only inputs: the source build folder (`$1` - install dependencies into `$1` or `$2` (no `pip install` inside `$1`/`$2`, no `npm install` inside them, no `mvn install` writing into them, no Cargo build artifacts ending up under them), - write a virtualenv / `node_modules` / `.m2` / `.gocache` / `.cargo` directory inside `$1` or `$2`, -- run the test command with its `cwd` set to `$1` or `$2` (every test command runs from inside `.tmp/_` after the `cd` in step 6 / activate-only step 5), +- run the test command with its `cwd` set to `$1` or `$2` (every test command runs from inside `/_` after the `cd` in step 6 / activate-only step 5), - create logs, caches, build outputs, JUnit XML, coverage reports, compiled test classes, or temp files inside `$1` or `$2`. Why each input is read-only: -- **`$1` (build folder)** is shared with the renderer (`plain_modules/...` by default) and downstream tooling. Writing into it corrupts the renderer's view of "what was generated" and breaks subsequent renders. The whole point of staging into `.tmp/` is so the source folder stays a clean, reproducible artifact of the render. +- **`$1` (build folder)** is shared with the renderer (`plain_modules/...` by default) and downstream tooling. Writing into it corrupts the renderer's view of "what was generated" and breaks subsequent renders. The whole point of staging into the system temp directory is so the source folder stays a clean, reproducible artifact of the render. - **`$2` (conformance tests folder)** is the user's authored test source — typically checked into version control. Writing into it pollutes the working tree, churns git status, and (with frameworks that auto-discover) can make subsequent runs pick up generated files as if they were tests. -If you find yourself about to issue any command whose `cwd` is `$1` or `$2`, or whose target path starts with `$1/` or `$2/`, **stop**. Either move the operation into `.tmp/_`, or you're doing something the script must not do. +If you find yourself about to issue any command whose `cwd` is `$1` or `$2`, or whose target path starts with `$1/` or `$2/`, **stop**. Either move the operation into `/_`, or you're doing something the script must not do. ### "No tests discovered" detection @@ -140,19 +148,20 @@ Shared across both shell flavors **and** both variants: - `69` — unrecoverable invocation error: missing argument, missing toolchain, can't enter working folder, can't create venv (install-inline), or prepared environment missing/broken (activate-only). Matches the reference scripts' `UNRECOVERABLE_ERROR_EXIT_CODE`. - `1` — "no tests discovered" guard tripped (see above). - Any other non-zero code — propagated from the underlying test command. -- **Working folder naming:** `.tmp/_` where `` is a short identifier for the language (`java`, `python`, `node`, `go`, `rust`, ...). Use the *first* argument (the build folder) in the path, never the conformance tests folder. All dependency installs, build outputs, caches, test runner artifacts, and the test invocation itself live inside this folder. Nothing the script does should touch `$1` after step 5 (install-inline) / step 4 (activate-only), or `$2` at any point. +- **Working folder naming:** `/_` — the system temp directory (`/tmp` in Bash, `[System.IO.Path]::GetTempPath()` in PowerShell) plus a short identifier for the language (`java`, `python`, `node`, `go`, `rust`, ...) and the **basename** of the *first* argument (the build folder — `$(basename "$1")` / `Split-Path $BuildFolder -Leaf`), never the conformance tests folder and never the raw argument (it may be absolute; concatenating it produces broken doubled paths). All dependency installs, build outputs, caches, test runner artifacts, and the test invocation itself live inside this folder. Nothing the script does should touch `$1` after step 5 (install-inline) / step 4 (activate-only), or `$2` at any point. +- **Cleanup ownership:** the **install-inline** variant removes its working folder on exit, success and failure alike (`trap 'rm -rf "$WORKING_FOLDER"' EXIT` in Bash; `Remove-Item -Recurse -Force` in the PowerShell `finally` block, after `Pop-Location`). The **activate-only** variant **never deletes the working folder** — prepare owns its lifecycle and re-creates it once per render; deleting it here would break every subsequent per-spec invocation. - **Logging — be as verbose as possible.** The script is the only thing the operator sees between a `codeplain` render and a green/red conformance result; when it fails, the only forensic evidence anyone has is its stdout/stderr. Treat the script like a production runbook, not a quiet helper. Concretely: - - **Announce every step before doing it**, with the exact value of every variable involved — the resolved build folder, the resolved conformance-tests folder, the captured `current_dir`, the working folder, the language, the toolchain version detected, the isolation root being activated (install-inline) or verified (activate-only), the variant in use, and the test command about to run. "Running tests" alone is useless; "Running `python -m unittest discover -b -s /abs/path/to/tests` from working folder `.tmp/python_build/` with venv activated at `./.venv` (Python 3.11.6)" is triage-ready. + - **Announce every step before doing it**, with the exact value of every variable involved — the resolved build folder, the resolved conformance-tests folder, the working folder, the language, the toolchain version detected, the isolation root being activated (install-inline) or verified (activate-only), the variant in use, and the test command about to run. "Running tests" alone is useless; "Running `python -m unittest discover -b -s /abs/path/to/tests` from working folder `/tmp/python_build/` with venv activated at `./.venv` (Python 3.11.6)" is triage-ready. - **Echo every non-trivial command before executing it.** In Bash, use `set -x` for the body of the script (or `echo "+ "` immediately before each call); in PowerShell, set `$VerbosePreference = 'Continue'` and use `Write-Verbose` / `Write-Host` to print each command line with its arguments. - **Print clear section banners.** Each pattern step gets a banner like `===== [6/8] Activate prepared environment =====` (activate-only) or `===== [7/8] Install dependencies =====` (install-inline) so a long log can be navigated by eye, **and** the banner names the variant so the operator immediately knows which branch is executing. - - **Print resolved absolute paths**, not just the relative names. Operators reading the log on a different machine need to know where the venv / `.m2` / `node_modules` / `.gocache` / `.cargo` actually landed, and where `$2` resolved to relative to `current_dir`. + - **Print resolved absolute paths**, not just the relative names. Operators reading the log on a different machine need to know where the venv / `.m2` / `node_modules` / `.gocache` / `.cargo` actually landed, and what `$2` resolved to. - **Print the toolchain's own `--version` output** (and the path it resolved from) during step 1, even on success. This is the single most useful piece of forensic data when conformance passes locally but fails in CI. - **Surface the install / activation output verbatim** — do not pipe it to `/dev/null`, do not redirect to a log file inside the working folder. The operator must see every dependency-resolver line (install-inline) or every "venv missing" / "\.m2 missing" guard message (activate-only) in the script's own output stream. Drop the `VERBOSE`-gated wrapping from the Python reference — conformance output is where forensic data is most valuable. - - **Print which variant the script is** (install-inline vs. activate-only) and **why** — e.g. "activate-only variant: detected `.tmp/python_build/.venv/bin/activate` from a prior `prepare_environment_python.sh` run". When the verify step trips, the operator must see both the path that was checked and the expected source ("did you run `prepare_environment_python.sh ` first?"). - - **On failure, print what was about to run before exiting** — the exact test command, the working directory, the captured `current_dir`, the resolved `$2`, and the relevant environment variables (`PYTHONPATH`, `NODE_PATH`, `GOMODCACHE`, `CARGO_HOME`, `MAVEN_OPTS`, `PATH`, plus `VIRTUAL_ENV` for Python). - - **Print a final summary line** that names the variant, the test command, the exit code, the captured `current_dir`, and the working folder so the operator knows exactly what to re-run by hand if needed. + - **Print which variant the script is** (install-inline vs. activate-only) and **why** — e.g. "activate-only variant: detected `/tmp/python_build/.venv/bin/activate` from a prior `prepare_environment_python.sh` run". When the verify step trips, the operator must see both the path that was checked and the expected source ("did you run `prepare_environment_python.sh ` first?"). + - **On failure, print what was about to run before exiting** — the exact test command, the working directory, the resolved `$CONFORMANCE_TESTS_FOLDER`, and the relevant environment variables (`PYTHONPATH`, `NODE_PATH`, `GOMODCACHE`, `CARGO_HOME`, `MAVEN_OPTS`, `PATH`, plus `VIRTUAL_ENV` for Python). + - **Print a final summary line** that names the variant, the test command, the exit code, the resolved conformance-tests folder, and the working folder so the operator knows exactly what to re-run by hand if needed. Verbosity is not noise here — a chatty script that documents itself in its own output is the difference between a 30-second triage and a 30-minute one. Never trade verbosity for terseness. -- **Capture `current_dir` before `cd`.** This is the single most common bug in hand-written conformance scripts: forgetting that the conformance tests folder argument is relative to the *invocation* directory, not the working folder. The captured value must also appear in the verbose log (per the logging rule above) so misresolutions of `$2` are visible at a glance. +- **Resolve `$2` before any `cd`.** This is the single most common bug in hand-written conformance scripts: using the conformance-tests argument after changing directory. Resolve it to an absolute path in step 3 (absolute → use as-is; relative → prefix the invocation directory) and use only the resolved variable afterwards. The resolved value must also appear in the verbose log (per the logging rule above) so misresolutions of `$2` are visible at a glance. ### Dependency isolation (install-inline) @@ -160,17 +169,17 @@ This section applies to **install-inline scripts only.** For activate-only scrip The dependency environment must live **inside** `$WORKING_FOLDER` so the test run can't be polluted by — or pollute — the user's global caches. Pick the most idiomatic isolation mechanism for the language: -| Language | Isolation mechanism | Install command (run inside `$WORKING_FOLDER`) | Test command (point at `$current_dir/$2`) | +| Language | Isolation mechanism | Install command (run inside `$WORKING_FOLDER`) | Test command (point at `$CONFORMANCE_TESTS_FOLDER`) | |---|---|---|---| -| Python | `venv` at `./.venv` | `python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt` | `python -m unittest discover -b -s "$current_dir/$2"` (or `pytest "$current_dir/$2"`) | -| Node.js | local `./node_modules` (default) | `npm ci` (preferred) or `npm install` | `npx jest --rootDir "$current_dir/$2"` | -| Java | project-scoped Maven repo at `./.m2` | `mvn -Dmaven.repo.local=./.m2 install -DskipTests` (build + install artifact so the test pom can resolve it) | `mvn -f "$current_dir/$2/pom.xml" -Dmaven.repo.local="$(pwd)/.m2" test` | -| Go | module cache at `./.gocache` | `GOMODCACHE="$PWD/.gocache" go mod download` (optional pre-warm) | `GOMODCACHE="$PWD/.gocache" go test "$current_dir/$2/..."` | -| Rust | cargo home at `./.cargo` | `CARGO_HOME="$PWD/.cargo" cargo fetch` (optional pre-warm) | `CARGO_HOME="$PWD/.cargo" cargo test --manifest-path "$current_dir/$2/Cargo.toml"` | +| Python | `venv` at `./.venv` | `python3 -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt` | `python -m unittest discover -b -s "$CONFORMANCE_TESTS_FOLDER"` (or `pytest "$CONFORMANCE_TESTS_FOLDER"`) | +| Node.js | local `./node_modules` (default) | `npm ci` (preferred) or `npm install` | `npx jest --rootDir "$CONFORMANCE_TESTS_FOLDER"` | +| Java | project-scoped Maven repo at `./.m2` | `mvn -Dmaven.repo.local=./.m2 install -DskipTests` (build + install artifact so the test pom can resolve it) | `mvn -f "$CONFORMANCE_TESTS_FOLDER/pom.xml" -Dmaven.repo.local="$(pwd)/.m2" test` | +| Go | module cache at `./.gocache` | `GOMODCACHE="$PWD/.gocache" go mod download` (optional pre-warm) | `GOMODCACHE="$PWD/.gocache" go test "$CONFORMANCE_TESTS_FOLDER/..."` | +| Rust | cargo home at `./.cargo` | `CARGO_HOME="$PWD/.cargo" cargo fetch` (optional pre-warm) | `CARGO_HOME="$PWD/.cargo" cargo test --manifest-path "$CONFORMANCE_TESTS_FOLDER/Cargo.toml"` | Notes: -- **Every path in the install command and test command is relative to `.tmp/_`.** That's why the script `cd`s into the working folder in step 6 — from that point on, `./.venv`, `./node_modules`, `./.m2`, etc. all resolve under `.tmp/_`, never under `$1` or `$2`. +- **Every path in the install command and test command is relative to `/_`.** That's why the script `cd`s into the working folder in step 6 — from that point on, `./.venv`, `./node_modules`, `./.m2`, etc. all resolve under `/_`, never under `$1` or `$2`. - **Always pass the isolation flag/env var to both the install command and the test command.** They must agree on where deps live, otherwise the test command will silently fall back to the global cache **or** (worse) write into `$1` / `$2`. - **Python is the only ecosystem where the venv is mandatory** to satisfy "into a virtual environment" literally. The others use language-native equivalents that achieve the same isolation. - **Propagate the install exit code immediately.** In Bash: ` || exit $?`. In PowerShell: check `$LASTEXITCODE` and `exit $LASTEXITCODE` if non-zero. @@ -180,13 +189,15 @@ Notes: This section applies to **activate-only scripts only.** The isolation location was created by prepare; conformance just needs to attach to it and pass the right flags to the test command. -| Language | Verify exists in step 4 | Activate in step 6 | Test command in step 8 (point at `$current_dir/$2`) | +In the table below, `$WORKING_FOLDER` is `/_` — e.g. `/tmp/python_$(basename "$1")` in Bash. + +| Language | Verify exists in step 4 | Activate in step 6 | Test command in step 8 (point at `$CONFORMANCE_TESTS_FOLDER`) | |---|---|---|---| -| Python | `.tmp/_$1/.venv/bin/activate` | `source .venv/bin/activate` (after `cd`-ing into the working folder) | `python -m unittest discover -b -s "$current_dir/$2"` | -| Node.js | `.tmp/_$1/node_modules/` | (nothing) | `npx jest --rootDir "$current_dir/$2"` | -| Java | `.tmp/_$1/.m2/` | `MAVEN_LOCAL_REPO="$(pwd)/.m2"` | `mvn -f "$current_dir/$2/pom.xml" -Dmaven.repo.local="$MAVEN_LOCAL_REPO" test` | -| Go | `.tmp/_$1/.gocache/` | `export GOMODCACHE="$(pwd)/.gocache"` | `go test "$current_dir/$2/..."` | -| Rust | `.tmp/_$1/.cargo/` | `export CARGO_HOME="$(pwd)/.cargo"` | `cargo test --manifest-path "$current_dir/$2/Cargo.toml"` | +| Python | `$WORKING_FOLDER/.venv/bin/activate` | `source .venv/bin/activate` (after `cd`-ing into the working folder) | `python -m unittest discover -b -s "$CONFORMANCE_TESTS_FOLDER"` | +| Node.js | `$WORKING_FOLDER/node_modules/` | (nothing) | `npx jest --rootDir "$CONFORMANCE_TESTS_FOLDER"` | +| Java | `$WORKING_FOLDER/.m2/` | `MAVEN_LOCAL_REPO="$(pwd)/.m2"` | `mvn -f "$CONFORMANCE_TESTS_FOLDER/pom.xml" -Dmaven.repo.local="$MAVEN_LOCAL_REPO" test` | +| Go | `$WORKING_FOLDER/.gocache/` | `export GOMODCACHE="$(pwd)/.gocache"` | `go test "$CONFORMANCE_TESTS_FOLDER/..."` | +| Rust | `$WORKING_FOLDER/.cargo/` | `export CARGO_HOME="$(pwd)/.cargo"` | `cargo test --manifest-path "$CONFORMANCE_TESTS_FOLDER/Cargo.toml"` | Notes: @@ -209,7 +220,7 @@ Notes: - **Exit codes:** use `exit 69` etc. (PowerShell honors them just like Bash). - **Toolchain check:** prefer `Get-Command -ErrorAction SilentlyContinue` and, where a specific version is needed, parse the tool's `--version` output. - **Filesystem:** use `Test-Path`, `Remove-Item -Recurse -Force`, `New-Item -ItemType Directory`, `Copy-Item -Recurse`, `Set-Location`. Quote paths to handle spaces. -- **Capture original cwd:** `$currentDir = (Get-Location).Path` **before** any `Set-Location` call. +- **Resolve `$ConformanceTestsFolder` to absolute** with `[System.IO.Path]::IsPathRooted` + `Join-Path (Get-Location) ...` **before** any `Set-Location` call. - **No `chmod` step needed.** If execution policy is likely to block the script, mention `Set-ExecutionPolicy -Scope CurrentUser RemoteSigned` to the user — don't bake it into the script. ## Workflow @@ -229,19 +240,21 @@ Notes: ## Anti-Patterns -- **(Hard mistake) Don't install into, build into, or otherwise write to the source build folder (`$1`) or the conformance tests folder (`$2`).** Both arguments are read-only input. Every install, cache, build artifact, log, JUnit XML, coverage report, compiled test class, and temp file must land in `.tmp/_`. This includes never running `pip install`, `npm install`, `mvn install`, or `cargo build` with `$1` or `$2` as their `cwd` or target; never letting a venv / `node_modules` / `.m2` / `.gocache` / `.cargo` directory appear inside `$1` or `$2`; and never running the test command from inside either folder. The whole point of staging into `.tmp/` is so the build folder remains a clean artifact of the render and the conformance tests folder remains a clean tree under the user's version control — writing to either one corrupts those guarantees. -- **Don't emit the install-inline variant when a `prepare_environment_` script already exists.** The conformance script's `rm -rf .tmp/_$1` will wipe everything prepare did, and the inline install will redo it from scratch on every run. Always run the [Variant decision](#variant-decision-install-inline-vs-activate-only) check first. +- **(Hard mistake) Don't install into, build into, or otherwise write to the source build folder (`$1`) or the conformance tests folder (`$2`).** Both arguments are read-only input. Every install, cache, build artifact, log, JUnit XML, coverage report, compiled test class, and temp file must land in `/_`. This includes never running `pip install`, `npm install`, `mvn install`, or `cargo build` with `$1` or `$2` as their `cwd` or target; never letting a venv / `node_modules` / `.m2` / `.gocache` / `.cargo` directory appear inside `$1` or `$2`; and never running the test command from inside either folder. The whole point of staging into the system temp directory is so the build folder remains a clean artifact of the render and the conformance tests folder remains a clean tree under the user's version control — writing to either one corrupts those guarantees. +- **(Hard mistake) Don't build any path by concatenating a raw argument.** `$1` and `$2` arrive as absolute paths from current renderers. Patterns like `WORKING_FOLDER=.tmp/$1`, `WORKING_FOLDER=_$1`, or `"$current_dir/$2"` silently produce doubled paths (`/project//abs/path/...`) — the classic symptom is `ImportError: Start directory is not importable` or "folder does not exist" pointing at a path that contains the project root twice. Always derive the working folder from `$(basename "$1")` / `Split-Path -Leaf`, and use the `$CONFORMANCE_TESTS_FOLDER` resolved in step 3. +- **(Hard mistake — activate-only) Don't delete the working folder.** No `trap 'rm -rf …' EXIT`, no `Remove-Item` in `finally`, no cleanup on failure. The working folder is prepare's deliverable and is attached to N times per render; deleting it after the first invocation makes every later invocation fail the "prepared environment missing" guard. Cleanup belongs to the install-inline variant only. +- **Don't emit the install-inline variant when a `prepare_environment_` script already exists.** The conformance script's staging `rm -rf` and exit cleanup will wipe everything prepare did, and the inline install will redo it from scratch on every run. Always run the [Variant decision](#variant-decision-install-inline-vs-activate-only) check first. - **Don't emit the activate-only variant when no prepare script exists.** The "verify prepared environment" check will fail on every run because nothing has populated the working folder. - **Don't silently fall back from activate-only to install-inline** when the prepared environment is missing. Exit `69` with a clear error so the misconfiguration is visible. Silent fallback hides the real bug and produces inconsistent behavior between runs. -- **Don't copy the conformance tests folder into `.tmp/`.** Only the build folder is staged (and only in install-inline). The test folder is read in place from `$current_dir/$2`. -- **Don't compute the test path after `cd`.** Capture `current_dir` first; otherwise `$2` will be resolved relative to the working folder and silently miss the tests. +- **Don't copy the conformance tests folder into the working folder.** Only the build folder is staged (and only in install-inline). The test folder is read in place from `$CONFORMANCE_TESTS_FOLDER`. +- **Don't compute the test path after `cd`.** Resolve `$2` to an absolute path in step 3; otherwise a relative `$2` will be resolved against the working folder and silently miss the tests. - **Don't skip the "no tests discovered" check.** A conformance suite that finds zero tests and exits `0` is the worst possible failure mode — it looks like success in CI. - **Don't skip the toolchain check**, even when "everyone has it installed" — exit code `69` is what the calling system relies on to detect a missing runtime. -- **Don't reuse the source folder in place** (install-inline). Always copy into `.tmp/_` first; the renderer relies on this isolation. +- **Don't reuse the source folder in place** (install-inline). Always copy into `/_` first; the renderer relies on this isolation. - **Don't change the exit-code contract.** Other parts of the system branch on `69` and `1` specifically — and these codes must be identical between the Bash and PowerShell variants. - **Don't write a cross-shell hybrid** (e.g. a `.sh` that detects PowerShell, or vice versa). Ship one script per shell, named with the appropriate extension. - **Don't install dependencies into the user's global location** (`~/.m2`, system-wide `pip`, `~/.cargo`, etc.) in the install-inline variant. Always isolate inside `$WORKING_FOLDER` so concurrent runs and other projects can't interfere. - **Don't run the test command without first verifying the install / activation succeeded.** A failed install (or missing prepared env) followed by a "test" run produces misleading errors that look like test failures. - **Don't forget to update `config.yaml`.** After creating the conformance test script, always add or update the `conformance-tests-script:` key in `config.yaml` to reference the new script. Without this entry, the `codeplain` renderer won't know where to find the conformance test script. - **Don't forget to link the script as a resource in the base `.plain` files.** A script that is referenced only from `config.yaml` is invisible to the spec contract — the renderer treats `conformance-tests-script:` as a build-system pointer, not as part of the test-req contract the model reads. Use the `add-resource` skill to add a markdown link to the script from the `.plain` module that owns the conformance test reqs (see Workflow step 9). Omitting the linked resource means the model authors and reviews conformance-test code without ever seeing the actual runner it has to satisfy — and for activate-only scripts, the model also loses sight of which isolation paths must already be in place by the time tests run. -- **Don't write a terse script.** Silent steps, `>/dev/null` redirects on the install / activation output, missing `--version` prints, absent variant banners, an un-logged `current_dir`, and a missing final summary all make the script harder to debug than the tests it is running. Follow the *Logging — be as verbose as possible* rule under [Conventions](#conventions) literally: every step announces itself, every command is echoed, the variant is named in its banner, `current_dir` and the resolved `$2` are printed, every failure prints what was about to run and where, and the final summary names the variant, test command, exit code, and working folder. +- **Don't write a terse script.** Silent steps, `>/dev/null` redirects on the install / activation output, missing `--version` prints, absent variant banners, an un-logged resolved `$2`, and a missing final summary all make the script harder to debug than the tests it is running. Follow the *Logging — be as verbose as possible* rule under [Conventions](#conventions) literally: every step announces itself, every command is echoed, the variant is named in its banner, the resolved `$CONFORMANCE_TESTS_FOLDER` is printed, every failure prints what was about to run and where, and the final summary names the variant, test command, exit code, and working folder. diff --git a/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_cypress.ps1 b/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_cypress.ps1 index f372e94..089f268 100644 --- a/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_cypress.ps1 +++ b/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_cypress.ps1 @@ -69,6 +69,17 @@ function Cleanup { if ($script:build_output -and (Test-Path $script:build_output)) { Remove-Item $script:build_output -Force -ErrorAction SilentlyContinue } + + # Step out of any temp subfolder we may still be in before deleting it + Set-Location -Path ([System.IO.Path]::GetTempPath()) -ErrorAction SilentlyContinue + + # Remove temporary build subfolders if they exist + if ($script:NODE_SUBFOLDER -and (Test-Path $script:NODE_SUBFOLDER)) { + Remove-Item -Path $script:NODE_SUBFOLDER -Recurse -Force -ErrorAction SilentlyContinue + } + if ($script:NODE_CONFORMANCE_TESTS_SUBFOLDER -and (Test-Path $script:NODE_CONFORMANCE_TESTS_SUBFOLDER)) { + Remove-Item -Path $script:NODE_CONFORMANCE_TESTS_SUBFOLDER -Recurse -Force -ErrorAction SilentlyContinue + } } # Check for and kill any existing Node server from previous runs @@ -98,20 +109,21 @@ if ($args[2] -eq "-v" -or $args[2] -eq "--verbose") { $current_dir = Get-Location try { - # Define the path to the subfolder - $NODE_SUBFOLDER = "node_$BuildFolder" + # Define the path to the subfolder. It lives in the system temp directory; + # $BuildFolder may be an absolute path, so only its leaf name is used. + $script:NODE_SUBFOLDER = Join-Path ([System.IO.Path]::GetTempPath()) "node_$(Split-Path $BuildFolder -Leaf)" # Running React application - Write-Host "### Step 1: Starting the React application in folder $NODE_SUBFOLDER..." + Write-Host "### Step 1: Starting the React application in folder $script:NODE_SUBFOLDER..." if ($env:VERBOSE -eq "1") { - Write-Host "Preparing Node subfolder: $NODE_SUBFOLDER" + Write-Host "Preparing Node subfolder: $script:NODE_SUBFOLDER" } # Check if the node subfolder exists - if (Test-Path $NODE_SUBFOLDER) { + if (Test-Path $script:NODE_SUBFOLDER) { # Delete all files and folders except "node_modules", "plain_modules", and "package-lock.json" - Get-ChildItem -Path $NODE_SUBFOLDER -Force | + Get-ChildItem -Path $script:NODE_SUBFOLDER -Force | Where-Object { $_.Name -ne "node_modules" -and $_.Name -ne "plain_modules" -and @@ -126,18 +138,18 @@ try { Write-Host "Subfolder does not exist. Creating it..." } - New-Item -ItemType Directory -Path $NODE_SUBFOLDER -Force | Out-Null + New-Item -ItemType Directory -Path $script:NODE_SUBFOLDER -Force | Out-Null } - Copy-Item -Path "$BuildFolder/*" -Destination $NODE_SUBFOLDER -Recurse -Force + Copy-Item -Path "$BuildFolder/*" -Destination $script:NODE_SUBFOLDER -Recurse -Force # Move to the subfolder - if (-not (Test-Path $NODE_SUBFOLDER)) { - Write-Host "Error: Node build folder '$NODE_SUBFOLDER' does not exist." + if (-not (Test-Path $script:NODE_SUBFOLDER)) { + Write-Host "Error: Node build folder '$script:NODE_SUBFOLDER' does not exist." exit $UNRECOVERABLE_ERROR_EXIT_CODE } - Push-Location $NODE_SUBFOLDER + Push-Location $script:NODE_SUBFOLDER # Temporarily allow stderr output without throwing (npm may write warnings to stderr) # ForEach-Object converts ErrorRecord objects (from stderr) to plain strings to avoid verbose error formatting @@ -245,17 +257,18 @@ try { # Move back to the original directory Set-Location $current_dir - # Define the path to the conformance tests subfolder - $NODE_CONFORMANCE_TESTS_SUBFOLDER = "node_$ConformanceTestsFolder" + # Define the path to the conformance tests subfolder. It lives in the system + # temp directory; $ConformanceTestsFolder may be absolute, so only its leaf name is used. + $script:NODE_CONFORMANCE_TESTS_SUBFOLDER = Join-Path ([System.IO.Path]::GetTempPath()) "node_$(Split-Path $ConformanceTestsFolder -Leaf)" if ($env:VERBOSE -eq "1") { - Write-Host "Preparing conformance tests Node subfolder: $NODE_CONFORMANCE_TESTS_SUBFOLDER" + Write-Host "Preparing conformance tests Node subfolder: $script:NODE_CONFORMANCE_TESTS_SUBFOLDER" } # Check if the conformance tests node subfolder exists - if (Test-Path $NODE_CONFORMANCE_TESTS_SUBFOLDER) { + if (Test-Path $script:NODE_CONFORMANCE_TESTS_SUBFOLDER) { # Delete all files and folders except "node_modules", "plain_modules", and "package-lock.json" - Get-ChildItem -Path $NODE_CONFORMANCE_TESTS_SUBFOLDER -Force | + Get-ChildItem -Path $script:NODE_CONFORMANCE_TESTS_SUBFOLDER -Force | Where-Object { $_.Name -ne "node_modules" -and $_.Name -ne "plain_modules" -and @@ -270,18 +283,18 @@ try { Write-Host "Subfolder does not exist. Creating it..." } - New-Item -ItemType Directory -Path $NODE_CONFORMANCE_TESTS_SUBFOLDER -Force | Out-Null + New-Item -ItemType Directory -Path $script:NODE_CONFORMANCE_TESTS_SUBFOLDER -Force | Out-Null } - Copy-Item -Path "$ConformanceTestsFolder/*" -Destination $NODE_CONFORMANCE_TESTS_SUBFOLDER -Recurse -Force + Copy-Item -Path "$ConformanceTestsFolder/*" -Destination $script:NODE_CONFORMANCE_TESTS_SUBFOLDER -Recurse -Force # Move to the subfolder with Cypress tests - if (-not (Test-Path $NODE_CONFORMANCE_TESTS_SUBFOLDER)) { - Write-Host "Error: conformance tests Node folder '$NODE_CONFORMANCE_TESTS_SUBFOLDER' does not exist." + if (-not (Test-Path $script:NODE_CONFORMANCE_TESTS_SUBFOLDER)) { + Write-Host "Error: conformance tests Node folder '$script:NODE_CONFORMANCE_TESTS_SUBFOLDER' does not exist." exit $UNRECOVERABLE_ERROR_EXIT_CODE } - Push-Location $NODE_CONFORMANCE_TESTS_SUBFOLDER + Push-Location $script:NODE_CONFORMANCE_TESTS_SUBFOLDER # Temporarily allow stderr output without throwing (npm may write warnings to stderr) # ForEach-Object converts ErrorRecord objects (from stderr) to plain strings to avoid verbose error formatting diff --git a/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_golang.ps1 b/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_golang.ps1 index 631f5e4..24a9193 100644 --- a/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_golang.ps1 +++ b/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_golang.ps1 @@ -23,7 +23,15 @@ $ConformanceTestsFolder = $args[1] $current_dir = Get-Location -$GO_BUILD_SUBFOLDER = "go_$BuildFolder" +# Resolve conformance tests folder to an absolute path so it can be used +# from the build subfolder (where we'll Push-Location to next). +if (-not [System.IO.Path]::IsPathRooted($ConformanceTestsFolder)) { + $ConformanceTestsFolder = Join-Path $current_dir $ConformanceTestsFolder +} + +# Working folder lives in the system temp directory. $BuildFolder may be an +# absolute path, so only its leaf name is used to build the working folder name. +$GO_BUILD_SUBFOLDER = Join-Path ([System.IO.Path]::GetTempPath()) "go_$(Split-Path $BuildFolder -Leaf)" if ($env:VERBOSE -eq "1") { Write-Host "Preparing Go build subfolder: $GO_BUILD_SUBFOLDER" @@ -60,9 +68,9 @@ try { go get # Move to conformance tests folder - Set-Location "$current_dir/$ConformanceTestsFolder" + Set-Location $ConformanceTestsFolder if ($LASTEXITCODE -and $LASTEXITCODE -ne 0) { - Write-Host "Error: Conformance tests folder '$current_dir/$ConformanceTestsFolder' does not exist." + Write-Host "Error: Conformance tests folder '$ConformanceTestsFolder' does not exist." exit $UNRECOVERABLE_ERROR_EXIT_CODE } @@ -75,7 +83,7 @@ try { } # Move back to build directory - Set-Location "$current_dir/$GO_BUILD_SUBFOLDER" + Set-Location $GO_BUILD_SUBFOLDER # Execute Go lang conformance tests Write-Host "Running Golang conformance tests...`n" @@ -83,7 +91,7 @@ try { # Temporarily allow stderr output without throwing (Go may write to stderr) # ForEach-Object converts ErrorRecord objects (from stderr) to plain strings to avoid verbose error formatting $ErrorActionPreference = 'Continue' - $output = go run "$current_dir/$ConformanceTestsFolder/conformance_tests.go" 2>&1 | ForEach-Object { if ($_ -is [System.Management.Automation.ErrorRecord]) { $_.Exception.Message } else { $_ } } | Out-String + $output = go run (Join-Path $ConformanceTestsFolder "conformance_tests.go") 2>&1 | ForEach-Object { if ($_ -is [System.Management.Automation.ErrorRecord]) { $_.Exception.Message } else { $_ } } | Out-String $exit_code = $LASTEXITCODE $ErrorActionPreference = 'Stop' @@ -97,4 +105,7 @@ try { exit $exit_code } finally { Pop-Location + if (Test-Path $GO_BUILD_SUBFOLDER) { + Remove-Item -Path $GO_BUILD_SUBFOLDER -Recurse -Force -ErrorAction SilentlyContinue + } } diff --git a/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_java.sh b/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_java.sh index a9b2706..e522ac2 100755 --- a/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_java.sh +++ b/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_java.sh @@ -19,18 +19,31 @@ fi current_dir=$(pwd) printf "Current directory: $current_dir\n" -tree $2 +# Resolve the conformance tests folder to an absolute path so it can be used +# after changing into the working folder ($2 may be relative to the invocation +# directory on older renderers; newer renderers pass it as an absolute path). +case "$2" in + /*) CONFORMANCE_TESTS_SOURCE="$2" ;; + *) CONFORMANCE_TESTS_SOURCE="$current_dir/$2" ;; +esac -JAVA_BUILD_SUBFOLDER=,tmp/$1 +tree "$CONFORMANCE_TESTS_SOURCE" + +# Working folders live in the system temp directory. $1 and $2 may be absolute +# paths, so only their basenames are used to build the working folder names. +JAVA_BUILD_SUBFOLDER="/tmp/java_$(basename "$1")" +CONFORMANCE_TESTS_FOLDER="/tmp/java_conformance_$(basename "$2")" + +trap 'rm -rf "$JAVA_BUILD_SUBFOLDER" "$CONFORMANCE_TESTS_FOLDER"' EXIT if [ "${VERBOSE:-}" -eq 1 ] 2>/dev/null; then printf "Preparing Java build subfolder: $JAVA_BUILD_SUBFOLDER\n" fi -rm -rf $JAVA_BUILD_SUBFOLDER -mkdir -p $JAVA_BUILD_SUBFOLDER +rm -rf "$JAVA_BUILD_SUBFOLDER" +mkdir -p "$JAVA_BUILD_SUBFOLDER" -cp -R $1/* $JAVA_BUILD_SUBFOLDER +cp -R "$1"/* "$JAVA_BUILD_SUBFOLDER" printf "Copied from $1 to $JAVA_BUILD_SUBFOLDER...\n" # Move to the subfolder @@ -54,17 +67,15 @@ if [ $exit_code -ne 0 ]; then exit $exit_code fi -CONFORMANCE_TESTS_FOLDER=.tmp/java_conformance - cd "$current_dir" 2>/dev/null printf "Moved to $current_dir...\n" printf "Preparing Java conformance tests subfolder: $CONFORMANCE_TESTS_FOLDER\n" -rm -rf $CONFORMANCE_TESTS_FOLDER -mkdir -p $CONFORMANCE_TESTS_FOLDER +rm -rf "$CONFORMANCE_TESTS_FOLDER" +mkdir -p "$CONFORMANCE_TESTS_FOLDER" -cp -R $2/* $CONFORMANCE_TESTS_FOLDER -printf "Copied from $2 to $CONFORMANCE_TESTS_FOLDER...\n" +cp -R "$CONFORMANCE_TESTS_SOURCE"/* "$CONFORMANCE_TESTS_FOLDER" +printf "Copied from $CONFORMANCE_TESTS_SOURCE to $CONFORMANCE_TESTS_FOLDER...\n" # Move to the subfolder cd "$CONFORMANCE_TESTS_FOLDER" 2>/dev/null diff --git a/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_python.ps1 b/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_python.ps1 index 68c8972..83d736b 100644 --- a/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_python.ps1 +++ b/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_python.ps1 @@ -33,7 +33,15 @@ if (Get-Command python3 -ErrorAction SilentlyContinue) { $current_dir = Get-Location -$PYTHON_BUILD_SUBFOLDER = "python_$BuildFolder" +# Resolve conformance tests folder to an absolute path so it can be used +# from the build subfolder (where we'll Push-Location to next). +if (-not [System.IO.Path]::IsPathRooted($ConformanceTestsFolder)) { + $ConformanceTestsFolder = Join-Path $current_dir $ConformanceTestsFolder +} + +# Working folder lives in the system temp directory. $BuildFolder may be an +# absolute path, so only its leaf name is used to build the working folder name. +$PYTHON_BUILD_SUBFOLDER = Join-Path ([System.IO.Path]::GetTempPath()) "python_$(Split-Path $BuildFolder -Leaf)" if ($env:VERBOSE -eq "1") { Write-Host "Preparing Python build subfolder: $PYTHON_BUILD_SUBFOLDER" @@ -72,7 +80,7 @@ try { # Temporarily allow stderr output without throwing (Python unittest writes progress to stderr) # ForEach-Object converts ErrorRecord objects (from stderr) to plain strings to avoid verbose error formatting $ErrorActionPreference = 'Continue' - $output = & $PYTHON_CMD -m unittest discover -b -s "$current_dir/$ConformanceTestsFolder" 2>&1 | ForEach-Object { if ($_ -is [System.Management.Automation.ErrorRecord]) { $_.Exception.Message } else { $_ } } | Out-String + $output = & $PYTHON_CMD -m unittest discover -b -s $ConformanceTestsFolder 2>&1 | ForEach-Object { if ($_ -is [System.Management.Automation.ErrorRecord]) { $_.Exception.Message } else { $_ } } | Out-String $exit_code = $LASTEXITCODE $ErrorActionPreference = 'Stop' @@ -89,4 +97,7 @@ try { exit $exit_code } finally { Pop-Location + if (Test-Path $PYTHON_BUILD_SUBFOLDER) { + Remove-Item -Path $PYTHON_BUILD_SUBFOLDER -Recurse -Force -ErrorAction SilentlyContinue + } } diff --git a/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_python.sh b/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_python.sh index 0c3cc2f..cf75766 100644 --- a/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_python.sh +++ b/forge/skills/implement-conformance-testing-script/assets/run_conformance_tests_python.sh @@ -26,18 +26,28 @@ else exit $UNRECOVERABLE_ERROR_EXIT_CODE fi -current_dir=$(pwd) +# Resolve the conformance tests folder to an absolute path so it can be used +# after changing into the working folder ($2 may be relative to the invocation +# directory on older renderers; newer renderers pass it as an absolute path). +case "$2" in + /*) CONFORMANCE_TESTS_FOLDER="$2" ;; + *) CONFORMANCE_TESTS_FOLDER="$(pwd)/$2" ;; +esac -PYTHON_BUILD_SUBFOLDER=".tmp/$1" +# Working folder lives in the system temp directory. $1 may be an absolute +# path, so only its basename is used to build the working folder name. +PYTHON_BUILD_SUBFOLDER="/tmp/python_$(basename "$1")" + +trap 'rm -rf "$PYTHON_BUILD_SUBFOLDER"' EXIT if [ "${VERBOSE:-}" -eq 1 ] 2>/dev/null; then printf "Preparing Python build subfolder: $PYTHON_BUILD_SUBFOLDER\n" fi -rm -rf $PYTHON_BUILD_SUBFOLDER -mkdir -p $PYTHON_BUILD_SUBFOLDER +rm -rf "$PYTHON_BUILD_SUBFOLDER" +mkdir -p "$PYTHON_BUILD_SUBFOLDER" -cp -R $1/* $PYTHON_BUILD_SUBFOLDER +cp -R "$1"/* "$PYTHON_BUILD_SUBFOLDER" # Move to the subfolder cd "$PYTHON_BUILD_SUBFOLDER" 2>/dev/null @@ -84,7 +94,7 @@ printf "Requirements setup completed in %.2f seconds\n\n" "$duration" # Execute all Python conformance tests in the build folder printf "Running Python conformance tests...\n\n" -output=$($PYTHON_CMD -m unittest discover -b -s "$current_dir/$2" 2>&1) +output=$($PYTHON_CMD -m unittest discover -b -s "$CONFORMANCE_TESTS_FOLDER" 2>&1) exit_code=$? # Echo the original output diff --git a/forge/skills/implement-prepare-environment-script/SKILL.md b/forge/skills/implement-prepare-environment-script/SKILL.md index 79b9298..8f43e34 100644 --- a/forge/skills/implement-prepare-environment-script/SKILL.md +++ b/forge/skills/implement-prepare-environment-script/SKILL.md @@ -33,7 +33,7 @@ Without a prepare script, every one of those N invocations does the full depende `prepare_environment_` exists to **amortize that cost to one** install per render: -- Prepare runs **once**, installs everything, populates the project-local isolation location inside `.tmp/_/` (`./.venv`, `./node_modules`, `./.m2`, `./.gocache`, `./.cargo`, `./.pub-cache`). +- Prepare runs **once**, installs everything, populates the project-local isolation location inside the working folder `/_/` (`./.venv`, `./node_modules`, `./.m2`, `./.gocache`, `./.cargo`, `./.pub-cache`). Throughout this skill, `/_` means: the system temp directory (`/tmp` in Bash, `[System.IO.Path]::GetTempPath()` in PowerShell), a short language identifier, and the **basename** of the build-folder argument (`$(basename "$1")` / `Split-Path -Leaf`) — never the raw argument, which arrives as an absolute path from current renderers. - The conformance script then runs **N times**, each invocation in its [activate-only variant](../implement-conformance-testing-script/SKILL.md#variant-decision-install-inline-vs-activate-only), attaching to the already-populated working folder and skipping the install step entirely. - Net effect: install cost goes from `N × install-cost` to `1 × install-cost + N × cheap-attach-cost`. @@ -45,8 +45,8 @@ This is the **whole reason** the prepare-then-conformance split exists. If a pro `prepare_environment_` exists **solely** to set up the working folder that `run_conformance_tests_` then attaches to (via the activate-only variant of the conformance script). It has **no relationship** to [`run_unittests_`](../implement-unit-testing-script/SKILL.md): -- The unit-test runner is **fully self-contained**. It does its own staging into its **own** `.tmp/_/` working folder, and it installs its own dependencies inline (`pip install -r requirements.txt`, `npm ci`, `mvn`, `cargo fetch`, ...) every run. -- The unit-test runner **never reads from** the working folder `prepare_environment` populates. The two scripts use independent working folders even when they happen to share a `.tmp/_/` naming convention — each script wipes and rebuilds its own copy. +- The unit-test runner is **fully self-contained**. It does its own staging into its **own** `/_/` working folder, and it installs its own dependencies inline (`pip install -r requirements.txt`, `npm ci`, `mvn`, `cargo fetch`, ...) every run — and it removes that folder when it exits. +- The unit-test runner **never reads from** the working folder `prepare_environment` populates. The two scripts use independent working folders even when they happen to share a `/_/` naming convention — each script wipes and rebuilds its own copy. - The unit-test runner **does not require** `prepare_environment` to have run first. Users and CI systems routinely run unit tests as a smoke check without ever invoking conformance, and that must keep working. - There is **no activate-only variant** of the unit-test runner. [`implement-unit-testing-script`](../implement-unit-testing-script/SKILL.md) emits a self-contained script every time — the two-variant pattern is exclusive to the conformance runner. @@ -86,13 +86,13 @@ The same pattern applies to both shell flavors. Only the syntax changes. Every prepare-environment script must implement these steps **in this order**: 1. **Toolchain check.** Verify that the required language runtime / build tool (and the required version, if any) is installed. If not, print an error and exit with code `69`. -2. **Argument validation.** Require **one** positional argument: ``. If missing, print usage and exit with code `69`. -3. **Working directory setup.** Define a working folder at `.tmp/_` — **identical** to the path the conformance script will use. Wipe it (`rm -rf` / `Remove-Item -Recurse -Force`) and recreate it. This folder — and **only** this folder — is where every subsequent write must land. +2. **Argument validation.** Require **one** positional argument: ``. If missing, print usage and exit with code `69`. The renderer passes this argument as an **absolute path** (older renderer versions passed a relative one) — the script must work with both, which is why the working-folder name in step 3 is built from the argument's *basename*, never from the raw argument. +3. **Working directory setup.** Define a working folder at `/_` — **identical** to the path the conformance script will use (`/tmp/_$(basename "$1")` in Bash, `Join-Path ([System.IO.Path]::GetTempPath()) "_$(Split-Path $BuildFolder -Leaf)"` in PowerShell). Wipe it (`rm -rf` / `Remove-Item -Recurse -Force`) and recreate it. This folder — and **only** this folder — is where every subsequent write must land. **Do not register a cleanup trap / `finally` deletion** — unlike its two siblings, prepare deliberately leaves the working folder populated on exit; that folder *is* the deliverable the conformance script attaches to. 4. **Copy the build.** Recursively copy everything from `` (`$1`) into the working folder. After this step the source folder (`$1`) is treated as **read-only** for the rest of the script. -5. **Enter the working directory.** `cd` / `Set-Location` into `.tmp/_`. If that fails, exit with code `69`. All remaining steps run from inside the working folder; they must never write back to `$1`. -6. **Install dependencies / pre-build artifacts into an isolated environment inside `.tmp/_`.** Set up a per-working-folder dependency location (a Python venv at `./.venv`, a local `./node_modules`, a project-scoped Maven repo at `./.m2`, etc.) and install/resolve all dependencies into it. Where the language requires building before tests can run (Java, Rust, Go), also produce the build artifact and place it where the conformance script can find it — **inside the working folder**, never inside `$1` and never in the user's home directory. **Never** install into the source build folder (`$1`), the user's global cache (`~/.m2`, system-wide `pip`, `~/.cargo`, `~/.npm`, ...), or anywhere outside `.tmp/_`. If any sub-step fails, exit with code `69` (do **not** propagate Maven/pip/npm exit codes — a half-prepared environment is itself an unrecoverable error). See [Dependency isolation](#dependency-isolation) for per-language specifics. +5. **Enter the working directory.** `cd` / `Set-Location` into `/_`. If that fails, exit with code `69`. All remaining steps run from inside the working folder; they must never write back to `$1`. +6. **Install dependencies / pre-build artifacts into an isolated environment inside `/_`.** Set up a per-working-folder dependency location (a Python venv at `./.venv`, a local `./node_modules`, a project-scoped Maven repo at `./.m2`, etc.) and install/resolve all dependencies into it. Where the language requires building before tests can run (Java, Rust, Go), also produce the build artifact and place it where the conformance script can find it — **inside the working folder**, never inside `$1` and never in the user's home directory. **Never** install into the source build folder (`$1`), the user's global cache (`~/.m2`, system-wide `pip`, `~/.cargo`, `~/.npm`, ...), or anywhere outside `/_`. If any sub-step fails, exit with code `69` (do **not** propagate Maven/pip/npm exit codes — a half-prepared environment is itself an unrecoverable error). See [Dependency isolation](#dependency-isolation) for per-language specifics. -That's it. There is no step 7. Once dependencies are installed and (where applicable) the build artifact is in the local repo, this script's job is done. +That's it. There is no step 7. Once dependencies are installed and (where applicable) the build artifact is in the local repo, this script's job is done — and the populated working folder is intentionally left in place for the conformance runner. ### The build folder is read-only — hard rule @@ -100,14 +100,14 @@ The source build folder passed in as `$1` is **input only**. Prepare reads from - install dependencies into it (no `pip install` inside `$1`, no `npm install` inside `$1`, no `mvn install` writing into `$1`, no Cargo build artifacts ending up under `$1`), - write a virtualenv / `node_modules` / `.m2` / `.gocache` / `.cargo` / `.pub-cache` directory inside it, -- pre-build into it (every compile output — `target/`, `build/`, `dist/`, native binaries, generated sources — lives inside `.tmp/_`), +- pre-build into it (every compile output — `target/`, `build/`, `dist/`, native binaries, generated sources — lives inside `/_`), - create logs, caches, or temp files inside it. -The build folder is shared with the renderer (`plain_modules/...` by default) and with the conformance script, which staging-checks it via `if [ ! -d ".tmp/_$1" ]` and expects `$1` itself to look the same as it did right after rendering. Writing into `$1` corrupts the renderer's view of "what was generated", churns git status if the project commits `$1`, and (if the conformance script ever does an `rm -rf $1` during its own setup) silently destroys work prepare did. +The build folder is shared with the renderer (`plain_modules/...` by default) and with the conformance script, which staging-checks the working folder via `if [ ! -d "/tmp/_$(basename "$1")" ]` and expects `$1` itself to look the same as it did right after rendering. Writing into `$1` corrupts the renderer's view of "what was generated", churns git status if the project commits `$1`, and (if the conformance script ever does an `rm -rf $1` during its own setup) silently destroys work prepare did. -The whole point of staging via `.tmp/_` is so the source build folder stays a clean, reproducible artifact of the render. Every dependency, every compiled class, every binary, every cache must land inside the working folder — because that is exactly what the conformance script's activate-only variant attaches to. +The whole point of staging into `/_` is so the source build folder stays a clean, reproducible artifact of the render and no build debris lands inside the user's project. Every dependency, every compiled class, every binary, every cache must land inside the working folder — because that is exactly what the conformance script's activate-only variant attaches to. -If you find yourself about to issue any command whose `cwd` is `$1`, or whose target path starts with `$1/`, **stop**. Either move the operation into `.tmp/_`, or you're doing something the script must not do. +If you find yourself about to issue any command whose `cwd` is `$1`, or whose target path starts with `$1/`, **stop**. Either move the operation into `/_`, or you're doing something the script must not do. ## Coordination contract @@ -117,7 +117,8 @@ The two scripts must agree on: | What | Why it matters | |---|---| -| **Working folder path** (`.tmp/_$1`) | The conformance script `cd`s into this folder. If prepare uses a different name, the conformance script sees an empty / freshly-staged folder and re-does all the work. | +| **Working folder path** (`/_`) | The conformance script `cd`s into this folder. If prepare uses a different name, the conformance script sees an empty / freshly-staged folder and re-does all the work. Both scripts must derive it the same way: system temp dir + `_` + basename of `$1`. | +| **Working folder lifecycle** | Prepare wipes-and-recreates the folder at the start and **leaves it populated on exit**. The activate-only conformance script attaches to it and **never deletes it** — not on success, not on failure. Only prepare (on its next run) recreates it. A cleanup trap in either script destroys the amortization this pair exists for. | | **Dependency isolation location** (`./.venv`, `./.m2`, `./node_modules`, `./.gocache`, `./.cargo`, …) — relative to the working folder | If prepare populates `~/.m2` and conformance reads from `./.m2`, the warm cache is invisible. Always use the **project-local** path inside the working folder, in both scripts. | | **Build artifact location** (Java/Rust/Go) | The conformance test project depends on the build's artifact. It must be findable at the exact coordinates the conformance script expects (e.g. installed into the same project-local `./.m2` for Java). | | **Toolchain version** | If prepare runs under Java 21 and conformance runs under Java 17, classfile incompatibilities will surface at test time, not prepare time. Both scripts should perform the same toolchain check. | @@ -131,7 +132,8 @@ Shared across both shell flavors: - **Exit codes:** - `69` — unrecoverable: missing argument, missing toolchain, can't enter working folder, dependency install / build failed. Treat **all** failures as unrecoverable here — there is no "soft" failure mode for prepare. - `0` — success. -- **Working folder naming:** `.tmp/_` where `` is a short identifier for the language (`java`, `python`, `node`, `go`, `rust`, ...). Use the *first* (and only) argument in the path. All dependency installs, build outputs, caches, and pre-built artifacts live inside this folder. Nothing the script does should touch `$1` after step 4. +- **Working folder naming:** `/_` — the system temp directory (`/tmp` in Bash, `[System.IO.Path]::GetTempPath()` in PowerShell) plus a short identifier for the language (`java`, `python`, `node`, `go`, `rust`, ...) and the **basename** of the first (and only) argument (`$(basename "$1")` / `Split-Path -Leaf`). Never embed the raw argument in the path — it may be absolute, and concatenating it produces broken doubled paths. All dependency installs, build outputs, caches, and pre-built artifacts live inside this folder. Nothing the script does should touch `$1` after step 4. +- **No cleanup on exit.** Prepare leaves the working folder populated — that folder is its deliverable. Do not add a `trap … EXIT` / `finally` deletion; the folder is reclaimed the next time prepare wipes-and-recreates it (and the OS eventually reclaims the temp dir). - **Logging — be as verbose as possible.** The script is the only thing the operator sees between a `codeplain` render and a green/red conformance result; when prepare fails, the only forensic evidence anyone has is its stdout/stderr. Treat the script like a production runbook, not a quiet helper. Concretely: - **Announce every step before doing it**, with the exact value of every variable involved — the resolved source folder, the working folder, the language, the toolchain version detected, the isolation root, the dependency-manifest path, the install / pre-build command about to run. "Installing dependencies" alone is useless; "Installing dependencies from `./requirements.txt` into venv `./.venv` (Python 3.11.6 at `/usr/local/bin/python3`)" is triage-ready. - **Echo every non-trivial command before executing it.** In Bash, use `set -x` for the body of the script (or `echo "+ "` immediately before each call); in PowerShell, set `$VerbosePreference = 'Continue'` and use `Write-Verbose` / `Write-Host` to print each command line with its arguments. @@ -159,7 +161,7 @@ The dependency environment must live **inside** `$WORKING_FOLDER` so the conform Notes: -- **Every path in the install / pre-build command is relative to `.tmp/_`.** That's why the script `cd`s into the working folder in step 5 — from that point on, `./.venv`, `./node_modules`, `./.m2`, `./.gocache`, `./.cargo`, `./.pub-cache`, and any compile output (`target/`, `build/`, `dist/`, native binaries) all resolve under `.tmp/_`, never under `$1` and never under the user's home directory. +- **Every path in the install / pre-build command is relative to `/_`.** That's why the script `cd`s into the working folder in step 5 — from that point on, `./.venv`, `./node_modules`, `./.m2`, `./.gocache`, `./.cargo`, `./.pub-cache`, and any compile output (`target/`, `build/`, `dist/`, native binaries) all resolve under `/_`, never under `$1` and never under the user's home directory. - **Java/Rust/Go must compile, not just download.** The conformance script will time-out / re-compile from scratch if you only resolve metadata. Use `mvn install`, `cargo build --tests`, `go build ./...` (not just `dependency:resolve` / `cargo fetch` / `go mod download`). - **Python and Node.js only need to install** — they're interpreted/JIT-compiled at test time, so `pip install` / `npm ci` is sufficient. - **Always pass the isolation flag/env var.** `mvn` without `-Dmaven.repo.local`, `cargo` without `CARGO_HOME`, etc., write to the user's home directory instead of `$WORKING_FOLDER`. The conformance script will look in the wrong place and the warming was wasted — and the user's home dir gets polluted. @@ -207,9 +209,10 @@ Adding a `prepare_environment_` script changes the contract for the corres | Step in conformance | If prepare exists, you must... | |---|---| -| Staging block (`rm -rf .tmp/_$1` + `mkdir -p` + `cp -R $1/* .tmp/...`) | **Remove entirely.** Replace with a guard: `if [ ! -d ".tmp/_$1" ]; then echo "Error: build folder missing — run prepare_environment_.sh first."; exit 69; fi` | +| Staging block (`rm -rf` of the working folder + `mkdir -p` + `cp -R $1/*` into it) | **Remove entirely.** Replace with a guard: `WORKING_FOLDER="/tmp/_$(basename "$1")"; if [ ! -d "$WORKING_FOLDER" ]; then echo "Error: build folder missing — run prepare_environment_.sh first."; exit 69; fi` | | Dependency install / pre-build (`pip install`, `mvn install -DskipTests`, `npm ci`, `cargo build --tests`, etc.) | **Remove entirely.** Replace with a guard that the isolation location exists (`.venv/bin/activate` for Python, `.m2/` for Java, `node_modules/` for Node, etc.) and exit `69` if missing. | | Activation step (Python `source .venv/bin/activate`, Java `-Dmaven.repo.local=$(pwd)/.m2`) | **Keep.** Without it the test command can't see the prepared deps. | +| Working-folder cleanup on exit (`trap 'rm -rf "$WORKING_FOLDER"' EXIT` / `Remove-Item` in `finally`) | **Remove entirely.** The activate-only variant never deletes the working folder — prepare owns its lifecycle. Leaving the trap in wipes the warmed environment after the first conformance run, so every later run fails the "missing prepared environment" guard. | | Test execution + "no tests discovered" guard + exit-code propagation | **Keep unchanged.** | 4. **Verify the conformance script's exit codes still follow [`implement-conformance-testing-script`](../implement-conformance-testing-script/SKILL.md)** — the new "missing prepared environment" guard should exit `69` (unrecoverable invocation error), the no-tests guard should still exit `1`, and the test command's exit code should still propagate. @@ -224,14 +227,16 @@ Adding a `prepare_environment_` script changes the contract for the corres ## Anti-Patterns - **(Hard mistake) Don't pre-warm the unit-test runner from this script.** `prepare_environment_` is for the **conformance** script only. The unit-test runner ([`implement-unit-testing-script`](../implement-unit-testing-script/SKILL.md)) is always fully self-contained — it stages, installs, and runs in one shot, every invocation, regardless of whether a prepare script exists. Do not add a unit-test dependency-install step to `prepare_environment` "to save time"; the unit-test runner will not read what you produce, and the coupling breaks the activate-only contract between prepare and conformance. See [`prepare_environment` is conformance-only — NOT for unit tests](#prepare_environment-is-conformance-only--not-for-unit-tests) above. -- **(Hard mistake) Don't install into, build into, or otherwise write to the source build folder (`$1`).** The build folder passed as `$1` is read-only input after step 4. Every install, cache, build artifact (`target/`, `build/`, `dist/`, native binaries, generated sources), log, and temp file must land in `.tmp/_`. This includes never running `pip install`, `npm install`, `mvn install`, `cargo build`, or `go build` with `$1` as their `cwd` or target; never letting a venv / `node_modules` / `.m2` / `.gocache` / `.cargo` / `.pub-cache` directory appear inside `$1`; and never producing a pre-built artifact at any path under `$1/`. The whole point of staging into `.tmp/` is so the build folder remains a clean artifact of the render — writing to it corrupts the renderer's view, churns git status, and can be silently destroyed if the conformance script ever re-stages `$1` on its own. +- **(Hard mistake) Don't install into, build into, or otherwise write to the source build folder (`$1`).** The build folder passed as `$1` is read-only input after step 4. Every install, cache, build artifact (`target/`, `build/`, `dist/`, native binaries, generated sources), log, and temp file must land in `/_`. This includes never running `pip install`, `npm install`, `mvn install`, `cargo build`, or `go build` with `$1` as their `cwd` or target; never letting a venv / `node_modules` / `.m2` / `.gocache` / `.cargo` / `.pub-cache` directory appear inside `$1`; and never producing a pre-built artifact at any path under `$1/`. The whole point of staging into the system temp directory is so the build folder remains a clean artifact of the render — writing to it corrupts the renderer's view, churns git status, and can be silently destroyed if the conformance script ever re-stages `$1` on its own. +- **(Hard mistake) Don't build any path by concatenating the raw build-folder argument.** `$1` arrives as an absolute path from current renderers. Patterns like `WORKING_FOLDER=.tmp/$1` or `WORKING_FOLDER=_$1` silently produce doubled paths (`/project//abs/path/...`) — the classic symptom is a "folder does not exist" or `ImportError: Start directory is not importable` error pointing at a path that contains the project root twice. Always take `$(basename "$1")` / `Split-Path -Leaf`. +- **(Hard mistake) Don't add a cleanup trap that deletes the working folder on exit.** The populated working folder is this script's deliverable — the activate-only conformance runner attaches to it N times per render. Deleting it on exit turns every conformance run into a "missing prepared environment" failure. (The unit-test and install-inline conformance scripts *do* clean up after themselves; prepare and activate-only conformance never do.) - **Don't write to the user's global dependency cache** (`~/.m2`, system-wide `pip`, `~/.cargo`, `~/.npm`, etc.). The conformance script reads from the project-local cache; a global write is invisible to it and pollutes the user's home dir. - **Don't use a different working folder name than the conformance script.** They must match exactly. If you change one, change the other. - **Don't run tests** — not unit tests, not conformance tests, not smoke tests. Prepare's contract is "set up the environment"; running tests belongs to its siblings. - **Don't propagate non-`69` exit codes from `mvn` / `pip` / `npm`.** A failed install means the environment isn't usable. Treat every failure as `exit 69` so the orchestrator can tell "prepare failed" apart from "tests failed". - **Don't skip the toolchain check**, even when "everyone has it installed" — exit code `69` is what the calling system relies on to detect a missing runtime, and prepare is usually the *first* script to run, so it's the cheapest place to surface a missing JDK / Python / Node. - **Don't print the "moved to ..." line before the `cd` success check.** The reference script does this and it lies on failure. Put the log *after* the guard, or print "attempting to enter ..." instead. -- **Don't reuse the source folder in place.** Always copy into `.tmp/_` first; the conformance script relies on this isolation. +- **Don't reuse the source folder in place.** Always copy into `/_` first; the conformance script relies on this isolation. - **Don't change the exit-code contract between Bash and PowerShell variants.** The `.sh` and `.ps1` for the same language must use identical exit codes for identical failure modes. - **Don't write a cross-shell hybrid** (e.g. a `.sh` that detects PowerShell, or vice versa). Ship one script per shell, named with the appropriate extension. - **Don't forget to time the install.** Without the duration log, there's no way to tell whether prepare is actually saving wall-clock time vs. doing the same work the conformance script would have done inline. diff --git a/forge/skills/implement-prepare-environment-script/assets/prepare_environment_java.sh b/forge/skills/implement-prepare-environment-script/assets/prepare_environment_java.sh index d8b16cd..4fb5537 100644 --- a/forge/skills/implement-prepare-environment-script/assets/prepare_environment_java.sh +++ b/forge/skills/implement-prepare-environment-script/assets/prepare_environment_java.sh @@ -16,16 +16,20 @@ if [ -z "$1" ]; then exit 1 fi -JAVA_BUILD_SUBFOLDER=.tmp/$1 +# Working folder lives in the system temp directory. $1 may be an absolute +# path, so only its basename is used to build the working folder name. +# No cleanup trap here on purpose: this script deliberately leaves the working +# folder populated so the conformance runner can attach to it on every invocation. +JAVA_BUILD_SUBFOLDER="/tmp/java_$(basename "$1")" if [ "${VERBOSE:-}" -eq 1 ] 2>/dev/null; then printf "Copying generated code to main project folder: $JAVA_BUILD_SUBFOLDER\n" fi -rm -rf $JAVA_BUILD_SUBFOLDER -mkdir -p $JAVA_BUILD_SUBFOLDER +rm -rf "$JAVA_BUILD_SUBFOLDER" +mkdir -p "$JAVA_BUILD_SUBFOLDER" -cp -R $1/* $JAVA_BUILD_SUBFOLDER +cp -R "$1"/* "$JAVA_BUILD_SUBFOLDER" printf "Copied from $1 to $JAVA_BUILD_SUBFOLDER...\n" # Move to the subfolder diff --git a/forge/skills/implement-prepare-environment-script/assets/prepare_environment_python.sh b/forge/skills/implement-prepare-environment-script/assets/prepare_environment_python.sh index 1afc4b6..c917e04 100644 --- a/forge/skills/implement-prepare-environment-script/assets/prepare_environment_python.sh +++ b/forge/skills/implement-prepare-environment-script/assets/prepare_environment_python.sh @@ -27,16 +27,20 @@ fi current_dir=$(pwd) -PYTHON_BUILD_SUBFOLDER=".tmp/$1" +# Working folder lives in the system temp directory. $1 may be an absolute +# path, so only its basename is used to build the working folder name. +# No cleanup trap here on purpose: this script deliberately leaves the working +# folder populated so the conformance runner can attach to it on every invocation. +PYTHON_BUILD_SUBFOLDER="/tmp/python_$(basename "$1")" if [ "${VERBOSE:-}" -eq 1 ] 2>/dev/null; then printf "Preparing Python build subfolder: $PYTHON_BUILD_SUBFOLDER\n" fi -rm -rf $PYTHON_BUILD_SUBFOLDER -mkdir -p $PYTHON_BUILD_SUBFOLDER +rm -rf "$PYTHON_BUILD_SUBFOLDER" +mkdir -p "$PYTHON_BUILD_SUBFOLDER" -cp -R $1/* $PYTHON_BUILD_SUBFOLDER +cp -R "$1"/* "$PYTHON_BUILD_SUBFOLDER" # Move to the subfolder cd "$PYTHON_BUILD_SUBFOLDER" 2>/dev/null diff --git a/forge/skills/implement-unit-testing-script/SKILL.md b/forge/skills/implement-unit-testing-script/SKILL.md index 2919b60..912f377 100644 --- a/forge/skills/implement-unit-testing-script/SKILL.md +++ b/forge/skills/implement-unit-testing-script/SKILL.md @@ -31,11 +31,11 @@ The same seven-step pattern applies to both. Only the syntax changes. Every testing script must implement these steps **in this order**: 1. **Toolchain check.** Verify that the required language runtime / build tool (and the required version, if any) is installed. If not, print an error and exit with code `69`. -2. **Argument validation.** Require exactly one positional argument: the source build folder name. If missing, print usage and exit with code `1`. -3. **Working directory setup.** Define a working folder at `.tmp/_`. If it exists, wipe its contents; otherwise create it. This folder — and **only** this folder — is where every subsequent write must land. +2. **Argument validation.** Require exactly one positional argument: the source build folder. If missing, print usage and exit with code `1`. The renderer passes this argument as an **absolute path** (older renderer versions passed a relative one) — the script must work with both, which is why the working-folder name in step 3 is built from the argument's *basename*, never from the raw argument. +3. **Working directory setup.** Define a working folder in the **system temp directory**: `/tmp/_$(basename "$1")` in Bash, `Join-Path ([System.IO.Path]::GetTempPath()) "_$(Split-Path $BuildFolder -Leaf)"` in PowerShell (written as `/_` below). If it exists, wipe its contents; otherwise create it. Register cleanup so the folder is removed when the script exits — `trap 'rm -rf "$WORKING_FOLDER"' EXIT` in Bash, a `Remove-Item -Recurse -Force` in the `finally` block in PowerShell. This folder — and **only** this folder — is where every subsequent write must land. 4. **Copy the build.** Recursively copy everything from the source folder into the working folder. After this step the source folder (`$1`) is treated as **read-only** for the rest of the script. -5. **Enter the working directory.** `cd` / `Set-Location` into `.tmp/_`. If that fails, exit with code `2`. All remaining steps run from inside the working folder; they must never write back to the source build folder. -6. **Install dependencies into an isolated environment inside `.tmp/_`.** Set up a per-working-folder dependency location (a Python venv at `./.venv`, a local `./node_modules`, a project-scoped Maven repo at `./.m2`, etc.) and install/resolve all dependencies into it. **Never** install into the source build folder, the user's global cache (`~/.m2`, system-wide `pip`, `~/.cargo`, `~/.npm`, ...), or anywhere outside `.tmp/_`. If the install command fails, propagate its exit code immediately and **do not** proceed to step 7. See [Dependency isolation](#dependency-isolation) for per-language specifics. +5. **Enter the working directory.** `cd` / `Set-Location` into `/_`. If that fails, exit with code `2`. All remaining steps run from inside the working folder; they must never write back to the source build folder. +6. **Install dependencies into an isolated environment inside `/_`.** Set up a per-working-folder dependency location (a Python venv at `./.venv`, a local `./node_modules`, a project-scoped Maven repo at `./.m2`, etc.) and install/resolve all dependencies into it. **Never** install into the source build folder, the user's global cache (`~/.m2`, system-wide `pip`, `~/.cargo`, `~/.npm`, ...), or anywhere outside `/_`. If the install command fails, propagate its exit code immediately and **do not** proceed to step 7. See [Dependency isolation](#dependency-isolation) for per-language specifics. 7. **Run the tests.** Invoke the language's standard test command (e.g. `mvn test`, `pytest`, `npm test`, `go test ./...`, `cargo test`), pointed at the same isolated environment from step 6. The script's final exit code is whatever the test command returns. ### The build folder is read-only — hard rule @@ -44,12 +44,12 @@ The source build folder passed in as `$1` is **input only**. The script must nev - install dependencies into it (no `pip install` inside `$1`, no `npm install` inside `$1`, no `mvn install` writing into `$1`, no Cargo build artifacts ending up under `$1`), - write a virtualenv / `node_modules` / `.m2` / `.gocache` / `.cargo` directory inside it, -- run the test command from inside it (every test command runs from inside `.tmp/_` after the `cd` in step 5), +- run the test command from inside it (every test command runs from inside `/_` after the `cd` in step 5), - create logs, caches, build outputs, or temp files inside it. -The build folder is shared with the renderer (`plain_modules/...` by default) and downstream tooling. Writing into it corrupts the renderer's view of "what was generated" and breaks subsequent renders. Every write must go into `.tmp/_` — the whole point of staging via `.tmp` is so the source build folder stays a clean, reproducible artifact of the render. +The build folder is shared with the renderer (`plain_modules/...` by default) and downstream tooling. Writing into it corrupts the renderer's view of "what was generated" and breaks subsequent renders. Every write must go into `/_` — the whole point of staging into the system temp directory is so the source build folder stays a clean, reproducible artifact of the render and no build debris is left inside the user's project. -If you find yourself about to issue any command whose `cwd` is the source folder, or whose target path starts with `$1/`, **stop**. Either move the operation into `.tmp/_`, or you're doing something the script must not do. +If you find yourself about to issue any command whose `cwd` is the source folder, or whose target path starts with `$1/`, **stop**. Either move the operation into `/_`, or you're doing something the script must not do. ## Conventions @@ -60,7 +60,8 @@ Shared across both shell flavors: - `2` — filesystem problem (couldn't enter the working folder). - `69` — required toolchain / runtime is not installed. - Any other non-zero code — propagated from the underlying test command. -- **Working folder naming:** `.tmp/_` where `` is a short identifier for the language (`java`, `python`, `node`, `go`, `rust`, ...). All dependency installs, build outputs, caches, and the test run itself live inside this folder. Nothing the script does should touch the source build folder after step 4. +- **Working folder naming:** `/_` — the system temp directory (`/tmp` in Bash, `[System.IO.Path]::GetTempPath()` in PowerShell) plus a short identifier for the language (`java`, `python`, `node`, `go`, `rust`, ...) and the **basename** of the build-folder argument (`$(basename "$1")` / `Split-Path $BuildFolder -Leaf`). Never embed the raw argument in the path — it may be absolute, and concatenating it produces broken doubled paths. All dependency installs, build outputs, caches, and the test run itself live inside this folder. Nothing the script does should touch the source build folder after step 4. +- **Cleanup on exit:** the working folder is temporary — remove it when the script exits, on success and failure alike (`trap 'rm -rf "$WORKING_FOLDER"' EXIT` in Bash; `Remove-Item -Recurse -Force` in the PowerShell `finally` block, after `Pop-Location`). No leftover build folders may accumulate in the temp directory or the user's project. - **Logging — be as verbose as possible.** The script is the only thing the operator sees between a `codeplain` render and a green/red test result; when it fails, the only forensic evidence anyone has is its stdout/stderr. Treat the script like a production runbook, not a quiet helper. Concretely: - **Announce every step before doing it**, with the exact value of every variable involved — the resolved source folder, the working folder, the language, the toolchain version detected, the isolation root, the dependency-manifest path, the test command about to run. "Installing dependencies" alone is useless; "Installing dependencies from `./requirements.txt` into venv `./.venv` (Python 3.11.6 at `/usr/local/bin/python3`)" is triage-ready. - **Echo every non-trivial command before executing it.** In Bash, use `set -x` for the body of the script (or `echo "+ "` immediately before each call); in PowerShell, set `$VerbosePreference = 'Continue'` and use `Write-Verbose` / `Write-Host` to print each command line with its arguments. @@ -86,7 +87,7 @@ The dependency environment must live **inside** `$WORKING_FOLDER` so the test ru Notes: -- **Every path in the install command and test command is relative to `.tmp/_`.** That's why the script `cd`s into the working folder in step 5 — from that point on, `./.venv`, `./node_modules`, `./.m2`, etc. all resolve under `.tmp/_`, never under the source build folder. +- **Every path in the install command and test command is relative to `/_`.** That's why the script `cd`s into the working folder in step 5 — from that point on, `./.venv`, `./node_modules`, `./.m2`, etc. all resolve under `/_`, never under the source build folder. - **Always pass the isolation flag/env var to both the install command and the test command** — they must agree on where deps live, otherwise the test command will silently fall back to the global cache **or** (worse) the source build folder. - **Python is the only ecosystem where the venv is mandatory** to satisfy "into a virtual environment" literally. The others use language-native equivalents that achieve the same isolation. - **Pre-warming is optional for Java/Go/Rust** — their test commands will fetch deps on demand. Doing it as a separate step makes failures easier to diagnose and gives a clean "install failed vs test failed" signal. @@ -121,9 +122,11 @@ Notes: ## Anti-Patterns -- **(Hard mistake) Don't install into, build into, or otherwise write to the source build folder.** The build folder passed as `$1` is read-only input. Every install, cache, build artifact, log, and temp file must land in `.tmp/_`. This includes never running `pip install`, `npm install`, `mvn install`, or `cargo build` with the source folder as their `cwd` or target, never letting a venv / `node_modules` / `.m2` / `.gocache` / `.cargo` directory appear inside the source folder, and never running the test command from inside it. The whole point of staging the build into `.tmp/` is so the source folder remains a clean, reproducible artifact of the render — writing to it corrupts the renderer's view and breaks subsequent renders. +- **(Hard mistake) Don't install into, build into, or otherwise write to the source build folder.** The build folder passed as `$1` is read-only input. Every install, cache, build artifact, log, and temp file must land in `/_`. This includes never running `pip install`, `npm install`, `mvn install`, or `cargo build` with the source folder as their `cwd` or target, never letting a venv / `node_modules` / `.m2` / `.gocache` / `.cargo` directory appear inside the source folder, and never running the test command from inside it. The whole point of staging the build into the system temp directory is so the source folder remains a clean, reproducible artifact of the render — writing to it corrupts the renderer's view and breaks subsequent renders. +- **(Hard mistake) Don't build any path by concatenating the raw build-folder argument.** `$1` arrives as an absolute path from current renderers. Patterns like `WORKING_FOLDER=.tmp/$1`, `WORKING_FOLDER=_$1`, or `"$current_dir/$1"` silently produce doubled paths (`/project//abs/path/...`) — the classic symptom is an `ImportError: Start directory is not importable` or "folder does not exist" pointing at a path that contains the project root twice. Always take `$(basename "$1")` / `Split-Path -Leaf`. - Don't skip the toolchain check, even when "everyone has it installed" — exit code `69` is what the calling system relies on to detect a missing runtime. -- Don't reuse the source folder in place. Always copy into `.tmp/_` first; the renderer relies on this isolation. +- Don't reuse the source folder in place. Always copy into `/_` first; the renderer relies on this isolation. +- Don't leave the working folder behind. Register the cleanup (`trap … EXIT` / `finally`) before the first write so even an early failure removes it. - Don't change the exit-code contract. Other parts of the system branch on `1`, `2`, and `69` specifically — and these codes must be identical between the Bash and PowerShell variants. - Don't write a cross-shell hybrid (e.g. a `.sh` that detects PowerShell, or vice versa). Ship one script per shell, named with the appropriate extension. - Don't install dependencies into the user's global location (`~/.m2`, system-wide `pip`, `~/.cargo`, etc.). Always isolate inside `$WORKING_FOLDER` so concurrent runs and other projects can't interfere. diff --git a/forge/skills/implement-unit-testing-script/assets/run_unittests_flutter.ps1 b/forge/skills/implement-unit-testing-script/assets/run_unittests_flutter.ps1 index b8ed3c4..71c136c 100644 --- a/forge/skills/implement-unit-testing-script/assets/run_unittests_flutter.ps1 +++ b/forge/skills/implement-unit-testing-script/assets/run_unittests_flutter.ps1 @@ -16,7 +16,9 @@ if (-not (Get-Command flutter -ErrorAction SilentlyContinue)) { } $SOURCE_FOLDER = $args[0] -$BUILD_SUBFOLDER = ".tmp/flutter_build_unittests" +# Working folder lives in the system temp directory. $SOURCE_FOLDER may be an +# absolute path, so only its leaf name is used to build the working folder name. +$BUILD_SUBFOLDER = Join-Path ([System.IO.Path]::GetTempPath()) "flutter_$(Split-Path $SOURCE_FOLDER -Leaf)" Write-Host "Current directory: $(Get-Location)" Write-Host "Source folder: $SOURCE_FOLDER" @@ -79,4 +81,7 @@ try { Remove-Item -Path "flutter_test_stdout.txt" -ErrorAction SilentlyContinue Remove-Item -Path "flutter_test_stderr.txt" -ErrorAction SilentlyContinue Pop-Location + if (Test-Path $BUILD_SUBFOLDER) { + Remove-Item -Path $BUILD_SUBFOLDER -Recurse -Force -ErrorAction SilentlyContinue + } } diff --git a/forge/skills/implement-unit-testing-script/assets/run_unittests_golang.ps1 b/forge/skills/implement-unit-testing-script/assets/run_unittests_golang.ps1 index 0f9b03b..cadce30 100644 --- a/forge/skills/implement-unit-testing-script/assets/run_unittests_golang.ps1 +++ b/forge/skills/implement-unit-testing-script/assets/run_unittests_golang.ps1 @@ -13,7 +13,9 @@ if (-not $args[0]) { $BuildFolder = $args[0] -$GO_BUILD_SUBFOLDER = "go_$BuildFolder" +# Working folder lives in the system temp directory. $BuildFolder may be an +# absolute path, so only its leaf name is used to build the working folder name. +$GO_BUILD_SUBFOLDER = Join-Path ([System.IO.Path]::GetTempPath()) "go_$(Split-Path $BuildFolder -Leaf)" if ($env:VERBOSE -eq "1") { Write-Host "Preparing Go build subfolder: $GO_BUILD_SUBFOLDER" @@ -65,4 +67,7 @@ try { exit $exit_code } finally { Pop-Location + if (Test-Path $GO_BUILD_SUBFOLDER) { + Remove-Item -Path $GO_BUILD_SUBFOLDER -Recurse -Force -ErrorAction SilentlyContinue + } } diff --git a/forge/skills/implement-unit-testing-script/assets/run_unittests_java.sh b/forge/skills/implement-unit-testing-script/assets/run_unittests_java.sh index 46db1cc..182fd8b 100644 --- a/forge/skills/implement-unit-testing-script/assets/run_unittests_java.sh +++ b/forge/skills/implement-unit-testing-script/assets/run_unittests_java.sh @@ -16,8 +16,11 @@ if [ -z "$1" ]; then exit 1 fi -# Define the path to the java build subfolder -WORKING_FOLDER=.tmp/$1 +# Working folder lives in the system temp directory. $1 may be an absolute +# path, so only its basename is used to build the working folder name. +WORKING_FOLDER="/tmp/java_$(basename "$1")" + +trap 'rm -rf "$WORKING_FOLDER"' EXIT # Check if the java subfolder exists if [ -d "$WORKING_FOLDER" ]; then @@ -30,7 +33,7 @@ fi # copy all folders and files from the build folder to the subfolder -cp -R $1/* $WORKING_FOLDER +cp -R "$1"/* "$WORKING_FOLDER" printf "Copied from $1 to $WORKING_FOLDER...\n" # Move to the subfolder cd "$WORKING_FOLDER" 2>/dev/null diff --git a/forge/skills/implement-unit-testing-script/assets/run_unittests_python.ps1 b/forge/skills/implement-unit-testing-script/assets/run_unittests_python.ps1 index 80eac98..a9588e8 100644 --- a/forge/skills/implement-unit-testing-script/assets/run_unittests_python.ps1 +++ b/forge/skills/implement-unit-testing-script/assets/run_unittests_python.ps1 @@ -23,7 +23,9 @@ if (Get-Command python3 -ErrorAction SilentlyContinue) { exit $UNRECOVERABLE_ERROR_EXIT_CODE } -$PYTHON_BUILD_SUBFOLDER = "python_$BuildFolder" +# Working folder lives in the system temp directory. $BuildFolder may be an +# absolute path, so only its leaf name is used to build the working folder name. +$PYTHON_BUILD_SUBFOLDER = Join-Path ([System.IO.Path]::GetTempPath()) "python_$(Split-Path $BuildFolder -Leaf)" if ($env:VERBOSE -eq "1") { Write-Host "Preparing Python build subfolder: $PYTHON_BUILD_SUBFOLDER" @@ -73,4 +75,7 @@ try { exit $exit_code } finally { Pop-Location + if (Test-Path $PYTHON_BUILD_SUBFOLDER) { + Remove-Item -Path $PYTHON_BUILD_SUBFOLDER -Recurse -Force -ErrorAction SilentlyContinue + } } diff --git a/forge/skills/implement-unit-testing-script/assets/run_unittests_python.sh b/forge/skills/implement-unit-testing-script/assets/run_unittests_python.sh index 8524b2a..e715d42 100644 --- a/forge/skills/implement-unit-testing-script/assets/run_unittests_python.sh +++ b/forge/skills/implement-unit-testing-script/assets/run_unittests_python.sh @@ -14,16 +14,20 @@ echo "Current directory: $current_dir" echo "Build folder name: $1" echo "--------------------------------" -PYTHON_BUILD_SUBFOLDER=.tmp/$1 +# Working folder lives in the system temp directory. $1 may be an absolute +# path, so only its basename is used to build the working folder name. +PYTHON_BUILD_SUBFOLDER="/tmp/python_$(basename "$1")" + +trap 'rm -rf "$PYTHON_BUILD_SUBFOLDER"' EXIT if [ "${VERBOSE:-}" -eq 1 ] 2>/dev/null; then printf "Preparing Python build subfolder: $PYTHON_BUILD_SUBFOLDER\n" fi -rm -rf $PYTHON_BUILD_SUBFOLDER -mkdir -p $PYTHON_BUILD_SUBFOLDER +rm -rf "$PYTHON_BUILD_SUBFOLDER" +mkdir -p "$PYTHON_BUILD_SUBFOLDER" -cp -R $1/* $PYTHON_BUILD_SUBFOLDER +cp -R "$1"/* "$PYTHON_BUILD_SUBFOLDER" # Move to the subfolder cd "$PYTHON_BUILD_SUBFOLDER" 2>/dev/null diff --git a/forge/skills/implement-unit-testing-script/assets/run_unittests_react.ps1 b/forge/skills/implement-unit-testing-script/assets/run_unittests_react.ps1 index c7c69f0..b0f3e32 100644 --- a/forge/skills/implement-unit-testing-script/assets/run_unittests_react.ps1 +++ b/forge/skills/implement-unit-testing-script/assets/run_unittests_react.ps1 @@ -16,8 +16,9 @@ if (-not $args[0]) { $BuildFolder = $args[0] -# Define the path to the subfolder -$NODE_SUBFOLDER = "node_$BuildFolder" +# Define the path to the subfolder. It lives in the system temp directory; +# $BuildFolder may be an absolute path, so only its leaf name is used. +$NODE_SUBFOLDER = Join-Path ([System.IO.Path]::GetTempPath()) "node_$(Split-Path $BuildFolder -Leaf)" if ($env:VERBOSE -eq "1") { Write-Host "Preparing Node subfolder: $NODE_SUBFOLDER" @@ -80,4 +81,7 @@ try { exit $TEST_EXIT_CODE } finally { Pop-Location + if (Test-Path $NODE_SUBFOLDER) { + Remove-Item -Path $NODE_SUBFOLDER -Recurse -Force -ErrorAction SilentlyContinue + } } diff --git a/forge/skills/init-config-file/SKILL.md b/forge/skills/init-config-file/SKILL.md index 121b10e..a4dd530 100644 --- a/forge/skills/init-config-file/SKILL.md +++ b/forge/skills/init-config-file/SKILL.md @@ -29,7 +29,7 @@ If you only fixed a typo inside a `.plain` file and the testing surface didn't m 1. Determines **how many** `config.yaml` files the project needs (one per part — see [Per-part split](#per-part-split)). 2. For each config, gathers the decided values from the current project state (existing scripts under `test_scripts/`, the template directory, the build/dest folder choices). 3. Emits a clean, alphabetically-grouped `config.yaml` containing **only** keys that are actually in use, using the canonical key names from the [Valid keys reference](#valid-keys-reference). -4. Verifies that every `*-script` value points at a file that exists on disk under `test_scripts/` (or wherever the user placed it), using the same lookup rule the renderer uses: absolute path → path relative to the config file's directory → path relative to the renderer's directory. +4. Verifies that every `*-script` value points at a file that exists on disk under `test_scripts/` (or wherever the user placed it), using the same resolution rule the renderer uses: an absolute path (or a `~` path) is used as-is; a relative path in `config.yaml` resolves against the **config file's directory**. There is no other fallback — the renderer fails fast with a `FileNotFoundError` when a script path doesn't resolve to an existing file. 5. Hands off to `plain-healthcheck` for the full validation pass. ## What this skill does NOT do @@ -51,7 +51,7 @@ These keys reflect choices made in Phase 3 of `forge-plain` and are the bread an | Key | Type | Default | When to include | |---|---|---|---| -| `unittests-script` | path (string) | — | **Required.** Every project gets a unit-test runner. Path resolves relative to the config file's directory (preferred) or the renderer directory. | +| `unittests-script` | path (string) | — | **Required.** Every project gets a unit-test runner. A relative path resolves against the config file's directory; the renderer fails fast if the file doesn't exist. | | `conformance-tests-script` | path (string) | — | Include when the user opted into conformance testing in Phase 3. | | `prepare-environment-script` | path (string) | — | Include only when both (a) the user opted into a prepare-environment script and (b) `conformance-tests-script` is also set. Setting prepare without conformance is a hard `plain-healthcheck` failure. | | `test-script-timeout` | int (seconds) | `120` | Include only when the user explicitly raised/lowered the default. | @@ -72,7 +72,7 @@ These are useful but the defaults are almost always fine. Only include them when | `copy-conformance-tests` | bool | `false` | Requires `conformance-tests-script` to also be set. | | `conformance-tests-dest` | string | `dist_conformance_tests` | Target folder for the conformance-test copy. Must differ from `conformance-tests-folder`. | | `log-to-file` | bool | `true` | Disable only when the user explicitly does not want a log file. Controls whether logs are mirrored to disk — it does **not** set the log level (that's `logging-config-path`'s job). | -| `log-file-name` | string | `codeplain.log` | If `log-to-file` is `false`, this key must be left out. Resolved relative to the `.plain` file directory. | +| `log-file-name` | string | `codeplain.log` | If `log-to-file` is `false`, this key must be left out. A relative value set here resolves against the config file's directory; when the key is left out entirely, the default resolves against the `.plain` file's directory. | | `render-machine-graph` | bool | `false` | Include only when the user wants the state-machine graph rendered. | | `headless` | bool | `false` | Include only when the project is meant to run in CI / non-interactive mode by default. | | `force-render` | bool | `false` | Almost never belongs in `config.yaml`; prefer the CLI flag for one-off forced renders. | @@ -117,7 +117,7 @@ In other words: `logging-config-path` is the **only** knob that changes the leve ### Default behavior -- The CLI default value for `logging-config-path` is `logging_config.yaml`. If a file by that exact name exists in the current working directory, it will be loaded automatically — even without `logging-config-path` being set in `config.yaml`. +- The CLI default value for `logging-config-path` is `logging_config.yaml`, resolved against the **`.plain` file's directory** (like every defaulted path parameter). If a file by that name exists next to the plain file, it will be loaded automatically — even without `logging-config-path` being set in `config.yaml`. A value set in `config.yaml` resolves against the config file's directory; a value passed on the CLI resolves against the invocation directory. - If the file does not exist, the renderer silently keeps the baseline levels from step 1 above (no warning). - If the file exists but fails to parse / apply, the renderer warns (`Failed to load logging configuration from …`) and falls back to the baseline. @@ -201,7 +201,7 @@ For each part: - `conformance-tests-folder` ≠ `conformance-tests-dest`. - `copy-conformance-tests: true` requires `conformance-tests-script`. - `log-file-name` is set ⇒ `log-to-file` is not `false`. - - All `*-script` paths resolve on disk (absolute → relative to config dir → relative to renderer dir). + - All `*-script` paths resolve on disk (absolute / `~` paths as-is; relative paths against the config file's directory — no other fallback; the renderer fails fast on a missing script). - No script path crosses stacks (e.g. `backend/config.yaml` must not reference `run_unittests_js.sh`). ### Step 3 — Emit `config.yaml` diff --git a/forge/skills/load-plain-reference/SKILL.md b/forge/skills/load-plain-reference/SKILL.md index fe6ebfc..e2c9f9a 100644 --- a/forge/skills/load-plain-reference/SKILL.md +++ b/forge/skills/load-plain-reference/SKILL.md @@ -360,7 +360,7 @@ The **primary** purpose of these scripts is **automated execution by the rendere - **Conformance tests are per-functional-spec.** Each functional spec in a module has its own folder under `conformance_tests///`. After the renderer finishes generating code for a new functional spec and the unit tests and refactoring passes, it runs the conformance tests of **all previous functional spec** to detect regressions — see [Conformance Test Workflow](#conformance-test-workflow). For a single module (with 0 `requires` modules) with N functional specs, the conformance script gets invoked **on the order of N times per render**, each invocation pointing at a different spec's test folder. - **Each per-spec invocation is independent.** The conformance script does not know that it's the second invocation in a long sequence; from its point of view, each invocation is a cold start against a single spec's tests. - **Per-spec independence is also what makes dependency installation expensive.** A naive conformance runner would re-install all of the project's runtime dependencies (Python venv + `pip install`, Maven dependency tree, `npm ci`, `cargo build`, ...) on every one of those N invocations. That's `N × install-cost` of wasted work for every render. -- **That is exactly why `prepare_environment_` exists.** Its **only** job is to amortize the install cost: install once at the start of a render, populate `.tmp/_/` with the warmed dependency cache and build artifacts, then let the conformance runner **attach** to that working folder on each of the N per-spec invocations instead of re-installing. The conformance runner's [activate-only variant](../implement-conformance-testing-script/SKILL.md#variant-decision-install-inline-vs-activate-only) does precisely that. When no prepare script exists, the conformance runner falls back to the install-inline variant and pays the install cost N times — acceptable for tiny projects, costly for anything realistic. +- **That is exactly why `prepare_environment_` exists.** Its **only** job is to amortize the install cost: install once at the start of a render, populate the working folder in the system temp directory (`/tmp/_/` on macOS/Linux, the `GetTempPath()` equivalent on Windows) with the warmed dependency cache and build artifacts, then let the conformance runner **attach** to that working folder on each of the N per-spec invocations instead of re-installing. The conformance runner's [activate-only variant](../implement-conformance-testing-script/SKILL.md#variant-decision-install-inline-vs-activate-only) does precisely that — and it **never deletes** that working folder; prepare owns its lifecycle. When no prepare script exists, the conformance runner falls back to the install-inline variant and pays the install cost N times — acceptable for tiny projects, costly for anything realistic. - **The unit-test runner has a different execution model, because unit tests live in a different place.** Unit tests are part of the generated codebase itself — they sit directly inside `plain_modules//` next to the implementation they exercise — whereas conformance tests live *outside* the codebase, in their own per-spec folders under `conformance_tests///`. As a result, the unit-test runner doesn't have a per-spec axis to iterate over: it just runs against the whole `plain_modules//` build in one go, gets invoked far fewer times per render, and has no amortization gain to chase. That's why the unit-test runner is always self-contained and there is no `prepare_environment`-equivalent for it. Keep this framing in mind when you author or adapt any of these scripts. The decisions about working-folder paths, isolation locations, exit codes, and the activate-only-vs-install-inline split are not arbitrary house style — they are what makes the renderer's per-spec loop tractable. @@ -369,7 +369,7 @@ Keep this framing in mind when you author or adapt any of these scripts. The dec - **`run_unittests_.sh` / `.ps1`** — runs the auto-generated unit tests in `plain_modules//`. Authored by the [`implement-unit-testing-script`](../implement-unit-testing-script/SKILL.md) skill. Receives one positional argument: the source build folder name. Invoked roughly once per render. **Fully self-contained:** it installs its own dependencies inline (via `pip install -r requirements.txt`, `npm ci`, `mvn`, `cargo fetch`, etc.) and never relies on any other script having run first. - **`run_conformance_tests_.sh` / `.ps1`** — runs the conformance tests in `conformance_tests///` against the generated implementation. Authored by the [`implement-conformance-testing-script`](../implement-conformance-testing-script/SKILL.md) skill. Receives two positional arguments: the source build folder and the conformance tests folder. **Invoked once per previous functional spec on every render** — i.e. roughly N times for a module with N functional specs. -- **`prepare_environment_.sh` / `.ps1`** — *optional* one-time setup that runs **before** the conformance script and **only the conformance script**. Invoked **once per render** to install the build's dependencies and pre-warm build artifacts into `.tmp/_/` so the N subsequent conformance invocations can attach to the warmed environment instead of re-installing. Authored by the [`implement-prepare-environment-script`](../implement-prepare-environment-script/SKILL.md) skill. Receives one positional argument: the source build folder name. **It does not feed the unit-test script** — see [`prepare_environment` is conformance-only](#prepare_environment-is-conformance-only-common-mistake) below. +- **`prepare_environment_.sh` / `.ps1`** — *optional* one-time setup that runs **before** the conformance script and **only the conformance script**. Invoked **once per render** to install the build's dependencies and pre-warm build artifacts into the system-temp working folder (`/tmp/_/`) so the N subsequent conformance invocations can attach to the warmed environment instead of re-installing. Authored by the [`implement-prepare-environment-script`](../implement-prepare-environment-script/SKILL.md) skill. Receives one positional argument: the source build folder name. **It does not feed the unit-test script** — see [`prepare_environment` is conformance-only](#prepare_environment-is-conformance-only-common-mistake) below. ### `prepare_environment` is conformance-only (common mistake) @@ -379,7 +379,7 @@ It is a **common and costly mistake** to assume that `prepare_environment_ Why: -- **Unit tests run against `plain_modules//`, conformance tests run against `.tmp/_/`.** The two scripts stage into different places. `prepare_environment` populates `.tmp/_/` for conformance; the unit-test script does its own staging into its **own** `.tmp/_/` working folder and installs its own dependencies there. +- **Unit tests run against `plain_modules//`, conformance tests run against the system-temp working folder.** The two scripts stage into different places. `prepare_environment` populates `/tmp/_/` for conformance; the unit-test script does its own staging into its **own** system-temp working folder, installs its own dependencies there, and removes that folder when it exits. - **The unit-test runner must work in isolation.** Users and CI systems run unit tests as a quick smoke check without ever invoking conformance. If `run_unittests_` depended on `prepare_environment` having run, those one-off unit-test invocations would silently fail (or be "fixed" by a misguided edit to make it depend on `prepare`). - **The skill contract enforces it.** [`implement-unit-testing-script`](../implement-unit-testing-script/SKILL.md) emits a fully self-contained script every time: toolchain check → stage → install dependencies inline → run tests. It never emits an activate-only variant. The two-variant pattern is exclusive to the conformance runner. @@ -389,11 +389,13 @@ If you find yourself authoring (or asked to author) a `prepare_environment` scri Anything not listed here is documented in the individual skill file: -- **Input folders are read-only — hard rule.** The build folder (and, for conformance, the conformance tests folder too) is **input only**. Every install, build artifact, cache, log, JUnit XML, coverage report, compiled test class, and temp file must land inside `.tmp/_`, never inside the input folder. The build folder is shared with the renderer and with the user's version control; writing into it corrupts both. If you find yourself about to issue a command whose `cwd` is an input folder, or whose target path starts with the input folder, **stop** — the write has to go into `.tmp/_`. +- **Input folders are read-only — hard rule.** The build folder (and, for conformance, the conformance tests folder too) is **input only**. Every install, build artifact, cache, log, JUnit XML, coverage report, compiled test class, and temp file must land inside the working folder in the system temp directory, never inside the input folder. The build folder is shared with the renderer and with the user's version control; writing into it corrupts both. If you find yourself about to issue a command whose `cwd` is an input folder, or whose target path starts with the input folder, **stop** — the write has to go into the working folder. +- **Arguments arrive as absolute paths.** Current renderers resolve the build folder and conformance-tests folder to absolute paths before invoking the scripts (older renderers passed relative ones). Scripts therefore build their working-folder name from the **basename** of the argument (`/tmp/_$(basename "$1")` in Bash, `Join-Path ([System.IO.Path]::GetTempPath()) "_$(Split-Path $arg -Leaf)"` in PowerShell) and resolve the conformance-tests argument to absolute before any `cd`. Concatenating a raw argument (`.tmp/$1`, `_$1`, `$current_dir/$2`) produces doubled paths and errors like `ImportError: Start directory is not importable`. +- **Cleanup ownership.** The unit-test runner and the install-inline conformance runner remove their working folder on exit (`trap 'rm -rf …' EXIT` / `finally`). `prepare_environment` and the activate-only conformance runner **never delete** the working folder — prepare wipes-and-recreates it once per render and the activate-only runner attaches to it N times; deleting it in between destroys the amortization. - **Shell flavor matches the host.** `.sh` on macOS / Linux, `.ps1` on Windows. A project intended for both OSes ships both files in matching pairs (`prepare` + `conformance` for each language must agree on working-folder name and isolation paths). - **Exit codes are part of the contract.** `69` for unrecoverable errors (missing toolchain, bad args, can't enter the working folder, install failed); `1` for the "no tests discovered" guard in the conformance runner (and bad usage in the unit-test runner); any other non-zero code is propagated from the underlying test command. Other skills — notably [`plain-healthcheck`](../plain-healthcheck/SKILL.md) and [`check-plain-env`](../check-plain-env/SKILL.md) — branch on these codes. - **Wired in via `config.yaml`.** Each script that is actually generated must be referenced from the relevant `config.yaml` via the `unittests-script:`, `conformance-tests-script:`, and `prepare-environment-script:` keys respectively. See the [`init-config-file`](../init-config-file/SKILL.md) skill for the canonical assembly. **If `prepare-environment-script` is declared, `conformance-tests-script` must be declared too** — a prepare script only makes sense in service of conformance, and `plain-healthcheck` will hard-fail a project that violates this. -- **Conformance scripts come in two variants — unit-test scripts do not.** When a `prepare_environment_` script exists, the conformance script is the **activate-only** variant (it attaches to the env prepare populated in `.tmp/`). When no prepare exists, the conformance script is the **install-inline** variant (it stages and installs in one shot). The `implement-conformance-testing-script` skill picks the right variant automatically based on whether a prepare script is already on disk. **The unit-test script has no activate-only variant** — it is always self-contained, regardless of whether a `prepare_environment_` script exists. +- **Conformance scripts come in two variants — unit-test scripts do not.** When a `prepare_environment_` script exists, the conformance script is the **activate-only** variant (it attaches to the env prepare populated in the system-temp working folder). When no prepare exists, the conformance script is the **install-inline** variant (it stages and installs in one shot). The `implement-conformance-testing-script` skill picks the right variant automatically based on whether a prepare script is already on disk. **The unit-test script has no activate-only variant** — it is always self-contained, regardless of whether a `prepare_environment_` script exists. - **Dependency isolation is project-local.** Each language's package cache / virtual env / build repo lives inside the working folder (`./.venv` for Python, `./node_modules` for Node, `./.m2` for Java, `./.gocache` for Go, `./.cargo` for Rust, `./.pub-cache` for Flutter, ...) — never in the user's home directory. The conformance script reads from the same project-local location prepare wrote to; the unit-test script uses its **own** working folder and its **own** copy of the isolated dependencies. - **No language-package checks live in these scripts.** The scripts themselves install language packages via `pip install -r requirements.txt`, `npm ci`, `mvn -Dmaven.repo.local=...`, `go mod download`, `cargo fetch`, etc. They do **not** pre-verify individual packages; that's the package manager's job. The host-level checks for the toolchains and external dependencies belong in `check-plain-env`, not in these scripts. - **Scripts are verbose**. They print out every step they take, including toolchain checks, dependency installations, and test results. This makes it easier to debug and understand what's going on. @@ -569,13 +571,37 @@ GOOD — split into two independent root modules ## `codeplain` CLI reference +### Path parameter resolution + +Every path argument the renderer accepts (`--build-folder`, `--conformance-tests-folder`, `--unittests-script`, `--conformance-tests-script`, `--prepare-environment-script`, `--log-file-name`, `--logging-config-path`, `--template-dir`, `--base-folder`, `--build-dest`, `--conformance-tests-dest`) resolves **based on where the value was written**: + +- **CLI argument** → resolved against the **current working directory** of the `codeplain` invocation. +- **`config.yaml` value** → resolved against the **directory containing that config file**. +- **Default (not provided anywhere)** → resolved against the **directory containing the plain file**. +- **Absolute paths** and paths starting with `~` are used as-is (`~` is expanded). + +Consequences worth knowing when authoring or debugging a project: + +- Script paths are **validated at parse time** — the renderer fails fast with a `FileNotFoundError` if a `*-script` value doesn't resolve to an existing file. There is no fallback lookup (older renderer versions fell back to the renderer's own directory; that behavior is gone). +- The renderer passes the **resolved absolute paths** to the test scripts as their positional arguments — the scripts must not assume relative paths (see [Testing Scripts](#testing-scripts)). +- Since defaults resolve against the plain file's directory, `plain_modules/`, `conformance_tests/`, and `codeplain.log` land next to the spec regardless of where `codeplain` is invoked from — but an explicitly passed relative path lands relative to where it was written (CLI → cwd, config → config dir). + +### CLI help + ```txt -Render ***plain specs to target code. +Render plain code to target code. Path arguments resolve based on where they +were written: values given on the command line are resolved against the +current working directory, values read from config.yaml are resolved against +the config file's directory, and defaults are resolved against the directory +containing the plain file. Absolute paths (and paths starting with '~') are +used as-is. positional arguments: - filename Path to the plain file to render. The directory containing this file has highest precedence for template loading, so - you can place custom templates here to override the defaults. See --template-dir for more details about template - loading. + filename Path to the plain file to render. The directory + containing this file has highest precedence for + template loading, so you can place custom templates + here to override the defaults. See --template-dir for + more details about template loading. options: -h, --help show this help message and exit @@ -585,61 +611,90 @@ options: --build-folder BUILD_FOLDER Folder for build files --log-to-file, --no-log-to-file - Enable logging to a file. Defaults to True. Set to False to disable. + Enable logging to a file. Defaults to True. Set to + False to disable. --log-file-name LOG_FILE_NAME - Name of the log file. Defaults to 'codeplain.log'.Always resolved relative to the plain file directory.If file on - this path already exists, the already existing log file will be overwritten by the current logs. + Name of the log file. Defaults to 'codeplain.log'. If + a file already exists at the resolved path, it will be + overwritten by the current logs. --render-range RENDER_RANGE - Specify a range of functionalities to render (e.g. `1` , `2`, `3`). Use comma to separate start and end IDs. If only - one functionality ID is provided, only that functionality is rendered. Range is inclusive of both start and end IDs. + Specify a range of functionalities to render (e.g. `1` + , `2`, `3`). Use comma to separate start and end IDs. + If only one functionality ID is provided, only that + functionality is rendered. Range is inclusive of both + start and end IDs. --render-from RENDER_FROM - Continue generation starting from this specific functionality (e.g. `2`). The functionality with this ID will be - included in the output. The functionality ID must match one of the functionalities in your plain file. + Continue generation starting from this specific + functionality (e.g. `2`). The functionality with this + ID will be included in the output. The functionality + ID must match one of the functionalities in your plain + file. --force-render Force re-render of all the required modules. --unittests-script UNITTESTS_SCRIPT - Shell script to run unit tests on generated code. Receives the build folder path as its first argument (default: - 'plain_modules'). + Shell script to run unit tests on generated code. + Receives the build folder path as its first argument + (default: 'plain_modules'). --conformance-tests-folder CONFORMANCE_TESTS_FOLDER Folder for conformance test files --conformance-tests-script CONFORMANCE_TESTS_SCRIPT - Path to conformance tests shell script. Every conformance test script should accept two arguments: 1) Path to a - folder (e.g. `plain_modules/module_name`) containing generated source code, 2) Path to a subfolder of the conformance - tests folder (e.g. `conformance_tests/subfoldername`) containing test files. + Path to conformance tests shell script. Every + conformance test script should accept two arguments: + 1) Path to a folder (e.g. `plain_modules/module_name`) + containing generated source code, 2) Path to a + subfolder of the conformance tests folder (e.g. + `conformance_tests/subfoldername`) containing test + files. --prepare-environment-script PREPARE_ENVIRONMENT_SCRIPT - Path to a shell script that prepares the testing environment. The script should accept the source code folder path as - its first argument. + Path to a shell script that prepares the testing + environment. The script should accept the source code + folder path as its first argument. --test-script-timeout TEST_SCRIPT_TIMEOUT - Timeout for test scripts in seconds. If not provided, the default timeout of 120 seconds is used. - --api [API] Alternative base URL for the API. Default: `https://api.codeplain.ai` - --api-key API_KEY API key used to access the API. If not provided, the `CODEPLAIN_API_KEY` environment variable is used. - --full-plain Full preview ***plain specification before code generation.Use when you want to preview context of all ***plain - primitives that are going to be included in order to render the given module. - --dry-run Dry run preview of the code generation (without actually making any changes). + Timeout for test scripts in seconds. If not provided, + the default timeout of 120 seconds is used. + --api [API] Alternative base URL for the API. Default: + `https://api.codeplain.ai` + --api-key API_KEY API key used to access the API. If not provided, the + `CODEPLAIN_API_KEY` environment variable is used. + --full-plain Full preview ***plain specification before code + generation.Use when you want to preview context of all + ***plain primitives that are going to be included in + order to render the given module. + --dry-run Dry run preview of the code generation (without + actually making any changes). --replay-with REPLAY_WITH --template-dir TEMPLATE_DIR - Path to a custom template directory. Templates are searched in the following order: 1) Directory containing the plain - file, 2) Custom template directory (if provided through this argument), 3) Built-in standard_template_library - directory - --copy-build If set, copy the rendered contents of code in `--base-folder` folder to `--build-dest` folder after successful - rendering. + Path to a custom template directory. Templates are + searched in the following order: 1) Directory + containing the plain file, 2) Custom template + directory (if provided through this argument), 3) + Built-in standard_template_library directory + --copy-build If set, copy the rendered contents of code in `--base- + folder` folder to `--build-dest` folder after + successful rendering. --build-dest BUILD_DEST - Target folder to copy rendered contents of code to (used only if --copy-build is set). + Target folder to copy rendered contents of code to + (used only if --copy-build is set). --copy-conformance-tests - If set, copy the conformance tests of code in `--conformance-tests-folder` folder to `--conformance-tests-dest` - folder successful rendering. Requires --conformance-tests-script. + If set, copy the conformance tests of code in + `--conformance-tests-folder` folder to `--conformance- + tests-dest` folder successful rendering. Requires + --conformance-tests-script. --conformance-tests-dest CONFORMANCE_TESTS_DEST - Target folder to copy conformance tests of code to (used only if --copy-conformance-tests is set). + Target folder to copy conformance tests of code to + (used only if --copy-conformance-tests is set). --render-machine-graph If set, render the state machine graph. --logging-config-path LOGGING_CONFIG_PATH Path to the logging configuration file. - --headless Run in headless mode: no TUI, no terminal output except a single render-started message. All logs are written to the - log file. + --headless Run in headless mode: no TUI, no terminal output + except a single render-started message. All logs are + written to the log file. configuration: --config-name CONFIG_NAME - Name of the config file to look for. Looked up in the plain file directory and the current working directory. - Defaults to config.yaml. + Name of the config file to look for. Looked up in the + plain file directory and the current working + directory. Defaults to config.yaml. ``` diff --git a/forge/skills/plain-healthcheck/SKILL.md b/forge/skills/plain-healthcheck/SKILL.md index a60573b..26d6cc0 100644 --- a/forge/skills/plain-healthcheck/SKILL.md +++ b/forge/skills/plain-healthcheck/SKILL.md @@ -52,7 +52,7 @@ For each `config.yaml` in the inventory, check **all** of the following. Collect 2. **At minimum `unittests-script` is present.** Every project gets a unit-test runner. 3. **For every script field that is present** (`unittests-script`, `conformance-tests-script`, `prepare-environment-script`): - The path is a string ending in `.sh` (macOS/Linux) or `.ps1` (Windows). The extension must match the rest of the project — do not mix `.sh` and `.ps1` in a single config. - - The referenced file actually exists on disk under `test_scripts/`. + - The referenced file actually exists on disk, resolved the way the renderer resolves it: absolute / `~` paths as-is, relative paths against the **config file's directory** (typically landing under `test_scripts/`). There is no other fallback — the renderer fails fast with a `FileNotFoundError` on a script path that doesn't resolve, so a path that only "works" from some other directory is a failure here. - On Unix, the script has the executable bit set (`-x`). If not, that is a fixable failure. 4. **No mixed stacks per config.** Every script referenced from a single `config.yaml` must target the same language/stack. For example, `backend/config.yaml` should not reference `run_unittests_js.sh`. If a config crosses stacks, that is a failure — the project should have been split into multiple configs per the rule in `PLAIN_REFERENCE.md`. 5. **No dangling fields.** Any `*-script` field whose target file does not exist is a failure. diff --git a/forge/skills/run-codeplain/SKILL.md b/forge/skills/run-codeplain/SKILL.md index 562312c..029e2bb 100644 --- a/forge/skills/run-codeplain/SKILL.md +++ b/forge/skills/run-codeplain/SKILL.md @@ -65,9 +65,9 @@ Before launching anything: Do **not** prompt for flags the user did not signal interest in; the empty command line is the right default. -5. **Locate and reset the log file.** It is `codeplain.log` in the same directory as the `.plain` file unless `--log-file-name` was overridden. **The file is overwritten** at the start of each run (per the `--log-file-name` help text). So: +5. **Locate and reset the log file.** By default it is `codeplain.log` in the same directory as the `.plain` file. If `--log-file-name` was overridden, the override resolves based on **where it was written**: a relative value on the CLI resolves against the invocation directory; a relative value in `config.yaml` resolves against the config file's directory; absolute and `~` paths are used as-is. **The file is overwritten** at the start of each run (per the `--log-file-name` help text). So: - Before launch, note the **absolute path** of the log file and its current size (`wc -c `) — this lets you detect the moment the new run truncates and starts writing. - - After launch, every log read must be against this same path. Re-derive it if the user passes `--log-file-name` or runs from a different working directory. + - After launch, every log read must be against this same path. Re-derive it (using the resolution rule above) if the user passes `--log-file-name`, sets `log-file-name` in `config.yaml`, or runs from a different working directory. - The log is **per-run**: every byte you read during this run was written by this run. You don't have to worry about stale lines from prior runs. 6. **Read every test script referenced by the governing `config.yaml`.** This is non-negotiable. The scripts are the contract between the renderer and the project; you cannot judge what the renderer is "fixing" without knowing what `pass` and `fail` actually mean in this project. For each of `unittests-script`, `prepare-environment-script`, and `conformance-tests-script` (those that are declared), read the script end-to-end **before** launching, and write down for yourself: @@ -75,7 +75,7 @@ Before launching anything: - **Which framework is invoked** (e.g. `vitest`, `jest`, `pytest`, `go test`, `cargo test`, `phpunit`, etc.) and any framework flags (`--noEmit`, `--reporter=verbose`, custom configs). - **What "pass" means**, exit-code-wise. Some scripts combine multiple checks (e.g. `unit_testing_typescript.sh` here runs `tsc --noEmit` *and* `vitest`, and its final exit code is the worst of the two — so a type error alone fails the unit-test gate). - **What "fail" means**, including any failures that the script deliberately downgrades to warnings (e.g. `prepare_environment_typescript.sh` here treats a failed `npm run build` as a non-fatal warning). Failures the script swallows are failures the renderer will **never see**, so a spec defect that depends on them will silently slip through. - - **Where the scripts materialize their working environment** (e.g. `.tmp/typescript_/`, `build/`, `target/`). This is where the renderer's iteration actually runs; if you need to look at what is breaking, this is the directory — not `plain_modules//` directly. + - **Where the scripts materialize their working environment** (current-convention scripts stage into the system temp directory, e.g. `/tmp/typescript_/`; older scripts used project-local folders like `.tmp/typescript_/`, `build/`, `target/`). This is where the renderer's iteration actually runs; if you need to look at what is breaking, this is the directory — not `plain_modules//` directly. - **How conformance tests are discovered and staged** (glob, naming convention, alias setup, etc.). Some scripts stage tests into the source tree (this repo flattens `conformance_tests///` into `src//`); some run them in place; some require a specific file extension. This tells you exactly which conformance test file is in play for a given functionality. - **Toolchain prerequisites** the script asserts (Node version, Python version, specific binaries). If any are missing on the user's machine, stop now and tell them — the renderer will burn credits on phantom failures otherwise. @@ -380,6 +380,15 @@ Renderer surfaces a conflict, or you can see two specs in the same module that c Symptom: log shows the unit-test or conformance-test **script** itself failing (non-zero exit, syntax error, missing dependency) on every attempt, rather than the renderer's code failing the tests. The fix is **not** a spec change — it is a `test_scripts/` change. Stop and hand off to the matching `implement-*-testing-script` skill, or have the user repair the script directly. +**Special case — stale scripts after a renderer upgrade (doubled paths).** The renderer now passes the build folder and conformance-tests folder to the scripts as **absolute paths**; older test scripts built paths by concatenating those arguments (`.tmp/$1`, `python_$1`, `$current_dir/$2`). The tell is a test-log path that contains the project root **twice**, e.g.: + +``` +ImportError: Start directory is not importable +/Users/x/project//Users/x/project/conformance_tests/module/spec +``` + +The render then gets stuck "fixing" tests that can never pass. The fix is never a spec edit: stop the render and regenerate the affected scripts via the matching `implement-*-testing-script` skill (current convention: system-temp working folder from `basename "$1"`, `$2` resolved to absolute before any `cd`). + ### Pathology F — Unit-test retry loop Less common than the conformance loop, but the same shape: `Running unit tests script ... (attempt: )` with `` climbing for the same functionality. Threshold: **3 attempts**. From `ATTEMPT_COUNTER >= 2` onwards, run the same [Spec-deviation classification](#spec-deviation-classification) classifier and act on the bucket it returns. The most common root causes for unit-test loops are an implementation req that doesn't match the unit-test scaffolding (Bucket 2), or test reqs / framework usage that the renderer is satisfying by weakening assertions (Bucket 3). Hand-off is typically `add-implementation-requirement` or `debug-specs`.