diff --git a/AGENTS.md b/AGENTS.md index c7b9da0..b85a09f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -32,3 +32,5 @@ Read `docs/PRODUCT_SPEC.md`, `docs/ARCHITECTURE.md`, `docs/BOUNDARY.md`, and `do Run `npm run verify` after meaningful changes. The repository is a public alpha only after the owner-authorized repository and package publication have been verified. Keep statistical, adoption, grant, and production claims fail-closed until their evidence exists. + +Follow [docs/agent-verification-map.md](docs/agent-verification-map.md) and [docs/agent-evaluation-protocol.md](docs/agent-evaluation-protocol.md) for bounded agent work, task scope, acceptance commands and evidence labels. diff --git a/CHANGELOG.md b/CHANGELOG.md index 22358e1..a0f1126 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,26 @@ All notable changes are documented here, newest first. Entries mirror the tagged releases (dates UTC); the release page for each tag carries the same text plus its assets. +## [Unreleased] + +- Added a deterministic TAM workflow descriptor and a second service-quality + workflow covering instrument, topic card, coverage, audit, recode, + measurement, and all supported emitters. +- Extended `npm run first-use` to verify the complete workflow and preserve + rows/raw columns without executing a statistical engine. +- Added a named `--workflow service-quality` entrypoint mode with concise help + and a truthful local summary for first-time command-line use. +- Added a deterministic `workflow-summary.md` beside the JSON summary, with + step statuses, row/raw-column invariants, emitted filenames, and the explicit + no-statistical-engine boundary. +- Added an offline clean-room consumer smoke that packs, installs, imports, and + runs the CLI with a shipped example without contacting the registry. +- Added release/update/rollback guidance. This entry is not a published + version and does not claim statistical validity or engine execution. + +Release candidate: `0.1.0-alpha.6` is prepared locally. It is not published +until the owner completes npm authentication and the release gates pass. + ## [v0.1.0-alpha.5](https://github.com/daichunghy/quant-research/releases/tag/v0.1.0-alpha.5) — 2026-08-23 Public alpha release of @agentbiz/quant-research.\n\nInstall:\n\n npm install @agentbiz/quant-research@alpha\n\nThis release provides deterministic instrument, coverage, dataset, recode, measurement-spec, receipt, and emitter contracts. It does not estimate SEM, compute fit indices, write spreadsheets, call model providers, or claim adoption. @@ -21,4 +41,3 @@ Current public source alpha. The GitHub tag is verified by npm ci and npm run ve ## [v0.1.0-alpha.1](https://github.com/daichunghy/quant-research/releases/tag/v0.1.0-alpha.1) — 2026-08-23 Public alpha. Deterministic instrument, coverage, dataset, recode, measurement-spec, receipt, and SPSS/lavaan/AMOS/SmartPLS emitter contracts. This release does not run SEM, write spreadsheets, claim ethical approval, or claim adoption. Verify with npm run verify; external consumers are not yet recorded. - diff --git a/README.md b/README.md index 7392173..b7d80f5 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ Large language models do not reliably keep Likert bounds, reverse-keyed items, c It compiles JSON in and emits dictionaries, coverage maps, recode artifacts, SPSS syntax, lavaan specs, AMOS path lists, and SmartPLS indicator maps. It does **not** estimate models, compute fit indices, write Excel or Google Sheets, or call a model provider. -**Live status (2026-08-24):** public alpha (`0.1.0-alpha.5` on GitHub, 0 stars, 0 forks). No external user, downstream repository, or pilot is verified. npm `alpha` resolves `0.1.0-alpha.5` while `latest` remains `0.1.0-alpha.4`. This release is a deterministic contract and emitter library. It does not estimate SEM, compute fit indices, write Excel or Google Sheets, call a model provider, or claim adoption, downloads, or dependent repositories. +**Live status (2026-08-28):** alpha.6 is prepared locally but not yet published; the public GitHub and npm alpha remain alpha.5, with `latest` at alpha.4. No external user, downstream repository, or pilot is verified. This release is a deterministic contract and emitter library. It does not estimate SEM, compute fit indices, write Excel or Google Sheets, call a model provider, or claim adoption, downloads, or dependent repositories. > If it caught one inconsistent recode before analysis, > [star it](https://github.com/daichunghy/quant-research/stargazers). That is @@ -20,6 +20,24 @@ Install the public alpha from the npm registry: npm install @agentbiz/quant-research@alpha ``` +After the prepared alpha.6 package is published, run the complete packaged +workflow without cloning the repository: + +```bash +npm install --save-exact @agentbiz/quant-research@0.1.0-alpha.6 +npx agentbiz-quant-workflow --workflow service-quality --out ./quant-output +``` + +Then run the complete packaged workflow without cloning the repository: + +```bash +npx agentbiz-quant-workflow --workflow service-quality --out ./quant-output +``` + +The command writes a reproducible bundle with a human-readable +`workflow-summary.md`; it preserves rows and raw columns and reports that no +statistical engine executed. + The package targets Node.js 20 and 22. Use the GitHub repository for release notes, boundary documents, examples, and issue reporting. @@ -51,6 +69,7 @@ npm run verify ```bash node dist/cli.js compile examples/tam-instrument.json +node dist/cli.js compile examples/service-quality-instrument.json node dist/cli.js codebook examples/tam-instrument.json node dist/cli.js recode examples/tam-instrument.json examples/tam-dataset.json node dist/cli.js gap examples/tam-instrument.json examples/tam-observed.json @@ -60,6 +79,18 @@ node dist/cli.js emit-spss examples/tam-bundle.json node dist/cli.js csv examples/tam-dataset.json ``` +Run the complete service-quality example through the named local workflow mode: + +```bash +node scripts/research-workflow.mjs --workflow service-quality --out /tmp/quant-research-service-quality-workflow +``` + +Use `node scripts/research-workflow.mjs --help` to see the named modes and the +descriptor form. The output directory includes a human-readable +`workflow-summary.md` and machine-readable JSON summary. The workflow emits +specifications only; its summary reports row/raw-column preservation and +`statistical engine executed: false`. + MCP stdio: `node dist/mcp.js` (`agentbiz-quant-mcp`). Tools are executable through `executeTool`, not documentation-only. Agent skill: copy `skills/quant-research/SKILL.md` into the target agent's skill directory, or use the installed package with `AGENTBIZ_QUANT_RESEARCH_ROOT` pointing at a clone. The skill is portable and does not assume the maintainer's filesystem path. @@ -99,7 +130,7 @@ Agent tool JSON Schemas live on `@agentbiz/quant-research/tools`. They describe | `agentbiz.measurement.v1` | Declared CB-SEM / PLS-SEM / reliability spec | | emitters | SPSS syntax, lavaan text, AMOS JSON paths, SmartPLS indicator map | -The TAM example items are **demonstration items**. They are not published scales. +The TAM and service-quality example items are **demonstration items**. They are not published scales. ## Non-goals @@ -127,6 +158,10 @@ other researchers find the contracts. Release history: [CHANGELOG.md](CHANGELOG.md). +Agent-assisted changes follow the [verification map](docs/agent-verification-map.md) and [evaluation protocol](docs/agent-evaluation-protocol.md). Run `npm run agent-eval -- QR-01` for a manifest-backed local acceptance task. + +The current local evidence is recorded in the [agent scaling checkpoint](docs/agent-scaling-checkpoint.md). + ## License Apache-2.0. See [LICENSE](LICENSE). diff --git a/docs/REPRODUCIBILITY.md b/docs/REPRODUCIBILITY.md index 195718d..e10cbf5 100644 --- a/docs/REPRODUCIBILITY.md +++ b/docs/REPRODUCIBILITY.md @@ -26,6 +26,7 @@ digests; the package content check also requires the manifest to be present in t ```sh node dist/cli.js compile examples/tam-instrument.json +node dist/cli.js compile examples/service-quality-instrument.json node dist/cli.js codebook examples/tam-instrument.json node dist/cli.js recode examples/tam-instrument.json examples/tam-dataset.json node dist/cli.js audit examples/tam-instrument.json examples/tam-dataset.json @@ -34,8 +35,9 @@ node dist/cli.js emit-spss examples/tam-bundle.json node dist/cli.js csv examples/tam-dataset.json ``` -Use the JSON fixtures as demonstration inputs only. The TAM items are not presented as a published -scale, and the emitted syntax is not a fitted model. A reproducibility record should include the +Use the JSON fixtures as demonstration inputs only. The TAM and service-quality items are not +presented as published scales, and the emitted syntax is not a fitted model. A reproducibility +record should include the release tag, Node.js version, command, exit status, and output artifact hash where the artifact is retained. diff --git a/docs/agent-evaluation-protocol.md b/docs/agent-evaluation-protocol.md new file mode 100644 index 0000000..bc9a1e6 --- /dev/null +++ b/docs/agent-evaluation-protocol.md @@ -0,0 +1,63 @@ +# quant-research Agent Evaluation Protocol + +**Status:** local protocol +**Purpose:** measure whether agents produce reproducible research artifacts +without inventing statistical evidence or mutating source data + +Each evaluation uses a fresh worktree, explicit paths, fixed acceptance +commands, no credentials and parent-maintainer review. + +## Task corpus + +| ID | Task | Acceptance | Owner | Risk | +| --- | --- | --- | --- | --- | +| QR-01 | Contract and schema | schema and typecheck pass | contract | high | +| QR-02 | Instrument compiler | module and determinism tests pass | instrument | high | +| QR-03 | Provenance and topic cards | citation and binding tests pass | provenance | high | +| QR-04 | Gap map and dataset audit | coverage and audit tests pass | dataset | high | +| QR-05 | Recode safety | new-column/null/source-preservation tests pass | recode | critical | +| QR-06 | Measurement specification | measurement and boundary tests pass | measurement | critical | +| QR-07 | Emitters and tools | golden output and tool tests pass | emit | high | +| QR-08 | CLI/package surface | smoke, first-use, pack and errors pass | release | high | + +The machine-readable seed is +[`fixtures/agent-evals/manifest.json`](../fixtures/agent-evals/manifest.json). + +## Rubric + +Score 0–2 for each dimension: + +| Dimension | 0 | 1 | 2 | +| --- | --- | --- | --- | +| Correctness | acceptance fails | partial/rescue needed | acceptance passes | +| Scientific boundary | invented statistic or claim | boundary unclear | non-claims and warnings preserved | +| Data safety | drops/overwrites source | safe but incomplete | immutable/new-column behavior tested | +| Determinism | digest/output drifts | unclear evidence | stable artifact and receipt | +| Scope | unrelated or irreversible | minor drift | atomic and reversible | +| Verification | unsupported claim | partial checks | reproducible checks and artifacts | + +Maximum is 12. Promotion requires at least 10/12, correctness/scientific +boundary/data safety all equal to 2, no P0/P1 issue and parent verification. + +## Procedure + +1. Select one task and record paths, owner, risk and acceptance commands. +2. Create a fresh worktree from the intended base revision. +3. Require inspection of contracts, schemas, examples and tests before editing. +4. Run acceptance commands and inspect artifact, receipt, diff and non-claims. +5. Integrate only after parent review, then run `npm run verify`. +6. Convert recurring failures into fixtures, tests or stable diagnostics. + +`npm run agent-eval -- QR-01` runs the manifest's allowlisted commands. It does +not authorize engine execution, external data access or a scientific claim. + +## Wave policy + +| Wave | Scope | Quantity | Promotion | +| --- | --- | ---: | --- | +| A | contracts, instruments, coverage and CLI | 4 | all acceptance commands pass | +| B | recode, measurement and emitters | 4 | no data/scientific-boundary regression | +| C | package and external researcher feedback | as available | consented evidence and documented fixes | + +Do not create artificial datasets, dependents, downloads, citations or research +results to increase the score. diff --git a/docs/agent-scaling-checkpoint.md b/docs/agent-scaling-checkpoint.md new file mode 100644 index 0000000..73c3821 --- /dev/null +++ b/docs/agent-scaling-checkpoint.md @@ -0,0 +1,44 @@ +# Agent scaling checkpoint — 2026-08-27 + +**Evidence level:** local and fixture-verified + +The repository now contains a quant-research-specific [verification +map](agent-verification-map.md), [evaluation protocol](agent-evaluation-protocol.md), +machine-readable corpus at `fixtures/agent-evals/manifest.json`, and an +allowlisted `npm run agent-eval -- ` runner. The contract checker is +part of `npm run verify`. + +## Acceptance baseline + +All eight manifest tasks passed on the current tree: + +```text +QR-01 contract/schema: pass +QR-02 instrument compiler: pass +QR-03 provenance/topic cards: pass +QR-04 gap map/dataset audit: pass +QR-05 recode safety: pass +QR-06 measurement specification: pass +QR-07 emitters/tools: pass +QR-08 CLI/package surface: pass +``` + +The complete repository verification also passed: 55 tests, schema +reconciliation, error-reference checks, package/secret checks, offline +clean-room consumer smoke and CLI smoke. +The first-use command now runs the complete typed workflow and checks that +rows and raw columns are preserved without executing a statistical engine. +Each run now also writes a deterministic `workflow-summary.md` beside the JSON +summary for quick human review. The checked examples now include both TAM and +service-quality workflow families. + +## Limits + +This proves reproducible local artifacts only. It does not prove ethical +approval, construct validity, SEM correctness, executed SPSS/lavaan/AMOS/ +SmartPLS jobs, external researcher use, grant eligibility or production +adoption. The package still must not drop rows, overwrite source columns, +invoke engines, write spreadsheets or import OpenSheet-AI. + +The next useful evidence is a consented external researcher workflow and a +qualified review of any future statistical-engine adapter. diff --git a/docs/agent-verification-map.md b/docs/agent-verification-map.md new file mode 100644 index 0000000..6e85fd5 --- /dev/null +++ b/docs/agent-verification-map.md @@ -0,0 +1,103 @@ +# quant-research Agent Verification Map + +**Status:** local operating contract +**Authority:** `docs/PRODUCT_SPEC.md`, `docs/ARCHITECTURE.md`, +`docs/BOUNDARY.md`, `docs/THREAT_MODEL.md`, exported contracts, schemas and +tests + +quant-research applies the transcript's verification-first workflow to typed +research artifacts. It produces instruments, coverage maps, recodes, +measurement specifications and emitter outputs. It does not run statistics, +write spreadsheets or turn a model's prose into an unvalidated research claim. + +## Trust curve + +```text +validate typed research input + -> compile deterministic artifact + -> verify provenance and contract + -> inspect recode/audit findings + -> emit reproducible engine syntax/specification + -> bounded agent work + -> external researcher evidence +``` + +Receipts prove the input and artifact digests under the package contract. They +do not prove ethical approval, construct validity, SEM correctness or an +executed external engine. + +## Surface map + +| Surface | Start here | Minimum verification | Invariant | +| --- | --- | --- | --- | +| Contract/schema | `src/core/`, `schemas/` | schema, receipt and reconciliation tests | unknown versions fail closed | +| Domain compilers | `src/instrument`, `src/topic-card`, `src/gap-map`, `src/dataset`, `src/recode`, `src/measurement` | module, negative and deterministic tests | typed inputs only; no network or clock | +| Emitters | `src/emit/` | golden emitter tests | emit text/specs; never invoke engines | +| Tools/MCP | `src/tools/`, `src/mcp.ts` | tool, boundary and CLI tests | no `ai` dependency or arbitrary execution | +| Provenance | examples and citation validation | provenance/error tests | `cited` requires `source.citation` | +| Dataset/recode safety | `src/dataset`, `src/recode` | boundary and immutability tests | report issues; add columns; never drop rows | +| Package/release | `package.json`, `scripts/check-pack.mjs` | pack, secrets and smoke checks | public surface matches declared files | + +## Verification ladder + +```bash +npm run typecheck +npm run test +npm run build +npm run check:schema +npm run check:errors +npm run check:secrets +npm run check:smoke +npm run check:pack +npm run check:agent-contract +npm run verify +npm run agent-eval -- QR-01 +``` + +`first-use` and local quickstarts are local evidence only. They do not prove +external researcher use, engine execution, publication quality or grant +eligibility. + +## Non-negotiable PR invariants + +- compilers accept validated typed JSON and use no network, credentials, + model calls, filesystem reads, randomness or implicit time; +- no compiler estimates SEM, fit indices, alpha, AVE, HTMT, p-values or other + statistical results; +- dataset audits report out-of-bound/missing/duplicate/straight-line findings + without dropping rows; +- recodes create new columns, preserve nulls and never overwrite source data; +- demonstration items remain labelled `demonstration`; `cited` requires a + citation; +- emitters produce SPSS/lavaan/AMOS/SmartPLS specifications but never execute + an engine or claim its output; +- no Excel/Google Sheets write, OpenSheet-AI import, payment or ERP connector + is added to this package; +- receipts bind exact input and artifact digests but do not claim validation; +- one agent task has one purpose, explicit paths and a reversible diff; +- local CI and package checks are not adoption, production or grant evidence. + +## Bounded work + +Use two or three independent tasks only when write sets are disjoint. Serialize +core schemas, recode semantics, emitters and package exports. The parent +maintainer inspects each diff, reruns targeted checks, then runs +`npm run verify`. Timeouts and unreviewed agent summaries are not evidence. + +## Failure-to-guardrail loop + +Convert every recurring failure into a negative fixture, deterministic test, +error reference, secret scan or package check. Prefer an explicit unsupported +result over a silently broader scientific or platform claim. + +## Handoff + +```text +Scope: +Files changed: +Invariant protected: +Targeted checks: +Aggregate check: +Evidence level: +Known non-claims: +``` diff --git a/docs/first-use.md b/docs/first-use.md index 3db9211..784db94 100644 --- a/docs/first-use.md +++ b/docs/first-use.md @@ -11,6 +11,42 @@ npm ci npm run first-use ``` +For an installed package, the shortest path is: + +```bash +npm install @agentbiz/quant-research@alpha +npx agentbiz-quant-workflow --workflow service-quality --out ./quant-output +``` + +This uses the workflow script and demonstration inputs shipped in the package; +no source checkout is required. + +To run the complete TAM workflow and receive a readable output directory: + +```bash +npm run build +node scripts/research-workflow.mjs --out /tmp/quant-research-workflow +``` + +This creates a codebook, coverage map, dataset audit, recode, measurement +specification, and SPSS/lavaan/AMOS/SmartPLS emissions. The emissions are +specifications only; the workflow also writes `workflow-summary.json` and the +concise newcomer-facing `workflow-summary.md`. It does not run a statistical +engine or delete rows or items. + +To run a second complete workflow for service quality and customer experience: + +```bash +node scripts/research-workflow.mjs --workflow service-quality --out /tmp/quant-research-service-quality-workflow +``` + +The entrypoint lists this mode with `node scripts/research-workflow.mjs --help`. +The descriptor form remains supported for replaying or supplying another +workflow JSON. This service-quality example keeps its instrument, topic card, +observed columns, small dataset, recode plan, and measurement input together +in `examples/service-quality-workflow/`; it follows the same audit, recode, +and emitter path as the TAM example. + The outputs should make the following visible: - construct and item codes with Likert bounds and reverse flags; @@ -18,7 +54,7 @@ The outputs should make the following visible: - covered, partial, missing, mismatch, and unexpected indicators; - declared syntax for lavaan or SPSS without pretending that estimation ran. -The TAM items are demonstrations, not published scales. This package does not +The TAM and service-quality inputs are demonstrations, not published scales. This package does not estimate SEM, compute fit indices, write spreadsheets, call a model provider, or decide whether a measurement model is valid. @@ -31,3 +67,6 @@ without attaching participant data, credentials, or unpublished instruments. This path proves deterministic local output. It does not prove external users, ethical approval, statistical validity, or package adoption. + +For package release, consumer updates, and rollback, see +[release-and-rollback.md](release-and-rollback.md). diff --git a/docs/release-and-rollback.md b/docs/release-and-rollback.md new file mode 100644 index 0000000..a6fb8bf --- /dev/null +++ b/docs/release-and-rollback.md @@ -0,0 +1,58 @@ +# Release, update, and rollback + +This runbook covers the current deterministic alpha. It does not authorize a +publish, a stable statistical package claim, an executed SEM analysis, or +external adoption. + +## Before release + +1. Freeze the source commit and package version. +2. From a clean checkout, run `npm ci`, `npm run verify`, and + `npm pack --dry-run --json`. The verify command includes an offline + clean-room consumer smoke that installs the package, imports it, starts the + CLI, and runs a shipped example. +3. Confirm that `examples/reproducibility-manifest.json` pins every distributed + JSON fixture and that the replay digests still match. +4. Run `npm run first-use` and retain the summary. It must show preserved rows, + preserved raw columns, and `statisticalEngineExecuted: false`. +5. Recheck the registry with + `npm view @agentbiz/quant-research dist-tags versions --json`. Source tags + and registry tags may not be synchronized. +6. Only the package owner may publish or create a release. Record the exact + version, tarball file list, manifest hash, replay output, and gate output. + +## Consumer update + +Use an exact version and replay the deterministic workflow: + +```sh +npm install --save-exact @agentbiz/quant-research@ +npm run first-use +``` + +Keep the instrument, dataset, recode, and emitted specifications under the +consumer's own provenance rules. Demonstration items remain demonstrations; +the package does not estimate models or run SPSS, lavaan, AMOS, or SmartPLS. + +## Rollback + +1. Stop the update and record package version, input digest, artifact digest, + emitted syntax version, and the redacted symptom. +2. Restore the last known-good exact package version or lockfile, then rerun + the verification and first-use workflow: + + ```sh + npm install --save-exact @agentbiz/quant-research@ + npm run verify + npm run first-use + ``` + +3. Keep raw datasets and prior emitted artifacts unchanged. Recode operations + must continue to add columns rather than overwrite source columns or drop + rows. +4. Preserve faulty tags and release history. Do not force-push or silently + repoint a published dist-tag. A replacement release requires owner approval. +5. Compare the known-good receipt and artifact digests before resuming work. + +Rollback restores the deterministic contract version. It does not establish +ethical approval, statistical validity, engine execution, or adoption. diff --git a/examples/reproducibility-manifest.json b/examples/reproducibility-manifest.json index 534f766..bf9377e 100644 --- a/examples/reproducibility-manifest.json +++ b/examples/reproducibility-manifest.json @@ -2,9 +2,13 @@ "manifestVersion": "agentbiz.reproducibility.v1", "package": { "name": "@agentbiz/quant-research", - "version": "0.1.0-alpha.5" + "version": "0.1.0-alpha.6" }, "fixtures": [ + { + "path": "examples/service-quality-instrument.json", + "sha256": "sha256:c24d330212534988fa87581a5d4336e89879c5413cc3e640afb24eb2ef314461" + }, { "path": "examples/tam-bundle.json", "sha256": "sha256:b2a95a6360ab34b81303678882742a5186600148cd546c9166ccec1f21f1d103" @@ -32,6 +36,10 @@ { "path": "examples/tam-topic-card.json", "sha256": "sha256:5ee4ca7bfc930c85fb9bbb7d956a5c13cb6b6c054c343d454ffe2f0619bea98a" + }, + { + "path": "examples/tam-workflow.json", + "sha256": "sha256:d9aeb588ab4311bef8e5e9de7c44f54fc111e66fbc779c4183a8dbb5f9174758" } ], "replays": [ diff --git a/examples/service-quality-instrument.json b/examples/service-quality-instrument.json new file mode 100644 index 0000000..4390061 --- /dev/null +++ b/examples/service-quality-instrument.json @@ -0,0 +1,85 @@ +{ + "schemaVersion": "agentbiz.instrument.v1", + "language": "en", + "constructs": [ + { + "code": "REL", + "name": "Service reliability", + "kind": "reflective", + "scale": { + "min": 1, + "max": 7, + "anchors": [ + { "value": 1, "label": "Strongly disagree" }, + { "value": 4, "label": "Neither agree nor disagree" }, + { "value": 7, "label": "Strongly agree" } + ] + }, + "items": [ + { + "code": "REL1", + "text": "This service performs consistently when I need it.", + "status": "demonstration" + }, + { + "code": "REL2", + "text": "This service delivers the outcome it promises.", + "status": "demonstration" + } + ] + }, + { + "code": "RESP", + "name": "Service responsiveness", + "kind": "reflective", + "scale": { "min": 1, "max": 7 }, + "items": [ + { + "code": "RESP1", + "text": "Requests are handled within a reasonable time.", + "status": "demonstration" + }, + { + "code": "RESP2", + "text": "The service communicates clearly when work is delayed.", + "status": "demonstration" + }, + { + "code": "RESP3", + "text": "Getting help from this service takes too much effort.", + "reverse": true, + "status": "demonstration" + } + ] + }, + { + "code": "SAT", + "name": "Service satisfaction", + "kind": "reflective", + "scale": { "min": 1, "max": 7 }, + "items": [ + { + "code": "SAT1", + "text": "Overall, I am satisfied with this service.", + "status": "demonstration" + }, + { + "code": "SAT2", + "text": "This service meets my expectations.", + "status": "demonstration" + }, + { + "code": "SAT3", + "text": "I would choose this service again for a similar need.", + "status": "demonstration" + }, + { + "code": "SAT4", + "text": "Using this service has been a frustrating experience.", + "reverse": true, + "status": "demonstration" + } + ] + } + ] +} diff --git a/examples/service-quality-workflow/dataset.json b/examples/service-quality-workflow/dataset.json new file mode 100644 index 0000000..69e8196 --- /dev/null +++ b/examples/service-quality-workflow/dataset.json @@ -0,0 +1,25 @@ +{ + "schemaVersion": "agentbiz.dataset.v1", + "columns": [ + { "name": "id", "role": "id" }, + { "name": "REL1", "role": "indicator", "itemCode": "REL1" }, + { "name": "REL2", "role": "indicator", "itemCode": "REL2" }, + { "name": "REL3", "role": "indicator", "itemCode": "REL3" }, + { "name": "RESP1", "role": "indicator", "itemCode": "RESP1" }, + { "name": "RESP2", "role": "indicator", "itemCode": "RESP2" }, + { "name": "RESP3", "role": "indicator", "itemCode": "RESP3" }, + { "name": "SAT1", "role": "indicator", "itemCode": "SAT1" }, + { "name": "SAT2", "role": "indicator", "itemCode": "SAT2" }, + { "name": "SAT3", "role": "indicator", "itemCode": "SAT3" }, + { "name": "channel", "role": "demographic" } + ], + "rowCount": 6, + "rows": [ + { "id": "C01", "REL1": 6, "REL2": 6, "REL3": 2, "RESP1": 5, "RESP2": 6, "RESP3": 2, "SAT1": 6, "SAT2": 6, "SAT3": 6, "channel": "email" }, + { "id": "C02", "REL1": 5, "REL2": 6, "REL3": 3, "RESP1": 6, "RESP2": 5, "RESP3": 2, "SAT1": 6, "SAT2": 5, "SAT3": 6, "channel": "chat" }, + { "id": "C03", "REL1": 4, "REL2": 4, "REL3": 4, "RESP1": 4, "RESP2": 4, "RESP3": 4, "SAT1": 4, "SAT2": 4, "SAT3": 4, "channel": "phone" }, + { "id": "C04", "REL1": 7, "REL2": 6, "REL3": 1, "RESP1": 6, "RESP2": 7, "RESP3": 1, "SAT1": 7, "SAT2": 6, "SAT3": 7, "channel": "email" }, + { "id": "C05", "REL1": 3, "REL2": 4, "REL3": null, "RESP1": 3, "RESP2": 4, "RESP3": 5, "SAT1": 3, "SAT2": 4, "SAT3": 3, "channel": "web" }, + { "id": "C06", "REL1": 2, "REL2": 3, "REL3": 6, "RESP1": 2, "RESP2": 3, "RESP3": 6, "SAT1": 2, "SAT2": 3, "SAT3": 2, "channel": "chat" } + ] +} diff --git a/examples/service-quality-workflow/instrument.json b/examples/service-quality-workflow/instrument.json new file mode 100644 index 0000000..2a074fc --- /dev/null +++ b/examples/service-quality-workflow/instrument.json @@ -0,0 +1,94 @@ +{ + "schemaVersion": "agentbiz.instrument.v1", + "language": "en", + "constructs": [ + { + "code": "REL", + "name": "Service reliability", + "kind": "reflective", + "scale": { + "min": 1, + "max": 7, + "anchors": [ + { "value": 1, "label": "Strongly disagree" }, + { "value": 4, "label": "Neither agree nor disagree" }, + { "value": 7, "label": "Strongly agree" } + ] + }, + "items": [ + { + "code": "REL1", + "text": "This service delivers what it promises.", + "status": "demonstration", + "source": { "note": "Demonstration item for a service-quality workflow; not a published scale." } + }, + { + "code": "REL2", + "text": "This service is dependable when I need it.", + "status": "demonstration", + "source": { "note": "Demonstration item for a service-quality workflow; not a published scale." } + }, + { + "code": "REL3", + "text": "This service often fails to deliver the expected result.", + "reverse": true, + "status": "demonstration", + "source": { "note": "Demonstration reverse-keyed item; not a published scale." } + } + ] + }, + { + "code": "RESP", + "name": "Service responsiveness", + "kind": "reflective", + "scale": { "min": 1, "max": 7 }, + "items": [ + { + "code": "RESP1", + "text": "My requests receive a timely response.", + "status": "demonstration", + "source": { "note": "Demonstration item for a service-quality workflow; not a published scale." } + }, + { + "code": "RESP2", + "text": "The service explains delays clearly.", + "status": "demonstration", + "source": { "note": "Demonstration item for a service-quality workflow; not a published scale." } + }, + { + "code": "RESP3", + "text": "Getting help from this service takes too much effort.", + "reverse": true, + "status": "demonstration", + "source": { "note": "Demonstration reverse-keyed item; not a published scale." } + } + ] + }, + { + "code": "SAT", + "name": "Overall service satisfaction", + "kind": "reflective", + "scale": { "min": 1, "max": 7 }, + "items": [ + { + "code": "SAT1", + "text": "Overall, I am satisfied with this service.", + "status": "demonstration", + "source": { "note": "Demonstration item for a service-quality workflow; not a published scale." } + }, + { + "code": "SAT2", + "text": "This service meets my expectations.", + "status": "demonstration", + "source": { "note": "Demonstration item for a service-quality workflow; not a published scale." } + }, + { + "code": "SAT3", + "text": "I would choose this service again for a similar need.", + "status": "demonstration", + "source": { "note": "Demonstration item for a service-quality workflow; not a published scale." } + } + ] + } + ] +} diff --git a/examples/service-quality-workflow/measurement.json b/examples/service-quality-workflow/measurement.json new file mode 100644 index 0000000..62ce4d2 --- /dev/null +++ b/examples/service-quality-workflow/measurement.json @@ -0,0 +1,5 @@ +{ + "schemaVersion": "agentbiz.measurement.v1", + "mode": "cb-sem", + "engineOptions": { "estimator": "ML" } +} diff --git a/examples/service-quality-workflow/observed.json b/examples/service-quality-workflow/observed.json new file mode 100644 index 0000000..76eca75 --- /dev/null +++ b/examples/service-quality-workflow/observed.json @@ -0,0 +1,14 @@ +{ + "schemaVersion": "agentbiz.gap-map.v1", + "observed": [ + { "column": "REL1", "itemCode": "REL1", "constructCode": "REL", "reverse": false, "scale": { "min": 1, "max": 7 } }, + { "column": "REL2", "itemCode": "REL2", "constructCode": "REL", "reverse": false, "scale": { "min": 1, "max": 7 } }, + { "column": "REL3", "itemCode": "REL3", "constructCode": "REL", "reverse": true, "scale": { "min": 1, "max": 7 } }, + { "column": "RESP1", "itemCode": "RESP1", "constructCode": "RESP", "reverse": false, "scale": { "min": 1, "max": 7 } }, + { "column": "RESP2", "itemCode": "RESP2", "constructCode": "RESP", "reverse": false, "scale": { "min": 1, "max": 7 } }, + { "column": "RESP3", "itemCode": "RESP3", "constructCode": "RESP", "reverse": true, "scale": { "min": 1, "max": 7 } }, + { "column": "SAT1", "itemCode": "SAT1", "constructCode": "SAT", "reverse": false, "scale": { "min": 1, "max": 7 } }, + { "column": "SAT2", "itemCode": "SAT2", "constructCode": "SAT", "reverse": false, "scale": { "min": 1, "max": 7 } }, + { "column": "SAT3", "itemCode": "SAT3", "constructCode": "SAT", "reverse": false, "scale": { "min": 1, "max": 7 } } + ] +} diff --git a/examples/service-quality-workflow/recode.json b/examples/service-quality-workflow/recode.json new file mode 100644 index 0000000..9a94b54 --- /dev/null +++ b/examples/service-quality-workflow/recode.json @@ -0,0 +1,11 @@ +{ + "schemaVersion": "agentbiz.recode.v1", + "operations": [ + { "kind": "reverse-score", "itemCode": "REL3", "as": "REL3_R" }, + { "kind": "reverse-score", "itemCode": "RESP3", "as": "RESP3_R" }, + { "kind": "construct-score", "constructCode": "REL", "as": "REL_MEAN" }, + { "kind": "construct-score", "constructCode": "RESP", "as": "RESP_MEAN" }, + { "kind": "construct-score", "constructCode": "SAT", "as": "SAT_MEAN" }, + { "kind": "flag-straight-line", "as": "STRAIGHT" } + ] +} diff --git a/examples/service-quality-workflow/topic-card.json b/examples/service-quality-workflow/topic-card.json new file mode 100644 index 0000000..d770320 --- /dev/null +++ b/examples/service-quality-workflow/topic-card.json @@ -0,0 +1,16 @@ +{ + "schemaVersion": "agentbiz.topic-card.v1", + "question": "How are service reliability and responsiveness associated with overall customer satisfaction in this demonstration service experience study?", + "claimClass": "association", + "population": "demonstration service users", + "unit": "survey respondent", + "timeframe": "after a completed service interaction", + "constructCodes": ["REL", "RESP", "SAT"], + "hypotheses": [ + { "id": "H1", "from": "REL", "to": "SAT", "sign": "+" }, + { "id": "H2", "from": "RESP", "to": "SAT", "sign": "+" }, + { "id": "H3", "from": "REL", "to": "RESP", "sign": "+" } + ], + "engine": "lavaan", + "mode": "confirmatory" +} diff --git a/examples/service-quality-workflow/workflow.json b/examples/service-quality-workflow/workflow.json new file mode 100644 index 0000000..835ee50 --- /dev/null +++ b/examples/service-quality-workflow/workflow.json @@ -0,0 +1,23 @@ +{ + "workflowVersion": "agentbiz.research-workflow.v1", + "name": "Service quality and customer experience demonstration workflow", + "provenance": "demonstration", + "inputs": { + "instrument": "instrument.json", + "topicCard": "topic-card.json", + "observed": "observed.json", + "dataset": "dataset.json", + "recode": "recode.json", + "measurement": "measurement.json" + }, + "emitters": [ + "codebook", + "dictionary-csv", + "dataset-csv", + "spss-data", + "spss", + "lavaan", + "amos", + "smartpls" + ] +} diff --git a/examples/tam-workflow.json b/examples/tam-workflow.json new file mode 100644 index 0000000..5fc0a99 --- /dev/null +++ b/examples/tam-workflow.json @@ -0,0 +1,23 @@ +{ + "workflowVersion": "agentbiz.research-workflow.v1", + "name": "TAM demonstration research workflow", + "provenance": "demonstration", + "inputs": { + "instrument": "tam-instrument.json", + "topicCard": "tam-topic-card.json", + "observed": "tam-observed.json", + "dataset": "tam-dataset.json", + "recode": "tam-recode.json", + "measurement": "tam-measurement.json" + }, + "emitters": [ + "codebook", + "dictionary-csv", + "dataset-csv", + "spss-data", + "spss", + "lavaan", + "amos", + "smartpls" + ] +} diff --git a/fixtures/agent-evals/manifest.json b/fixtures/agent-evals/manifest.json new file mode 100644 index 0000000..c94c043 --- /dev/null +++ b/fixtures/agent-evals/manifest.json @@ -0,0 +1,14 @@ +{ + "version": 1, + "evidenceBoundary": "local workflow evidence only", + "tasks": [ + { "id": "QR-01", "title": "Contract and schema", "paths": ["src/core", "schemas", "test/schema-reconciliation.test.ts"], "commands": ["npm run check:schema", "npm run typecheck"], "owner": "contract", "risk": "high", "humanReview": true }, + { "id": "QR-02", "title": "Instrument compiler", "paths": ["src/instrument", "src/core", "test/instrument.test.ts", "test/reproducibility.test.ts"], "commands": ["npm run test", "npm run typecheck"], "owner": "instrument", "risk": "high", "humanReview": true }, + { "id": "QR-03", "title": "Provenance and topic cards", "paths": ["src/topic-card", "src/core", "test/intent.test.ts", "test/modules.test.ts"], "commands": ["npm run test", "npm run check:errors"], "owner": "provenance", "risk": "high", "humanReview": true }, + { "id": "QR-04", "title": "Gap map and dataset audit", "paths": ["src/gap-map", "src/dataset", "test/pipeline.test.ts", "test/boundary.test.ts"], "commands": ["npm run test", "npm run check:smoke"], "owner": "dataset", "risk": "high", "humanReview": true }, + { "id": "QR-05", "title": "Recode safety", "paths": ["src/recode", "test/boundary.test.ts", "test/pipeline.test.ts"], "commands": ["npm run test", "npm run check:errors"], "owner": "recode", "risk": "critical", "humanReview": true }, + { "id": "QR-06", "title": "Measurement specification", "paths": ["src/measurement", "test/measurement.test.ts", "test/boundary.test.ts"], "commands": ["npm run test", "npm run check:errors"], "owner": "measurement", "risk": "critical", "humanReview": true }, + { "id": "QR-07", "title": "Emitters and tools", "paths": ["src/emit", "src/tools", "src/mcp.ts", "test/emitter-golden.test.ts", "test/tool.test.ts"], "commands": ["npm run test", "npm run check:secrets"], "owner": "emit", "risk": "high", "humanReview": true }, + { "id": "QR-08", "title": "CLI and package surface", "paths": ["src/cli.ts", "scripts", "package.json", "README.md"], "commands": ["npm run check:smoke", "npm run first-use", "npm run check:pack", "npm run check:errors"], "owner": "release", "risk": "high", "humanReview": true } + ] +} diff --git a/package-lock.json b/package-lock.json index 3889f5e..1b78f10 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@agentbiz/quant-research", - "version": "0.1.0-alpha.5", + "version": "0.1.0-alpha.6", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@agentbiz/quant-research", - "version": "0.1.0-alpha.5", + "version": "0.1.0-alpha.6", "license": "Apache-2.0", "bin": { "agentbiz-quant": "dist/cli.js", diff --git a/package.json b/package.json index 451aa49..85f4261 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@agentbiz/quant-research", - "version": "0.1.0-alpha.5", + "version": "0.1.0-alpha.6", "description": "Builds checked research instruments, recodes, coverage reports, and analysis specs from typed inputs.", "license": "Apache-2.0", "private": false, @@ -21,7 +21,8 @@ }, "bin": { "agentbiz-quant": "dist/cli.js", - "agentbiz-quant-mcp": "dist/mcp.js" + "agentbiz-quant-mcp": "dist/mcp.js", + "agentbiz-quant-workflow": "scripts/research-workflow.mjs" }, "exports": { ".": { @@ -42,7 +43,8 @@ "schemas", "README.md", "LICENSE", - "examples" + "examples", + "scripts/research-workflow.mjs" ], "scripts": { "build": "tsc -p tsconfig.json", @@ -56,7 +58,11 @@ "check:errors": "node scripts/check-errors.mjs", "check:secrets": "node scripts/check-secrets.mjs", "check:smoke": "node scripts/cli-smoke.mjs", - "verify": "npm run typecheck && npm run build && npm run test && npm run check:schema && npm run check:errors && npm run check:pack && npm run check:secrets && npm run check:smoke" + "check:agent-contract": "node scripts/check-agent-contract.mjs", + "agent-eval": "node scripts/run-agent-eval.mjs", + "check:release-candidate": "node scripts/check-release-candidate.mjs", + "check:clean-room": "node scripts/check-clean-room.mjs", + "verify": "npm run typecheck && npm run build && npm run test && npm run check:schema && npm run check:errors && npm run check:pack && npm run check:release-candidate && npm run check:clean-room && npm run check:secrets && npm run check:smoke && npm run check:agent-contract" }, "devDependencies": { "@types/node": "^22.15.3", diff --git a/scripts/check-agent-contract.mjs b/scripts/check-agent-contract.mjs new file mode 100644 index 0000000..f0d02be --- /dev/null +++ b/scripts/check-agent-contract.mjs @@ -0,0 +1,52 @@ +import { existsSync, readFileSync } from "node:fs"; +import { join } from "node:path"; + +const root = process.cwd(); +const failures = []; +function read(relativePath) { + const absolutePath = join(root, relativePath); + if (!existsSync(absolutePath)) { failures.push(`missing agent-contract file: ${relativePath}`); return ""; } + return readFileSync(absolutePath, "utf8"); +} + +const map = read("docs/agent-verification-map.md"); +const protocol = read("docs/agent-evaluation-protocol.md"); +const manifestText = read("fixtures/agent-evals/manifest.json"); +const agents = read("AGENTS.md"); +const product = read("docs/PRODUCT_SPEC.md"); +const architecture = read("docs/ARCHITECTURE.md"); +const boundary = read("docs/BOUNDARY.md"); +const packageJson = JSON.parse(read("package.json")); +let manifest = null; +try { manifest = JSON.parse(manifestText); } catch { failures.push("agent evaluation manifest is not valid JSON"); } + +for (const text of ["docs/PRODUCT_SPEC.md", "docs/ARCHITECTURE.md", "docs/BOUNDARY.md", "docs/THREAT_MODEL.md", "npm run verify", "deterministic", "receipts", "Excel/Google Sheets"]) if (!map.includes(text)) failures.push(`verification map is missing: ${text}`); +for (const text of ["QR-01", "QR-08", "Scientific boundary", "Data safety", "10/12", "parent review"]) if (!protocol.includes(text)) failures.push(`evaluation protocol is missing: ${text}`); +for (const text of ["deterministic", "receipt", "grant thresholds"]) if (!product.toLowerCase().includes(text.toLowerCase())) failures.push(`product spec anchor is missing: ${text}`); +for (const text of ["pure", "no network", "emitters"]) if (!architecture.toLowerCase().includes(text.toLowerCase())) failures.push(`architecture anchor is missing: ${text}`); +if (!boundary.includes("Github 3") || !boundary.includes("OpenSheet-AI")) failures.push("boundary document must preserve cross-project ownership"); +if (!agents.includes("npm run verify")) failures.push("AGENTS.md is missing npm run verify"); + +const expected = new Set(["QR-01", "QR-02", "QR-03", "QR-04", "QR-05", "QR-06", "QR-07", "QR-08"]); +if (!manifest || manifest.version !== 1 || !Array.isArray(manifest.tasks)) failures.push("agent evaluation manifest must declare version 1 and tasks"); +else { + const actual = new Set(); + for (const task of manifest.tasks) { + if (!task || typeof task.id !== "string" || actual.has(task.id)) failures.push(`missing or duplicate task ID: ${String(task?.id)}`); + if (typeof task?.id === "string") actual.add(task.id); + if (!Array.isArray(task?.paths) || !task.paths.length) failures.push(`${task?.id}: paths required`); + if (!Array.isArray(task?.commands) || !task.commands.length) failures.push(`${task?.id}: commands required`); + if (task?.risk === "critical" && task.humanReview !== true) failures.push(`${task.id}: critical tasks require human review`); + for (const command of task?.commands ?? []) { + const match = /^npm run ([a-z0-9:_-]+)$/.exec(command); + if (!match || !packageJson.scripts?.[match[1]]) failures.push(`${task?.id}: command is not a package script: ${String(command)}`); + } + } + for (const id of expected) if (!actual.has(id)) failures.push(`missing task: ${id}`); +} +if (packageJson.scripts?.["check:agent-contract"] !== "node scripts/check-agent-contract.mjs") failures.push("missing check:agent-contract script"); +if (packageJson.scripts?.["agent-eval"] !== "node scripts/run-agent-eval.mjs") failures.push("missing agent-eval script"); +if (typeof packageJson.scripts?.verify !== "string" || !packageJson.scripts.verify.includes("check:agent-contract")) failures.push("verify must include check:agent-contract"); + +if (failures.length) { console.error(failures.join("\n")); process.exitCode = 1; } +else console.log("agent contract ok: map, protocol, manifest and package hooks are present"); diff --git a/scripts/check-clean-room.mjs b/scripts/check-clean-room.mjs new file mode 100644 index 0000000..459ea53 --- /dev/null +++ b/scripts/check-clean-room.mjs @@ -0,0 +1,89 @@ +#!/usr/bin/env node + +import { execFileSync } from "node:child_process"; +import { existsSync } from "node:fs"; +import { mkdtemp, mkdir, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { dirname, isAbsolute, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +const root = resolve(dirname(fileURLToPath(import.meta.url)), ".."); +const forbiddenRoots = new Set(["src", "test", "fixtures", "scripts"]); + +function run(file, args, cwd) { + return execFileSync(file, args, { + cwd, + encoding: "utf8", + stdio: "pipe", + env: { ...process.env, npm_config_offline: "true", npm_config_ignore_scripts: "true" }, + }); +} + +function fail(message) { + throw new Error(`clean-room consumer smoke: ${message}`); +} + +function packPackage(directory) { + let payload; + try { + payload = JSON.parse( + run("npm", ["pack", "--offline", "--ignore-scripts", "--json", "--pack-destination", directory], root), + ); + } catch (error) { + fail(`npm pack failed: ${error instanceof Error ? error.message : String(error)}`); + } + const entry = Array.isArray(payload) ? payload[0] : payload; + if (entry === null || typeof entry !== "object" || typeof entry.filename !== "string") { + fail("npm pack returned no tarball"); + } + const files = Array.isArray(entry.files) ? entry.files : []; + const forbidden = files + .map((file) => (typeof file === "string" ? file : file?.path)) + .filter((file) => typeof file === "string") + .filter((file) => forbiddenRoots.has(file.split("/")[0] ?? "") && file !== "scripts/research-workflow.mjs"); + if (forbidden.length > 0) { + fail(`packed forbidden paths: ${forbidden.join(", ")}`); + } + return isAbsolute(entry.filename) ? entry.filename : join(directory, entry.filename); +} + +async function main() { + const temporary = await mkdtemp(join(tmpdir(), "quant-research-clean-room-")); + try { + const tarball = packPackage(temporary); + const consumer = join(temporary, "consumer"); + await mkdir(consumer); + run("npm", ["init", "-y", "--silent"], consumer); + run( + "npm", + ["install", "--offline", "--ignore-scripts", "--no-package-lock", "--no-save", tarball], + consumer, + ); + + const packageRoot = join(consumer, "node_modules", "@agentbiz", "quant-research"); + const binary = join(consumer, "node_modules", ".bin", "agentbiz-quant"); + const workflowBinary = join(consumer, "node_modules", ".bin", "agentbiz-quant-workflow"); + const help = run(binary, ["--help"], consumer); + if (!help.includes("foundation CLI")) fail("installed binary did not load the packaged CLI"); + const example = join(packageRoot, "examples", "tam-instrument.json"); + const codebook = run(binary, ["codebook", example], consumer); + if (!codebook.includes("# Instrument codebook")) fail("installed package example could not produce a codebook"); + const workflowOutput = join(temporary, "workflow-output"); + const workflow = run(workflowBinary, ["--workflow", "service-quality", "--out", workflowOutput], consumer); + if (!workflow.includes("statistical engine executed: false")) fail("installed workflow did not report its execution boundary"); + if (!existsSync(join(workflowOutput, "workflow-summary.md"))) fail("installed workflow did not write markdown summary"); + run( + process.execPath, + ["--input-type=module", "-e", "import('@agentbiz/quant-research').then(({ COMPILER_VERSION }) => { if (typeof COMPILER_VERSION !== 'string') process.exit(1); })"], + consumer, + ); + console.log("clean-room consumer smoke: pass (CLI, packaged workflow, shipped example, package import)"); + } finally { + await rm(temporary, { recursive: true, force: true }); + } +} + +main().catch((error) => { + console.error(error instanceof Error ? error.message : String(error)); + process.exitCode = 1; +}); diff --git a/scripts/check-pack.mjs b/scripts/check-pack.mjs index 4dd8bf6..e5903a5 100644 --- a/scripts/check-pack.mjs +++ b/scripts/check-pack.mjs @@ -25,7 +25,7 @@ if (paths.length === 0) { process.exit(1); } -const allowed = /^(package\.json|LICENSE|README\.md|dist\/|schemas\/|examples\/)/; +const allowed = /^(package\.json|LICENSE|README\.md|dist\/|schemas\/|examples\/|scripts\/research-workflow\.mjs$)/; const forbidden = paths.filter((path) => !allowed.test(path)); if (forbidden.length > 0) { process.stderr.write(`Packed unexpected files:\n${forbidden.join("\n")}\n`); @@ -41,6 +41,7 @@ const required = [ "dist/mcp.js", "dist/tools/index.js", "dist/tools/execute.js", + "scripts/research-workflow.mjs", "schemas/instrument.v1.schema.json", "schemas/topic-card.v1.schema.json", "schemas/gap-map.v1.schema.json", @@ -49,6 +50,7 @@ const required = [ "schemas/measurement.v1.schema.json", "schemas/receipt.v1.schema.json", "examples/tam-instrument.json", + "examples/service-quality-instrument.json", "examples/tam-bundle.json", "examples/reproducibility-manifest.json", ]; diff --git a/scripts/check-release-candidate.mjs b/scripts/check-release-candidate.mjs new file mode 100644 index 0000000..2608a8d --- /dev/null +++ b/scripts/check-release-candidate.mjs @@ -0,0 +1,186 @@ +#!/usr/bin/env node + +import { spawnSync } from "node:child_process"; +import { readFileSync, statSync } from "node:fs"; +import { dirname, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +const root = resolve(dirname(fileURLToPath(import.meta.url)), ".."); + +export const REQUIRED_SOURCE_FILES = Object.freeze([ + "package.json", + "package-lock.json", + "README.md", + "LICENSE", + "CHANGELOG.md", + "docs/first-use.md", + "docs/release-and-rollback.md", + "examples/reproducibility-manifest.json", + "examples/tam-workflow.json", + "examples/service-quality-workflow/workflow.json", + "scripts/research-workflow.mjs", +]); + +export const REQUIRED_PACK_FILES = Object.freeze([ + "examples/reproducibility-manifest.json", + "examples/tam-workflow.json", + "examples/service-quality-workflow/workflow.json", +]); + +const FORBIDDEN_PACK_ROOTS = new Set(["src", "test", "fixtures", "scripts"]); +const ALPHA_VERSION = /^(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)-alpha\.(0|[1-9]\d*)(?:\+[0-9A-Za-z-]+(?:\.[0-9A-Za-z-]+)*)?$/; + +function normalizePath(path) { + return String(path).replaceAll("\\", "/").replace(/^\.\//, ""); +} + +function asPathSet(paths) { + return new Set(Array.from(paths ?? [], (path) => normalizePath(path))); +} + +export function isValidAlphaVersion(version) { + return typeof version === "string" && ALPHA_VERSION.test(version); +} + +export function evaluateReleaseCandidate({ packageJson, sourcePaths, packPaths }) { + const failures = []; + const sourceSet = asPathSet(sourcePaths); + const packSet = asPathSet(packPaths); + + if (packageJson?.private !== false) { + failures.push("package.json must set private to false"); + } + if (!isValidAlphaVersion(packageJson?.version)) { + failures.push(`package.json version must be a valid alpha semver (received ${String(packageJson?.version)})`); + } + if (packageJson?.dependencies !== undefined && Object.keys(packageJson.dependencies ?? {}).length > 0) { + failures.push("package.json must not add runtime dependencies to the Node-built-in contract"); + } + + for (const file of REQUIRED_SOURCE_FILES) { + if (!sourceSet.has(file)) { + failures.push(`missing release-candidate file: ${file}`); + } + } + + if (packSet.size === 0) { + failures.push("npm pack --dry-run returned no files"); + } + + const forbidden = [...packSet] + .filter((path) => FORBIDDEN_PACK_ROOTS.has(path.split("/")[0] ?? "") && path !== "scripts/research-workflow.mjs") + .sort(); + for (const path of forbidden) { + failures.push(`packed archive exposes forbidden path: ${path}`); + } + + for (const file of REQUIRED_PACK_FILES) { + if (!packSet.has(file)) { + failures.push(`packed archive is missing required example: ${file}`); + } + } + + return { + ok: failures.length === 0, + failures, + packageName: typeof packageJson?.name === "string" ? packageJson.name : null, + version: typeof packageJson?.version === "string" ? packageJson.version : null, + packFileCount: packSet.size, + }; +} + +export function collectExistingSourceFiles(packageRoot) { + return REQUIRED_SOURCE_FILES.filter((file) => { + try { + return statSync(join(packageRoot, file)).isFile(); + } catch { + return false; + } + }); +} + +export function parsePackPaths(stdout) { + const text = stdout.trim(); + let payload; + try { + payload = JSON.parse(text); + } catch { + const jsonStart = text.indexOf("["); + if (jsonStart < 0) { + throw new Error("npm pack --json did not return a JSON array"); + } + payload = JSON.parse(text.slice(jsonStart)); + } + + const entry = Array.isArray(payload) ? payload[0] : payload; + if (entry === null || typeof entry !== "object" || !Array.isArray(entry.files)) { + throw new Error("npm pack --json returned no file list"); + } + + return entry.files + .map((file) => (typeof file === "string" ? file : file?.path)) + .filter((path) => typeof path === "string") + .map((path) => normalizePath(path)); +} + +export function runPackDryRun(packageRoot) { + const result = spawnSync("npm", ["pack", "--dry-run", "--json", "--ignore-scripts"], { + cwd: packageRoot, + encoding: "utf8", + env: { ...process.env, npm_config_offline: "true", npm_config_ignore_scripts: "true" }, + maxBuffer: 4 * 1024 * 1024, + }); + + if (result.error) { + throw new Error(`npm pack could not start: ${result.error.message}`); + } + if (result.status !== 0) { + const detail = (result.stderr || result.stdout || "").trim().slice(0, 2000); + throw new Error(`npm pack failed with status ${result.status ?? 1}${detail ? `: ${detail}` : ""}`); + } + return parsePackPaths(result.stdout); +} + +export function runReleaseCandidate(packageRoot = root) { + const failures = []; + let packageJson = {}; + try { + packageJson = JSON.parse(readFileSync(join(packageRoot, "package.json"), "utf8")); + } catch (error) { + failures.push(`package.json could not be read as JSON: ${error instanceof Error ? error.message : String(error)}`); + } + + let packPaths = []; + try { + packPaths = runPackDryRun(packageRoot); + } catch (error) { + failures.push(error instanceof Error ? error.message : String(error)); + } + + const evaluation = evaluateReleaseCandidate({ + packageJson, + sourcePaths: collectExistingSourceFiles(packageRoot), + packPaths, + }); + return { + ...evaluation, + failures: [...failures, ...evaluation.failures], + ok: failures.length === 0 && evaluation.ok, + }; +} + +function main() { + const result = runReleaseCandidate(); + if (!result.ok) { + process.stderr.write("release-candidate gate: FAIL\n"); + process.stderr.write(`${result.failures.join("\n")}\n`); + process.exitCode = 1; + return; + } + + process.stdout.write(`release-candidate gate: PASS (${result.packageName}@${result.version}, ${result.packFileCount} packed files)\n`); +} + +if (process.argv[1] !== undefined && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + main(); +} diff --git a/scripts/first-use.mjs b/scripts/first-use.mjs index 7baa8ec..13b05aa 100644 --- a/scripts/first-use.mjs +++ b/scripts/first-use.mjs @@ -1,4 +1,14 @@ import { spawnSync } from "node:child_process"; +import { mkdtemp, readFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; + +const root = fileURLToPath(new URL("..", import.meta.url)); +const workflowScript = join(root, "scripts", "research-workflow.mjs"); +const workflowDescriptor = join(root, "examples", "tam-workflow.json"); +const workflowStepIds = ["instrument", "topic-card", "gap-map", "dataset-audit", "recode", "measurement"]; +const workflowEmitterIds = ["codebook", "dictionary-csv", "dataset-csv", "spss-data", "spss", "lavaan", "amos", "smartpls"]; const commands = [ ["compile instrument", ["dist/cli.js", "compile", "examples/tam-instrument.json"]], @@ -9,17 +19,98 @@ const commands = [ ["emit SPSS", ["dist/cli.js", "emit-spss", "examples/tam-bundle.json"]], ]; -for (const [label, args] of commands) { - console.log(`\n== ${label} ==`); - const result = spawnSync(process.execPath, args, { stdio: "inherit" }); +function runCommand(label, args, options = { stdio: "inherit" }) { + const result = spawnSync(process.execPath, args, { + cwd: root, + encoding: "utf8", + maxBuffer: 2 * 1024 * 1024, + ...options, + stdio: "pipe", + }); if (result.error) { - console.error(`${label} failed to start: ${result.error.message}`); - process.exit(1); + throw new Error(`${label} failed to start: ${result.error.message}`); } if (result.status !== 0) { - console.error(`${label} exited with status ${result.status ?? 1}`); - process.exit(result.status ?? 1); + const detail = (result.stderr || result.stdout || "").trim().slice(0, 4000); + throw new Error(`${label} exited with status ${result.status ?? 1}${detail ? `: ${detail}` : ""}`); } + return result.stdout; } -console.log("\nfirst-use path passed: deterministic research artifacts were emitted from the demonstration fixtures"); +async function readAndCheckWorkflowSummary(outputDirectory) { + let summary; + try { + summary = JSON.parse(await readFile(join(outputDirectory, "workflow-summary.json"), "utf8")); + } catch (error) { + throw new Error(`workflow summary could not be read: ${error instanceof Error ? error.message : String(error)}`); + } + + if (summary === null || typeof summary !== "object") { + throw new Error("workflow summary is not a JSON object."); + } + const steps = summary.steps; + if (!Array.isArray(steps) || steps.map((step) => step?.id).join("|") !== workflowStepIds.join("|")) { + throw new Error(`workflow summary has an unexpected step sequence; expected ${workflowStepIds.join(" -> ")}.`); + } + + const invariants = summary.invariants; + if ( + invariants === null || + typeof invariants !== "object" || + invariants.rowsPreserved !== true || + invariants.rawRows !== invariants.auditedRows || + invariants.rawRows !== invariants.recodedRows || + invariants.rawColumnsPreserved !== true || + invariants.statisticalEngineExecuted !== false + ) { + throw new Error("workflow summary failed row-preservation or local-only invariants."); + } + + const emitters = summary.emitters; + if (emitters === null || typeof emitters !== "object" || Object.keys(emitters).join("|") !== workflowEmitterIds.join("|")) { + throw new Error(`workflow summary has an unexpected emitter set; expected ${workflowEmitterIds.join(", ")}.`); + } + return summary; +} + +async function runCompleteWorkflow() { + const outputDirectory = await mkdtemp(join(tmpdir(), "agentbiz-quant-first-use-")); + try { + console.log("\n== complete research workflow =="); + runCommand("complete research workflow", [workflowScript, workflowDescriptor, "--out", outputDirectory]); + const summary = await readAndCheckWorkflowSummary(outputDirectory); + console.log(`workflow summary: ${JSON.stringify({ + workflowVersion: summary.workflowVersion, + steps: summary.steps.map(({ id, status }) => ({ id, status })), + rows: { + raw: summary.invariants.rawRows, + audited: summary.invariants.auditedRows, + recoded: summary.invariants.recodedRows, + }, + rowsPreserved: summary.invariants.rowsPreserved, + rawColumnsPreserved: summary.invariants.rawColumnsPreserved, + statisticalEngineExecuted: summary.invariants.statisticalEngineExecuted, + emitters: Object.keys(summary.emitters), + })}`); + console.log("research workflow complete: temporary artifacts validated and removed"); + } finally { + await rm(outputDirectory, { recursive: true, force: true }); + } +} + +async function main() { + for (const [label, args] of commands) { + console.log(`\n== ${label} ==`); + runCommand(label, args); + } + + await runCompleteWorkflow(); + console.log("\nfirst-use path passed: deterministic research artifacts were emitted from the demonstration fixtures"); +} + +if (process.argv[1] !== undefined && import.meta.url === pathToFileURL(process.argv[1]).href) { + main().catch((error) => { + console.error(error instanceof Error ? error.message : String(error)); + process.exitCode = 1; + }); +} diff --git a/scripts/research-workflow.mjs b/scripts/research-workflow.mjs new file mode 100644 index 0000000..09e2ee9 --- /dev/null +++ b/scripts/research-workflow.mjs @@ -0,0 +1,405 @@ +#!/usr/bin/env node + +import { spawnSync } from "node:child_process"; +import { realpathSync } from "node:fs"; +import { mkdir, readFile, writeFile } from "node:fs/promises"; +import { dirname, isAbsolute, join, relative, resolve } from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; + +const root = fileURLToPath(new URL("..", import.meta.url)); +const cli = join(root, "dist", "cli.js"); +const REQUIRED_INPUTS = ["instrument", "topicCard", "observed", "dataset", "recode", "measurement"]; +const EMITTERS = new Set(["codebook", "dictionary-csv", "dataset-csv", "spss-data", "spss", "lavaan", "amos", "smartpls"]); +const DEFAULT_WORKFLOW = "tam"; +const NAMED_WORKFLOWS = Object.freeze({ + tam: join(root, "examples", "tam-workflow.json"), + "service-quality": join(root, "examples", "service-quality-workflow", "workflow.json"), +}); +const NAMED_WORKFLOW_NAMES = Object.keys(NAMED_WORKFLOWS); + +function usage() { + return [ + "Usage: node scripts/research-workflow.mjs [workflow.json] [--workflow ] [--out ]", + "", + "Runs a deterministic local research workflow from a workflow descriptor.", + "The descriptor paths are relative to the descriptor; all inputs must be typed", + "agentbiz JSON documents. The workflow emits specifications only and never runs", + "SPSS, lavaan, AMOS, SmartPLS, or any statistical engine.", + "", + "Named workflows:", + " tam examples/tam-workflow.json (default)", + " service-quality examples/service-quality-workflow/workflow.json", + "", + "Example: node scripts/research-workflow.mjs --workflow service-quality --out /tmp/service-quality-workflow", + "Default output: a new directory under the operating system temp directory.", + ].join("\n"); +} + +function fail(message) { + throw new Error(`research workflow: ${message}`); +} + +function assertRecord(value, label) { + if (value === null || typeof value !== "object" || Array.isArray(value)) { + fail(`${label} must be a JSON object.`); + } + return value; +} + +function assertString(value, label) { + if (typeof value !== "string" || value.trim() === "") { + fail(`${label} must be a non-empty string.`); + } + return value; +} + +function validateDescriptor(value) { + const descriptor = assertRecord(value, "workflow descriptor"); + if (descriptor.workflowVersion !== "agentbiz.research-workflow.v1") { + fail("workflowVersion must equal agentbiz.research-workflow.v1."); + } + assertString(descriptor.name, "name"); + if (descriptor.provenance !== "demonstration") { + fail("provenance must be demonstration; this runner does not invent citations."); + } + + const inputs = assertRecord(descriptor.inputs, "inputs"); + for (const key of REQUIRED_INPUTS) { + assertString(inputs[key], `inputs.${key}`); + } + + if (!Array.isArray(descriptor.emitters) || descriptor.emitters.length === 0) { + fail("emitters must be a non-empty array."); + } + const seen = new Set(); + for (const emitter of descriptor.emitters) { + assertString(emitter, "emitters[]"); + if (!EMITTERS.has(emitter)) { + fail(`unsupported emitter ${emitter}.`); + } + if (seen.has(emitter)) { + fail(`duplicate emitter ${emitter}.`); + } + seen.add(emitter); + } + return { name: descriptor.name, provenance: descriptor.provenance, inputs, emitters: descriptor.emitters }; +} + +async function readJson(path) { + return JSON.parse(await readFile(path, "utf8")); +} + +function runCli(args) { + const result = spawnSync(process.execPath, [cli, ...args], { + cwd: root, + encoding: "utf8", + }); + if (result.error) { + fail(`could not start ${args[0]}: ${result.error.message}`); + } + if (result.status !== 0) { + const detail = (result.stderr || result.stdout || `exit ${String(result.status)}`).trim(); + fail(`${args.join(" ")} failed: ${detail}`); + } + return result.stdout; +} + +function runJson(args) { + const output = runCli(args); + try { + return JSON.parse(output); + } catch (error) { + fail(`${args[0]} returned invalid JSON: ${error instanceof Error ? error.message : String(error)}`); + } +} + +async function writeJson(path, value) { + await writeFile(path, `${JSON.stringify(value, null, 2)}\n`, "utf8"); +} + +function markdownInline(value) { + return String(value) + .replaceAll("\\", "\\\\") + .replaceAll("`", "\\`") + .replaceAll("|", "\\|") + .replaceAll("\r", " ") + .replaceAll("\n", " "); +} + +function renderWorkflowSummary(summary) { + const artifactFilenames = Object.values(summary.outputs); + const emitterFilenames = Object.values(summary.emitters); + return [ + "# Research workflow summary", + "", + `- Workflow name: \`${markdownInline(summary.name)}\``, + `- Provenance: \`${markdownInline(summary.provenance)}\``, + `- Workflow version: \`${markdownInline(summary.workflowVersion)}\``, + "", + "## Six step statuses", + "", + "| Step | Status |", + "| --- | --- |", + ...summary.steps.map(({ id, status }) => `| \`${markdownInline(id)}\` | \`${markdownInline(status)}\` |`), + "", + "## Invariants", + "", + `- Rows: raw \`${summary.invariants.rawRows}\`, audited \`${summary.invariants.auditedRows}\`, recoded \`${summary.invariants.recodedRows}\`; rows preserved: \`${summary.invariants.rowsPreserved}\`.`, + `- Raw columns: \`${summary.invariants.rawColumns}\`; recoded columns: \`${summary.invariants.recodedColumns}\`; raw columns preserved: \`${summary.invariants.rawColumnsPreserved}\`.`, + "- No statistical engine executed.", + "", + "## Emitted filenames", + "", + "### Workflow artifacts", + "", + ...artifactFilenames.map((filename) => `- \`${markdownInline(filename)}\``), + "", + "### Emitter outputs", + "", + ...emitterFilenames.map((filename) => `- \`${markdownInline(filename)}\``), + "", + "### Summary files", + "", + "- `workflow-summary.json`", + "- `workflow-summary.md`", + "", + ].join("\n"); +} + +function pathFromDescriptor(descriptorPath, value) { + return isAbsolute(value) ? value : resolve(dirname(descriptorPath), value); +} + +function relativeInputPath(descriptorPath, inputPath) { + const value = relative(dirname(descriptorPath), inputPath); + return value.startsWith(".") ? value : `./${value}`; +} + +function parseArguments(args) { + let workflowPath = NAMED_WORKFLOWS[DEFAULT_WORKFLOW]; + let workflowMode = DEFAULT_WORKFLOW; + let outputDirectory; + let hasNamedWorkflow = false; + let hasDescriptor = false; + + for (let index = 0; index < args.length; index += 1) { + const arg = args[index]; + if (arg === "--out") { + const value = args[index + 1]; + if (!value) { + fail("--out requires a directory."); + } + outputDirectory = resolve(value); + index += 1; + } else if (arg === "--workflow") { + const value = args[index + 1]; + if (!value || value.startsWith("--")) { + fail("--workflow requires a name."); + } + if (hasNamedWorkflow || hasDescriptor) { + fail("provide only one named workflow or workflow descriptor."); + } + if (!Object.hasOwn(NAMED_WORKFLOWS, value)) { + fail(`unknown workflow ${value}; choose ${NAMED_WORKFLOW_NAMES.join(", ")}.`); + } + workflowPath = NAMED_WORKFLOWS[value]; + workflowMode = value; + hasNamedWorkflow = true; + index += 1; + } else if (arg.startsWith("--")) { + fail(`unknown option ${arg}.`); + } else { + if (hasNamedWorkflow || hasDescriptor) { + fail("provide only one named workflow or workflow descriptor."); + } + workflowPath = arg; + workflowMode = "descriptor"; + hasDescriptor = true; + } + } + + return { workflowPath, workflowMode, outputDirectory }; +} + +function countStatuses(rows) { + return rows.reduce((counts, row) => { + counts[row.status] = (counts[row.status] ?? 0) + 1; + return counts; + }, {}); +} + +function requireArtifact(payload, label) { + if (payload === null || typeof payload !== "object" || payload.artifact === undefined || payload.receipt === undefined) { + fail(`${label} did not return an artifact and receipt.`); + } + return payload; +} + +function requireRows(payload, label) { + if (!payload.artifact || !Array.isArray(payload.artifact.rows)) { + fail(`${label} artifact does not contain rows.`); + } + return payload.artifact.rows; +} + +async function runWorkflow(workflowPath, outputDirectory) { + const descriptorPath = resolve(workflowPath); + const descriptor = validateDescriptor(await readJson(descriptorPath)); + await mkdir(outputDirectory, { recursive: true }); + + const input = Object.fromEntries( + REQUIRED_INPUTS.map((key) => [key, pathFromDescriptor(descriptorPath, descriptor.inputs[key])]), + ); + const instrument = requireArtifact(runJson(["compile", input.instrument]), "instrument compile"); + const topicCard = requireArtifact( + runJson(["compile", input.topicCard, "--instrument", input.instrument]), + "topic-card compile", + ); + const gapMap = requireArtifact(runJson(["gap", input.instrument, input.observed]), "gap map"); + const audit = requireArtifact(runJson(["audit", input.instrument, input.dataset]), "dataset audit"); + const recode = requireArtifact(runJson(["recode", input.instrument, input.dataset, input.recode]), "recode"); + const measurement = requireArtifact( + runJson([ + "compile", + input.measurement, + "--instrument", + input.instrument, + "--topic-card", + input.topicCard, + ]), + "measurement compile", + ); + + const rawDataset = await readJson(input.dataset); + const auditDataset = audit.artifact.dataset; + if (!auditDataset || !Array.isArray(auditDataset.rows)) { + fail("dataset audit artifact does not contain dataset rows."); + } + const auditRows = auditDataset.rows; + const recodedRows = requireRows(recode, "recode"); + if (auditRows.length !== rawDataset.rows.length || recodedRows.length !== rawDataset.rows.length) { + fail("dataset audit or recode changed the number of rows."); + } + const rawColumnNames = rawDataset.columns.map((column) => column.name); + const recodedColumnNames = recode.artifact.columns.map((column) => column.name); + if (!rawColumnNames.every((name) => recodedColumnNames.includes(name))) { + fail("recode output is missing a raw dataset column."); + } + + const bundle = { + instrument: instrument.artifact.instrument, + topicCard: topicCard.artifact, + measurement: measurement.artifact, + }; + + const jsonArtifacts = [ + ["instrument.json", instrument], + ["topic-card.json", topicCard], + ["gap-map.json", gapMap], + ["dataset-audit.json", audit], + ["recode.json", recode], + ["measurement.json", measurement], + ["bundle.json", bundle], + ]; + for (const [name, value] of jsonArtifacts) { + await writeJson(join(outputDirectory, name), value); + } + + const emitterOutputs = {}; + for (const emitter of descriptor.emitters) { + const [args, filename, format] = { + codebook: [["codebook", input.instrument], "codebook.md", "text"], + "dictionary-csv": [["dictionary-csv", input.instrument], "dictionary.csv", "text"], + "dataset-csv": [["csv", input.dataset], "dataset.csv", "text"], + "spss-data": [["emit-spss-data", input.dataset], "spss-data.sps", "text"], + spss: [["emit-spss", join(outputDirectory, "bundle.json")], "spss.sps", "text"], + lavaan: [["emit-lavaan", join(outputDirectory, "bundle.json")], "lavaan.R", "text"], + amos: [["emit-amos", join(outputDirectory, "bundle.json")], "amos.json", "json"], + smartpls: [["emit-smartpls", join(outputDirectory, "bundle.json")], "smartpls.json", "json"], + }[emitter]; + const output = runCli(args); + if (format === "json") { + await writeJson(join(outputDirectory, filename), JSON.parse(output)); + } else { + await writeFile(join(outputDirectory, filename), output, "utf8"); + } + emitterOutputs[emitter] = filename; + } + + const summary = { + workflowVersion: "agentbiz.research-workflow.v1", + name: descriptor.name, + provenance: descriptor.provenance, + inputs: Object.fromEntries( + REQUIRED_INPUTS.map((key) => [key, relativeInputPath(descriptorPath, input[key])]), + ), + steps: [ + { id: "instrument", status: instrument.receipt.status, artifactDigest: instrument.receipt.artifactDigest, counts: instrument.receipt.counts }, + { id: "topic-card", status: topicCard.receipt.status, artifactDigest: topicCard.receipt.artifactDigest, counts: topicCard.receipt.counts }, + { id: "gap-map", status: gapMap.receipt.status, artifactDigest: gapMap.receipt.artifactDigest, counts: gapMap.receipt.counts, statusCounts: countStatuses(gapMap.artifact.rows) }, + { id: "dataset-audit", status: audit.receipt.status, artifactDigest: audit.receipt.artifactDigest, counts: audit.receipt.counts }, + { id: "recode", status: recode.receipt.status, artifactDigest: recode.receipt.artifactDigest, counts: recode.receipt.counts }, + { id: "measurement", status: measurement.receipt.status, artifactDigest: measurement.receipt.artifactDigest, counts: measurement.receipt.counts }, + ], + invariants: { + rawRows: rawDataset.rows.length, + auditedRows: auditRows.length, + recodedRows: recodedRows.length, + rawColumns: rawColumnNames.length, + recodedColumns: recodedColumnNames.length, + rowsPreserved: true, + rawColumnsPreserved: true, + statisticalEngineExecuted: false, + }, + emitters: emitterOutputs, + outputs: Object.fromEntries(jsonArtifacts.map(([name]) => [name.replace(/\.json$/, ""), name])), + }; + await writeJson(join(outputDirectory, "workflow-summary.json"), summary); + await writeFile(join(outputDirectory, "workflow-summary.md"), renderWorkflowSummary(summary), "utf8"); + return summary; +} + +async function main() { + const args = process.argv.slice(2); + if (args.includes("--help") || args.includes("-h")) { + console.log(usage()); + return; + } + + const selection = parseArguments(args); + let { outputDirectory } = selection; + + if (!outputDirectory) { + const { mkdtemp } = await import("node:fs/promises"); + const { tmpdir } = await import("node:os"); + outputDirectory = await mkdtemp(join(tmpdir(), "agentbiz-quant-workflow-")); + } + const summary = await runWorkflow(selection.workflowPath, outputDirectory); + console.log(`research workflow complete: ${outputDirectory}`); + console.log(`workflow: ${selection.workflowMode} (${summary.provenance})`); + console.log( + `rows preserved: raw ${summary.invariants.rawRows}, audited ${summary.invariants.auditedRows}, recoded ${summary.invariants.recodedRows}; raw columns preserved: ${summary.invariants.rawColumnsPreserved}`, + ); + console.log(`emitters: ${Object.keys(summary.emitters).join(", ")}`); + console.log(`statistical engine executed: ${summary.invariants.statisticalEngineExecuted}`); +} + +function wasExecutedDirectly() { + if (process.argv[1] === undefined) { + return false; + } + try { + return realpathSync(process.argv[1]) === fileURLToPath(import.meta.url); + } catch { + return import.meta.url === pathToFileURL(process.argv[1]).href; + } +} + +if (wasExecutedDirectly()) { + main().catch((error) => { + console.error(error instanceof Error ? error.message : String(error)); + process.exitCode = 1; + }); +} + +export { runWorkflow, validateDescriptor }; diff --git a/scripts/run-agent-eval.mjs b/scripts/run-agent-eval.mjs new file mode 100644 index 0000000..07a6278 --- /dev/null +++ b/scripts/run-agent-eval.mjs @@ -0,0 +1,34 @@ +import { readFileSync } from "node:fs"; +import { spawnSync } from "node:child_process"; + +const manifest = JSON.parse(readFileSync("fixtures/agent-evals/manifest.json", "utf8")); +const taskId = process.argv[2]; +const dryRun = process.argv.includes("--dry-run"); + +if (!taskId || taskId === "--dry-run") { + console.error("Usage: npm run agent-eval -- [--dry-run]"); + process.exitCode = 2; +} else { + const task = manifest.tasks.find((candidate) => candidate.id === taskId); + if (!task) { + console.error(`Unknown agent evaluation task: ${taskId}`); + process.exitCode = 2; + } else { + console.log(`Agent evaluation ${task.id}: ${task.title}`); + console.log(`risk=${task.risk} owner=${task.owner} humanReview=${task.humanReview}`); + console.log(`Allowed paths: ${task.paths.join(", ")}`); + for (const command of task.commands) { + const match = /^npm run ([a-z0-9:_-]+)$/.exec(command); + if (!match) { console.error(`${task.id}: non-allowlisted command: ${command}`); process.exitCode = 2; break; } + console.log(`\n$ ${command}`); + if (dryRun) continue; + const result = spawnSync("npm", ["run", match[1]], { stdio: "inherit" }); + if (result.error || result.status !== 0) { + console.error(`${task.id}: acceptance command failed: ${result.error?.message ?? `exit ${String(result.status)}`}`); + process.exitCode = result.status ?? 1; + break; + } + } + if (!process.exitCode) console.log(`\n${task.id}: acceptance commands passed; parent diff review remains required`); + } +} diff --git a/test/clean-room.test.ts b/test/clean-room.test.ts new file mode 100644 index 0000000..f06bc8e --- /dev/null +++ b/test/clean-room.test.ts @@ -0,0 +1,15 @@ +import { spawnSync } from "node:child_process"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { describe, expect, it } from "vitest"; + +const root = fileURLToPath(new URL("../", import.meta.url)); +const script = join(root, "scripts", "check-clean-room.mjs"); + +describe("clean-room consumer", () => { + it("installs the packed package and uses the shipped example offline", () => { + const result = spawnSync(process.execPath, [script], { cwd: root, encoding: "utf8" }); + expect(result.status, result.stderr || result.stdout).toBe(0); + expect(result.stdout).toContain("clean-room consumer smoke: pass (CLI, packaged workflow, shipped example, package import)"); + }); +}); diff --git a/test/first-use-workflow.test.ts b/test/first-use-workflow.test.ts new file mode 100644 index 0000000..7a5a537 --- /dev/null +++ b/test/first-use-workflow.test.ts @@ -0,0 +1,59 @@ +import { spawnSync } from "node:child_process"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { describe, expect, it } from "vitest"; + +const root = fileURLToPath(new URL("..", import.meta.url)); +const script = join(root, "scripts", "first-use.mjs"); + +describe("first-use complete workflow", () => { + it("exits successfully, prints a complete summary, and preserves rows", () => { + const result = spawnSync(process.execPath, [script], { + cwd: root, + encoding: "utf8", + }); + + expect(result.status, result.stderr || result.stdout).toBe(0); + expect(result.stdout).toContain("first-use path passed: deterministic research artifacts were emitted from the demonstration fixtures"); + expect(result.stdout).toContain("== complete research workflow =="); + expect(result.stdout).not.toContain('"artifact":'); + + const summaryLine = result.stdout.split(/\r?\n/).find((line) => line.startsWith("workflow summary: ")); + expect(summaryLine).toBeDefined(); + const summary = JSON.parse(summaryLine?.slice("workflow summary: ".length) ?? "null") as { + workflowVersion: string; + steps: { id: string; status: string }[]; + rows: { raw: number; audited: number; recoded: number }; + rowsPreserved: boolean; + rawColumnsPreserved: boolean; + statisticalEngineExecuted: boolean; + emitters: string[]; + }; + + expect(summary.workflowVersion).toBe("agentbiz.research-workflow.v1"); + expect(summary.steps.map((step) => step.id)).toEqual([ + "instrument", + "topic-card", + "gap-map", + "dataset-audit", + "recode", + "measurement", + ]); + expect(summary.rows).toEqual({ raw: 12, audited: 12, recoded: 12 }); + expect(summary.rowsPreserved).toBe(true); + expect(summary.rawColumnsPreserved).toBe(true); + expect(summary.statisticalEngineExecuted).toBe(false); + expect(summary.emitters).toEqual([ + "codebook", + "dictionary-csv", + "dataset-csv", + "spss-data", + "spss", + "lavaan", + "amos", + "smartpls", + ]); + + expect(result.stdout).toContain("research workflow complete: temporary artifacts validated and removed"); + }); +}); diff --git a/test/instrument.test.ts b/test/instrument.test.ts index d239f83..4d41f9f 100644 --- a/test/instrument.test.ts +++ b/test/instrument.test.ts @@ -20,6 +20,19 @@ describe("instrument", () => { ]); }); + it("compiles the multi-factor service-quality demonstration instrument", () => { + const compiled = compileInstrument(loadExample("service-quality-instrument.json")); + expect(compiled.receipt.status).toBe("compiled"); + expect(compiled.artifact.instrument.constructs.map((item) => item.code)).toEqual(["REL", "RESP", "SAT"]); + expect(compiled.artifact.dictionary).toHaveLength(9); + expect(compiled.artifact.dictionary.filter((row) => row.reverse).map((row) => row.itemCode)).toEqual([ + "RESP3", + "SAT4", + ]); + expect(compiled.artifact.instrument.constructs[0]?.scale.max).toBe(7); + expect(compiled.artifact.instrument.constructs.map((item) => item.items.length)).toEqual([2, 3, 4]); + }); + it("is digest-stable", () => { const input = loadExample("tam-instrument.json"); const first = compileInstrument(input); diff --git a/test/release-candidate.test.ts b/test/release-candidate.test.ts new file mode 100644 index 0000000..d9fb759 --- /dev/null +++ b/test/release-candidate.test.ts @@ -0,0 +1,88 @@ +import { readFileSync } from "node:fs"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { spawnSync } from "node:child_process"; +import { describe, expect, it } from "vitest"; +import { + evaluateReleaseCandidate, + REQUIRED_PACK_FILES, + REQUIRED_SOURCE_FILES, + isValidAlphaVersion, +} from "../scripts/check-release-candidate.mjs"; + +const root = fileURLToPath(new URL("../", import.meta.url)); +const script = join(root, "scripts", "check-release-candidate.mjs"); + +function passingInput() { + return { + packageJson: { name: "@agentbiz/quant-research", private: false, version: "0.1.0-alpha.6" }, + sourcePaths: REQUIRED_SOURCE_FILES, + packPaths: ["package.json", "README.md", "LICENSE", ...REQUIRED_PACK_FILES, "scripts/research-workflow.mjs", "dist/index.js"], + }; +} + +describe("release-candidate gate", () => { + it("passes the real package through the offline npm pack dry-run", () => { + const result = spawnSync(process.execPath, [script], { cwd: root, encoding: "utf8" }); + expect(result.status, result.stderr).toBe(0); + expect(result.stdout).toContain("release-candidate gate: PASS"); + expect(result.stdout).toContain("0.1.0-alpha.6"); + }); + + it("requires a public alpha package and the release input files", () => { + const input = passingInput(); + const result = evaluateReleaseCandidate({ + ...input, + packageJson: { ...input.packageJson, private: true, version: "0.1.0" }, + sourcePaths: REQUIRED_SOURCE_FILES.filter((file) => file !== "docs/first-use.md"), + }); + + expect(result.ok).toBe(false); + expect(result.failures).toEqual(expect.arrayContaining([ + "package.json must set private to false", + "package.json version must be a valid alpha semver (received 0.1.0)", + "missing release-candidate file: docs/first-use.md", + ])); + }); + + it("rejects accidental runtime dependency pollution", () => { + const result = evaluateReleaseCandidate({ + ...passingInput(), + packageJson: { ...passingInput().packageJson, dependencies: { cac: "^6.7.14" } }, + }); + + expect(result.ok).toBe(false); + expect(result.failures).toContain( + "package.json must not add runtime dependencies to the Node-built-in contract", + ); + }); + + it("rejects forbidden packed roots and missing distributed examples", () => { + const input = passingInput(); + const result = evaluateReleaseCandidate({ + ...input, + packPaths: ["package.json", "src/index.ts", "test/example.test.ts", "fixtures/input.json"], + }); + + expect(result.ok).toBe(false); + expect(result.failures).toEqual(expect.arrayContaining([ + "packed archive exposes forbidden path: fixtures/input.json", + "packed archive exposes forbidden path: src/index.ts", + "packed archive exposes forbidden path: test/example.test.ts", + "packed archive is missing required example: examples/reproducibility-manifest.json", + "packed archive is missing required example: examples/tam-workflow.json", + "packed archive is missing required example: examples/service-quality-workflow/workflow.json", + ])); + }); + + it("keeps the package hook in verify and validates the alpha shape", () => { + const packageJson = JSON.parse(readFileSync(join(root, "package.json"), "utf8")) as { + scripts: Record; + }; + expect(packageJson.scripts["check:release-candidate"]).toBe("node scripts/check-release-candidate.mjs"); + expect(packageJson.scripts.verify).toContain("check:release-candidate"); + expect(isValidAlphaVersion("0.1.0-alpha.6")).toBe(true); + expect(isValidAlphaVersion("0.1.0")).toBe(false); + expect(isValidAlphaVersion("0.1.0-beta.1")).toBe(false); + }); +}); diff --git a/test/research-workflow.test.ts b/test/research-workflow.test.ts new file mode 100644 index 0000000..1522c40 --- /dev/null +++ b/test/research-workflow.test.ts @@ -0,0 +1,103 @@ +import { readFile, readdir } from "node:fs/promises"; +import { mkdtemp } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { spawnSync } from "node:child_process"; +import { describe, expect, it } from "vitest"; + +const root = fileURLToPath(new URL("..", import.meta.url)); +const script = join(root, "scripts", "research-workflow.mjs"); +const workflow = join(root, "examples", "tam-workflow.json"); + +async function run(outputDirectory: string): Promise> { + const result = spawnSync(process.execPath, [script, workflow, "--out", outputDirectory], { + cwd: root, + encoding: "utf8", + }); + expect(result.status, result.stderr || result.stdout).toBe(0); + return JSON.parse(await readFile(join(outputDirectory, "workflow-summary.json"), "utf8")) as Record; +} + +async function files(directory: string): Promise> { + const names = await readdir(directory); + const values = await Promise.all( + names.map(async (name) => [name, await readFile(join(directory, name), "utf8")] as const), + ); + return new Map(values); +} + +describe("research workflow example", () => { + it("replays the complete typed workflow without dropping rows or claiming statistics", async () => { + const first = await mkdtemp(join(tmpdir(), "agentbiz-quant-workflow-test-")); + const second = await mkdtemp(join(tmpdir(), "agentbiz-quant-workflow-test-")); + const firstSummary = await run(first); + const secondSummary = await run(second); + + expect(firstSummary).toEqual(secondSummary); + expect(firstSummary.provenance).toBe("demonstration"); + expect(firstSummary.invariants).toEqual({ + rawRows: 12, + auditedRows: 12, + recodedRows: 12, + rawColumns: 11, + recodedColumns: 14, + rowsPreserved: true, + rawColumnsPreserved: true, + statisticalEngineExecuted: false, + }); + expect(firstSummary.steps).toEqual( + expect.arrayContaining([ + expect.objectContaining({ id: "instrument", status: "compiled" }), + expect.objectContaining({ id: "topic-card", status: "compiled" }), + expect.objectContaining({ id: "gap-map", status: "compiled" }), + expect.objectContaining({ id: "dataset-audit", status: "audited" }), + expect.objectContaining({ id: "recode", status: "compiled" }), + expect.objectContaining({ id: "measurement", status: "compiled" }), + ]), + ); + + const output = await files(first); + expect(output.has("workflow-summary.json")).toBe(true); + const markdown = output.get("workflow-summary.md"); + expect(markdown).toBeDefined(); + expect(markdown).toContain("# Research workflow summary"); + expect(markdown).toContain("## Six step statuses"); + expect(markdown).toContain("| `instrument` | `compiled` |"); + expect(markdown).toContain("| `measurement` | `compiled` |"); + expect(markdown).toContain("Rows: raw `12`, audited `12`, recoded `12`; rows preserved: `true`."); + expect(markdown).toContain("Raw columns: `11`; recoded columns: `14`; raw columns preserved: `true`."); + expect(markdown).toContain("- No statistical engine executed."); + expect(markdown).toContain("- `workflow-summary.json`"); + expect(markdown).toContain("- `workflow-summary.md`"); + expect(markdown).toContain("- `instrument.json`"); + expect(markdown).toContain("- `smartpls.json`"); + expect(output.get("codebook.md")).toContain("Demonstration items are not published"); + expect(output.get("lavaan.R")).toContain("not a fitted model"); + expect(output.get("spss.sps")).toContain("does not contain estimated results"); + expect(output.get("amos.json")).toContain("No estimator was executed"); + expect(output.get("smartpls.json")).toContain("No estimator was executed"); + + const secondOutput = await files(second); + expect(output.get("workflow-summary.md")).toBe(secondOutput.get("workflow-summary.md")); + expect([...output.entries()]).toEqual([...secondOutput.entries()]); + }); + + it("rejects a descriptor that tries to relabel a demonstration workflow", async () => { + const { validateDescriptor } = await import("../scripts/research-workflow.mjs"); + expect(() => validateDescriptor({ + workflowVersion: "agentbiz.research-workflow.v1", + name: "unverified scale", + provenance: "cited", + inputs: { + instrument: "tam-instrument.json", + topicCard: "tam-topic-card.json", + observed: "tam-observed.json", + dataset: "tam-dataset.json", + recode: "tam-recode.json", + measurement: "tam-measurement.json", + }, + emitters: ["lavaan"], + })).toThrow(/provenance must be demonstration/); + }); +}); diff --git a/test/service-quality-workflow.test.ts b/test/service-quality-workflow.test.ts new file mode 100644 index 0000000..44837a9 --- /dev/null +++ b/test/service-quality-workflow.test.ts @@ -0,0 +1,148 @@ +import { readFile, readdir, rm, mkdtemp } from "node:fs/promises"; +import { spawnSync } from "node:child_process"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { describe, expect, it } from "vitest"; + +const root = fileURLToPath(new URL("..", import.meta.url)); +const script = join(root, "scripts", "research-workflow.mjs"); +const workflow = join(root, "examples", "service-quality-workflow", "workflow.json"); + +interface WorkflowSummary { + readonly provenance: string; + readonly steps: readonly { readonly id: string; readonly status: string }[]; + readonly invariants: { + readonly rawRows: number; + readonly auditedRows: number; + readonly recodedRows: number; + readonly rawColumns: number; + readonly recodedColumns: number; + readonly rowsPreserved: boolean; + readonly rawColumnsPreserved: boolean; + readonly statisticalEngineExecuted: boolean; + }; + readonly emitters: Readonly>; +} + +async function runWorkflow(outputDirectory: string): Promise { + const result = spawnSync(process.execPath, [script, workflow, "--out", outputDirectory], { + cwd: root, + encoding: "utf8", + }); + expect(result.status, result.stderr || result.stdout).toBe(0); + expect(result.stdout).toContain("statistical engine executed: false"); + return JSON.parse(await readFile(join(outputDirectory, "workflow-summary.json"), "utf8")) as WorkflowSummary; +} + +async function outputFiles(directory: string): Promise> { + const names = await readdir(directory); + const values = await Promise.all( + names.map(async (name) => [name, await readFile(join(directory, name), "utf8")] as const), + ); + return new Map(values); +} + +describe("service-quality research workflow example", () => { + it("lists the named mode in help and runs it with a concise truthful summary", async () => { + const help = spawnSync(process.execPath, [script, "--help"], { + cwd: root, + encoding: "utf8", + }); + expect(help.status, help.stderr || help.stdout).toBe(0); + expect(help.stdout).toContain("--workflow "); + expect(help.stdout).toContain("service-quality"); + expect(help.stdout).toContain("never runs"); + + const outputDirectory = await mkdtemp(join(tmpdir(), "agentbiz-quant-service-quality-named-test-")); + try { + const result = spawnSync(process.execPath, [script, "--workflow", "service-quality", "--out", outputDirectory], { + cwd: root, + encoding: "utf8", + }); + + expect(result.status, result.stderr || result.stdout).toBe(0); + expect(result.stdout).toContain("workflow: service-quality (demonstration)"); + expect(result.stdout).toContain("rows preserved: raw 6, audited 6, recoded 6; raw columns preserved: true"); + expect(result.stdout).toContain("statistical engine executed: false"); + + const summary = JSON.parse(await readFile(join(outputDirectory, "workflow-summary.json"), "utf8")) as WorkflowSummary; + expect(summary.provenance).toBe("demonstration"); + expect(summary.invariants).toMatchObject({ + rawRows: 6, + auditedRows: 6, + recodedRows: 6, + rowsPreserved: true, + rawColumnsPreserved: true, + statisticalEngineExecuted: false, + }); + } finally { + await rm(outputDirectory, { recursive: true, force: true }); + } + }); + + it("runs all steps deterministically while preserving data and raw text outputs", async () => { + const first = await mkdtemp(join(tmpdir(), "agentbiz-quant-service-quality-test-")); + const second = await mkdtemp(join(tmpdir(), "agentbiz-quant-service-quality-test-")); + try { + const firstSummary = await runWorkflow(first); + const secondSummary = await runWorkflow(second); + + expect(firstSummary).toEqual(secondSummary); + expect(firstSummary.provenance).toBe("demonstration"); + expect(firstSummary.steps.map(({ id, status }) => ({ id, status }))).toEqual([ + { id: "instrument", status: "compiled" }, + { id: "topic-card", status: "compiled" }, + { id: "gap-map", status: "compiled" }, + { id: "dataset-audit", status: "audited" }, + { id: "recode", status: "compiled" }, + { id: "measurement", status: "compiled" }, + ]); + expect(firstSummary.invariants).toEqual({ + rawRows: 6, + auditedRows: 6, + recodedRows: 6, + rawColumns: 11, + recodedColumns: 17, + rowsPreserved: true, + rawColumnsPreserved: true, + statisticalEngineExecuted: false, + }); + expect(Object.keys(firstSummary.emitters)).toEqual([ + "codebook", + "dictionary-csv", + "dataset-csv", + "spss-data", + "spss", + "lavaan", + "amos", + "smartpls", + ]); + + const firstFiles = await outputFiles(first); + const secondFiles = await outputFiles(second); + expect([...firstFiles.entries()]).toEqual([...secondFiles.entries()]); + const markdown = firstFiles.get("workflow-summary.md"); + expect(markdown).toBeDefined(); + expect(markdown).toContain("# Research workflow summary"); + expect(markdown).toContain("## Six step statuses"); + expect(markdown).toContain("| `dataset-audit` | `audited` |"); + expect(markdown).toContain("Rows: raw `6`, audited `6`, recoded `6`"); + expect(markdown).toContain("Raw columns: `11`; recoded columns: `17`"); + expect(markdown).toContain("No statistical engine executed."); + expect(markdown).toContain("`smartpls.json`"); + for (const name of ["codebook.md", "dictionary.csv", "dataset.csv", "spss-data.sps", "spss.sps", "lavaan.R"]) { + const output = firstFiles.get(name); + expect(output).toBeDefined(); + expect(output).toContain("\n"); + expect(output).not.toContain("\\n"); + expect(output).not.toContain('\\"'); + } + } finally { + await Promise.all([ + rm(first, { recursive: true, force: true }), + rm(second, { recursive: true, force: true }), + ]); + } + }); +});