diff --git a/AGENTS.md b/AGENTS.md index e6c30e6..9745145 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -22,7 +22,7 @@ This repository follows a **plugin-only** policy: implement fixes in `dsh-github 2. Read this file, then `node scripts/agent.mjs describe --json`. `agent-contract.json` maps each task to its owning files, focused tests, risks and approval boundaries; package scripts and release URLs are derived from current metadata. 3. Run `node scripts/agent.mjs doctor --json` before install. It is a dependency-free, read-only repository preflight: exit 0 means preflight passed, 1 means prerequisites missing, 2 means invalid arguments/metadata. Build presence is not build freshness, and no live DSH or credential readiness is claimed. 4. Choose a task with `node scripts/agent.mjs plan models --json` (or authorization/search/client/compatibility/tooling/release). Output is an unexecuted argv plan, never implicit permission to run destructive operations. -5. Use Node 24 LTS for development and the exact pnpm version in `package.json`; the runtime dependency floor is Node 22.19.0. Run frozen install in this checkout. Dependencies/build output are not carried into worktrees. +5. Use Node 24 LTS for development and the exact pnpm version in `package.json`; the runtime dependency floor is Node 22.19.0. Select the execution environment and task explicitly with `node scripts/agent.mjs guard`. Local auxiliary checks are limited to the guarded dependency-free script tests; dependency installation and full verification belong to GitHub Actions. Do not retry local dependency restoration, change registries, or treat local reports/idle sessions as CI evidence. The guard applies only to operations routed through it; it cannot prevent direct shell bypass. 6. Create/reuse a tracking issue and feature branch, implement a regression first, run the full gate, inspect the diff, and open a PR. For requested delivery, merge after required review/CI and branch protection without a separate approval prompt, unless the user limits scope. Important updates continue through the release-delivery rule below. Live-profile install and worktree checkout still require explicit user approval. Use native DSH tools for goals, background jobs and scoped subagents. Use local Git and the GitHub REST API for delivery; no external orchestration daemon, roster or `vcs_*` tool is required. Do not assume a model provider's identity is the assisting agent. @@ -317,6 +317,60 @@ Desktop plugin upgrades do not necessarily reconcile profile-local copies of Hos ## Mechanical verification +When a task explicitly requires post-upgrade Desktop/UI acceptance, use acceptance +contract/report schema 2 described in `docs/agent-acceptance.md`. Exact-head CI +and installed-on-disk evidence cannot replace live behavior or the public loaded +plugin version. Attempt safe authorized checks on an already-running Desktop; +unreliable local npm is not a reason to skip them. Use existing computer-use +tools, never shell activation or a repository script as an App controller. +Begin with current app discovery, not a historical "already open" claim. If not +running, start only with explicit launch authorization via controlled `start_app` +using the exact approved target; then verify discovery and fresh UI, not merely +the launch receipt. Do not start an already-running app again. A welcome screen +is not automatically an auth blocker: inspect an official safe skip first. DSH's +Add API Key -> Set up later path does not submit/store a key or authenticate; +use it only when actually exposed, within tool permission rules. Never infer +Copilot account invalidity from welcome UI or bypass real authentication/warnings. +Welcome dismissal may replace the window: rediscover app/window and read its fresh +tree before acting, rather than retrying a stale window on an empty immediate diff. +Honor user-input interruption: reread UI before retrying the same permitted action. +Before handoff, freshly discover the current app and inspect the target panel or +workspace, reporting the observation time only; past success does not establish +current running state or guarantee the app remains running later. +An action receipt or empty immediate diff is not a navigation failure: obtain +a fresh post-action `get_window_state` observation first. Keep one attempt per +supported navigation method, at most two different methods, then report precise +`automation-blocked` and request the named panel once. User absence or an +unanswered request never grants permission or proves the panel opened. +App unavailable, restart/install, account/OAuth/paid actions outside explicit +permission stay blocked. Do not repeat exhausted methods on ordinary continuation. +Record only fixed action/outcome codes, numeric times and version/source facts; +never upload identities, credentials or conversation content. Unknown loaded +version permits qualified observations only, not a version-specific fix claim. +For formal upgrade acceptance, prefer the published exact-version GitHub Release +asset canonical URL, never a local source directory, build, link or worktree. +A downloaded official tgz is a verified GitHub cache, not a local build; use it +only as a documented fallback when native URL delivery is unavailable/rejected. +Record exact source URL/version, archive and Release SHA-256, numeric observation +time and fixed fallback reason separately. Matching hashes do not grant install +or restart permission or prove the resulting runtime loaded. The consistency +checker never downloads/installs or independently authenticates this provenance. +Missing/not-run/blocked/error/timeout mandatory evidence cannot pass acceptance; even a +consistent complete report is not trusted proof of execution or automatic closure. +Keep required-check completeness separate from issue disposition. Controlled +regressions can exercise timers, never-resolving requests, late results and +simulated disconnects; production Desktop fault injection is different. Do not +disconnect networking, alter proxies/accounts or trigger paid requests to fill +coverage without specific permission. After a specific defect is fixed, relevant +controlled regressions/exact CI pass, release/install/load premises are met and +normal live checks do not reproduce it, explicit user risk acceptance may close +with "fixed/not reproduced; untested faults listed; reopen on recurrence". +Do not claim the original root cause or production fault coverage was proven. +Lack of safe/non-destructive production injection alone is not an indefinite +closure blocker. Known failures/reproduction, critical safety gaps and mandatory +requirements remain blockers; residual-risk disposition cannot relabel them +passed or authorize automatic issue closure. + For the opt-in task acceptance pilot, define observable requirements before implementation and validate the clean candidate with `node scripts/agent.mjs accept --contract CONTRACT_FILE --report REPORT_FILE`. @@ -327,10 +381,20 @@ live diagnostics automatically. Failed, blocked and not-run requirements cannot pass; new commits require new evidence. Existing review/CI/release gates remain authoritative. This tooling does not activate agents or access live profiles. -Run from the repository root: +Choose a bounded local auxiliary check or authoritative remote evidence explicitly +with `node scripts/agent.mjs guard --task TASK_ID --environment local-auxiliary +--operation scripts-test` or `--environment github-actions-evidence +--operation ci-evidence --head FULL_COMMIT_SHA`. The dependency-free scripts +tests are the only guarded local verification here. Do not run dependency +installation/recovery locally; full install, build, types, integration and +package verification use the existing GitHub Actions workflow. See +[environment and CI evidence guardrails](./docs/environment-evidence-guardrails.md). +The guard covers only operations invoked through this entry point and is not a +shell sandbox. + +The full gate and package commands, run by the existing CI workflow, are: ```sh -pnpm install --frozen-lockfile pnpm verify pnpm pack --pack-destination artifacts ``` @@ -339,7 +403,7 @@ Then run `pnpm verify:tarball -- artifacts/dsh-github-copilot-. `pnpm verify` checks the Agent contract, source and local test types, baseline markers, a clean build, Vitest tests, Node tooling tests, and real built Host import/Client-loader/Remote smoke. `tests/fixtures` are intentionally excluded from local test typecheck because they import source from a separate pinned Core checkout. The checked-in code must pass; never suppress compiler errors or weaken a test to get a green report. -If the exact frozen install is blocked solely by an enterprise registry or network policy while the repository's GitHub Actions can obtain the unchanged pinned packages from an authorized official registry, record the attempted command and exact environmental failure as **local validation blocked**, not passed. A PR may proceed with that limitation disclosed; the complete required GitHub CI matrix, including its install, tests, build and package checks, is then the authoritative gate before merge. Never weaken or skip CI, substitute dependencies, bypass organizational restrictions, claim unrun local gates succeeded, or tag/release a revision whose required CI is not green. If CI has the same failure, resolve it through approved infrastructure rather than bypassing it. +If a historical/approved local installation attempt is blocked solely by an enterprise registry or network policy while GitHub Actions can obtain unchanged pinned packages from an authorized official registry, record it as **local validation blocked**, not passed. Do not repeat dependency recovery, switch sources, or expand permissions. A PR may proceed with that limitation disclosed; the complete required GitHub CI matrix, including install, tests, build and package checks, is authoritative before merge. Never weaken or skip CI, substitute dependencies, bypass organizational restrictions, claim unrun local gates succeeded, or tag/release a revision whose required CI is not green. If CI has the same failure, resolve it through approved infrastructure rather than bypassing it. The release CI matrix targets only exact official `0.2.1-rc.3` on Windows and Linux; earlier pins are not support gates. It runs the unchanged tagged-source runtime fixture, optional native Chat fixture, exact published-artifact adapter fixture and all 28 native reviewer regressions without skips. This is a required gate, not evidence that an unexecuted qualification passed. `verify:upstream` is static seam-marker evidence. `verify:controlled-core` exclusively installs a temporary config fixture, refuses an existing target, and removes only its own file; it is not full plugin activation. Published-artifact synthetic adapter fixtures do not prove live provider transport. The release job must wait for the complete reusable CI matrix on the tagged revision, then verify its own packed bytes before publishing. diff --git a/agent-contract.json b/agent-contract.json index c58477a..07d82a2 100644 --- a/agent-contract.json +++ b/agent-contract.json @@ -1,6 +1,6 @@ { "schemaVersion": 1, - "entrypoints": ["AGENTS.md", "CONTRIBUTING.md", "README.md", "README.zh.md", "SECURITY.md", "deployment-baseline.json"], + "entrypoints": ["AGENTS.md", "CONTRIBUTING.md", "README.md", "README.zh.md", "SECURITY.md", "deployment-baseline.json", "agent-execution-policy.json", "docs/environment-evidence-guardrails.md"], "acceptance": { "mode": "evidence-consistency-only", "policy": "docs/agent-acceptance.md", @@ -10,7 +10,19 @@ "cleanHeadRequired": true, "executionVerified": false, "independenceVerified": false, - "automaticUpload": false + "automaticUpload": false, + "liveUi": { + "schemaVersion": 2, + "ci": "guarded-exact-head", + "source": "report-consistency-only", + "browserManualRequireLiveContract": true, + "upgradeSource": "exact-github-release-url-or-verified-cache", + "appLifecycle": "fresh-discovery-consented-launch-safe-skip-rediscovery-final", + "disposition": "separate-explicit-residual-risk-no-required-waiver", + "automaticClosure": false, + "maxNavigationMethods": 2, + "maxObservationMs": 120000 + } }, "boundaries": { "credentialOwner": "DSH credentials and pi-ai OAuth; canonical compatibility record is preserved. Approved #297 plugin account records are directly authorized, never copied; immutable request binding and managed-only CAS switching. Only an exact single committed plugin-owned native refresh with unchanged account, entitlement and endpoint may retain already-dispatched HTTP streams; all reusable proof is invalidated. Unknown/external/Core-only notifications still revoke. Never request, log or export grants.", @@ -164,10 +176,10 @@ "risk": "Compatibility remains plugin-only: inspect Core APIs read-only and test unchanged pinned source/published artifacts; never modify, patch or publish Core. The current exact prerelease target and development dependencies are official dsh-v0.2.1-rc.3 at 1f81c047fc48fa190a683e469daa699857e9afe8; retain all thirteen earlier exact baselines. Rc.3 changes public WebRuntime provider configuration to required Volatile getters; plugin routing supplies read-only plugin-owned Volatile references. Reviewer qualification requires the exact rc.3 package version and imported apply callback identity. Tagged-source and isolated published-npm-artifact fixtures are required CI gates and remain unverified until they pass. Candidate Desktop artifacts/signatures remain unverified; historical signed rc.2 evidence is shared peer ownership only. No plugin activation, profile upgrade, live authorization, endpoint or model-call proof may be claimed. Follow docs/official-first-021-alpha5.md for complete/partial/unverified support, retain/migrate/defer decisions, gaps and retirement triggers; unverified parity is not absence. Use exact plugin pi-ai 1.1.0 and official framework/schema peers, preserve public stream identity, native transcript normalization and system/tool prefixes without trimming history. Native gateway fixtures require an explicit public owner Peer and interactive Client carrier; no synthetic principal is live authorization. Managed Auto-review sampling must not use prompt/error/model detection, mutate frozen requests, intercept canonical routes, add retries or weaken native authorization. Public bundlePatchPaths must preserve ordered patches and reject unsafe composition. Strict Remote create factories retain the legacy schema bridge, never src-json. Native role descriptors remain v3 with cache stateVersion 2 refolding, not conversion; unknown/v1/v2 histories fail closed unchanged. Keep dsh-authorization and schemastery as shared peers, not private runtime dependencies; React remains a Client ModuleLoader external outside the Node peer graph. Audit a materialized hash-verified Desktop package graph before Desktop acceptance; source metadata cannot substitute. Do not delete/replace profile-local shadowing peers as compatibility work. Retain serialized agent/created, projected Session history and MCP/PTC/Workflow/Sandbox/Shell/image-offload/Team ownership. Preserve modelErrors, Anthropic system-before-probe, native final token admission, compaction transactions, Usage, diagnostics/gzip consent and byte/semantic liveness. Experimental plugin records are deferred, never emitted or adopted implicitly. Source/synthetic evidence is not live endpoint, publication or loaded-runtime proof. Existing public Host, Client and Remote seams remain the boundary." }, "tooling": { - "purpose": "Agent contract, preflight, acceptance evidence consistency, verification, packaging and attribution", - "read": ["agent-contract.json", "scripts/agent.mjs", "scripts/acceptance.mjs", "docs/agent-acceptance.md", "tests/scripts/acceptance.test.mjs", "scripts/verify-agent-contract.mjs", "scripts/verify-package.mjs", ".github/workflows/ci.yml"], - "tests": [], - "risk": "Machine reports must distinguish not checked from failed and never claim live runtime or release success." + "purpose": "Agent contract, deterministic environment choice, bounded guarded operations, authoritative CI evidence, acceptance consistency, verification, packaging and attribution", + "read": ["agent-contract.json", "agent-execution-policy.json", "scripts/agent.mjs", "scripts/environment-guard.mjs", "scripts/acceptance.mjs", "docs/agent-acceptance.md", "docs/environment-evidence-guardrails.md", "tests/scripts/agent.test.mjs", "tests/scripts/acceptance.test.mjs", "tests/scripts/environment-guardrails.test.mjs", "scripts/verify-agent-contract.mjs", "scripts/verify-package.mjs", ".github/workflows/ci.yml"], + "tests": ["tests/scripts/environment-guardrails.test.mjs"], + "risk": "The entry point cannot prevent direct shell bypass; only routed operations are enforced. Local dependency-network work is rejected before execution and never auto-retried or redirected. Budgets persist per task/operation in the Git worktree metadata and do not reset on success or continuation. CI evidence must come from authenticated gh Actions API results for the configured repository, exact clean HEAD, exact CI workflow, and every required verification job; skipped release is optional, not a substitute for required work. Local reports, self-declared JSON, idle sessions, and CI-green do not prove business/field acceptance or roadmap completion." }, "release": { "purpose": "Version alignment and default immutable GitHub plus npm dual distribution", diff --git a/agent-execution-policy.json b/agent-execution-policy.json new file mode 100644 index 0000000..40017df --- /dev/null +++ b/agent-execution-policy.json @@ -0,0 +1,54 @@ +{ + "schemaVersion": 1, + "repository": "cloga/dsh-github-copilot", + "ownerTask": "tooling", + "maxTaskScopes": 128, + "environments": { + "local-auxiliary": { + "operations": ["scripts-test"] + }, + "github-actions-evidence": { + "operations": ["ci-evidence"] + } + }, + "operations": { + "dependency-install": { + "kind": "network", + "maxFailures": 0 + }, + "scripts-test": { + "kind": "local-command", + "environment": "local-auxiliary", + "maxFailures": 2, + "argv": [ + "node", + "--test", + "tests/scripts/agent.test.mjs", + "tests/scripts/acceptance.test.mjs", + "tests/scripts/environment-guardrails.test.mjs" + ] + }, + "ci-evidence": { + "kind": "github-actions-read", + "environment": "github-actions-evidence", + "maxFailures": 2, + "maxApiCalls": 3, + "requestTimeoutMs": 15000, + "workflowPath": ".github/workflows/ci.yml", + "workflowName": "CI", + "requiredJobs": [ + "verify (ubuntu-latest)", + "verify (windows-latest)", + "Published adapter (ubuntu-latest)", + "Published adapter (windows-latest)", + "release-ready" + ], + "optionalJobs": [ + { + "name": "release", + "allowedConclusions": ["success", "skipped"] + } + ] + } + } +} diff --git a/docs/agent-acceptance.md b/docs/agent-acceptance.md index 55751bb..b074679 100644 --- a/docs/agent-acceptance.md +++ b/docs/agent-acceptance.md @@ -34,12 +34,13 @@ The plugin-only and approval boundaries in [AGENTS.md](../AGENTS.md) still apply A later source commit requires a new report and appropriate revalidation. `agent.mjs describe` and `plan ` expose this policy and the unexecuted -command with input placeholders. The checker reads files and fixed Git commands -only; it never executes report-supplied commands or follows URLs. +command with input placeholders. Schema 1 reads files and fixed Git commands only. Schema 2 additionally invokes +the existing bounded GitHub CI reader. Neither executes report-supplied commands, +follows report URLs, or operates a Desktop application. ## Contract format -Version 1 accepts exactly these fields. This example is illustrative, not +Version 1 accepts exactly these fields for non-live checks. This example is illustrative, not completed acceptance or evidence that the behavior has been exercised: ```json @@ -65,11 +66,279 @@ uniquely identified requirements and 1-16 non-goals. IDs are lowercase alphanumeric/hyphen identifiers beginning with a letter, at most 64 characters. Text is nonempty and at most 2,000 characters. Methods are `static`, `unit`, `integration`, `browser`, `review` or `manual`; a method is a declared evidence -category, never an instruction to execute code. +category, never an instruction to execute code. `browser` and `manual` requirements +now require schema 2; schema 1 returns `ACCEPTANCE_LIVE_CONTRACT_REQUIRED`. + +## Required live Desktop acceptance (schema 2) + +This focused #451 extension also contributes to agentic engineering improvement; +it does not depend on #442's evaluator platform or multi-agent loop. A task that +explicitly requests post-upgrade UI acceptance must use schema 2, not avoid the +requirement by labeling it `unit`. Local npm restrictions still prohibit local +dependency recovery; they do not excuse skipping safe checks on an already-running +Desktop. Real UI actions use the existing computer-use tools and their permissions, +not this repository checker. + +Use the same `agent.mjs accept` command and base contract/report fields, with +`schemaVersion: 2` in both documents. Add this contract field (illustrative): + +```json +{ + "liveUi": { + "expectedPluginVersion": "0.1.0-alpha.1", + "panel": "models-manage", + "checks": [ + { "id": "manage-reopen", "action": "reopen-manage", "outcome": "ready" }, + { "id": "refresh-settles", "action": "refresh-models", "outcome": "ready" } + ] + } +} +``` + +Every live check ID must exist in `requirements`; every `browser`/`manual` +requirement must occur in `liveUi.checks`. Define the required scenarios before +capture: success in a signed-in case cannot substitute for a missing no-account +or failure scenario. Report actions must match each required action; observing +a panel cannot satisfy a required refresh or reopen. There are 1-16 unique live checks. Expected outcomes are +`ready` or `actionable-failure` (the latter is for an explicitly required error +presentation scenario, never a generic failure waiver). Panels are only +`models-manage`, `plugin-details` or `about`. + +The report adds exactly `liveUi` with: + +| Field | Strict shape / allowed values | +|---|---| +| `appState` | `running`, `not-running`, `unknown` | +| `desktop` | `{version, source, observedAtMs}`; source `about` or `unknown` | +| `installed` | Same shape; source `disk-manifest` or `unknown` | +| `loaded` | Same shape; source `public-runtime-status`, `runtime-ui` or `unknown` | +| `upgradeSource` | `{kind, version, url, sha256, releaseSha256, fallbackReason, observedAtMs}` as below | +| `navigation` | 0-2 unique-method `{method, outcome, atMs, observedAtMs, observationSource}` rows | +| `handoff` | `{state, panel}`; state `none`, `requested`, `user-absent`; panel equals contract | +| `permission` | `{state, action}`; state `safe-read-only`, `approval-required`, `user-absent` | +| `steps` | 0-32 unique-check `{checkId, action, observation, startedAtMs, finishedAtMs, observedAtMs, observationSource}` rows | +| `lifecycle` | Current discovery, permitted launch, welcome, rediscovery and final observation chain below | + +### Current app lifecycle, not historical readiness + +Schema 2 also requires exactly this content-free lifecycle shape. This is an +illustrative shape, **not a completed report or proof that these tools ran**: + +```json +{ + "discovery": { "state": "running", "atMs": 1000, "source": "list-apps" }, + "launch": { "state": "not-needed", "atMs": null }, + "welcome": { "state": "not-seen", "atMs": null }, + "rediscovery": { "atMs": 1001, "source": "list-apps", "window": "same" }, + "final": { + "atMs": 2000, "source": "list-apps-and-get-window-state", + "appState": "running", "surface": "target-panel" + }, + "interaction": "clear" +} +``` + +Discover the app **now** using `list_apps`, never infer it is running from old +conversation evidence. Discovery `state` is `running`, `not-running` or `unknown`. +When actually not running, explicit authorization for the exact launch target +permits one controlled `start_app`, recorded as `authorized-start-app` with a +time after discovery. Otherwise use `not-authorized` and stop. `not-needed` and +`not-authorized` use null launch times. A launch when discovery already reports +running is invalid: do not create duplicate processes or use shell launch/focus +workarounds. The reported authorization is not an approval grant. + +Welcome `state` is `not-seen` (null time), `safe-skip-observed`, `safe-skip-used`, +`auth-required` or `unknown` (positive observed time after discovery/launch). +Do not automatically label welcome as authentication failure. Inspect official +safe skip controls when present; DSH can expose **Add API Key -> Set up later** +without entering/submitting/storing any key or signing in. This is observation +guidance, not permission to click through real authentication, unsafe warnings, +or legal agreements. An actual credential requirement with no permitted safe +skip remains `LIVE_UI_AUTH_REQUIRED`; a skip not yet used/unknown welcome state +is `LIVE_UI_WELCOME_UNRESOLVED`, with next action to inspect the official skip. + +After navigation or welcome dismissal, rediscover the app/window with `list_apps` +and read a fresh tree. A welcome window may disappear and be replaced by the +main window; do not retry the old window just because an action returned done +or its immediate diff was empty. `rediscovery.window` is only `same`, +`replacement` or `unobserved`, never a real window identifier/title/path. +Its positive time must be after discovery/launch/welcome; missing observation +uses null and cannot pass. Navigating and checking the target panel follows this +rediscovery. `start_app` success is not main-window readiness. + +Before delivering, freshly perform app discovery **and** a UI observation. +`final` records its positive UTC epoch-millisecond time, app state and surface +(`target-panel`, `workspace`, `welcome`, `unobserved`). Only running and the +target-panel/workspace can pass, after rediscovery, navigation and all required +checks. This final observation must be no older than 120 seconds at checker +completion, no later than its clock, and within 120 seconds of the last check. +The checker re-evaluates this after its bounded CI reads. This is a bounded +report-consistency freshness rule, not proof of clock accuracy or a promise the +application remains running afterwards. Recheck if waiting/CI made it stale. + +`interaction` is `clear` or `interrupted`. Interruption remains +`LIVE_UI_INPUT_INTERRUPTED`; reread the app UI before the same tool/action retry +under its existing consent and limits. Unapproved launch, missing rediscovery, +and stale/stopped/welcome-only final state return distinct blocked codes, not +product failure or success. Existing two navigation-method limits, one panel +handoff, permission rules, installed/loaded/CI separation and GitHub-first +source constraints remain unchanged. This checker still only validates supplied +rows; it cannot know whether a real host executed any of these steps. + +### Required completeness versus risk-accepted issue disposition + +Controlled automatic regressions can deterministically exercise timeouts, +never-resolving promises, late responses and simulated disconnects. That is +distinct from reproducing the same faults in a production Desktop. Do not alter +system networking/proxies, credentials/accounts or trigger paid calls merely to +complete fault coverage. When only destructive or unavailable live injection +remains, record why rather than indefinitely holding a fixed issue open. + +After a specific defect is fixed, relevant controlled regressions and exact-head +CI pass, published/installed/loaded-version prerequisites are satisfied and +required normal live checks no longer reproduce it, the user may accept residual +production uncertainty. Communicate "fixed / not reproduced after repair; +unmeasured faults remain; reopen on recurrence", not proven original root cause, +complete production coverage or arbitrary assumption of success. + +Schema 2 reports may add an optional `disposition` object, separate from checks: + +```json +{ + "decision": "accept-residual-risk", + "userAuthorized": true, + "defect": "specific-fix", + "controlledRegression": "passed", + "criticalBlocker": false, + "productionFaults": [ + { "scenario": "timeout", "status": "not-tested", "reason": "no-safe-control" } + ] +} +``` + +Only these fields are accepted. Decision is `keep-open` or `accept-residual-risk`; +defect is `specific-fix` or `unconfirmed`; regression is `passed`, `failed` or +`not-run`. Boolean authorization/blocker fields are self-reported, not permission. +The 1-4 unique production scenarios are `disconnect`, `never-resolves`, `timeout`, +`late-response`; status is `passed`, `failed`, `not-tested`. Untested reasons are +`no-safe-control` or `destructive-only`; observed pass/fail uses `observed`. +No raw errors, identities, network configuration or product content is accepted. + +Only all required checks/live/version/source/real CI gates plus a specific fix, +passed controlled regressions, reported explicit authorization, no critical +blocker and no known production failure can yield +`close-with-residual-risk-and-reopen-on-recurrence`. Coverage remains `incomplete` +when a production fault is untested. Missing mandatory requirements never become +passed; known failure or critical blocker also prevents final `ok/closeEligible`. +Other dispositions recommend `keep-open`. Outputs always retain +`requiredChecksWaived: false`, `rootCauseVerified: false`, +`authorizationVerified: false`, `automaticClosure: false`. This is a bounded +decision-consistency aid, not an authenticated risk signoff, automatic issue +operation or relaxation of required CI/review/approval rules. + +Versions are exact SemVer-like version strings, not paths, account names or raw +status output. Unknown version/source uses `version: null, source: "unknown", +observedAtMs: null`; known versions need a positive integer UTC epoch-millisecond +observation time. Desktop About establishes only the Desktop version. A disk +manifest establishes installation, **not loaded runtime**. A matching public +loaded-plugin version is mandatory for version-specific completion; otherwise +observations may be qualified, but final `ok`/`closeEligible` remain false. +Known different loaded versions return `LIVE_UI_VERSION_PREREQUISITE_UNMET` +with `request-exact-upgrade-permission`, not automation failure. A matching +download/checksum does not prove installation or loading. If the exact required +install/restart needs specific consent, an unavailable user leaves it blocked; +a broad request to continue acceptance never fabricates that consent. + +### Formal upgrade source selection + +Prefer the published **exact-version GitHub Release asset canonical URL**: +`https://github.com//releases/download/v/dsh-github-copilot-.tgz`. +The repository comes from `agent-execution-policy.json`; both version positions +must equal `expectedPluginVersion`. Do not use local source directories, links, +worktrees or locally built archives as formal upgrade acceptance sources. + +`upgradeSource.kind` is `github-release-url` (preferred), +`verified-github-cache` (fallback only), `local-build`, `source-directory`, +`link-worktree` or `unknown`. A tgz downloaded from the canonical Release URL and +checked against the published SHA-256 is a **GitHub download cache**, not a local +build. Keep those labels distinct even though both reside on disk. + +For either eligible source, record the exact version, canonical originating URL, +archive `sha256`, published `releaseSha256` and positive integer `observedAtMs`. +Both lowercase SHA-256 values must be present and equal. Preferred URL use +requires `fallbackReason: "none"`. Cache use requires the fixed reason +`native-url-unavailable` or `native-url-rejected`; convenience or a local build +is not a fallback reason. Reuse existing Release/SHA256SUMS evidence for these +facts. Never substitute a source path, mutable branch URL, npm alias, token, +signed redirect URL, account name or raw download error. Unknown values may be +null and local/unknown source kinds can be recorded, but cannot pass final +acceptance. Malformed/foreign URLs are invalid; disallowed/incomplete source +evidence yields `LIVE_UI_UPGRADE_SOURCE_REJECTED`. + +This is executable **source-selection report consistency**, not an installer +or new provenance attestation. The checker does not download assets, read local +installation paths, authenticate a reported hash, prove Release existence, or +operate native UI. Results retain `upgradeSourceKind` and +`provenanceVerified: false`; an invented matching hash cannot be detected by +this checker. Actual Release/readback, archive integrity, install permission, +installed-on-disk and loaded-runtime evidence remain distinct. Do not reinstall, +restart or bypass warnings just to populate the report. + +### Navigation and outcomes + +Navigation methods are `accessible-control`, `supported-shortcut` or +`user-opened-panel`; outcomes are `opened`, `unchanged`, `no-candidate`. Do not +retry a method just because its action returned done with no immediate tree diff. +First obtain a fresh `get_window_state` observation. Both navigation and step +rows require `observationSource: "get-window-state"` and an observation time +strictly after the action. Observations must settle within 120,000ms and be in +recorded order; no looping or repeated check/method IDs. An opened navigation +ends navigation attempts. A second method is one supported safe alternate, not +a shell/window-manager workaround. + +Step actions are `observe-panel`, `reopen-manage`, `refresh-models`; observations +are `ready`, `actionable-failure`, `error`, `timeout`, `unchanged`. For example, +record the bounded refresh interval and fresh settled ready observation; do not +interpret a transient Checking state or an action receipt as success/failure. +Actions remain subject to the real task and tool permissions. Reported permission +does not grant any. `permission.action` is `none`, `restart`, `install`, +`account-switch`, `oauth` or `paid-call`; any non-`none` or non-safe state blocks +this safe-check flow. App not running requires a blocked result unless the +separate explicit launch authorization and controlled lifecycle above apply. + +If both distinct navigation attempts fail after fresh observations, return +`LIVE_UI_AUTOMATION_BLOCKED`, not product failure. The deterministic `nextAction` +requests the specified panel once; after `handoff.state: requested/user-absent`, +it waits for the user instead of asking repeatedly. Never infer consent/panel +opening from autopilot, user absence, or an automatically unanswered `ask_user`. +After the user really opens the panel, inspect it afresh and record +`user-opened-panel`; retain prior failure context locally, without resetting the +existing task/operation failure ledger. The report is not a persistent UI +executor or an interception mechanism: hiding attempts in a fabricated report +cannot be detected. + +Schema 2 check `notes` are fixed codes only: `observed`, `not-run`, `blocked`, +`failed`; all `evidence` arrays must be empty. The structured live rows replace +arbitrary files/free text. Unknown fields/values are rejected; no identities, +credentials, raw errors, conversation content, screenshots or ad-hoc logs belong +in this report. Keep reports local; the tool never uploads them. + +Schema 2 always invokes the existing `ci-evidence` operation under task +`String(issue)`, exact clean current commit and its existing persistent budget. +It never trusts a report's CI boolean/JSON or resets failures. Non-green/pending +CI, missing live checks, errors/timeouts, blocked permission, unconfirmed loaded +version, or incomplete handoff makes `state: "blocked", ok: false, +closeEligible: false`. A complete consistent report plus real exact-head CI can +return `state: "reported-complete", ok: true, closeEligible: true`, but this is +**report eligibility for human review**, not proof of UI execution or permission +to close an issue. `executionVerified/environmentVerified/deliveryAuthorized` +remain false; `liveUi.source` is always `report-consistency-only`. `ciVerified` +alone is real Actions-result evidence. No trusted Desktop execution source is +invented here. ## Report format -The report has exactly `schemaVersion`, `commit`, `contractSha256`, `producer`, +The schema 1 report has exactly `schemaVersion`, `commit`, `contractSha256`, `producer`, `environment` and `checks`. Version is 1. `commit` is the full 40-character lowercase Git SHA-1 of the candidate; SHA-256 Git repositories are not supported by this version. `contractSha256` is the lowercase SHA-256 of the exact contract diff --git a/docs/environment-evidence-guardrails.md b/docs/environment-evidence-guardrails.md new file mode 100644 index 0000000..8cc816c --- /dev/null +++ b/docs/environment-evidence-guardrails.md @@ -0,0 +1,116 @@ +# Environment and CI evidence guardrails + +Tracking: [issue #451](https://github.com/cloga/dsh-github-copilot/issues/451). + +The guarded entry point makes the execution choice explicit. It is deliberately +separate from the opt-in acceptance-evidence consistency checker and the #442 +engineering roadmap; it is not an evaluator, agent loop, daemon, or product +runtime feature. + +## Select one bounded operation + +Use a stable task identifier (normally the issue number) and choose the +environment explicitly: + +```sh +node scripts/agent.mjs guard --task 451 --environment local-auxiliary --operation scripts-test +node scripts/agent.mjs guard --task 451 --environment github-actions-evidence --operation ci-evidence --head FULL_40_CHARACTER_COMMIT +``` + +`local-auxiliary` runs only the fixed, dependency-free Node script test set. +It does not install or restore packages, build, typecheck, or run integration +tests. `dependency-install` is refused before a child process can start; the +guard never changes registry sources, retries installs, widens permissions, or +changes machine/network/credential configuration. + +`github-actions-evidence` reads the real GitHub Actions REST results through +the authenticated `gh` CLI. It requires the explicit head to equal the current +clean checkout HEAD, the origin to identify this repository, the active +`.github/workflows/ci.yml` workflow, a completed successful run for that exact +head, and successful results for both Windows/Linux `verify` and +`Published adapter` matrix jobs plus `release-ready`. Distinct runs are ordered +by workflow run number, then run ID; attempt numbers order only attempts of +the same run. An older successful rerun cannot mask a newer failed or pending +run. Missing or invalid run ordering metadata is rejected. The normal `release` job +may be absent or skipped; it is not a required verification job. Failed, +cancelled, in-progress, stale, missing, mismatched, or skipped required results +are rejected. No report JSON, local `passed` label, session status, or inferred +workflow name is accepted as a substitute. + +Each allowed operation has two execution failures per task/operation in the +worktree's Git metadata. Failed local checks and unavailable/malformed GitHub +reads consume this budget. Successfully read but unacceptable CI evidence does +not: missing or running results return `state: "blocked"` with +`evidenceState: "not-ready"`; failed, cancelled, stale or mismatched evidence +returns `evidenceState: "rejected"`. Neither state passes verification, but later +explicit reads can accept newly completed evidence without a pending-state +lockout. There is no automatic polling or retry. + +Execution-failure counts survive ordinary continuation, pending results and +successful calls; there is no reset option. Once exhausted, the entry point returns +`GUARD_FAILURE_BUDGET_EXHAUSTED` without executing the operation. State is local +to that Git worktree and capped at 128 task/operation records; it is not a +defense against deliberate deletion or tampering with Git metadata. Each +evidence check stops after at most three non-paginated `gh api` calls, each +with a 15-second process timeout; it performs no application-level retry loop. + +## Enforcement and evidence limits + +These checks apply only when an operation is routed through +`scripts/agent.mjs guard`. They cannot intercept an agent or user running an +arbitrary shell command directly, and are not a sandbox or permission boundary. +The local tests validate the guard's behavior, not external workflow execution. +CI-green proves only that the named required jobs passed for the exact commit; +it does not establish live DSH configuration, loaded plugin state, account or +supplier behavior, business/field acceptance, or completion of a broader +engineering roadmap. + +The full dependency install, build, type, unit/integration and package gates +remain the existing GitHub CI workflow. On a local checkout, run only the +guarded dependency-free script tests; do not repeatedly attempt local dependency +recovery when the configured network cannot reach npm. + +## Autonomous field acceptance is a separate required evidence track + +For explicitly requested post-upgrade Desktop acceptance, use schema 2 of +[the existing acceptance contract](./agent-acceptance.md#required-live-desktop-acceptance-schema-2). +It composes this exact-head CI reader (same issue/operation budget, no reset) +with bounded content-free live evidence. CI success alone, installed-but-not-loaded, +not-run UI, automation-blocked, errors/timeouts and unapproved actions cannot +produce final acceptance or closure eligibility. A complete report is still +only a consistency judgment, not trusted proof that a UI tool ran. + +Safe checks on an already-running application remain required despite local npm +restrictions. An action returning done or no immediate tree diff needs a fresh +post-action observation before any failure conclusion. Use at most two distinct +supported navigation methods, then request the specified panel once; do not +equate user absence with permission or with a panel actually opened. Repository +scripts never control Desktop, activate windows via shell, install/restart, or +authorize account/OAuth/paid operations. See AGENTS.md for the controlled workflow. + +Formal upgrade evidence also enforces GitHub-first source selection: the exact +published Release asset URL is preferred; a verified GitHub-downloaded tgz is +only a separately labeled cache fallback with matching recorded archive/Release +SHA-256 and a fixed native-URL failure reason. Local builds/source directories/ +links/worktrees are not eligible. This validates reported selection, not remote +asset existence, authenticated provenance, installation permission or loading. + +Current-app evidence must begin with fresh discovery, not a historical "opened" +claim. Explicitly authorized startup uses controlled `start_app` once, followed +by real rediscovery/observation; no shell/window-manager workaround. Welcome UI +requires inspection for an official safe skip before concluding authentication +is required. A skip may replace the welcome window with a new main window: +rediscover and observe it instead of repeating actions on the old window. +Final handoff requires a fresh timestamped current running app and target-panel +or workspace observation, not the launch receipt or old success. The validator +checks this bounded lifecycle report, not actual host execution; it never stores +real app paths, window IDs, workspace titles, credentials or conversation content. + +Mandatory acceptance completeness is separate from issue disposition. Untested +production fault injection may remain explicit residual risk after a specific +fix, passing controlled regressions/CI, satisfied release/install/load premises +and nonreproducing required live checks. With explicit user risk acceptance, +recommend closure with residuals listed and reopen-on-recurrence; never rewrite +untested results, claim a proven root cause or auto-close issues. Known failures, +critical gaps and mandatory unmet checks cannot use this exception. Optional +schema 2 disposition records enforce that distinction without an extra platform. diff --git a/scripts/acceptance.mjs b/scripts/acceptance.mjs index f7d3602..6ab9f99 100644 --- a/scripts/acceptance.mjs +++ b/scripts/acceptance.mjs @@ -3,6 +3,7 @@ import { createHash } from 'node:crypto' import { constants } from 'node:fs' import { lstat, open, realpath } from 'node:fs/promises' import { dirname, join, resolve } from 'node:path' +import executionPolicy from '../agent-execution-policy.json' with { type: 'json' } const statuses = ['passed', 'failed', 'blocked', 'not-run'] const methods = ['static', 'unit', 'integration', 'browser', 'review', 'manual'] @@ -26,8 +27,10 @@ function safePath(value) { } function validateDocuments(contract, report) { - keys(contract, ['schemaVersion', 'issue', 'task', 'objective', 'nonGoals', 'requirements']) - require(contract.schemaVersion === 1 && Number.isSafeInteger(contract.issue) && contract.issue > 0 + const liveRequired = contract.schemaVersion === 2 + keys(contract, ['schemaVersion', 'issue', 'task', 'objective', 'nonGoals', 'requirements', + ...liveRequired ? ['liveUi'] : []]) + require([1, 2].includes(contract.schemaVersion) && Number.isSafeInteger(contract.issue) && contract.issue > 0 && id(contract.task) && text(contract.objective)) list(contract.nonGoals, 1, 16) require(contract.nonGoals.every(text)) @@ -36,10 +39,12 @@ function validateDocuments(contract, report) { for (const row of contract.requirements) { keys(row, ['id', 'expected', 'method']) require(id(row.id) && !required.has(row.id) && text(row.expected) && methods.includes(row.method)) + require(liveRequired || !['browser', 'manual'].includes(row.method), 'ACCEPTANCE_LIVE_CONTRACT_REQUIRED') required.add(row.id) } - keys(report, ['schemaVersion', 'commit', 'contractSha256', 'producer', 'environment', 'checks']) - require(report.schemaVersion === 1 && typeof report.commit === 'string' && /^[a-f0-9]{40}$/.test(report.commit) + keys(report, ['schemaVersion', 'commit', 'contractSha256', 'producer', 'environment', 'checks', + ...liveRequired ? ['liveUi', ...Object.hasOwn(report, 'disposition') ? ['disposition'] : []] : []]) + require(report.schemaVersion === contract.schemaVersion && typeof report.commit === 'string' && /^[a-f0-9]{40}$/.test(report.commit) && hash(report.contractSha256) && ['implementation', 'independent-review', 'human', 'ci'].includes(report.producer)) keys(report.environment, ['node', 'platform', 'core']) require(typeof report.environment.node === 'string' && /^\d+\.\d+\.\d+$/.test(report.environment.node) @@ -52,8 +57,10 @@ function validateDocuments(contract, report) { for (const check of report.checks) { keys(check, ['id', 'status', 'notes', 'evidence']) require(required.has(check.id) && !seen.has(check.id) && statuses.includes(check.status) && text(check.notes)) + if (liveRequired) require(['observed', 'not-run', 'blocked', 'failed'].includes(check.notes) + && check.evidence.length === 0) seen.add(check.id) - list(check.evidence, check.status === 'passed' ? 1 : 0, 8) + list(check.evidence, check.status === 'passed' && !liveRequired ? 1 : 0, 8) const paths = new Set() for (const evidence of check.evidence) { keys(evidence, ['path', 'sha256']) @@ -67,6 +74,206 @@ function validateDocuments(contract, report) { return artifacts } +const panels = ['models-manage', 'plugin-details', 'about'] +const version = value => value === null || (typeof value === 'string' + && /^\d+\.\d+\.\d+(?:-[A-Za-z0-9.-]+)?$/.test(value) && value.length <= 100) +const timestamp = value => Number.isSafeInteger(value) && value > 0 + +export function validateDisposition(value, requiredComplete) { + keys(value, ['decision', 'userAuthorized', 'defect', 'controlledRegression', + 'criticalBlocker', 'productionFaults']) + require(['keep-open', 'accept-residual-risk'].includes(value.decision) + && typeof value.userAuthorized === 'boolean' && ['specific-fix', 'unconfirmed'].includes(value.defect) + && ['passed', 'failed', 'not-run'].includes(value.controlledRegression) + && typeof value.criticalBlocker === 'boolean') + list(value.productionFaults, 1, 4) + const seen = new Set() + for (const row of value.productionFaults) { + keys(row, ['scenario', 'status', 'reason']) + require(['disconnect', 'never-resolves', 'timeout', 'late-response'].includes(row.scenario) + && !seen.has(row.scenario) && ['passed', 'failed', 'not-tested'].includes(row.status) + && ['observed', 'no-safe-control', 'destructive-only'].includes(row.reason) + && (row.status === 'not-tested') === (row.reason !== 'observed')) + seen.add(row.scenario) + } + const eligible = requiredComplete && value.decision === 'accept-residual-risk' + && value.userAuthorized && value.defect === 'specific-fix' && value.controlledRegression === 'passed' + && !value.criticalBlocker && value.productionFaults.every(row => row.status !== 'failed') + return { recommendation: eligible ? 'close-with-residual-risk-and-reopen-on-recurrence' : 'keep-open', + productionFaultCoverage: value.productionFaults.some(row => row.status === 'not-tested') + ? 'incomplete' : 'reported-tested', + requiredChecksWaived: false, rootCauseVerified: false, automaticClosure: false, + authorizationVerified: false } +} + +function validateAppLifecycle(value, now) { + keys(value, ['discovery', 'launch', 'welcome', 'rediscovery', 'final', 'interaction']) + keys(value.discovery, ['state', 'atMs', 'source']) + require(['running', 'not-running', 'unknown'].includes(value.discovery.state) + && timestamp(value.discovery.atMs) && value.discovery.source === 'list-apps') + keys(value.launch, ['state', 'atMs']) + require(['not-needed', 'authorized-start-app', 'not-authorized'].includes(value.launch.state) + && (value.launch.state === 'authorized-start-app' + ? timestamp(value.launch.atMs) && value.launch.atMs > value.discovery.atMs + && value.discovery.state === 'not-running' + : value.launch.atMs === null)) + keys(value.welcome, ['state', 'atMs']) + require(['not-seen', 'safe-skip-observed', 'safe-skip-used', 'auth-required', 'unknown'].includes(value.welcome.state) + && (value.welcome.state === 'not-seen' ? value.welcome.atMs === null : timestamp(value.welcome.atMs) + && value.welcome.atMs >= (value.launch.atMs ?? value.discovery.atMs))) + keys(value.rediscovery, ['atMs', 'source', 'window']) + require(['same', 'replacement', 'unobserved'].includes(value.rediscovery.window) + && value.rediscovery.source === 'list-apps' + && (value.rediscovery.atMs === null || timestamp(value.rediscovery.atMs) + && value.rediscovery.atMs > (value.welcome.atMs ?? value.launch.atMs ?? value.discovery.atMs))) + keys(value.final, ['atMs', 'source', 'appState', 'surface']) + require((value.final.atMs === null || timestamp(value.final.atMs)) + && value.final.source === 'list-apps-and-get-window-state' + && ['running', 'not-running', 'unknown'].includes(value.final.appState) + && ['target-panel', 'workspace', 'welcome', 'unobserved'].includes(value.final.surface) + && ['clear', 'interrupted'].includes(value.interaction)) + if (value.interaction === 'interrupted') return 'LIVE_UI_INPUT_INTERRUPTED' + if (value.discovery.state !== 'running' && value.launch.state !== 'authorized-start-app') { + return 'LIVE_UI_LAUNCH_BLOCKED' + } + if (value.welcome.state === 'auth-required') return 'LIVE_UI_AUTH_REQUIRED' + if (['safe-skip-observed', 'unknown'].includes(value.welcome.state)) return 'LIVE_UI_WELCOME_UNRESOLVED' + if (value.rediscovery.atMs === null || value.rediscovery.window === 'unobserved') return 'LIVE_UI_WINDOW_UNCONFIRMED' + if (value.final.atMs === null || value.final.atMs <= value.rediscovery.atMs + || value.final.atMs > now || now - value.final.atMs > 120000 + || value.final.appState !== 'running' || !['target-panel', 'workspace'].includes(value.final.surface)) { + return 'LIVE_UI_CURRENT_APP_UNCONFIRMED' + } + return null +} + +export function validateLiveUi(contract, report, now = Date.now()) { + keys(contract, ['expectedPluginVersion', 'panel', 'checks']) + require(version(contract.expectedPluginVersion) && contract.expectedPluginVersion !== null + && panels.includes(contract.panel)) + list(contract.checks, 1, 16) + const required = new Map() + for (const row of contract.checks) { + keys(row, ['id', 'action', 'outcome']) + require(id(row.id) && !required.has(row.id) && ['ready', 'actionable-failure'].includes(row.outcome) + && ['observe-panel', 'reopen-manage', 'refresh-models'].includes(row.action)) + required.set(row.id, row) + } + keys(report, ['appState', 'desktop', 'installed', 'loaded', 'upgradeSource', + 'navigation', 'handoff', 'permission', 'steps', 'lifecycle']) + const lifecycleCode = validateAppLifecycle(report.lifecycle, now) + require(['running', 'not-running', 'unknown'].includes(report.appState)) + for (const [field, sources] of [ + ['desktop', ['about', 'unknown']], + ['installed', ['disk-manifest', 'unknown']], + ['loaded', ['public-runtime-status', 'runtime-ui', 'unknown']], + ]) { + keys(report[field], ['version', 'source', 'observedAtMs']) + require(version(report[field].version) && sources.includes(report[field].source) + && (report[field].source === 'unknown') === (report[field].version === null) + && (report[field].version === null ? report[field].observedAtMs === null : timestamp(report[field].observedAtMs))) + } + const source = report.upgradeSource + keys(source, ['kind', 'version', 'url', 'sha256', 'releaseSha256', 'fallbackReason', 'observedAtMs']) + require(['github-release-url', 'verified-github-cache', 'local-build', 'source-directory', + 'link-worktree', 'unknown'].includes(source.kind) && version(source.version) + && (source.url === null || source.url === `https://github.com/${executionPolicy.repository}/releases/download/v${source.version}/dsh-github-copilot-${source.version}.tgz`) + && (source.sha256 === null || hash(source.sha256)) + && (source.releaseSha256 === null || hash(source.releaseSha256)) + && ['none', 'native-url-unavailable', 'native-url-rejected'].includes(source.fallbackReason) + && (source.observedAtMs === null || timestamp(source.observedAtMs))) + keys(report.permission, ['state', 'action']) + require(['safe-read-only', 'approval-required', 'user-absent'].includes(report.permission.state) + && ['none', 'restart', 'install', 'account-switch', 'oauth', 'paid-call'].includes(report.permission.action)) + keys(report.handoff, ['state', 'panel']) + require(['none', 'requested', 'user-absent'].includes(report.handoff.state) + && report.handoff.panel === contract.panel) + list(report.navigation, 0, 2) + const methods = new Set() + for (const attempt of report.navigation) { + keys(attempt, ['method', 'outcome', 'atMs', 'observedAtMs', 'observationSource']) + require(['accessible-control', 'supported-shortcut', 'user-opened-panel'].includes(attempt.method) + && !methods.has(attempt.method) && ['opened', 'unchanged', 'no-candidate'].includes(attempt.outcome) + && timestamp(attempt.atMs) && timestamp(attempt.observedAtMs) + && attempt.observedAtMs > attempt.atMs && attempt.observedAtMs - attempt.atMs <= 120000 + && attempt.observationSource === 'get-window-state') + require(!report.navigation.slice(0, methods.size).some(row => row.outcome === 'opened')) + require(report.navigation.slice(0, methods.size).every(row => row.observedAtMs <= attempt.atMs)) + methods.add(attempt.method) + } + list(report.steps, 0, 32) + const observed = new Map() + let lastEnd = 0 + for (const step of report.steps) { + keys(step, ['checkId', 'action', 'observation', 'startedAtMs', 'finishedAtMs', + 'observedAtMs', 'observationSource']) + require(required.has(step.checkId) && !observed.has(step.checkId) + && ['observe-panel', 'reopen-manage', 'refresh-models'].includes(step.action) + && ['ready', 'actionable-failure', 'error', 'timeout', 'unchanged'].includes(step.observation) + && timestamp(step.startedAtMs) && timestamp(step.finishedAtMs) + && step.startedAtMs >= lastEnd && step.finishedAtMs >= step.startedAtMs + && timestamp(step.observedAtMs) && step.observedAtMs > step.finishedAtMs + && step.observedAtMs - step.startedAtMs <= 120000 + && step.observationSource === 'get-window-state') + lastEnd = step.observedAtMs + observed.set(step.checkId, step) + } + const opened = report.navigation.at(-1)?.outcome === 'opened' + require(!opened || report.steps.every(step => step.startedAtMs >= report.navigation.at(-1).observedAtMs)) + const qualifiedObservation = opened && report.steps.length > 0 + const result = code => ({ code, ok: code === 'LIVE_UI_REPORTED_COMPLETE', qualifiedObservation, + upgradeSourceKind: source.kind, provenanceVerified: false, + currentAppObservedAtMs: report.lifecycle.final.atMs, + nextAction: ({ + LIVE_UI_LAUNCH_BLOCKED: 'stop-for-explicit-launch-permission', + LIVE_UI_AUTH_REQUIRED: 'stop-at-authentication-boundary', + LIVE_UI_WELCOME_UNRESOLVED: 'inspect-official-safe-skip', + LIVE_UI_WINDOW_UNCONFIRMED: 'rediscover-app-window', + LIVE_UI_CURRENT_APP_UNCONFIRMED: 'fresh-discovery-and-observation', + LIVE_UI_INPUT_INTERRUPTED: 'reread-ui-before-same-action-retry', + })[code] ?? (code === 'LIVE_UI_AUTOMATION_BLOCKED' + ? report.handoff.state === 'none' ? 'request-specified-panel-once' : 'wait-for-user-panel' + : code === 'LIVE_UI_NOT_RUN' + ? report.navigation.length === 0 ? 'inspect-running-app' : 'try-one-supported-alternate' + : code === 'LIVE_UI_PERMISSION_BLOCKED' ? 'stop-for-explicit-permission' + : code === 'LIVE_UI_APP_UNAVAILABLE' ? 'report-app-unavailable' + : code === 'LIVE_UI_HANDOFF_PENDING' ? 'wait-for-user-panel' + : code === 'LIVE_UI_LOADED_VERSION_UNCONFIRMED' ? 'read-public-loaded-version' + : code === 'LIVE_UI_VERSION_PREREQUISITE_UNMET' ? 'request-exact-upgrade-permission' + : code === 'LIVE_UI_UPGRADE_SOURCE_REJECTED' ? 'select-exact-github-release-url' + : code === 'LIVE_UI_INCOMPLETE' ? 'perform-remaining-authorized-checks' : 'report-result'), + versionFixReportedConsistent: code === 'LIVE_UI_REPORTED_COMPLETE', executionVerified: false, + environmentVerified: false, source: 'report-consistency-only' }) + if (report.permission.state !== 'safe-read-only' || report.permission.action !== 'none') { + return result('LIVE_UI_PERMISSION_BLOCKED') + } + if (lifecycleCode) return result(lifecycleCode) + if (report.navigation.some(row => row.atMs < report.lifecycle.rediscovery.atMs) + || report.lifecycle.final.atMs < lastEnd + || (lastEnd > 0 && report.lifecycle.final.atMs - lastEnd > 120000) + || report.navigation.some(row => row.observedAtMs > report.lifecycle.final.atMs)) { + return result('LIVE_UI_CURRENT_APP_UNCONFIRMED') + } + if (report.appState !== 'running') return result('LIVE_UI_APP_UNAVAILABLE') + if (!opened) return result(report.navigation.length === 2 + ? 'LIVE_UI_AUTOMATION_BLOCKED' : 'LIVE_UI_NOT_RUN') + if (report.handoff.state !== 'none') return result('LIVE_UI_HANDOFF_PENDING') + if ([...observed.values()].some(value => ['error', 'timeout', 'unchanged'].includes(value.observation))) { + return result('LIVE_UI_OBSERVATION_FAILED') + } + if ([...required].some(([key, expected]) => observed.get(key)?.observation !== expected.outcome + || observed.get(key)?.action !== expected.action)) return result('LIVE_UI_INCOMPLETE') + if (report.loaded.version === null) return result('LIVE_UI_LOADED_VERSION_UNCONFIRMED') + if (report.loaded.version !== contract.expectedPluginVersion) return result('LIVE_UI_VERSION_PREREQUISITE_UNMET') + if (!['github-release-url', 'verified-github-cache'].includes(source.kind) + || source.version !== contract.expectedPluginVersion || source.url === null + || source.sha256 === null || source.sha256 !== source.releaseSha256 || source.observedAtMs === null + || (source.kind === 'github-release-url' ? source.fallbackReason !== 'none' : source.fallbackReason === 'none')) { + return result('LIVE_UI_UPGRADE_SOURCE_REJECTED') + } + return result('LIVE_UI_REPORTED_COMPLETE') +} + async function readBounded(path, limit) { let handle try { @@ -154,13 +361,18 @@ async function artifactPath(root, path) { } } -export async function verifyAcceptance({ root = process.cwd(), contractPath, reportPath }) { +export async function verifyAcceptance({ root = process.cwd(), contractPath, reportPath, execute }) { const commit = cleanHead(root) const contractBytes = await readBounded(resolve(contractPath), 65536) const reportBytes = await readBounded(resolve(reportPath), 65536) const contract = json(contractBytes) const report = json(reportBytes) const artifacts = validateDocuments(contract, report) + if (Object.hasOwn(report, 'disposition')) validateDisposition(report.disposition, false) + let liveUi = contract.schemaVersion === 2 ? validateLiveUi(contract.liveUi, report.liveUi) : null + if (liveUi) require(contract.liveUi.checks.every(row => contract.requirements.some(required => required.id === row.id))) + if (liveUi) require(contract.requirements.filter(row => ['browser', 'manual'].includes(row.method)) + .every(row => contract.liveUi.checks.some(live => live.id === row.id))) require(report.commit === commit, 'ACCEPTANCE_STALE_COMMIT') require(report.contractSha256 === digest(contractBytes), 'ACCEPTANCE_STALE_CONTRACT') const evidenceRoot = await realpath(dirname(resolve(reportPath))) @@ -177,12 +389,30 @@ export async function verifyAcceptance({ root = process.cwd(), contractPath, rep && digest(await readBounded(resolve(reportPath), 65536)) === digest(reportBytes), 'ACCEPTANCE_FILE_CHANGED') require(cleanHead(root) === commit, 'ACCEPTANCE_STALE_COMMIT') const counts = Object.fromEntries(statuses.map(status => [status, report.checks.filter(check => check.status === status).length])) + let ci = null + if (liveUi) { + const { runGuardedOperation } = await import('./environment-guard.mjs') + ci = await runGuardedOperation({ task: String(contract.issue), environment: 'github-actions-evidence', + operation: 'ci-evidence', head: commit, root, execute }) + require(cleanHead(root) === commit, 'ACCEPTANCE_STALE_COMMIT') + require(digest(await readBounded(resolve(contractPath), 65536)) === digest(contractBytes) + && digest(await readBounded(resolve(reportPath), 65536)) === digest(reportBytes), 'ACCEPTANCE_FILE_CHANGED') + liveUi = validateLiveUi(contract.liveUi, report.liveUi) + } + const ok = counts.passed === contract.requirements.length && (!liveUi || (liveUi.ok && ci.ok)) + && (!report.disposition || (!report.disposition.criticalBlocker + && report.disposition.productionFaults.every(row => row.status !== 'failed'))) + const disposition = report.disposition ? validateDisposition(report.disposition, ok) : null return { schemaVersion: 1, kind: 'acceptance-evidence-consistency', - ok: counts.passed === contract.requirements.length, + ok, issue: contract.issue, commit, contractSha256: report.contractSha256, counts, evidenceIntegrityVerified: true, executionVerified: false, independenceVerified: false, environmentVerified: false, deliveryAuthorized: false, + ...liveUi ? { state: ok ? 'reported-complete' : 'blocked', + closeEligible: ok && (!disposition || disposition.recommendation !== 'keep-open'), + liveUi, ci, ciVerified: ci.ok } : {}, + ...disposition ? { disposition } : {}, } } diff --git a/scripts/agent.mjs b/scripts/agent.mjs index 7114bec..74cc501 100644 --- a/scripts/agent.mjs +++ b/scripts/agent.mjs @@ -8,15 +8,17 @@ const readJson = async (root, path) => JSON.parse(await readFile(resolve(root, p /** Read repository-owned metadata only. Never inspect DSH_HOME, credentials or live services. */ export async function describeRepository(root = repositoryRoot) { - const [contract, pkg, baseline] = await Promise.all([ + const [contract, pkg, baseline, executionPolicy] = await Promise.all([ readJson(root, 'agent-contract.json'), readJson(root, 'package.json'), readJson(root, 'deployment-baseline.json'), + readJson(root, 'agent-execution-policy.json'), ]) return { schemaVersion: 1, kind: 'repository-contract', package: { name: pkg.name, version: pkg.version, node: pkg.engines.node, packageManager: pkg.packageManager }, + executionPolicy, entrypoints: contract.entrypoints, boundaries: contract.boundaries, acceptance: contract.acceptance, @@ -41,17 +43,12 @@ export async function planTask(task, root = repositoryRoot) { const description = await describeRepository(root) const area = Object.hasOwn(description.tasks, task) ? description.tasks[task] : undefined if (area === undefined) throw new Error(`Unknown task. Choose: ${Object.keys(description.tasks).join(', ')}`) - const scriptTests = area.tests.filter(path => path.startsWith('tests/scripts/')) - const unitTests = area.tests.filter(path => !path.startsWith('tests/scripts/')) const commands = [ - ...area.iterationReview ? [['node', 'scripts/iteration-review.mjs', '--report', area.iterationReview.report]] : [], - ['pnpm', 'install', '--frozen-lockfile'], - ...(area.tests.length === 0 ? [['pnpm', 'test:scripts']] : []), - ...(unitTests.length ? [['pnpm', 'exec', 'vitest', 'run', ...unitTests]] : []), - ...(scriptTests.length ? [['node', '--test', ...scriptTests]] : []), - ['pnpm', 'verify'], - ['pnpm', 'pack', '--pack-destination', 'artifacts'], - ['pnpm', 'verify:tarball', '--', `artifacts/${description.release.tarball}`], + ['node', 'scripts/agent.mjs', 'guard', '--task', 'TASK_ID', + '--environment', 'local-auxiliary', '--operation', 'scripts-test'], + ['node', 'scripts/agent.mjs', 'guard', '--task', 'TASK_ID', + '--environment', 'github-actions-evidence', '--operation', 'ci-evidence', + '--head', 'FULL_40_CHARACTER_COMMIT'], ] return { schemaVersion: 1, @@ -59,8 +56,14 @@ export async function planTask(task, root = repositoryRoot) { task, ...area, cwd: 'repository root', + preflightChecks: area.iterationReview ? [{ + argv: ['node', 'scripts/iteration-review.mjs', '--report', area.iterationReview.report], + executed: false, + requiredBeforeImplementation: area.iterationReview.requiredBeforeImplementation, + }] : [], commands: commands.map(argv => ({ argv, executed: false })), acceptance: description.acceptance, + executionPolicy: description.executionPolicy, delivery: 'Issue -> feature branch -> focused tests -> complete gate -> reviewed merge without a separate approval prompt; important updates continue to verified Release and npm integrity without a second release prompt, unless user scope excludes it. Other changes need an explicit release request. Ask before profile installation or interrupting restart; report any publication blocker.', boundaries: description.boundaries, } @@ -77,7 +80,8 @@ export async function doctor(root = repositoryRoot, nodeVersion = process.versio const pkg = await readJson(root, `node_modules/${name}/package.json`) checks.push({ id: `dependency:${name}`, status: 'pass', detail: pkg.version, next: null }) } catch { - checks.push({ id: `dependency:${name}`, status: 'fail', detail: 'missing-or-unreadable', next: 'pnpm install --frozen-lockfile' }) + checks.push({ id: `dependency:${name}`, status: 'fail', detail: 'missing-or-unreadable', + next: 'Use the guarded GitHub Actions evidence check for full verification; do not restore dependencies on a local checkout.' }) } } for (const path of ['lib/index.js', 'lib/client.js', 'lib/remote.js']) { @@ -92,7 +96,7 @@ export async function doctor(root = repositoryRoot, nodeVersion = process.versio package: description.package, checks, notChecked: ['GitHub authentication/CI/Release', 'npm ownership/authentication/publication', 'live DSH profile and loaded plugin', 'Copilot account availability', 'model transport', 'hosted search', 'build freshness', 'pnpm executable version'], - next: 'Run pnpm verify. Do not treat this report as runtime or release validation.', + next: 'Choose an explicit guarded environment and task. Use local-auxiliary only for dependency-free script tests; full verification requires actual GitHub Actions evidence.', } } @@ -111,6 +115,14 @@ export async function main(args = process.argv.slice(2)) { const { acceptanceMain } = await import('./acceptance.mjs') return acceptanceMain(rest.slice(1)) } + if (command === 'guard') { + const { guardErrorResult, guardMain } = await import('./environment-guard.mjs') + try { return await guardMain(rest.slice(1)) } + catch (error) { + console.log(JSON.stringify(guardErrorResult(error))) + return 2 + } + } try { let result if (extra !== undefined) throw new Error('Too many arguments.') @@ -118,7 +130,7 @@ export async function main(args = process.argv.slice(2)) { else if (command === 'doctor' && value === undefined) result = await doctor() else if (command === 'plan' && value !== undefined) result = await planTask(value) else if (command === 'attribution' && value !== undefined) result = { schemaVersion: 1, trailer: attribution(value) } - else throw new Error('Usage: node scripts/agent.mjs describe|doctor|plan |attribution "Actual Tool"|accept --contract FILE --report FILE [--json]') + else throw new Error('Usage: node scripts/agent.mjs describe|doctor|plan |attribution "Actual Tool"|accept --contract FILE --report FILE [--json]|guard --task TASK_ID --environment local-auxiliary|github-actions-evidence --operation NAME [--head SHA]') if (json || command !== 'attribution') console.log(JSON.stringify(result, null, 2)) else console.log(result.trailer) return result.ok === false ? 1 : 0 diff --git a/scripts/environment-guard.mjs b/scripts/environment-guard.mjs new file mode 100644 index 0000000..7f671c8 --- /dev/null +++ b/scripts/environment-guard.mjs @@ -0,0 +1,361 @@ +import { spawnSync } from 'node:child_process' +import { randomUUID } from 'node:crypto' +import { open, readFile, rename, unlink, writeFile } from 'node:fs/promises' +import { dirname, isAbsolute, resolve } from 'node:path' +import { fileURLToPath } from 'node:url' + +const scriptRoot = resolve(dirname(fileURLToPath(import.meta.url)), '..') +const policyPath = resolve(scriptRoot, 'agent-execution-policy.json') +const taskId = value => typeof value === 'string' && value.length <= 64 + && /^[a-z0-9]+(?:-[a-z0-9]+)*$/.test(value) +const commitId = value => typeof value === 'string' && /^[a-f0-9]{40}$/.test(value) +const isRecord = value => value !== null && typeof value === 'object' && !Array.isArray(value) +const requiredJobs = [ + 'verify (ubuntu-latest)', + 'verify (windows-latest)', + 'Published adapter (ubuntu-latest)', + 'Published adapter (windows-latest)', + 'release-ready', +] +const scriptsTestArgv = [ + 'node', '--test', 'tests/scripts/agent.test.mjs', 'tests/scripts/acceptance.test.mjs', + 'tests/scripts/environment-guardrails.test.mjs', +] + +class GuardError extends Error { + constructor(code, details = {}) { + super(code) + this.code = code + this.details = details + } +} + +class CiEvidenceError extends GuardError { + constructor(code, details = {}, evidenceState = 'rejected') { + super(code, { ...details, evidenceState }) + } +} + +function validPolicy(policy) { + return isRecord(policy) && policy.schemaVersion === 1 + && policy.repository === 'cloga/dsh-github-copilot' && policy.ownerTask === 'tooling' + && policy.maxTaskScopes === 128 + && JSON.stringify(policy.environments?.['local-auxiliary']?.operations) === JSON.stringify(['scripts-test']) + && JSON.stringify(policy.environments?.['github-actions-evidence']?.operations) === JSON.stringify(['ci-evidence']) + && policy.operations?.['dependency-install']?.kind === 'network' + && policy.operations['dependency-install'].maxFailures === 0 + && policy.operations?.['scripts-test']?.kind === 'local-command' + && policy.operations['scripts-test'].environment === 'local-auxiliary' + && policy.operations['scripts-test'].maxFailures === 2 + && JSON.stringify(policy.operations['scripts-test'].argv) === JSON.stringify(scriptsTestArgv) + && policy.operations?.['ci-evidence']?.kind === 'github-actions-read' + && policy.operations['ci-evidence'].environment === 'github-actions-evidence' + && policy.operations['ci-evidence'].maxFailures === 2 + && policy.operations['ci-evidence'].maxApiCalls === 3 + && policy.operations['ci-evidence'].requestTimeoutMs === 15000 + && policy.operations['ci-evidence'].workflowPath === '.github/workflows/ci.yml' + && policy.operations['ci-evidence'].workflowName === 'CI' + && JSON.stringify(policy.operations['ci-evidence'].requiredJobs) === JSON.stringify(requiredJobs) + && JSON.stringify(policy.operations['ci-evidence'].optionalJobs) === JSON.stringify([ + { name: 'release', allowedConclusions: ['success', 'skipped'] }, + ]) +} + +function executeGit(root, args, execute) { + const result = execute('git', ['-C', root, ...args], { + encoding: 'utf8', timeout: 10000, maxBuffer: 1024 * 1024, windowsHide: true, + env: { ...process.env, GIT_TERMINAL_PROMPT: '0' }, + }) + if (result.error || result.status !== 0) throw new GuardError('GUARD_GIT_UNAVAILABLE') + return result.stdout.trim() +} + +async function readPolicy() { + let policy + try { policy = JSON.parse(await readFile(policyPath, 'utf8')) } + catch { throw new GuardError('GUARD_POLICY_UNAVAILABLE') } + if (!validPolicy(policy)) throw new GuardError('GUARD_POLICY_INVALID') + return policy +} + +async function budgetPath(root, execute) { + const path = executeGit(root, ['rev-parse', '--git-path', 'copilot-agent-guard-state.json'], execute) + return isAbsolute(path) ? path : resolve(root, path) +} + +async function readBudget(path) { + try { + const state = JSON.parse(await readFile(path, 'utf8')) + if (!isRecord(state) || state.schemaVersion !== 1 || !isRecord(state.failures) + || Object.keys(state.failures).length > 128 + || Object.values(state.failures).some(value => !Number.isSafeInteger(value) || value < 0)) { + throw new GuardError('GUARD_BUDGET_STATE_INVALID') + } + return state + } catch (error) { + if (error instanceof GuardError) throw error + if (error.code === 'ENOENT') return { schemaVersion: 1, failures: {} } + throw new GuardError('GUARD_BUDGET_STATE_INVALID') + } +} + +async function writeBudget(path, state) { + const temporary = `${path}.${randomUUID()}.tmp` + try { + await writeFile(temporary, JSON.stringify(state), { flag: 'wx' }) + await rename(temporary, path) + } catch { + throw new GuardError('GUARD_BUDGET_STATE_UNAVAILABLE') + } +} + +function githubRepository(remote) { + if (typeof remote !== 'string') return null + const match = remote.match(/^(?:https:\/\/github\.com\/|git@github\.com:|ssh:\/\/git@github\.com\/)([^/]+\/[^/]+?)(?:\.git)?$/) + return match?.[1] ?? null +} + +function ghJson(endpoint, execute, limits) { + limits.calls++ + if (limits.calls > limits.maxApiCalls) throw new GuardError('CI_API_CALL_LIMIT') + const result = execute('gh', ['api', endpoint], { + encoding: 'utf8', timeout: limits.requestTimeoutMs, maxBuffer: 4 * 1024 * 1024, windowsHide: true, + env: { ...process.env, GH_PROMPT_DISABLED: '1', GH_PAGER: 'cat', GH_NO_UPDATE_NOTIFIER: '1' }, + }) + if (result.error || result.status !== 0) throw new GuardError('CI_GITHUB_READ_UNAVAILABLE') + try { return JSON.parse(result.stdout) } + catch { throw new GuardError('CI_GITHUB_RESPONSE_INVALID') } +} + +function failedConclusion(conclusion, prefix) { + const status = typeof conclusion === 'string' ? conclusion.toLowerCase() : 'unknown' + const suffix = { + failure: 'FAILED', + cancelled: 'CANCELLED', + timed_out: 'TIMED_OUT', + skipped: 'SKIPPED', + neutral: 'NEUTRAL', + action_required: 'ACTION_REQUIRED', + }[status] ?? 'NOT_SUCCESSFUL' + return `${prefix}_${suffix}` +} + +function requireSuccessfulJob(job, name) { + if (!job) throw new CiEvidenceError('CI_REQUIRED_JOB_MISSING', { job: name }, 'not-ready') + if (job.status !== 'completed') { + throw new CiEvidenceError('CI_REQUIRED_JOB_IN_PROGRESS', { job: name }, 'not-ready') + } + if (job.conclusion !== 'success') { + throw new CiEvidenceError(failedConclusion(job.conclusion, 'CI_REQUIRED_JOB'), { job: name }) + } +} + +function findUniqueJob(jobs, name) { + const matches = jobs.filter(job => job.name === name) + if (matches.length > 1) throw new CiEvidenceError('CI_REQUIRED_JOB_DUPLICATE', { job: name }) + return matches[0] +} + +function compareRuns(a, b) { + if (a.id === b.id) return b.run_attempt - a.run_attempt + return b.run_number - a.run_number || b.id - a.id +} + +function verifyCi({ root, head, policy, execute }) { + if (!commitId(head)) throw new CiEvidenceError('CI_HEAD_INVALID') + const actualHead = executeGit(root, ['rev-parse', '--verify', 'HEAD'], execute) + if (actualHead !== head) throw new CiEvidenceError('CI_HEAD_MISMATCH') + if (executeGit(root, ['status', '--porcelain=v1', '--untracked-files=all'], execute) !== '') { + throw new CiEvidenceError('CI_WORKTREE_DIRTY') + } + const repository = githubRepository(executeGit(root, ['config', '--get', 'remote.origin.url'], execute)) + if (repository !== policy.repository) throw new CiEvidenceError('CI_REPOSITORY_MISMATCH') + + const limits = { calls: 0, ...policy.operations['ci-evidence'] } + const workflow = ghJson(`/repos/${repository}/actions/workflows/ci.yml`, execute, limits) + if (!isRecord(workflow) || String(workflow.path ?? '').replaceAll('\\', '/') !== '.github/workflows/ci.yml' + || workflow.name !== 'CI' || !Number.isSafeInteger(workflow.id) || workflow.state !== 'active') { + throw new CiEvidenceError('CI_WORKFLOW_MISMATCH') + } + const runsResponse = ghJson(`/repos/${repository}/actions/runs?head_sha=${head}&per_page=100`, execute, limits) + if (!isRecord(runsResponse) || !Array.isArray(runsResponse.workflow_runs)) { + throw new GuardError('CI_GITHUB_RESPONSE_INVALID') + } + const exactHead = runsResponse.workflow_runs.filter(run => isRecord(run) && run.head_sha === head) + const runs = exactHead.filter(run => String(run.workflow_id) === String(workflow.id)) + if (runs.length === 0) { + throw new CiEvidenceError(exactHead.length ? 'CI_WORKFLOW_MISMATCH' : 'CI_RUN_MISSING', + {}, exactHead.length ? 'rejected' : 'not-ready') + } + if (runs.some(run => !['id', 'run_number', 'run_attempt'].every( + key => Number.isSafeInteger(run[key]) && run[key] > 0))) { + throw new GuardError('CI_GITHUB_RESPONSE_INVALID') + } + runs.sort(compareRuns) + const run = runs[0] + if (run.status !== 'completed') throw new CiEvidenceError('CI_RUN_IN_PROGRESS', {}, 'not-ready') + if (run.conclusion !== 'success') throw new CiEvidenceError(failedConclusion(run.conclusion, 'CI_RUN')) + + const jobsResponse = ghJson( + `/repos/${repository}/actions/runs/${run.id}/jobs?filter=latest&per_page=100`, execute, limits) + if (!isRecord(jobsResponse) || !Array.isArray(jobsResponse.jobs)) { + throw new GuardError('CI_GITHUB_RESPONSE_INVALID') + } + const jobs = jobsResponse.jobs.filter(isRecord) + for (const name of policy.operations['ci-evidence'].requiredJobs) { + requireSuccessfulJob(findUniqueJob(jobs, name), name) + } + const release = findUniqueJob(jobs, 'release') + if (release && (release.status !== 'completed' + || !policy.operations['ci-evidence'].optionalJobs[0].allowedConclusions.includes(release.conclusion))) { + throw new CiEvidenceError(release.status !== 'completed' + ? 'CI_RELEASE_JOB_IN_PROGRESS' : failedConclusion(release.conclusion, 'CI_RELEASE_JOB'), + {}, release.status !== 'completed' ? 'not-ready' : 'rejected') + } + return { + repository, + head, + runId: run.id, + runUrl: typeof run.html_url === 'string' ? run.html_url : null, + workflow: { id: workflow.id, name: workflow.name, path: workflow.path }, + event: typeof run.event === 'string' ? run.event : null, + jobs: [ + ...requiredJobs.map(name => ({ name, status: 'completed', conclusion: 'success' })), + ...(release ? [{ name: release.name, status: release.status, conclusion: release.conclusion }] : []), + ], + verification: 'github-actions-results-only', + } +} + +function output(state, task, environment, operation, code, executed, details = {}) { + return { + schemaVersion: 1, kind: 'guarded-operation-result', state, ok: state === 'passed', + code, task, environment, operation, executed, ...details, + } +} + +export async function runGuardedOperation({ + task, environment, operation, head, root = scriptRoot, execute = spawnSync, +}) { + if (!taskId(task)) throw new GuardError('GUARD_TASK_REQUIRED') + if (!['local-auxiliary', 'github-actions-evidence'].includes(environment)) { + throw new GuardError('GUARD_ENVIRONMENT_REQUIRED') + } + if (typeof operation !== 'string') throw new GuardError('GUARD_OPERATION_REQUIRED') + const policy = await readPolicy() + const spec = policy.operations[operation] + if (!spec) throw new GuardError('GUARD_OPERATION_UNKNOWN') + if (operation === 'dependency-install') { + return output('blocked', task, environment, operation, environment === 'local-auxiliary' + ? 'GUARD_LOCAL_NETWORK_BLOCKED' : 'GUARD_CI_IS_EXTERNAL_ONLY', false) + } + if (spec.environment !== environment || !policy.environments[environment].operations.includes(operation)) { + return output('blocked', task, environment, operation, 'GUARD_ENVIRONMENT_OPERATION_MISMATCH', false) + } + const path = await budgetPath(root, execute) + const lockPath = `${path}.lock` + let lock + try { lock = await open(lockPath, 'wx') } + catch (error) { + if (error.code === 'EEXIST') { + return output('blocked', task, environment, operation, 'GUARD_BUDGET_BUSY', false) + } + throw new GuardError('GUARD_BUDGET_STATE_UNAVAILABLE') + } + try { + const state = await readBudget(path) + const key = `${task}:${operation}` + const failures = state.failures[key] ?? 0 + if (failures === 0 && Object.keys(state.failures).length >= policy.maxTaskScopes) { + return output('blocked', task, environment, operation, 'GUARD_BUDGET_SCOPE_LIMIT', false, { + maxTaskScopes: policy.maxTaskScopes, + }) + } + if (failures >= spec.maxFailures) { + return output('blocked', task, environment, operation, 'GUARD_FAILURE_BUDGET_EXHAUSTED', false, { + failedAttempts: failures, maxFailures: spec.maxFailures, + }) + } + try { + if (operation === 'scripts-test') { + const [command, ...args] = spec.argv + const child = execute(command === 'node' ? process.execPath : command, args, { + cwd: root, encoding: 'utf8', timeout: 120000, maxBuffer: 8 * 1024 * 1024, windowsHide: true, + stdio: 'inherit', + }) + if (child.error || child.status !== 0) { + const next = failures + 1 + state.failures[key] = next + await writeBudget(path, state) + return output('failed', task, environment, operation, 'GUARD_LOCAL_CHECK_FAILED', true, { + exitCode: Number.isInteger(child.status) ? child.status : null, + failedAttempts: next, maxFailures: spec.maxFailures, remainingFailures: spec.maxFailures - next, + }) + } + return output('passed', task, environment, operation, 'GUARD_LOCAL_CHECK_PASSED', true, { + failedAttempts: failures, maxFailures: spec.maxFailures, remainingFailures: spec.maxFailures - failures, + }) + } + const evidence = verifyCi({ root, head, policy, execute }) + return output('passed', task, environment, operation, 'CI_EVIDENCE_VERIFIED', true, { + failedAttempts: failures, maxFailures: spec.maxFailures, remainingFailures: spec.maxFailures - failures, + evidence, + }) + } catch (error) { + if (error instanceof CiEvidenceError) { + return output('blocked', task, environment, operation, error.code, true, { + failedAttempts: failures, maxFailures: spec.maxFailures, + remainingFailures: spec.maxFailures - failures, ...error.details, + }) + } + const next = failures + 1 + state.failures[key] = next + await writeBudget(path, state) + return output('blocked', task, environment, operation, + error instanceof GuardError ? error.code : 'GUARD_OPERATION_UNAVAILABLE', true, { + failedAttempts: next, maxFailures: spec.maxFailures, remainingFailures: spec.maxFailures - next, + ...(error instanceof GuardError ? error.details : {}), + }) + } + } finally { + try { + await lock.close() + await unlink(lockPath) + } catch (error) { + if (error.code !== 'ENOENT') { + throw new GuardError('GUARD_BUDGET_LOCK_RELEASE_FAILED', { executed: null }) + } + } + } +} + +export async function guardMain(args) { + const options = {} + for (let index = 0; index < args.length; index += 2) { + const key = args[index] + const value = args[index + 1] + if (!['--task', '--environment', '--operation', '--head'].includes(key) + || Object.hasOwn(options, key) || typeof value !== 'string' || value.startsWith('--')) { + throw new GuardError('GUARD_ARGUMENTS') + } + options[key] = value + } + if (!options['--task'] || !options['--environment'] || !options['--operation']) { + throw new GuardError('GUARD_ARGUMENTS') + } + const value = await runGuardedOperation({ + task: options['--task'], environment: options['--environment'], + operation: options['--operation'], head: options['--head'], + }) + console.log(JSON.stringify(value)) + return value.ok ? 0 : 1 +} + +export function guardErrorResult(error) { + return { + schemaVersion: 1, kind: 'guarded-operation-result', state: 'blocked', ok: false, + code: error instanceof GuardError ? error.code : 'GUARD_OPERATION_UNAVAILABLE', + executed: error instanceof GuardError ? error.details.executed ?? false : false, + ...(error instanceof GuardError ? error.details : {}), + } +} diff --git a/scripts/verify-agent-contract.mjs b/scripts/verify-agent-contract.mjs index c16e965..0dd4843 100644 --- a/scripts/verify-agent-contract.mjs +++ b/scripts/verify-agent-contract.mjs @@ -148,8 +148,52 @@ export async function verifyAgentContract(root = repositoryRoot) { && acceptance.cleanHeadRequired === true && acceptance.executionVerified === false && acceptance.independenceVerified === false && acceptance.automaticUpload === false, 'acceptance evidence must retain clean-HEAD integrity and explicit unverified execution/independence') + require(JSON.stringify(acceptance.liveUi) === JSON.stringify({ + schemaVersion: 2, ci: 'guarded-exact-head', source: 'report-consistency-only', + browserManualRequireLiveContract: true, + upgradeSource: 'exact-github-release-url-or-verified-cache', + appLifecycle: 'fresh-discovery-consented-launch-safe-skip-rediscovery-final', + disposition: 'separate-explicit-residual-risk-no-required-waiver', + automaticClosure: false, + maxNavigationMethods: 2, maxObservationMs: 120000, + }), 'live UI acceptance must retain bounded navigation, exact-head CI and unverified report semantics') await file(acceptance.policy) await file(acceptance.implementation) + const executionPolicy = JSON.parse(await readFile(resolve(root, 'agent-execution-policy.json'), 'utf8')) + require(executionPolicy.schemaVersion === 1 && executionPolicy.repository === 'cloga/dsh-github-copilot' + && executionPolicy.ownerTask === 'tooling' && executionPolicy.maxTaskScopes === 128 + && JSON.stringify(executionPolicy.environments?.['local-auxiliary']?.operations) === JSON.stringify(['scripts-test']) + && JSON.stringify(executionPolicy.environments?.['github-actions-evidence']?.operations) === JSON.stringify(['ci-evidence']) + && executionPolicy.operations?.['dependency-install']?.kind === 'network' + && executionPolicy.operations['dependency-install'].maxFailures === 0 + && executionPolicy.operations?.['scripts-test']?.environment === 'local-auxiliary' + && executionPolicy.operations['scripts-test'].maxFailures === 2 + && JSON.stringify(executionPolicy.operations['scripts-test'].argv) === JSON.stringify([ + 'node', '--test', 'tests/scripts/agent.test.mjs', 'tests/scripts/acceptance.test.mjs', + 'tests/scripts/environment-guardrails.test.mjs', + ]) + && executionPolicy.operations?.['ci-evidence']?.environment === 'github-actions-evidence' + && executionPolicy.operations['ci-evidence'].maxFailures === 2 + && executionPolicy.operations['ci-evidence'].maxApiCalls === 3 + && executionPolicy.operations['ci-evidence'].requestTimeoutMs === 15000 + && executionPolicy.operations['ci-evidence'].workflowPath === '.github/workflows/ci.yml' + && JSON.stringify(executionPolicy.operations['ci-evidence'].requiredJobs) === JSON.stringify([ + 'verify (ubuntu-latest)', 'verify (windows-latest)', + 'Published adapter (ubuntu-latest)', 'Published adapter (windows-latest)', 'release-ready', + ]) + && JSON.stringify(executionPolicy.operations['ci-evidence'].optionalJobs) === JSON.stringify([ + { name: 'release', allowedConclusions: ['success', 'skipped'] }, + ]), 'environment policy must retain explicit environments, refusal, bounded budgets and exact CI jobs') + await file('agent-execution-policy.json') + await file('docs/environment-evidence-guardrails.md') + require(contract.tasks.tooling.read.includes('agent-execution-policy.json') + && contract.tasks.tooling.read.includes('scripts/environment-guard.mjs') + && contract.tasks.tooling.read.includes('docs/environment-evidence-guardrails.md') + && contract.tasks.tooling.tests.includes('tests/scripts/environment-guardrails.test.mjs') + && contract.tasks.tooling.risk.includes('cannot prevent direct shell bypass') + && agentGuide.includes('cannot prevent direct shell bypass') + && agentGuide.includes('Do not retry local dependency restoration'), + 'environment and CI guardrails must remain documented, bounded and registered') require(contract.tasks.tooling.read.includes(acceptance.policy) && contract.tasks.tooling.read.includes(acceptance.implementation) && template.includes('### Acceptance evidence'), 'acceptance evidence entrypoints are missing') @@ -165,7 +209,12 @@ export async function verifyAgentContract(root = repositoryRoot) { } const ciWorkflow = await readFile(resolve(root, '.github/workflows/ci.yml'), 'utf8') const releaseWorkflow = await readFile(resolve(root, '.github/workflows/release.yml'), 'utf8') - require(ciWorkflow.includes('release-ready:') && ciWorkflow.includes('uses: ./.github/workflows/release.yml'), 'main CI must gate and call reusable release delivery') + require(ciWorkflow.includes('verify:') && ciWorkflow.includes('runs-on: ${{ matrix.os }}') + && ciWorkflow.includes('os: [ubuntu-latest, windows-latest]') + && ciWorkflow.includes('name: Published adapter (${{ matrix.os }})') + && ciWorkflow.includes('release-ready:') + && ciWorkflow.includes('uses: ./.github/workflows/release.yml'), + 'CI evidence job names must match the required Windows/Linux matrix and published-adapter checks') require(releaseWorkflow.includes('workflow_call:') && releaseWorkflow.includes('scripts/release-policy.mjs --plan') && releaseWorkflow.includes('scripts/publish-release.mjs'), 'release workflow must plan and publish the exact main commit') require(releaseWorkflow.includes('scripts/publish-npm.mjs') && releaseWorkflow.includes('scripts/prepare-release-artifact.mjs') diff --git a/tests/scripts/acceptance.test.mjs b/tests/scripts/acceptance.test.mjs index f8dccfc..bb3e288 100644 --- a/tests/scripts/acceptance.test.mjs +++ b/tests/scripts/acceptance.test.mjs @@ -5,7 +5,7 @@ import { mkdtemp, mkdir, readFile, writeFile, rm, symlink, link } from 'node:fs/ import { tmpdir } from 'node:os' import { join, resolve } from 'node:path' import { test } from 'node:test' -import { verifyAcceptance } from '../../scripts/acceptance.mjs' +import { validateDisposition, validateLiveUi, verifyAcceptance } from '../../scripts/acceptance.mjs' import { repositoryRoot } from '../../scripts/agent.mjs' import { verifyCurrentCoreTemplate } from '../../scripts/verify-agent-contract.mjs' @@ -17,6 +17,230 @@ const git = (root, ...args) => { return result.stdout.trim() } +const riskDisposition = () => ({ + decision: 'accept-residual-risk', userAuthorized: true, defect: 'specific-fix', + controlledRegression: 'passed', criticalBlocker: false, + productionFaults: [{ scenario: 'timeout', status: 'not-tested', reason: 'no-safe-control' }], +}) + +test('residual production fault uncertainty permits a reviewed closure recommendation, never rewrites coverage', () => { + const result = validateDisposition(riskDisposition(), true) + assert.equal(result.recommendation, 'close-with-residual-risk-and-reopen-on-recurrence') + assert.equal(result.productionFaultCoverage, 'incomplete') + assert.equal(result.requiredChecksWaived, false) + assert.equal(result.rootCauseVerified, false) + assert.equal(result.automaticClosure, false) + assert.equal(result.authorizationVerified, false) +}) + +for (const [name, mutate] of [ + ['no authorization', r => { r.userAuthorized = false }], + ['unconfirmed fix', r => { r.defect = 'unconfirmed' }], + ['controlled regression failed', r => { r.controlledRegression = 'failed' }], + ['controlled regression unrun', r => { r.controlledRegression = 'not-run' }], + ['critical requirement missing', r => { r.criticalBlocker = true }], + ['known reproduction', r => { r.productionFaults[0] = { scenario: 'timeout', status: 'failed', reason: 'observed' } }], + ['explicit keep open', r => { r.decision = 'keep-open' }], +]) { + test(`risk disposition does not excuse ${name}`, () => { + const value = riskDisposition() + mutate(value) + assert.equal(validateDisposition(value, true).recommendation, 'keep-open') + }) +} + +test('risk disposition never waives mandatory acceptance or accepts raw fault evidence', () => { + assert.equal(validateDisposition(riskDisposition(), false).recommendation, 'keep-open') + const value = riskDisposition() + value.productionFaults[0].details = 'PRIVATE_SENTINEL' + assert.throws(() => validateDisposition(value, true), /ACCEPTANCE_INVALID/) +}) + +const liveContract = () => ({ + expectedPluginVersion: '0.1.0-alpha.1', panel: 'models-manage', + checks: [{ id: 'acceptance-binding', action: 'reopen-manage', outcome: 'ready' }], +}) +const timeBase = Date.now() - 10000 +const t = offset => timeBase + offset +const liveReport = () => ({ + appState: 'running', + lifecycle: { + discovery: { state: 'running', atMs: t(500), source: 'list-apps' }, + launch: { state: 'not-needed', atMs: null }, + welcome: { state: 'not-seen', atMs: null }, + rediscovery: { atMs: t(900), source: 'list-apps', window: 'same' }, + final: { atMs: t(1200), source: 'list-apps-and-get-window-state', + appState: 'running', surface: 'target-panel' }, + interaction: 'clear', + }, + desktop: { version: '0.2.1-rc.3', source: 'about', observedAtMs: t(1000) }, + installed: { version: '0.1.0-alpha.1', source: 'disk-manifest', observedAtMs: t(1000) }, + loaded: { version: '0.1.0-alpha.1', source: 'public-runtime-status', observedAtMs: t(1000) }, + upgradeSource: { + kind: 'github-release-url', version: '0.1.0-alpha.1', + url: 'https://github.com/cloga/dsh-github-copilot/releases/download/v0.1.0-alpha.1/dsh-github-copilot-0.1.0-alpha.1.tgz', + sha256: 'a'.repeat(64), releaseSha256: 'a'.repeat(64), fallbackReason: 'none', observedAtMs: t(1000), + }, + navigation: [{ method: 'accessible-control', outcome: 'opened', atMs: t(1001), + observedAtMs: t(1002), observationSource: 'get-window-state' }], + handoff: { state: 'none', panel: 'models-manage' }, + permission: { state: 'safe-read-only', action: 'none' }, + steps: [{ checkId: 'acceptance-binding', action: 'reopen-manage', observation: 'ready', + startedAtMs: t(1003), finishedAtMs: t(1100), observedAtMs: t(1101), observationSource: 'get-window-state' }], +}) + +for (const [name, mutate, code] of [ + ['not run', r => { r.navigation = []; r.steps = [] }, 'LIVE_UI_NOT_RUN'], + ['app stopped', r => { r.appState = 'not-running' }, 'LIVE_UI_APP_UNAVAILABLE'], + ['installed not loaded', r => { r.loaded = { version: null, source: 'unknown', observedAtMs: null } }, + 'LIVE_UI_LOADED_VERSION_UNCONFIRMED'], + ['wrong loaded version', r => { r.loaded.version = '0.1.0-alpha.2' }, 'LIVE_UI_VERSION_PREREQUISITE_UNMET'], + ['automation failure', r => { + r.navigation = [{ method: 'accessible-control', outcome: 'unchanged', atMs: t(1001), + observedAtMs: t(1002), observationSource: 'get-window-state' }, + { method: 'supported-shortcut', outcome: 'no-candidate', atMs: t(1003), + observedAtMs: t(1004), observationSource: 'get-window-state' }] + r.steps = []; r.handoff.state = 'requested' + }, 'LIVE_UI_AUTOMATION_BLOCKED'], + ['missing check', r => { r.steps = [] }, 'LIVE_UI_INCOMPLETE'], + ['wrong action for required check', r => { r.steps[0].action = 'observe-panel' }, 'LIVE_UI_INCOMPLETE'], + ['error', r => { r.steps[0].observation = 'error' }, 'LIVE_UI_OBSERVATION_FAILED'], + ['timeout', r => { r.steps[0].observation = 'timeout' }, 'LIVE_UI_OBSERVATION_FAILED'], + ['no change', r => { r.steps[0].observation = 'unchanged' }, 'LIVE_UI_OBSERVATION_FAILED'], + ['user absent', r => { r.permission.state = 'user-absent' }, 'LIVE_UI_PERMISSION_BLOCKED'], + ['handoff not opened', r => { r.handoff.state = 'requested' }, 'LIVE_UI_HANDOFF_PENDING'], + ['historical discovery without startup approval', r => { + r.lifecycle.discovery.state = 'not-running' + }, 'LIVE_UI_LAUNCH_BLOCKED'], + ['welcome requiring real authentication', r => { + r.lifecycle.welcome = { state: 'auth-required', atMs: t(600) } + }, 'LIVE_UI_AUTH_REQUIRED'], + ['safe welcome skip not yet used', r => { + r.lifecycle.welcome = { state: 'safe-skip-observed', atMs: t(600) } + }, 'LIVE_UI_WELCOME_UNRESOLVED'], + ['launch receipt without new window discovery', r => { + r.lifecycle.rediscovery = { atMs: null, source: 'list-apps', window: 'unobserved' } + }, 'LIVE_UI_WINDOW_UNCONFIRMED'], + ['welcome instead of current main interface', r => { + r.lifecycle.final.surface = 'welcome' + }, 'LIVE_UI_CURRENT_APP_UNCONFIRMED'], + ['current app stopped after past success', r => { + r.lifecycle.final.appState = 'not-running' + }, 'LIVE_UI_CURRENT_APP_UNCONFIRMED'], + ['missing final observation', r => { r.lifecycle.final.atMs = null }, 'LIVE_UI_CURRENT_APP_UNCONFIRMED'], + ['final observation predates action', r => { r.lifecycle.final.atMs = t(1000) }, 'LIVE_UI_CURRENT_APP_UNCONFIRMED'], + ['input interruption', r => { r.lifecycle.interaction = 'interrupted' }, 'LIVE_UI_INPUT_INTERRUPTED'], + ...['local-build', 'source-directory', 'link-worktree', 'unknown'].map(kind => [ + `upgrade from ${kind}`, r => { r.upgradeSource.kind = kind }, 'LIVE_UI_UPGRADE_SOURCE_REJECTED', + ]), + ['unexplained local cache', r => { r.upgradeSource.kind = 'verified-github-cache' }, 'LIVE_UI_UPGRADE_SOURCE_REJECTED'], + ['cache hash mismatch', r => { + r.upgradeSource.kind = 'verified-github-cache'; r.upgradeSource.fallbackReason = 'native-url-rejected' + r.upgradeSource.sha256 = 'b'.repeat(64) + }, 'LIVE_UI_UPGRADE_SOURCE_REJECTED'], + ['missing release digest', r => { r.upgradeSource.releaseSha256 = null }, 'LIVE_UI_UPGRADE_SOURCE_REJECTED'], + ['cache without archive hash', r => { + r.upgradeSource.kind = 'verified-github-cache'; r.upgradeSource.fallbackReason = 'native-url-unavailable' + r.upgradeSource.sha256 = null + }, 'LIVE_UI_UPGRADE_SOURCE_REJECTED'], + ['preferred URL mislabeled as fallback', r => { + r.upgradeSource.fallbackReason = 'native-url-rejected' + }, 'LIVE_UI_UPGRADE_SOURCE_REJECTED'], + ...['restart', 'install', 'account-switch', 'oauth', 'paid-call'].map(action => [ + `permission for ${action}`, r => { r.permission = { state: 'approval-required', action } }, + 'LIVE_UI_PERMISSION_BLOCKED', + ]), +]) { + test(`live consistency rejects ${name} rather than reporting completion`, () => { + const report = liveReport() + mutate(report) + const result = validateLiveUi(liveContract(), report) + assert.equal(result.ok, false) + assert.equal(result.code, code) + assert.equal(result.executionVerified, false) + assert.equal(result.versionFixReportedConsistent, false) + }) +} + +for (const [name, mutate] of [ + ['duplicate navigation method', r => { r.navigation.push({ ...r.navigation[0] }) }], + ['third navigation attempt', r => { r.navigation = Array(3).fill(r.navigation[0]) }], + ['unbounded wait', r => { r.steps[0].finishedAtMs = 200000 }], + ['wrong time order', r => { r.steps[0].finishedAtMs = 1 }], + ['raw account field', r => { r.account = 'PRIVATE_SENTINEL' }], + ['raw observation text', r => { r.steps[0].observation = 'PRIVATE_SENTINEL' }], + ['unknown version source', r => { r.loaded.source = 'installed-means-loaded' }], + ['unsupported activation', r => { r.steps[0].action = 'shell-activate' }], + ['action receipt instead of fresh UI observation', r => { r.steps[0].observationSource = 'action-done' }], + ['immediate diff instead of fresh navigation observation', r => { r.navigation[0].observationSource = 'action-diff' }], + ['stale post-action observation', r => { r.steps[0].observedAtMs = r.steps[0].finishedAtMs }], + ['local source path', r => { r.upgradeSource.url = 'C:\\local\\build.tgz' }], + ['wrong release version URL', r => { r.upgradeSource.url = r.upgradeSource.url.replaceAll('alpha.1', 'alpha.2') }], + ['noncanonical release URL', r => { r.upgradeSource.url = 'https://example.invalid/package.tgz' }], + ['shell launch source', r => { r.lifecycle.launch.state = 'shell-start' }], + ['raw workspace title', r => { r.lifecycle.final.title = 'PRIVATE_SENTINEL' }], + ['launch again when app already running', r => { + r.lifecycle.launch = { state: 'authorized-start-app', atMs: t(550) } + }], +]) { + test(`live consistency rejects ${name} with a fixed diagnostic`, () => { + const report = liveReport() + mutate(report) + assert.throws(() => validateLiveUi(liveContract(), report), /^Error: ACCEPTANCE_INVALID$/) + }) +} + +test('complete bounded live evidence remains a consistency judgment, not execution attestation', () => { + const result = validateLiveUi(liveContract(), liveReport()) + assert.equal(result.ok, true) + assert.equal(result.source, 'report-consistency-only') + assert.equal(result.executionVerified, false) + const contract = liveContract(), report = liveReport() + contract.checks[0].outcome = report.steps[0].observation = 'actionable-failure' + assert.equal(validateLiveUi(contract, report).ok, true) + report.upgradeSource.kind = 'verified-github-cache' + report.upgradeSource.fallbackReason = 'native-url-unavailable' + const cached = validateLiveUi(contract, report) + assert.equal(cached.ok, true) + assert.equal(cached.upgradeSourceKind, 'verified-github-cache') + assert.equal(cached.provenanceVerified, false) +}) + +test('navigation escalation asks for the specified panel once without inferring user response', () => { + const report = liveReport() + report.steps = [] + report.navigation[0].outcome = 'unchanged' + assert.equal(validateLiveUi(liveContract(), report).nextAction, 'try-one-supported-alternate') + report.navigation.push({ method: 'supported-shortcut', outcome: 'no-candidate', + atMs: t(1003), observedAtMs: t(1004), observationSource: 'get-window-state' }) + assert.equal(validateLiveUi(liveContract(), report).nextAction, 'request-specified-panel-once') + report.handoff.state = 'user-absent' + assert.equal(validateLiveUi(liveContract(), report).nextAction, 'wait-for-user-panel') + assert.equal(validateLiveUi(liveContract(), report).ok, false) + const unknown = liveReport() + unknown.loaded = { version: null, source: 'unknown', observedAtMs: null } + const qualified = validateLiveUi(liveContract(), unknown) + assert.equal(qualified.qualifiedObservation, true) + assert.equal(qualified.ok, false) +}) + +test('consented controlled launch and safe welcome skip require rediscovered main window and fresh final state', () => { + const report = liveReport() + report.lifecycle.discovery.state = 'not-running' + report.lifecycle.launch = { state: 'authorized-start-app', atMs: t(550) } + report.lifecycle.welcome = { state: 'safe-skip-used', atMs: t(600) } + report.lifecycle.rediscovery.window = 'replacement' + report.lifecycle.final.surface = 'workspace' + const result = validateLiveUi(liveContract(), report, t(1300)) + assert.equal(result.ok, true) + assert.equal(result.currentAppObservedAtMs, t(1200)) + assert.equal(result.executionVerified, false) + assert.equal(validateLiveUi(liveContract(), report, t(121201)).code, 'LIVE_UI_CURRENT_APP_UNCONFIRMED') + assert.equal(validateLiveUi(liveContract(), report, t(1199)).code, 'LIVE_UI_CURRENT_APP_UNCONFIRMED') + report.lifecycle.launch.state = 'not-authorized'; report.lifecycle.launch.atMs = null + assert.equal(validateLiveUi(liveContract(), report).code, 'LIVE_UI_LAUNCH_BLOCKED') +}) + async function fixture(run) { const root = await mkdtemp(join(tmpdir(), 'copilot-acceptance-')) const checkout = join(root, 'checkout') @@ -77,6 +301,91 @@ test('acceptance verifies exact clean HEAD, contract bytes and local artifact in assert.deepEqual(JSON.parse(invoked.stdout), result) })) +test('live contract composes real CI reader and persistent budget, never a supplied green report', () => fixture(async f => { + git(f.checkout, 'remote', 'add', 'origin', 'https://github.com/cloga/dsh-github-copilot.git') + f.contract.schemaVersion = f.report.schemaVersion = 2 + f.contract.requirements[0].method = 'manual' + f.contract.liveUi = liveContract() + f.report.liveUi = liveReport() + f.report.checks[0].notes = 'observed' + f.report.checks[0].evidence = [] + let apiFailure = false, conclusion = 'success', calls = 0 + const execute = (command, args, options) => { + if (command !== 'gh') return spawnSync(command, args, options) + calls++ + if (apiFailure) return { status: 1, stdout: '' } + const endpoint = args[1] + const value = endpoint.includes('/workflows/') ? { + id: 1, name: 'CI', path: '.github/workflows/ci.yml', state: 'active', + } : endpoint.includes('/jobs?') ? { jobs: [ + 'verify (ubuntu-latest)', 'verify (windows-latest)', 'Published adapter (ubuntu-latest)', + 'Published adapter (windows-latest)', 'release-ready', + ].map(name => ({ name, status: 'completed', conclusion: 'success' })) } : { + workflow_runs: [{ id: 1, workflow_id: 1, run_number: 1, run_attempt: 1, + head_sha: f.report.commit, status: 'completed', conclusion }], + } + return { status: 0, stdout: JSON.stringify(value) } + } + const verify = () => verifyAcceptance({ + root: f.checkout, contractPath: f.contractPath, reportPath: f.reportPath, execute, + }) + await f.save() + const complete = await verify() + assert.equal(complete.ok, true) + assert.equal(complete.ciVerified, true) + assert.equal(complete.executionVerified, false) + assert.equal(calls, 3) + f.report.disposition = riskDisposition() + await f.save() + const residual = await verify() + assert.equal(residual.ok, true) + assert.equal(residual.disposition.productionFaultCoverage, 'incomplete') + assert.equal(residual.disposition.recommendation, 'close-with-residual-risk-and-reopen-on-recurrence') + f.report.disposition.productionFaults[0] = { scenario: 'timeout', status: 'failed', reason: 'observed' } + await f.save() + assert.equal((await verify()).closeEligible, false) + delete f.report.disposition + f.report.checks[0].status = 'not-run' + await f.save() + assert.equal((await verify()).closeEligible, false) + f.report.checks[0].status = 'passed' + f.report.liveUi.steps = [] + await f.save() + const missing = await verify() + assert.equal(missing.ciVerified, true) + assert.equal(missing.ok, false) + assert.equal(missing.closeEligible, false) + f.report.liveUi = liveReport() + f.report.liveUi.loaded = { version: null, source: 'unknown', observedAtMs: null } + await f.save() + const installedOnly = await verify() + assert.equal(installedOnly.ciVerified, true) + assert.equal(installedOnly.closeEligible, false) + assert.equal(installedOnly.liveUi.qualifiedObservation, true) + f.report.liveUi = liveReport() + conclusion = 'failure' + await f.save() + const failedCi = await verify() + assert.equal(failedCi.liveUi.ok, true) + assert.equal(failedCi.ok, false) + assert.equal(failedCi.ci.code, 'CI_RUN_FAILED') + apiFailure = true + assert.equal((await verify()).ci.failedAttempts, 1) + apiFailure = false; conclusion = 'success' + assert.equal((await verify()).ci.failedAttempts, 1) + apiFailure = true + assert.equal((await verify()).ci.failedAttempts, 2) + const before = calls + assert.equal((await verify()).ci.code, 'GUARD_FAILURE_BUDGET_EXHAUSTED') + assert.equal(calls, before) +})) + +test('legacy report cannot claim required live acceptance without a live contract', () => fixture(async f => { + f.contract.requirements[0].method = 'browser' + await f.save() + await assert.rejects(f.verify(), /ACCEPTANCE_LIVE_CONTRACT_REQUIRED/) +})) + for (const status of ['failed', 'blocked', 'not-run']) { test(`acceptance preserves ${status} as a non-passing outcome`, () => fixture(async f => { f.report.checks[0].status = status diff --git a/tests/scripts/agent.test.mjs b/tests/scripts/agent.test.mjs index 0f89969..dd576a9 100644 --- a/tests/scripts/agent.test.mjs +++ b/tests/scripts/agent.test.mjs @@ -19,6 +19,14 @@ test('describes actual package metadata without claiming a release exists', asyn assert.equal(result.acceptance.executed, false) assert.equal(result.acceptance.executionVerified, false) assert.equal(result.acceptance.independenceVerified, false) + assert.deepEqual(result.acceptance.liveUi, { + schemaVersion: 2, ci: 'guarded-exact-head', source: 'report-consistency-only', + browserManualRequireLiveContract: true, automaticClosure: false, + upgradeSource: 'exact-github-release-url-or-verified-cache', + appLifecycle: 'fresh-discovery-consented-launch-safe-skip-rediscovery-final', + disposition: 'separate-explicit-residual-risk-no-required-waiver', + maxNavigationMethods: 2, maxObservationMs: 120000, + }) assert.deepEqual(result.admittedReleases, ['0.2.1-rc.3']) assert.deepEqual(result.baselines.map(baseline => baseline.release), [ '0.1.1-rc.2', '0.1.2-rc.1', '0.1.3-alpha.1', '0.1.5-alpha.1', '0.1.5-alpha.2', '0.1.5-rc.1', '0.1.5-rc.2', '0.1.6-alpha.1', '0.1.6-alpha.2', '0.2.0-rc.1', '0.2.0-rc.2', '0.2.1-alpha.4', '0.2.1-alpha.5', '0.2.1-rc.3', @@ -29,24 +37,41 @@ test('plans only known task IDs with unexecuted argument arrays', async () => { const plan = await planTask('models') assert.ok(plan.commands.every(command => command.executed === false && Array.isArray(command.argv))) for (const testFile of ['tests/account-model-catalog.spec.ts', 'tests/account-model-source.spec.ts', 'tests/preview-route.spec.ts', 'tests/pi-provider-bridge.spec.ts']) { - assert.ok(plan.commands.some(command => command.argv.includes(testFile)), `Missing general model gate: ${testFile}`) + assert.ok(plan.tests.includes(testFile), `Missing general model gate: ${testFile}`) } await assert.rejects(planTask('__proto__'), /Unknown task/) - assert.ok((await planTask('tooling')).commands.some(command => command.argv[1] === 'test:scripts')) + assert.deepEqual((await planTask('tooling')).commands.map(command => command.argv.slice(3)), [ + ['--task', 'TASK_ID', '--environment', 'local-auxiliary', '--operation', 'scripts-test'], + ['--task', 'TASK_ID', '--environment', 'github-actions-evidence', '--operation', 'ci-evidence', + '--head', 'FULL_40_CHARACTER_COMMIT'], + ]) + assert.ok(!(await planTask('tooling')).commands.some(command => command.argv.includes('pnpm'))) assert.deepEqual((await planTask('tooling')).acceptance, (await describeRepository()).acceptance) }) test('context diagnostics preserve shared usage and include native whole-turn evidence', async () => { const plan = await planTask('context') - assert.ok(plan.commands.some(command => command.argv.includes('tests/fixtures/turn-usage-core.fixture.ts'))) + assert.ok(plan.tests.includes('tests/fixtures/turn-usage-core.fixture.ts')) const contract = JSON.parse(await readFile(join(repositoryRoot, 'agent-contract.json'), 'utf8')) assert.ok(contract.tasks.context.risk.includes('Never filter or delay shared native usage')) await verifyAgentContract() }) +test('guarded plans retain required read-only iteration review preflight', async () => { + const plan = await planTask('autorouting') + assert.deepEqual(plan.preflightChecks, [{ + argv: ['node', 'scripts/iteration-review.mjs', '--report', 'docs/auto-iteration-review.json'], + executed: false, + requiredBeforeImplementation: true, + }]) + assert.ok(plan.commands.every(command => command.argv[1] === 'scripts/agent.mjs' + && command.argv[2] === 'guard' && command.executed === false)) + assert.deepEqual((await planTask('tooling')).preflightChecks, []) +}) + test('multi-account plan retains explicit direct-grant ownership and UI boundaries', async () => { const plan = await planTask('accounts') - assert.ok(plan.commands.some(command => command.argv.includes('tests/copilot-accounts-card.spec.ts'))) + assert.ok(plan.tests.includes('tests/copilot-accounts-card.spec.ts')) const contract = JSON.parse(await readFile(join(repositoryRoot, 'agent-contract.json'), 'utf8')) assert.match(contract.tasks.accounts.risk, /never grant copies/) assert.match(contract.tasks.accounts.risk, /Models alone/) @@ -63,7 +88,7 @@ test('multi-account plan retains explicit direct-grant ownership and UI boundari test('upload evidence retains request scope, cleanup and local-only observation boundaries', async () => { const plan = await planTask('authorization') - assert.ok(plan.commands.some(command => command.argv.includes('tests/request-upload-evidence.spec.ts'))) + assert.ok(plan.tests.includes('tests/request-upload-evidence.spec.ts')) const contract = JSON.parse(await readFile(join(repositoryRoot, 'agent-contract.json'), 'utf8')) assert.ok(contract.tasks.authorization.read.includes('src/request-upload-evidence.ts')) assert.match(contract.tasks.authorization.requestBodyEvidenceBoundary, /dispose subscriptions/) @@ -74,7 +99,7 @@ test('upload evidence retains request scope, cleanup and local-only observation test('automatic recovery plan preserves explicit composition and bounded capacity-only escalation', async () => { const plan = await planTask('compaction') for (const file of ['tests/preview-route.spec.ts', 'tests/fixtures/scoped-compaction-core.fixture.ts']) { - assert.ok(plan.commands.some(command => command.argv.includes(file))) + assert.ok(plan.tests.includes(file)) } assert.match(plan.automaticRecoveryBoundary, /Default-on only in the explicitly selected replacement/) assert.match(plan.automaticRecoveryBoundary, /at most 16 calls including the failed first attempt/) @@ -99,10 +124,9 @@ test('important updates carry release follow-through without a second approval p assert.deepEqual(policy.requiredConditions, ['reviewed-merge', 'green-required-ci', 'fresh-annotated-tag', 'verified-release-assets']) assert.equal(policy.localGateWhenRegistryBlocked, 'record-environment-blocker-and-require-complete-github-ci') assert.deepEqual(policy.completionEvidence, ['published-release-url', 'tag-and-commit', 'asset-and-sha256', 'npm-version-and-integrity']) - assert.ok(plan.commands.some(command => command.argv[0] === 'node' && command.argv[1] === '--test' - && command.argv.includes('tests/scripts/npm-distribution.test.mjs'))) - assert.ok(!plan.commands.some(command => command.argv.includes('vitest') - && command.argv.some(arg => arg.startsWith('tests/scripts/')))) + assert.ok(plan.tests.includes('tests/scripts/npm-distribution.test.mjs')) + assert.ok(plan.commands.every(command => command.argv[1] === 'scripts/agent.mjs' + && command.argv[2] === 'guard')) for (const boundary of ['merge', 'release']) { assert.ok(!plan.boundaries.approvalRequired.includes(boundary)) } @@ -230,6 +254,7 @@ test('every task plan carries the plugin-only implementation boundary', async () } for (const task of Object.keys(description.tasks)) { assert.deepEqual((await planTask(task)).boundaries.implementationScope, policy) + assert.ok((await planTask(task)).commands.every(command => command.argv[2] === 'guard')) } }) @@ -268,13 +293,16 @@ test('attribution names the actual tool without inventing a co-author email', () test('doctor fails missing dependencies but reports build and runtime evidence honestly', async () => { const root = await mkdtemp(join(tmpdir(), 'copilot-doctor-')) try { - for (const path of ['agent-contract.json', 'package.json', 'deployment-baseline.json']) { + for (const path of ['agent-contract.json', 'package.json', 'deployment-baseline.json', 'agent-execution-policy.json']) { await writeFile(join(root, path), await readFile(join(repositoryRoot, path))) } const missing = await doctor(root, '22.0.0') assert.equal(missing.ok, false) assert.equal(missing.checks.find(check => check.id === 'node-runtime').status, 'fail') assert.equal(missing.checks.find(check => check.id === 'build:lib/index.js').status, 'warning') + assert.match(missing.next, /guarded environment/) + assert.ok(missing.checks.filter(check => check.id.startsWith('dependency:')) + .every(check => !check.next.includes('pnpm install'))) for (const name of ['typescript', 'vitest', 'tsdown', '@earendil-works/pi-ai', '@deepseek-ai/dsh-llm-pi-ai']) { const path = join(root, 'node_modules', name) await mkdir(path, { recursive: true }) diff --git a/tests/scripts/environment-guardrails.test.mjs b/tests/scripts/environment-guardrails.test.mjs new file mode 100644 index 0000000..537903b --- /dev/null +++ b/tests/scripts/environment-guardrails.test.mjs @@ -0,0 +1,351 @@ +import assert from 'node:assert/strict' +import { spawnSync } from 'node:child_process' +import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { test } from 'node:test' +import { runGuardedOperation } from '../../scripts/environment-guard.mjs' + +const repository = 'cloga/dsh-github-copilot' +const requiredJobs = [ + 'verify (ubuntu-latest)', + 'verify (windows-latest)', + 'Published adapter (ubuntu-latest)', + 'Published adapter (windows-latest)', + 'release-ready', +] +const git = (cwd, ...args) => { + const result = spawnSync('git', ['-C', cwd, ...args], { encoding: 'utf8' }) + assert.equal(result.status, 0, result.stderr) + return result.stdout.trim() +} + +async function checkout(run) { + const root = await mkdtemp(join(tmpdir(), 'copilot-env-guard-')) + try { + git(root, 'init', '--quiet') + git(root, 'config', 'user.name', 'Environment guard fixture') + git(root, 'config', 'user.email', 'fixture@example.invalid') + await writeFile(join(root, 'source.txt'), 'fixture\n') + git(root, 'add', 'source.txt') + git(root, '-c', 'commit.gpgsign=false', 'commit', '--quiet', '-m', 'fixture') + git(root, 'remote', 'add', 'origin', `https://github.com/${repository}.git`) + const head = git(root, 'rev-parse', 'HEAD') + await run({ root, head }) + } finally { + await rm(root, { recursive: true, force: true }) + } +} + +function ciPayload({ workflow = {}, run = {}, jobs = [], includeRelease = true } = {}) { + return { + workflow: { + id: 42, name: 'CI', path: '.github/workflows/ci.yml', state: 'active', ...workflow, + }, + runs: { + workflow_runs: [{ + id: 100, workflow_id: 42, head_sha: '', status: 'completed', conclusion: 'success', + run_number: 12, run_attempt: 1, event: 'pull_request', + html_url: 'https://github.com/cloga/dsh-github-copilot/actions/runs/100', + ...run, + }], + }, + jobs: { + jobs: [ + ...requiredJobs.map(name => ({ name, status: 'completed', conclusion: 'success' })), + ...(includeRelease ? [{ name: 'release', status: 'completed', conclusion: 'skipped' }] : []), + ...jobs, + ], + }, + } +} + +function ciExecutor(payload, calls = []) { + return (command, args, options) => { + if (command === 'git') return spawnSync(command, args, options) + assert.equal(command, 'gh') + calls.push({ args, timeout: options.timeout }) + const endpoint = args[1] + const body = endpoint.endsWith('/actions/workflows/ci.yml') ? payload.workflow + : endpoint.includes('/actions/runs?') ? payload.runs + : endpoint.includes('/actions/runs/100/jobs?') ? payload.jobs + : null + if (body === null) return { status: 1, stdout: '', stderr: '' } + if (endpoint.includes('/actions/runs?') && payload.runs.workflow_runs[0]?.head_sha === '') { + payload.runs.workflow_runs[0].head_sha = payload.head + } + return { status: 0, stdout: JSON.stringify(body), stderr: '' } + } +} + +test('local dependency-network operation fails before executing any process', async () => checkout(async ({ root }) => { + let executions = 0 + const result = await runGuardedOperation({ + root, task: '451', environment: 'local-auxiliary', operation: 'dependency-install', + execute: () => { executions++; throw new Error('must not execute') }, + }) + assert.equal(executions, 0) + assert.equal(result.state, 'blocked') + assert.equal(result.code, 'GUARD_LOCAL_NETWORK_BLOCKED') + assert.equal(result.executed, false) +})) + +test('failed operation budget persists across calls and blocks before the next execution', async () => checkout(async ({ root }) => { + let executions = 0 + const execute = (command, args, options) => { + if (command === 'git') return spawnSync(command, args, options) + executions++ + return { status: 1, error: null, stdout: '', stderr: '' } + } + const invoke = () => runGuardedOperation({ + root, task: '451', environment: 'local-auxiliary', operation: 'scripts-test', execute, + }) + const first = await invoke() + const second = await invoke() + const blocked = await invoke() + assert.deepEqual([first.failedAttempts, second.failedAttempts], [1, 2]) + assert.equal(first.state, 'failed') + assert.equal(second.remainingFailures, 0) + assert.equal(blocked.code, 'GUARD_FAILURE_BUDGET_EXHAUSTED') + assert.equal(blocked.executed, false) + assert.equal(executions, 2) + const statePath = git(root, 'rev-parse', '--git-path', 'copilot-agent-guard-state.json') + const state = JSON.parse(await readFile(join(root, statePath), 'utf8')) + assert.equal(state.failures['451:scripts-test'], 2) +})) + +test('task and operation scopes do not share failure counts', async () => checkout(async ({ root }) => { + let executions = 0 + const execute = (command, args, options) => command === 'git' + ? spawnSync(command, args, options) + : (++executions, { status: 1, stdout: '', stderr: '' }) + await runGuardedOperation({ + root, task: '451', environment: 'local-auxiliary', operation: 'scripts-test', execute, + }) + const otherTask = await runGuardedOperation({ + root, task: '452', environment: 'local-auxiliary', operation: 'scripts-test', execute, + }) + assert.equal(otherTask.failedAttempts, 1) + assert.equal(executions, 2) +})) + +test('CI evidence accepts official Windows/Linux verification and an expected skipped release', async () => checkout(async ({ root, head }) => { + const calls = [] + const payload = ciPayload({ run: { head_sha: head } }) + payload.head = head + const result = await runGuardedOperation({ + root, task: '451', environment: 'github-actions-evidence', operation: 'ci-evidence', + head, execute: ciExecutor(payload, calls), + }) + assert.equal(result.state, 'passed') + assert.equal(result.code, 'CI_EVIDENCE_VERIFIED') + assert.equal(result.evidence.repository, repository) + assert.equal(result.evidence.head, head) + assert.equal(result.evidence.workflow.path, '.github/workflows/ci.yml') + assert.deepEqual(result.evidence.jobs.map(job => job.name), [...requiredJobs, 'release']) + assert.equal(result.evidence.jobs.at(-1).conclusion, 'skipped') + assert.equal(calls.length, 3) + assert.ok(calls.every(call => call.args[0] === 'api' && !call.args.includes('--paginate') + && call.timeout === 15000)) +})) + +for (const [label, configure, expectedCode] of [ + ['missing run', payload => { payload.runs.workflow_runs = [] }, 'CI_RUN_MISSING'], + ['old head', payload => { payload.runs.workflow_runs[0].head_sha = 'b'.repeat(40) }, 'CI_RUN_MISSING'], + ['wrong workflow', payload => { payload.runs.workflow_runs[0].workflow_id = 99 }, 'CI_WORKFLOW_MISMATCH'], + ['wrong workflow path', payload => { payload.workflow.path = '.github/workflows/release.yml' }, 'CI_WORKFLOW_MISMATCH'], + ['failed run', payload => { payload.runs.workflow_runs[0].conclusion = 'failure' }, 'CI_RUN_FAILED'], + ['cancelled run', payload => { payload.runs.workflow_runs[0].conclusion = 'cancelled' }, 'CI_RUN_CANCELLED'], + ['in-progress run', payload => { payload.runs.workflow_runs[0].status = 'in_progress' }, 'CI_RUN_IN_PROGRESS'], + ['skipped required job', payload => { payload.jobs.jobs[0].conclusion = 'skipped' }, 'CI_REQUIRED_JOB_SKIPPED'], + ['failed required job', payload => { payload.jobs.jobs[1].conclusion = 'failure' }, 'CI_REQUIRED_JOB_FAILED'], + ['cancelled required job', payload => { payload.jobs.jobs[1].conclusion = 'cancelled' }, 'CI_REQUIRED_JOB_CANCELLED'], + ['missing required job', payload => { payload.jobs.jobs = payload.jobs.jobs.filter(job => job.name !== requiredJobs[0]) }, 'CI_REQUIRED_JOB_MISSING'], + ['duplicate required job', payload => { payload.jobs.jobs.push({ ...payload.jobs.jobs[0] }) }, 'CI_REQUIRED_JOB_DUPLICATE'], + ['in-progress required job', payload => { payload.jobs.jobs[0].status = 'in_progress' }, 'CI_REQUIRED_JOB_IN_PROGRESS'], + ['failed optional release', payload => { payload.jobs.jobs.at(-1).conclusion = 'failure' }, 'CI_RELEASE_JOB_FAILED'], +]) { + test(`CI evidence rejects ${label}`, async () => checkout(async ({ root, head }) => { + const payload = ciPayload({ run: { head_sha: head } }) + payload.head = head + configure(payload) + const result = await runGuardedOperation({ + root, task: '451', environment: 'github-actions-evidence', operation: 'ci-evidence', + head, execute: ciExecutor(payload), + }) + assert.equal(result.state, 'blocked') + assert.equal(result.code, expectedCode) + })) +} + +test('CI evidence rejects a changed local HEAD before contacting GitHub', async () => checkout(async ({ root, head }) => { + await writeFile(join(root, 'source.txt'), 'changed\n') + git(root, 'add', 'source.txt') + git(root, '-c', 'commit.gpgsign=false', 'commit', '--quiet', '-m', 'move HEAD') + let ghCalls = 0 + const result = await runGuardedOperation({ + root, task: '451', environment: 'github-actions-evidence', operation: 'ci-evidence', + head, execute: (command, args, options) => command === 'git' + ? spawnSync(command, args, options) : (++ghCalls, { status: 1, stdout: '', stderr: '' }), + }) + assert.equal(result.code, 'CI_HEAD_MISMATCH') + assert.equal(result.executed, true) + assert.equal(ghCalls, 0) +})) + +test('CI evidence rejects a repository that does not match the configured origin', async () => checkout(async ({ root, head }) => { + git(root, 'remote', 'set-url', 'origin', 'https://github.com/example/another-repo.git') + let ghCalls = 0 + const result = await runGuardedOperation({ + root, task: '451', environment: 'github-actions-evidence', operation: 'ci-evidence', + head, execute: (command, args, options) => command === 'git' + ? spawnSync(command, args, options) : (++ghCalls, { status: 1, stdout: '', stderr: '' }), + }) + assert.equal(result.code, 'CI_REPOSITORY_MISMATCH') + assert.equal(ghCalls, 0) +})) + +test('CI evidence accepts a successful run when the optional release job is absent', async () => checkout(async ({ root, head }) => { + const payload = ciPayload({ run: { head_sha: head }, includeRelease: false }) + payload.head = head + const result = await runGuardedOperation({ + root, task: '451', environment: 'github-actions-evidence', operation: 'ci-evidence', + head, execute: ciExecutor(payload), + }) + assert.equal(result.state, 'passed') + assert.equal(result.evidence.jobs.length, requiredJobs.length) +})) + +test('CI query failures consume a persistent budget and stop before a third query', async () => checkout(async ({ root, head }) => { + let ghCalls = 0 + const execute = (command, args, options) => { + if (command === 'git') return spawnSync(command, args, options) + ghCalls++ + return { status: 1, stdout: '', stderr: '' } + } + const invoke = () => runGuardedOperation({ + root, task: '451', environment: 'github-actions-evidence', operation: 'ci-evidence', + head, execute, + }) + const first = await invoke() + const second = await invoke() + const third = await invoke() + assert.equal(first.code, 'CI_GITHUB_READ_UNAVAILABLE') + assert.equal(second.code, 'CI_GITHUB_READ_UNAVAILABLE') + assert.equal(second.remainingFailures, 0) + assert.equal(third.code, 'GUARD_FAILURE_BUDGET_EXHAUSTED') + assert.equal(third.executed, false) + assert.equal(ghCalls, 2) +})) + +for (const [status, conclusion, code] of [ + ['completed', 'failure', 'CI_RUN_FAILED'], + ['in_progress', null, 'CI_RUN_IN_PROGRESS'], +]) { + test(`an older successful rerun cannot mask a newer ${status} run`, () => checkout(async ({ root, head }) => { + const payload = ciPayload({ run: { head_sha: head, run_attempt: 2 } }) + payload.runs.workflow_runs.push({ + ...payload.runs.workflow_runs[0], + id: 101, run_number: 13, run_attempt: 1, status, conclusion, + }) + const calls = [] + const result = await runGuardedOperation({ + root, task: '451', environment: 'github-actions-evidence', operation: 'ci-evidence', + head, execute: ciExecutor(payload, calls), + }) + assert.equal(result.state, 'blocked') + assert.equal(result.code, code) + assert.equal(calls.length, 2) + })) +} + +for (const [label, prepare, code] of [ + ['missing run', payload => { payload.runs.workflow_runs = [] }, 'CI_RUN_MISSING'], + ['queued run', payload => { + payload.runs.workflow_runs[0].status = 'queued' + payload.runs.workflow_runs[0].conclusion = null + }, 'CI_RUN_IN_PROGRESS'], + ['running required job', payload => { + payload.jobs.jobs[0].status = 'in_progress' + payload.jobs.jobs[0].conclusion = null + }, 'CI_REQUIRED_JOB_IN_PROGRESS'], +]) { + test(`${label} does not consume failure budget or prevent later successful evidence`, () => checkout(async ({ root, head }) => { + const payload = ciPayload({ run: { head_sha: head } }) + const original = structuredClone(payload) + prepare(payload) + const invoke = () => runGuardedOperation({ + root, task: '451', environment: 'github-actions-evidence', operation: 'ci-evidence', + head, execute: ciExecutor(payload), + }) + for (let count = 0; count < 3; count++) { + const pending = await invoke() + assert.equal(pending.state, 'blocked') + assert.equal(pending.code, code) + assert.equal(pending.evidenceState, 'not-ready') + assert.equal(pending.failedAttempts, 0) + assert.equal(pending.remainingFailures, 2) + } + Object.assign(payload, original) + const completed = await invoke() + assert.equal(completed.state, 'passed') + assert.equal(completed.failedAttempts, 0) + })) +} + +test('rejected CI conclusions stay blocked without consuming execution failure budget', () => checkout(async ({ root, head }) => { + const payload = ciPayload({ run: { head_sha: head, conclusion: 'failure' } }) + const invoke = () => runGuardedOperation({ + root, task: '451', environment: 'github-actions-evidence', operation: 'ci-evidence', + head, execute: ciExecutor(payload), + }) + for (let count = 0; count < 3; count++) { + const result = await invoke() + assert.equal(result.code, 'CI_RUN_FAILED') + assert.equal(result.evidenceState, 'rejected') + assert.equal(result.ok, false) + assert.equal(result.failedAttempts, 0) + } + payload.runs.workflow_runs[0].conclusion = 'success' + assert.equal((await invoke()).state, 'passed') +})) + +test('attempt ordering applies only within the same CI run', () => checkout(async ({ root, head }) => { + const payload = ciPayload({ run: { head_sha: head, run_attempt: 1 } }) + payload.runs.workflow_runs.push({ + ...payload.runs.workflow_runs[0], run_attempt: 2, conclusion: 'failure', + }) + const result = await runGuardedOperation({ + root, task: '451', environment: 'github-actions-evidence', operation: 'ci-evidence', + head, execute: ciExecutor(payload), + }) + assert.equal(result.code, 'CI_RUN_FAILED') + assert.equal(result.ok, false) +})) + +test('pending and successful evidence reads do not reset an earlier API failure', () => checkout(async ({ root, head }) => { + const payload = ciPayload({ run: { head_sha: head } }) + let unavailable = true + const nativeExecute = ciExecutor(payload) + const execute = (command, args, options) => command === 'gh' && unavailable + ? { status: 1, stdout: '', stderr: '' } : nativeExecute(command, args, options) + const invoke = () => runGuardedOperation({ + root, task: '451', environment: 'github-actions-evidence', operation: 'ci-evidence', + head, execute, + }) + assert.equal((await invoke()).failedAttempts, 1) + unavailable = false + payload.runs.workflow_runs[0].status = 'in_progress' + const pending = await invoke() + assert.equal(pending.evidenceState, 'not-ready') + assert.equal(pending.failedAttempts, 1) + payload.runs.workflow_runs[0].status = 'completed' + const passed = await invoke() + assert.equal(passed.ok, true) + assert.equal(passed.failedAttempts, 1) + unavailable = true + assert.equal((await invoke()).failedAttempts, 2) + const blocked = await invoke() + assert.equal(blocked.code, 'GUARD_FAILURE_BUDGET_EXHAUSTED') + assert.equal(blocked.executed, false) +})) diff --git a/tests/scripts/iteration-review.test.mjs b/tests/scripts/iteration-review.test.mjs index 838cd9f..c2965e9 100644 --- a/tests/scripts/iteration-review.test.mjs +++ b/tests/scripts/iteration-review.test.mjs @@ -30,11 +30,11 @@ test('the current descriptive assessment review retains insufficient evidence an assert.equal(result.evidenceStatus, 'insufficient-sample') assert.equal(result.sampleCount, 63) assert.equal(result.evidenceTruthVerified, false) - assert.match(report.observation.limitations, /No Session IDs or sequence in aggregates/) - assert.equal(report.policyVersion, 'high-cost-v4') - assert.match(report.decision, /All hard-eligible fitting models participate/) - assert.match(report.decision, /uncalibrated biased trial coefficients/) - assert.match(report.decision, /without task-failure inference, extra calls or in-turn switches/) + assert.match(report.observation.limitations, /No Session IDs or sequence in aggregates/) + assert.equal(report.policyVersion, 'high-cost-v4') + assert.match(report.decision, /All hard-eligible fitting models participate/) + assert.match(report.decision, /uncalibrated biased trial coefficients/) + assert.match(report.decision, /without task-failure inference, extra calls or in-turn switches/) assert.match(report.followUp.trigger, /200 new decisions/) assert.match(report.followUp.action, /#334/) }) @@ -49,9 +49,11 @@ test('observed and insufficient evidence need ordered windows and positive sampl }) test('Auto plans begin with an unexecuted previous-review prerequisite', async () => { const plan = await planTask('autorouting') - assert.deepEqual(plan.commands[0], { argv: [ + assert.deepEqual(plan.preflightChecks[0], { argv: [ 'node', 'scripts/iteration-review.mjs', '--report', 'docs/auto-iteration-review.json', - ], executed: false }) + ], executed: false, requiredBeforeImplementation: true }) + assert.ok(plan.commands.every(command => command.argv[1] === 'scripts/agent.mjs' + && command.argv[2] === 'guard' && command.executed === false)) }) test('contract rejects disabled prerequisites, missing follow-up or automatic uploads', async () => { const root = await mkdtemp(join(tmpdir(), 'iteration-contract-'))