diff --git a/.github/skills/references/harness-tool-contract.md b/.github/skills/references/harness-tool-contract.md index 5c4454b4..b681ab0d 100644 --- a/.github/skills/references/harness-tool-contract.md +++ b/.github/skills/references/harness-tool-contract.md @@ -49,11 +49,13 @@ Installed to `~/.copilot/bin/harness` on every `harness install`. Add to PATH wi | `1` | Fail — stop before `editFiles` or compound | | `2` | Warn — may proceed with Activity log (strict profile: treat as block) | +**Envelope versioning (Phase 1 workbench, additive).** `--output` is a lane-bearing, opt-in surface, not a blanket capability every command gained — only the commands named below accept it; every other registered command now REJECTS `--output` with a structured `E_USAGE` error (exit 2) naming the lane-bearing commands, instead of silently ignoring it and rendering plain ledger output (fixed post-P1.6; see `lib/registry.mjs`'s `assertLaneSupported`). The lane-bearing set today: `orient`, `learnings`, and `status` accept `--output json-envelope|agent` (`--output=value` works too) — `json-envelope` wraps the command's result in a versioned `{schema, command, status, ...}` envelope, `agent` renders a budgeted plain-text summary for LLM callers; `verify` additionally accepts `--output jsonl`, streaming row-per-event with a terminal `result` row (no `json-envelope`/`agent` support of its own). `status` in all of these uses one shared vocabulary — `ok\|failed\|cancelled\|timed-out`, plus `blocked` where gate-like semantics already use it. These are opt-in, additive lanes: every legacy shape documented below (the default ledger render and `--json`) is **unversioned and unchanged** — no `schema` field, no wrapper, byte-identical output. Do not rely on `--output` for a command not named above — it will error, not silently no-op; a command's help (`harness help `) also states whether it supports a lane. + ## Command catalog -This table tracks only what differs in runtime character across commands — which turn a command runs on, whether it writes a lifecycle event, whether it mutates anything. Sigs and flags: `harness help ` (the CLI CATALOG is the single source of truth). +This table tracks only what differs in runtime character across commands — which turn a command runs on, whether it writes its own DOMAIN-SPECIFIC lifecycle event (the closed vocabulary in `lib/events.mjs#EVENT_TYPES` — `orient`, `gate`, `verify`, `compound`, `consolidate`, `remember`, `learning`, `knowledge`, plus the hook-only `session_start`/`pre_tool`/`post_tool`/`skill_activation`/`session_end`), and whether it mutates anything. This is separate from the dispatch-level `command.start`/`command.result` bracketing every registered command now also gets — see the footnote below the table. Sigs and flags: `harness help ` (the registry, `lib/registry.mjs`, is the single source of truth — the hand-written CLI CATALOG this line used to reference was retired in P1.6; `harness help`/`harness help ` render live from the same registry entries this doc is generated by hand from, so treat any drift you notice as a doc bug, not a second source of truth). -| Command | Tier | Events | Store | +| Command | Tier | Events (domain-specific) | Store | |---------|------|--------|-------| | `install` / `upgrade` | human/CI | none | mutates `~/.copilot/` | | `doctor` | human/CI | none | read-only (`--host vscode` runs an isolated hook-lifecycle fixture) | @@ -74,10 +76,14 @@ This table tracks only what differs in runtime character across commands — whi | `learnings` | agent-runtime | none | read-only | | `knowledge` | agent-runtime | writes | mutates `config.json`, cascade-deletes, or mirrors to the product repo | | `eval-knowledge` | agent-runtime | none | read-only | -| `events` | agent-runtime | none | read-only | +| `events` | agent-runtime | none³ | read-only | | `report` | agent-runtime | none | read-only (`--sync` writes `~/.harness/telemetry/`) | -¹ `init-repo`/`recall`/`validate-plan`/`index` historically called `writeEvent` (types `init_repo`/`recall`/`validate_plan`/`index`) while those four type strings were absent from the `EVENT_TYPES` allow-list (`events.mjs`), so the calls silently no-opped. The allow-list now includes all four (harness evolution Phase 1 hygiene) — the events record in `events.jsonl` like every other lifecycle write. +¹ `init-repo`/`recall`/`validate-plan`/`index` historically called `writeEvent` (types `init_repo`/`recall`/`validate_plan`/`index`) while those four type strings were absent from the `EVENT_TYPES` allow-list (`events.mjs`), so the calls silently no-opped. The allow-list now includes all four (harness evolution Phase 1 hygiene) — the events record in `events.jsonl` like every other lifecycle write, in ADDITION to the dispatch-level bracketing of footnote ². + +² **Every registered command dispatch (P1.6, `lib/registry.mjs`) is bracketed with `command.start`/`command.result` telemetry, independent of the domain-specific "Events" column above** — this is a dispatch-pipeline concern, not a per-command opt-in, so it applies uniformly regardless of whether that column says "writes" or "none". The one exception is `events` itself (footnote ³). A real, observable side effect: running ANY registered `harness` command — including ones whose Events column says "none", like `get`, `learnings`, `report`, or `eval-knowledge` — creates/appends `.harness/events.jsonl` under whatever `--workspace` resolves to (default: the current directory), in ANY cwd, even one with no other `.harness/` state. Opt out with `--no-events`, `--dry-run`, or `HARNESS_NO_EVENTS=1` (all three short-circuit `lib/events.mjs#writeEvent` before anything is written, for both the domain-specific events above and this dispatch-level bracketing). + +³ `events` itself is the one command excluded from footnote ² (`entry.instrument === false`, `lib/registry.mjs`): its own handler's job is reading and summarizing everything already in `events.jsonl`, so bracketing its own dispatch would append that very invocation's own `command.start` to the file an instant before the handler reads it back — a self-referential read-your-own-write that would inflate `harness events`'s own totals on every call. `report` opts out for the same reason (it reads `events.jsonl` via `loadReportEvents`, and `--sync` would copy the phantom row into the global store). **Query construction (deterministic-retrieval discipline):** build `--query` from the user's salient nouns and identifiers **verbatim** (e.g. `SYSTEM-OVERRIDE`, `payment`, `token`) — do not paraphrase intent into synonyms. The retrieval tokenizer normalizes identifier formats and morphology, but it cannot recover a term the query never contained. Passing the literal request terms is what keeps recall stable across phrasings. @@ -331,7 +337,7 @@ Every check in the `verify` payload carries its effective `severity`; non-passin } ``` -Lifecycle events are limited to `session_start`, `orient`, `gate`, `pre_tool`, `post_tool`, `skill_activation`, `verify`, `compound`, `consolidate`, `remember`, `learning`, `knowledge`, `session_end`, `init_repo`, `recall`, `validate_plan`, and `index` (the last four were formerly dropped by the allow-list despite their call sites — fixed as harness evolution Phase 1 hygiene; see the Command catalog table's footnote). Non-lifecycle commands `get`, `report`, `learnings`, and `eval-knowledge` never append events by design — they never call `writeEvent` at all. Every append-attempting command never stores prompt or query content; `skill_activation` stores only the skill and session binding. +DOMAIN-specific lifecycle events (the `EVENT_TYPES` allow-list a command names for its OWN event, distinct from the dispatch-level `command.start`/`command.result` bracketing — see the Command catalog table's footnote ²) are limited to `session_start`, `orient`, `gate`, `pre_tool`, `post_tool`, `skill_activation`, `verify`, `compound`, `consolidate`, `remember`, `learning`, `knowledge`, `session_end`, `init_repo`, `recall`, `validate_plan`, and `index` (the last four were formerly dropped by the allow-list despite their call sites — fixed as harness evolution Phase 1 hygiene; see the Command catalog table's footnote ¹). The dispatch pipeline adds three more types of its own — `command.start`, `command.result`, and `agent_lane` (P1.6, `lib/registry.mjs` / `lib/agent-lane.mjs`) — which are never named by a command. Non-lifecycle commands `get`, `report`, `learnings`, and `eval-knowledge` never call `writeEvent` under their OWN domain type — but post-P1.6 they are not event-free: every registered command's dispatch is still bracketed with `command.start`/`command.result` regardless (footnote ² again), so all four DO append to `events.jsonl`, just never under a `get`/`report`/`learnings`/`eval-knowledge`-named event. Every append-attempting command never stores prompt or query content; `skill_activation` stores only the skill and session binding. ## Host hook boundary diff --git a/.github/workflows/harness-windows.yml b/.github/workflows/harness-windows.yml new file mode 100644 index 00000000..75fb356d --- /dev/null +++ b/.github/workflows/harness-windows.yml @@ -0,0 +1,156 @@ +# Exercises the harness CLI on a real Windows runner. The repository's primary +# consumption platform is Windows (AGENTS.md), but every check until now ran on +# ubuntu-latest, so the win32 branches shipped untested: runner.mjs terminates +# descendants with `taskkill /T /F` instead of a POSIX process group, paths.mjs +# resolves LOCALAPPDATA/USERPROFILE, style.mjs only enables Unicode glyphs when +# a modern terminal announces itself, and global-bin.mjs writes a .cmd shim. +# +# Output is captured verbatim to artifacts so the real console rendering — not +# a reconstruction of it — can be read after the fact. +name: Harness on Windows + +on: + workflow_dispatch: + push: + branches: ['feat/**'] + +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + windows: + runs-on: windows-latest + steps: + - uses: actions/checkout@v4 + with: + persist-credentials: false + + - uses: actions/setup-node@v4 + with: + node-version: '22' + cache: npm + cache-dependency-path: packages/harness/package-lock.json + + - name: Install harness dependencies + run: npm ci --prefix packages/harness + + # The suite has never run on Windows. Failures here are the point of the + # job, so they must not hide the CLI scenarios below. + - name: Harness test suite (Windows) + id: suite + continue-on-error: true + run: npm --prefix packages/harness test + + - name: CLI scenarios + continue-on-error: true + shell: pwsh + run: | + $ErrorActionPreference = 'Continue' + New-Item -ItemType Directory -Force -Path win-output | Out-Null + $ws = Join-Path $env:RUNNER_TEMP 'ws' + $ch = Join-Path $env:RUNNER_TEMP 'copilot' + New-Item -ItemType Directory -Force -Path $ws, $ch | Out-Null + git -C $ws init -q 2>&1 | Out-Null + git -C $ws -c user.email=ci@example.com -c user.name=ci commit -q --allow-empty -m init 2>&1 | Out-Null + + function Probe($name, $cmdArgs) { + $header = "`n=== $name ===`n`$ harness $($cmdArgs -join ' ')" + Write-Host $header + $out = & node packages/harness/bin/harness.mjs @cmdArgs 2>&1 | Out-String + $code = $LASTEXITCODE + Write-Host $out + Write-Host "[exit $code]" + Add-Content -Path win-output/scenarios.txt -Value "$header`n$out[exit $code]" + } + + Probe 'help — design system on the Windows console' @('help') + Probe 'status — ledger rendering' @('status', '--workspace', $ws, '--copilot-home', $ch) + Probe 'status --json — legacy machine shape' @('status', '--json', '--workspace', $ws, '--copilot-home', $ch) + Probe 'status --output json-envelope — versioned envelope lane' @('status', '--output', 'json-envelope', '--workspace', $ws, '--copilot-home', $ch) + Probe 'status --output agent — budgeted agent lane' @('status', '--output', 'agent', '--workspace', $ws, '--copilot-home', $ch) + Probe 'orient' @('orient', '--query', 'windows smoke', '--workspace', $ws, '--copilot-home', $ch) + Probe 'index --structural (from main #42)' @('index', '--structural', '--workspace', $ws, '--copilot-home', $ch) + Probe 'knowledge status (from main #42)' @('knowledge', 'status', '--workspace', $ws, '--copilot-home', $ch) + Probe 'unknown flag — strict validation, expect E_USAGE exit 2' @('status', '--bogus', '--workspace', $ws) + Probe 'missing required arg — expect E_USAGE exit 2' @('recall', '--workspace', $ws, '--copilot-home', $ch) + Probe 'literal boundary — --json after -- must stay literal' @('status', '--workspace', $ws, '--copilot-home', $ch, '--', '--json') + + # Redaction and descendant termination are the two win32 paths with the + # least coverage, so they are exercised directly rather than inferred. + - name: Redaction on Windows + continue-on-error: true + shell: pwsh + env: + MY_API_TOKEN: 'ghp_windowssmoketest0123456789abcdefghij' + run: | + $ws = Join-Path $env:RUNNER_TEMP 'ws' + $out = & node packages/harness/bin/harness.mjs learnings --why $env:MY_API_TOKEN --workspace $ws 2>&1 | Out-String + Write-Host $out + Add-Content -Path win-output/scenarios.txt -Value "`n=== redaction (ledger) ===`n$out" + if ($out -match 'ghp_windowssmoketest') { + Write-Host '::error::SECRET LEAKED in ledger output on Windows' + Add-Content -Path win-output/scenarios.txt -Value 'RESULT: LEAKED' + } else { + Write-Host 'redaction held on Windows' + Add-Content -Path win-output/scenarios.txt -Value 'RESULT: masked' + } + + - name: Runner descendant termination via taskkill + continue-on-error: true + shell: pwsh + run: | + $script = @' + import { runProcess } from './packages/harness/lib/runner.mjs'; + const ctrl = new AbortController(); + // A parent that spawns a long-lived grandchild, so termination has to + // reach the whole tree rather than just the process we spawned. + const child = process.platform === 'win32' + ? { argv: ['cmd', '/c', 'start /b timeout /t 60 >nul & timeout /t 60 >nul'] } + : { argv: ['sh', '-c', 'sleep 60 & sleep 60'] }; + setTimeout(() => ctrl.abort(), 1200); + const started = Date.now(); + const res = await runProcess({ ...child, signal: ctrl.signal, timeoutMs: 30000 }); + console.log(JSON.stringify({ + status: res.status, + exitCode: res.exitCode, + settledMs: Date.now() - started, + platform: process.platform, + }, null, 2)); + if (res.status !== 'cancelled') { console.error('EXPECTED cancelled, GOT ' + res.status); process.exit(1); } + '@ + Set-Content -Path win-cancel.mjs -Value $script + $out = & node win-cancel.mjs 2>&1 | Out-String + Write-Host $out + Add-Content -Path win-output/scenarios.txt -Value "`n=== runner cancellation (taskkill path) ===`n$out" + + - name: Environment fingerprint + if: always() + shell: pwsh + run: | + $info = @{ + os = (Get-CimInstance Win32_OperatingSystem).Caption + version = [System.Environment]::OSVersion.VersionString + node = (node --version) + shell = 'pwsh' + WT_SESSION = "$env:WT_SESSION" + TERM_PROGRAM = "$env:TERM_PROGRAM" + } | ConvertTo-Json + Write-Host $info + Add-Content -Path win-output/scenarios.txt -Value "`n=== environment ===`n$info" + + - name: Upload captured output + if: always() + uses: actions/upload-artifact@v4 + with: + name: harness-windows-output + path: win-output/ + + - name: Report suite outcome + if: always() + shell: pwsh + run: | + Write-Host "test suite outcome: ${{ steps.suite.outcome }}" diff --git a/docs/architecture/harness-cli-workbench.md b/docs/architecture/harness-cli-workbench.md new file mode 100644 index 00000000..1e99d51a --- /dev/null +++ b/docs/architecture/harness-cli-workbench.md @@ -0,0 +1,536 @@ +# Harness CLI Workbench — Feature Plan + +Finalized 2026-07-29. Execution begins after PR #37 (knowledge layer M1–M4) merges. The live execution plan for the current phase is the single dated file under `docs/plans/` (Phase 1: `2026-07-29-harness-cli-phase1-core.md`); this document is the durable contract the phase plans derive from. + +## TL;DR + +The concrete target is five releases: **CLI modes → knowledge navigation → governed execution → durable runs/TUI → resource and plugin system**. + +The new command families are: `search`, `lookup`, `tree`, `checks`, `exec`, `bash`, `run`, `config`, `trust`, `resources`, `plugin`, and `tui`. + +## Boundary and invariants + +Harness owns deterministic capabilities; the host owns LLM reasoning. Constraints that bind every phase: + +- **CLI never calls an LLM.** An LLM host consumes Harness; Harness never consumes a model. +- **Knowledge-layer invariants (settled on PR #37, do not re-litigate):** read paths never create the learnings store; the ops-JSON sole writer is `consolidate --apply`; the store lives at `~/.harness/knowledge//` (local git, never pushed); promoted learnings are immutable targets in both lanes; human authority derives from on-disk verified evidence and must be at least as recent as any recorded governance decision. +- **Search, lookup, and tree comply with the read-path rule** — navigation never creates or mutates the store. +- **Resources and plugins never write the learnings store, the run journal, or evidence.** Plugins may contribute knowledge *sources*; ingestion still flows through the consolidation loop. +- **Gate (vocabulary):** a gate is the existing plan/verification gate mechanism (`lib/gate.mjs`) that blocks lifecycle transitions until named checks and evidence pass. Later phases expose and record gates; they do not reinvent them. +- **Enforcement classes:** every control in this plan is one of — **enforced** (blocks pre-execution), **detect-and-block** (best-effort detection that halts on trip), or **audit-only** (recorded, never blocks). Each control names its class where it is specified; "governed" always means at least detect-and-block. +- **TUI boundary:** the TUI consumes the kernel in-process through the same command registry as the CLI — one behavior path, no separate implementation, no CLI shell-out requirement. This is the settled answer to the earlier SDK question; the CLI JSON/JSONL contracts serve out-of-process consumers on their program side (CI, hooks, Copilot, Codex) — the model side of any host consumes the agent lane per the output-lanes contract below. +- **Three output lanes, produced at the source:** every command renders its one canonical result as ledger (human), envelope (programs/TUI), and agent (LLM) lanes — deterministically, never via a model pass, and never by converting one lane into another. See "Output lanes" below. + +## Output lanes: the three-audience contract + +Every workbench-dispatched command renders one canonical result three ways. All three +renderings are deterministic CLI work — never a model pass. No output is ever converted +from one audience's format into another's; each lane is produced at the source. + +| Lane | Audience | Format | Contract | +|---|---|---|---| +| Ledger | Human | Styled ledger rows (`lib/style.mjs` conventions: glyph/key/value, truecolor → 256 → ASCII degradation) | What the TUI panes and plain terminal output render | +| Envelope | Programs / TUI | Versioned JSON envelope; JSONL streaming for long-running operations | Summary scalars first, detail arrays after, so one payload serves both a one-line footer and an expanded view. Long operations stream row-per-event with distinct `cancelled` vs `timed-out` terminal outcomes | +| Agent | LLM | Budgeted plain text | Token/byte-capped at the source; injection-hardened; byte-metered | + +### Agent-lane requirements + +1. **Budgeted at the source.** Every agent rendering carries a hard local cap, following + the existing harness precedents: 2048-byte context pack, 220-token plan slice, + bounded `harness get` excerpts (40 lines / 2048 bytes), 1000-token repo map. + Truncation happens at item boundaries; the reported size never exceeds the budget. +2. **Deterministic.** Produced by the CLI from the canonical result — never by asking a + model to summarize the envelope. This preserves the system-wide guarantee that + `orient`/`recall` and every read surface involve no model, network, or embedding call. +3. **Hardened.** Wherever the content is retrieved text (knowledge, search hits, doc + excerpts), the rendering passes the existing data boundary: data-not-instructions + preamble framing, `inertLine` neutralization, and secret redaction. +4. **Metered.** Rendered bytes are measured and emitted with the command's event so + `harness report` token/utilization SLOs account for the agent lane's real cost. + +### The consumption rule + +Agents consume the agent lane, never the envelope. JSON is token-inefficient and its +arrays are unbounded, so envelope output must never enter model context. The envelope +belongs to the TUI and tooling; the ledger belongs to humans. This is what makes the +dual human/LLM promise real: zero tokens are ever spent translating tool output. + +### Boundary with the harness-evolution track + +The workbench defines the registry, envelope schema, and run-journal contracts; the +harness evolution blueprint (`knowledge/proposals/harness-evolution-blueprint.md`, §9) +commits its new surfaces (`knowledge status/promote/prune`, `index --structural`, the +structural query) to being conforming citizens of them — registry-dispatchable, +envelope-emitting, streaming-capable, and shipping an agent rendering per this contract. + +## Final command surface + +### Knowledge and workspace navigation + +```text +harness index [--status] +harness search +harness lookup +harness tree +harness get --docid | --path +harness orient --query +``` + +#### `search` + +```text +harness search "lease fencing" \ + --scope code,knowledge,plans \ + --match ranked \ + --explain +``` + +Features: + +- Scopes: + - `code` + - `knowledge` + - `learnings` + - `plans` + - `skills` + - `checks` + - `events` + - `runs` + - `all` +- Match modes: + - `ranked`: BM25/knowledge ranking + - `literal`: exact content search + - `regex`: regex content search + - `path`: file/path discovery + - `symbol`: indexed symbol search +- Filters: + - path/glob + - collection + - result type + - minimum score + - limit +- Pagination cursor +- Snapshot-specific search +- Retrieval explanation +- Explicit partial-result handling +- Source, freshness, score and provenance in every result +- Empty search returns success with zero results + +This command incorporates ranked search, grep-like content search, find-like path search, and symbol lookup without creating separate public commands for each tool. + +Federation semantics (`--scope` with multiple sources or `all`) are deterministic: per-source scores are normalized before merging; result identity for dedup is (source, entity id); ordering ties break stably by (score, source, id); cursors remain valid across sources; a failed source is reported explicitly in the result envelope, never silently dropped. + +#### `lookup` + +```text +harness lookup +``` + +Supported kinds: + +```text +file | symbol | document | plan | skill | check | run | event | resource | learning | episode +``` + +Features: + +- Exact entity retrieval +- Metadata and source provenance +- Bounded content preview +- Related entities +- Current index generation +- Structured not-found error + +`get` remains the document/file compatibility command; `lookup` handles the broader entity model. `lookup learning` and `lookup episode` are read-only views over the knowledge store and respect the read-path invariant. + +#### `tree` + +```text +harness tree workspace [path] --depth 3 +harness tree knowledge [collection] +harness tree run +harness tree resources +``` + +Features: + +- Workspace hierarchy respecting ignore rules +- Knowledge collections, documents, and the learnings store (episodes → learnings → primitives, with provenance and governance state) +- Run lifecycle and evidence hierarchy +- Resource origin and override hierarchy +- Depth, type and path filters +- Human, JSON and TUI rendering + +`tree` is the single navigation verb: `harness tree resources` is canonical, and the resources family does not carry a duplicate `resources tree` subcommand. + +### Checks and execution + +```text +harness checks list +harness checks show +harness checks run + +harness exec [options] -- +harness bash [options] --