Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions .github/skills/references/harness-tool-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,13 @@ Installed to `~/.copilot/bin/harness` on every `harness install`. Add to PATH wi
| `1` | Fail — stop before `editFiles` or compound |
| `2` | Warn — may proceed with Activity log (strict profile: treat as block) |

**Envelope versioning (Phase 1 workbench, additive).** `--output` is a lane-bearing, opt-in surface, not a blanket capability every command gained — only the commands named below accept it; every other registered command now REJECTS `--output` with a structured `E_USAGE` error (exit 2) naming the lane-bearing commands, instead of silently ignoring it and rendering plain ledger output (fixed post-P1.6; see `lib/registry.mjs`'s `assertLaneSupported`). The lane-bearing set today: `orient`, `learnings`, and `status` accept `--output json-envelope|agent` (`--output=value` works too) — `json-envelope` wraps the command's result in a versioned `{schema, command, status, ...}` envelope, `agent` renders a budgeted plain-text summary for LLM callers; `verify` additionally accepts `--output jsonl`, streaming row-per-event with a terminal `result` row (no `json-envelope`/`agent` support of its own). `status` in all of these uses one shared vocabulary — `ok\|failed\|cancelled\|timed-out`, plus `blocked` where gate-like semantics already use it. These are opt-in, additive lanes: every legacy shape documented below (the default ledger render and `--json`) is **unversioned and unchanged** — no `schema` field, no wrapper, byte-identical output. Do not rely on `--output` for a command not named above — it will error, not silently no-op; a command's help (`harness help <command>`) also states whether it supports a lane.

## Command catalog

This table tracks only what differs in runtime character across commands — which turn a command runs on, whether it writes a lifecycle event, whether it mutates anything. Sigs and flags: `harness help <command>` (the CLI CATALOG is the single source of truth).
This table tracks only what differs in runtime character across commands — which turn a command runs on, whether it writes its own DOMAIN-SPECIFIC lifecycle event (the closed vocabulary in `lib/events.mjs#EVENT_TYPES` — `orient`, `gate`, `verify`, `compound`, `consolidate`, `remember`, `learning`, `knowledge`, plus the hook-only `session_start`/`pre_tool`/`post_tool`/`skill_activation`/`session_end`), and whether it mutates anything. This is separate from the dispatch-level `command.start`/`command.result` bracketing every registered command now also gets — see the footnote below the table. Sigs and flags: `harness help <command>` (the registry, `lib/registry.mjs`, is the single source of truth — the hand-written CLI CATALOG this line used to reference was retired in P1.6; `harness help`/`harness help <command>` render live from the same registry entries this doc is generated by hand from, so treat any drift you notice as a doc bug, not a second source of truth).

| Command | Tier | Events | Store |
| Command | Tier | Events (domain-specific) | Store |
|---------|------|--------|-------|
| `install` / `upgrade` | human/CI | none | mutates `~/.copilot/` |
| `doctor` | human/CI | none | read-only (`--host vscode` runs an isolated hook-lifecycle fixture) |
Expand All @@ -74,10 +76,14 @@ This table tracks only what differs in runtime character across commands — whi
| `learnings` | agent-runtime | none | read-only |
| `knowledge` | agent-runtime | writes | mutates `config.json`, cascade-deletes, or mirrors to the product repo |
| `eval-knowledge` | agent-runtime | none | read-only |
| `events` | agent-runtime | none | read-only |
| `events` | agent-runtime | none³ | read-only |
| `report` | agent-runtime | none | read-only (`--sync` writes `~/.harness/telemetry/`) |

¹ `init-repo`/`recall`/`validate-plan`/`index` historically called `writeEvent` (types `init_repo`/`recall`/`validate_plan`/`index`) while those four type strings were absent from the `EVENT_TYPES` allow-list (`events.mjs`), so the calls silently no-opped. The allow-list now includes all four (harness evolution Phase 1 hygiene) — the events record in `events.jsonl` like every other lifecycle write.
¹ `init-repo`/`recall`/`validate-plan`/`index` historically called `writeEvent` (types `init_repo`/`recall`/`validate_plan`/`index`) while those four type strings were absent from the `EVENT_TYPES` allow-list (`events.mjs`), so the calls silently no-opped. The allow-list now includes all four (harness evolution Phase 1 hygiene) — the events record in `events.jsonl` like every other lifecycle write, in ADDITION to the dispatch-level bracketing of footnote ².

² **Every registered command dispatch (P1.6, `lib/registry.mjs`) is bracketed with `command.start`/`command.result` telemetry, independent of the domain-specific "Events" column above** — this is a dispatch-pipeline concern, not a per-command opt-in, so it applies uniformly regardless of whether that column says "writes" or "none". The one exception is `events` itself (footnote ³). A real, observable side effect: running ANY registered `harness` command — including ones whose Events column says "none", like `get`, `learnings`, `report`, or `eval-knowledge` — creates/appends `.harness/events.jsonl` under whatever `--workspace` resolves to (default: the current directory), in ANY cwd, even one with no other `.harness/` state. Opt out with `--no-events`, `--dry-run`, or `HARNESS_NO_EVENTS=1` (all three short-circuit `lib/events.mjs#writeEvent` before anything is written, for both the domain-specific events above and this dispatch-level bracketing).

³ `events` itself is the one command excluded from footnote ² (`entry.instrument === false`, `lib/registry.mjs`): its own handler's job is reading and summarizing everything already in `events.jsonl`, so bracketing its own dispatch would append that very invocation's own `command.start` to the file an instant before the handler reads it back — a self-referential read-your-own-write that would inflate `harness events`'s own totals on every call. `report` opts out for the same reason (it reads `events.jsonl` via `loadReportEvents`, and `--sync` would copy the phantom row into the global store).

**Query construction (deterministic-retrieval discipline):** build `--query` from the user's salient nouns and identifiers **verbatim** (e.g. `SYSTEM-OVERRIDE`, `payment`, `token`) — do not paraphrase intent into synonyms. The retrieval tokenizer normalizes identifier formats and morphology, but it cannot recover a term the query never contained. Passing the literal request terms is what keeps recall stable across phrasings.

Expand Down Expand Up @@ -331,7 +337,7 @@ Every check in the `verify` payload carries its effective `severity`; non-passin
}
```

Lifecycle events are limited to `session_start`, `orient`, `gate`, `pre_tool`, `post_tool`, `skill_activation`, `verify`, `compound`, `consolidate`, `remember`, `learning`, `knowledge`, `session_end`, `init_repo`, `recall`, `validate_plan`, and `index` (the last four were formerly dropped by the allow-list despite their call sites — fixed as harness evolution Phase 1 hygiene; see the Command catalog table's footnote). Non-lifecycle commands `get`, `report`, `learnings`, and `eval-knowledge` never append events by design — they never call `writeEvent` at all. Every append-attempting command never stores prompt or query content; `skill_activation` stores only the skill and session binding.
DOMAIN-specific lifecycle events (the `EVENT_TYPES` allow-list a command names for its OWN event, distinct from the dispatch-level `command.start`/`command.result` bracketing — see the Command catalog table's footnote ²) are limited to `session_start`, `orient`, `gate`, `pre_tool`, `post_tool`, `skill_activation`, `verify`, `compound`, `consolidate`, `remember`, `learning`, `knowledge`, `session_end`, `init_repo`, `recall`, `validate_plan`, and `index` (the last four were formerly dropped by the allow-list despite their call sites — fixed as harness evolution Phase 1 hygiene; see the Command catalog table's footnote ¹). The dispatch pipeline adds three more types of its own — `command.start`, `command.result`, and `agent_lane` (P1.6, `lib/registry.mjs` / `lib/agent-lane.mjs`) — which are never named by a command. Non-lifecycle commands `get`, `report`, `learnings`, and `eval-knowledge` never call `writeEvent` under their OWN domain type — but post-P1.6 they are not event-free: every registered command's dispatch is still bracketed with `command.start`/`command.result` regardless (footnote ² again), so all four DO append to `events.jsonl`, just never under a `get`/`report`/`learnings`/`eval-knowledge`-named event. Every append-attempting command never stores prompt or query content; `skill_activation` stores only the skill and session binding.

## Host hook boundary

Expand Down
156 changes: 156 additions & 0 deletions .github/workflows/harness-windows.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
# Exercises the harness CLI on a real Windows runner. The repository's primary
# consumption platform is Windows (AGENTS.md), but every check until now ran on
# ubuntu-latest, so the win32 branches shipped untested: runner.mjs terminates
# descendants with `taskkill /T /F` instead of a POSIX process group, paths.mjs
# resolves LOCALAPPDATA/USERPROFILE, style.mjs only enables Unicode glyphs when
# a modern terminal announces itself, and global-bin.mjs writes a .cmd shim.
#
# Output is captured verbatim to artifacts so the real console rendering — not
# a reconstruction of it — can be read after the fact.
name: Harness on Windows

on:
workflow_dispatch:
push:
branches: ['feat/**']

permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
windows:
runs-on: windows-latest
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false

- uses: actions/setup-node@v4
with:
node-version: '22'
cache: npm
cache-dependency-path: packages/harness/package-lock.json

- name: Install harness dependencies
run: npm ci --prefix packages/harness

# The suite has never run on Windows. Failures here are the point of the
# job, so they must not hide the CLI scenarios below.
- name: Harness test suite (Windows)
id: suite
continue-on-error: true
run: npm --prefix packages/harness test

- name: CLI scenarios
continue-on-error: true
shell: pwsh
run: |
$ErrorActionPreference = 'Continue'
New-Item -ItemType Directory -Force -Path win-output | Out-Null
$ws = Join-Path $env:RUNNER_TEMP 'ws'
$ch = Join-Path $env:RUNNER_TEMP 'copilot'
New-Item -ItemType Directory -Force -Path $ws, $ch | Out-Null
git -C $ws init -q 2>&1 | Out-Null
git -C $ws -c user.email=ci@example.com -c user.name=ci commit -q --allow-empty -m init 2>&1 | Out-Null

function Probe($name, $cmdArgs) {
$header = "`n=== $name ===`n`$ harness $($cmdArgs -join ' ')"
Write-Host $header
$out = & node packages/harness/bin/harness.mjs @cmdArgs 2>&1 | Out-String
$code = $LASTEXITCODE
Write-Host $out
Write-Host "[exit $code]"
Add-Content -Path win-output/scenarios.txt -Value "$header`n$out[exit $code]"
}

Probe 'help — design system on the Windows console' @('help')
Probe 'status — ledger rendering' @('status', '--workspace', $ws, '--copilot-home', $ch)
Probe 'status --json — legacy machine shape' @('status', '--json', '--workspace', $ws, '--copilot-home', $ch)
Probe 'status --output json-envelope — versioned envelope lane' @('status', '--output', 'json-envelope', '--workspace', $ws, '--copilot-home', $ch)
Probe 'status --output agent — budgeted agent lane' @('status', '--output', 'agent', '--workspace', $ws, '--copilot-home', $ch)
Probe 'orient' @('orient', '--query', 'windows smoke', '--workspace', $ws, '--copilot-home', $ch)
Probe 'index --structural (from main #42)' @('index', '--structural', '--workspace', $ws, '--copilot-home', $ch)
Probe 'knowledge status (from main #42)' @('knowledge', 'status', '--workspace', $ws, '--copilot-home', $ch)
Probe 'unknown flag — strict validation, expect E_USAGE exit 2' @('status', '--bogus', '--workspace', $ws)
Probe 'missing required arg — expect E_USAGE exit 2' @('recall', '--workspace', $ws, '--copilot-home', $ch)
Probe 'literal boundary — --json after -- must stay literal' @('status', '--workspace', $ws, '--copilot-home', $ch, '--', '--json')

# Redaction and descendant termination are the two win32 paths with the
# least coverage, so they are exercised directly rather than inferred.
- name: Redaction on Windows
continue-on-error: true
shell: pwsh
env:
MY_API_TOKEN: 'ghp_windowssmoketest0123456789abcdefghij'
run: |
$ws = Join-Path $env:RUNNER_TEMP 'ws'
$out = & node packages/harness/bin/harness.mjs learnings --why $env:MY_API_TOKEN --workspace $ws 2>&1 | Out-String
Write-Host $out
Add-Content -Path win-output/scenarios.txt -Value "`n=== redaction (ledger) ===`n$out"
if ($out -match 'ghp_windowssmoketest') {
Write-Host '::error::SECRET LEAKED in ledger output on Windows'
Add-Content -Path win-output/scenarios.txt -Value 'RESULT: LEAKED'
} else {
Write-Host 'redaction held on Windows'
Add-Content -Path win-output/scenarios.txt -Value 'RESULT: masked'
}

- name: Runner descendant termination via taskkill
continue-on-error: true
shell: pwsh
run: |
$script = @'
import { runProcess } from './packages/harness/lib/runner.mjs';
const ctrl = new AbortController();
// A parent that spawns a long-lived grandchild, so termination has to
// reach the whole tree rather than just the process we spawned.
const child = process.platform === 'win32'
? { argv: ['cmd', '/c', 'start /b timeout /t 60 >nul & timeout /t 60 >nul'] }
: { argv: ['sh', '-c', 'sleep 60 & sleep 60'] };
setTimeout(() => ctrl.abort(), 1200);
const started = Date.now();
const res = await runProcess({ ...child, signal: ctrl.signal, timeoutMs: 30000 });
console.log(JSON.stringify({
status: res.status,
exitCode: res.exitCode,
settledMs: Date.now() - started,
platform: process.platform,
}, null, 2));
if (res.status !== 'cancelled') { console.error('EXPECTED cancelled, GOT ' + res.status); process.exit(1); }
'@
Set-Content -Path win-cancel.mjs -Value $script
$out = & node win-cancel.mjs 2>&1 | Out-String
Write-Host $out
Add-Content -Path win-output/scenarios.txt -Value "`n=== runner cancellation (taskkill path) ===`n$out"

- name: Environment fingerprint
if: always()
shell: pwsh
run: |
$info = @{
os = (Get-CimInstance Win32_OperatingSystem).Caption
version = [System.Environment]::OSVersion.VersionString
node = (node --version)
shell = 'pwsh'
WT_SESSION = "$env:WT_SESSION"
TERM_PROGRAM = "$env:TERM_PROGRAM"
} | ConvertTo-Json
Write-Host $info
Add-Content -Path win-output/scenarios.txt -Value "`n=== environment ===`n$info"

- name: Upload captured output
if: always()
uses: actions/upload-artifact@v4
with:
name: harness-windows-output
path: win-output/

- name: Report suite outcome
if: always()
shell: pwsh
run: |
Write-Host "test suite outcome: ${{ steps.suite.outcome }}"
Loading
Loading