feat: verification probes with CI-aware execution policy

.github/workflows/ci.yml

@@ -14,7 +14,7 @@ jobs:
 
       - uses: oven-sh/setup-bun@v2
         with:
-          bun-version: latest
+          bun-version: 1.3.14
 
       - run: bun install --frozen-lockfile
 

.gitignore

@@ -46,3 +46,8 @@ yarn-error.log*
 # Kyora data
 .kyora
 
+
+# generated app artifacts
+.source/
+next-env.d.ts
+*.tsbuildinfo

README.md

@@ -2,16 +2,37 @@
 
 Tools that make coding agents trustworthy — by grounding them in what actually happens: at runtime, and in review.
 
-Two products, one repo:
+[kyora.sh](https://kyora.sh)
 
-## kyora state — queryable temporal runtime state
+## Contents
 
-Records state mutations, function calls, HTTP traffic, and errors over time, then exposes it all via MCP so agents can query what actually happened at runtime. Also indexes dependencies and docs semantically, minimizing hallucinations.
+- [kyora state — queryable temporal runtime state](#kyora-state)
+  - [Quick start](#quick-start)
+  - [Instrumentation](#instrumentation)
+  - [Auto-instrumentation (Bun plugin)](#auto-instrumentation-bun-plugin)
+  - [MCP tools](#mcp-tools)
+- [kyora review — multi-engine code review](#kyora-review)
+  - [Quick start](#quick-start-1)
+  - [How findings are ranked](#how-findings-are-ranked)
+  - [Engines](#engines)
+  - [CI](#ci)
+  - [Configuration](#configuration)
+- [Repo layout](#repo-layout)
+- [Development](#development)
+- [License](#license)
+
+## kyora state
+
+Queryable temporal runtime state for coding agents. Records state mutations, function calls, HTTP traffic, and errors over time, then exposes it all via MCP so agents can query what actually happened at runtime — instead of guessing from source. Also indexes dependencies and docs semantically, minimizing hallucinations.
+
+### Quick start
 
 ```bash
 bunx @kyora-sh/mcp
 ```
 
+Add to `.claude/settings.json`:
+
 ```json
 {
   "mcpServers": {
@@ -24,20 +45,118 @@ bunx @kyora-sh/mcp
 }
 ```
 
-Instrument with `@kyora/sdk` (`watch`, `trace`, auto-patching of fetch/console/errors, Bun plugin for annotation-driven instrumentation) and query it back through MCP tools (`kyora_query_state`, `kyora_get_recent_errors`, `kyora_get_http_log`, semantic doc search). Full docs: [`packages/state`](packages/state).
+### Instrumentation
+
+```ts
+import { init, watch, trace } from "@kyora/sdk"
+
+init({ dataDir: ".kyora" })
+
+// track state over time
+const cart = watch({ items: [], total: 0 }, "cart")
+cart.items.push({ name: "Widget", price: 9.99 })
+
+// record function calls (args, return values, errors, timing)
+const fetchUsers = trace(async function fetchUsers() {
+  return (await fetch("/api/users")).json()
+}, "fetchUsers")
+```
+
+`init()` automatically patches `fetch`, `console`, and error handlers.
+
+### Auto-instrumentation (Bun plugin)
+
+```toml
+# bunfig.toml
+preload = ["@kyora/sdk/plugin"]
+```
+
+```ts
+// @kyora.watch
+const state = { count: 0, users: [] }
+
+// @kyora.trace
+async function loadUsers() {
+  state.users = await (await fetch("/api/users")).json()
+}
+```
+
+Transforms at load time, no manual wrapping.
+
+### MCP tools
+
+| Tool | Description |
+|------|-------------|
+| `kyora_query_state` | query state snapshots over time |
+| `kyora_get_recent_errors` | recent errors with stack traces |
+| `kyora_get_http_log` | HTTP requests and responses |
+| `kyora_search_docs` | semantic search across indexed docs |
+| `kyora_list_indexed` | list indexed documentation sources |
+| `nora_index_source` | index npm packages, URLs, or local files |
+| `kyora_index_status` | check indexing progress |
 
-## kyora review — multi-engine code review on your own subscriptions
+## kyora review
 
-Codex, Claude Code, Kimi, Grok, and Qwen reviewing the same PR together — each vendor's own CLI, headless, under your login, inside the checkout so it can verify claims against real code. Findings merge with consensus ranking; single-engine findings can be adversarially cross-verified by another engine before they reach you.
+Multi-engine AI code review on the coding-agent subscriptions you already pay for. Codex, Claude Code, Kimi, Grok, and Qwen review the same diff independently — each vendor's own CLI running headless *inside your checkout*, under your own login, so it can grep callers and read tests instead of guessing from the diff. Findings are merged with consensus ranking and optional adversarial cross-verification, then land as one PR review with inline comments.
+
+No hosting, no accounts, no middleman keys: every engine is the vendor's own CLI authenticated with your subscription.
+
+### Quick start
+
+```bash
+bunx @kyora-sh/review doctor    # which engines are ready on this machine
+bunx @kyora-sh/review           # review your branch (diff vs main)
+```
+
+Review a PR and post the results:
 
 ```bash
-bunx @kyora-sh/review doctor          # which engines are ready
-bunx @kyora-sh/review --pr 123 --post # panel-review a PR
+bunx @kyora-sh/review review --pr 123 --post --verify
 ```
 
-CI: one workflow + your subscription tokens as repo secrets, auto-refreshed. See [`action/README.md`](action/README.md) and [`packages/review/cli`](packages/review/cli).
+### How findings are ranked
+
+- **consensus** — two or more engines independently flagged the same issue. Rare and almost always real.
+- **verified** — one engine flagged it, a *different* engine was asked to refute it against the actual code, and couldn't (`--verify`).
+- **single** — one engine, unchallenged. Refuted findings are dropped entirely.
+
+Findings cluster by file, overlapping lines, and description similarity, so five engines saying the same thing arrive as one finding with five votes — not five comments.
+
+### Engines
+
+| id | runs | auth |
+| --- | --- | --- |
+| `codex` | `codex exec` (read-only sandbox) | `codex login` (ChatGPT sub) or `OPENAI_API_KEY` |
+| `claude` | `claude -p` (probes allowed, writes and CI suites denied) | `claude` login or `CLAUDE_CODE_OAUTH_TOKEN` |
+| `kimi` | Claude Code against Kimi's Anthropic-compatible endpoint | `KIMI_API_KEY` |
+| `grok` | `grok -p` (grok-4.5, high reasoning effort) | `grok login` (SuperGrok / X Premium+) or `GROK_API_KEY` |
+| `qwen` | `qwen -p` | `qwen` login (Coding Plan) or API key |
+
+Every available engine runs by default; pick explicitly with `--engines codex,kimi`. An engine that's missing, rate-limited, or failing drops out and the review still lands with the rest. If *every* engine fails, the run exits non-zero instead of reporting a clean review.
+
+Engines may execute small targeted probes (snippet evaluation, single-function checks) to verify a suspicion, but never the repo's test suites, builds, or type checks — those are auto-detected from `.github/workflows/*` and declared off-limits, since CI runs them anyway. Writes, installs, and history-mutating git commands are denied.
+
+### CI
+
+One workflow plus your subscription tokens as repo secrets. Rotating credentials (Codex) are persisted via `actions/cache`, so you seed once and they keep themselves alive:
+
+```yaml
+- uses: eliahilse/kyora/action@main
+  with:
+    verify: "true"
+  env:
+    CODEX_AUTH_JSON: ${{ secrets.KYORA_CODEX_AUTH }}
+    CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.KYORA_CLAUDE_TOKEN }}
+    KIMI_API_KEY: ${{ secrets.KYORA_KIMI_KEY }}
+```
+
+Full setup, secret-seeding commands, and security notes: [`action/README.md`](action/README.md).
+
+### Configuration
+
+`kyora-review.config.json` at the repo root sets defaults (`engines`, `verify`, `post`, `base`, `failOn`, `maxDiffBytes`, `timeoutMs`), plus per-engine overrides — `bin`, `args` (with `{prompt}`/`{schema}`/`{schemaJson}`/`{out}` tokens), and `env` — so a vendor CLI changing its flags is a config edit, not a code change. CLI reference: [`packages/review/cli`](packages/review/cli).
 
-## Layout
+## Repo layout
 
 ```
 packages/state/sdk      @kyora/sdk        instrumentation (watch, trace, auto-patching)

action/README.md

@@ -68,4 +68,4 @@ Every engine uses *your* account through *its vendor's own CLI* — that's what
 ## Security notes
 
 - **Public repos:** Actions caches are readable by workflows in the same repo, including ones triggered from fork PRs. The `if:` guard above stops fork PRs from running this job, but if other workflows in your repo run untrusted code, set `persist-auth: "false"` and rely on the seeded secrets alone (Codex seeds then need re-seeding when the refresh token rotates out).
-- The review job needs only `contents: read` + `pull-requests: write`. Engines run with read-only sandboxes/tool allowlists — they review, they don't edit.
+- The review job needs only `contents: read` + `pull-requests: write`. Engines may execute targeted verification probes but are denied writes, installs, and the repo's CI-covered suites — they review, they don't edit.

packages/review/cli/README.md

@@ -28,7 +28,7 @@ bunx @kyora-sh/review review --pr 123 --post --verify
 | id | runs | auth |
 | --- | --- | --- |
 | `codex` | `codex exec` (read-only sandbox) | `codex login` (ChatGPT sub) or `OPENAI_API_KEY` |
-| `claude` | `claude -p` (read-only tool allowlist) | `claude` login or `CLAUDE_CODE_OAUTH_TOKEN` |
+| `claude` | `claude -p` (probes allowed, writes and CI suites denied) | `claude` login or `CLAUDE_CODE_OAUTH_TOKEN` |
 | `kimi` | Claude Code against Kimi's Anthropic-compatible endpoint | `KIMI_API_KEY` (+ optional `KIMI_BASE_URL`, `KIMI_MODEL`) |
 | `grok` | `grok -p` | `grok` login or `GROK_API_KEY` / `XAI_API_KEY` |
 | `qwen` | `qwen -p` | `qwen` login (Coding Plan) or API key |

packages/review/cli/src/ci.test.ts

@@ -0,0 +1,47 @@
+import { describe, expect, test } from "bun:test"
+import { extractRunCommands, isCiCovered } from "./ci"
+
+const WORKFLOW = `name: CI
+on: push
+jobs:
+  ci:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - run: bun install --frozen-lockfile
+      - run: bun run build
+      - name: multi
+        run: |
+          # comment
+          bun run check-types
+          bun run test
+      - run: echo "done"
+`
+
+describe("extractRunCommands", () => {
+  test("collects single-line and block-scalar run commands", () => {
+    expect(extractRunCommands(WORKFLOW)).toEqual([
+      "bun install --frozen-lockfile",
+      "bun run build",
+      "bun run check-types",
+      "bun run test",
+      'echo "done"',
+    ])
+  })
+})
+
+describe("isCiCovered", () => {
+  test("flags suites, builds, and checkers", () => {
+    expect(isCiCovered("bun run test")).toBe(true)
+    expect(isCiCovered("bun run build")).toBe(true)
+    expect(isCiCovered("bun run check-types")).toBe(true)
+    expect(isCiCovered("npx vitest run")).toBe(true)
+    expect(isCiCovered("tsc --noEmit")).toBe(true)
+  })
+
+  test("ignores installs and misc commands", () => {
+    expect(isCiCovered("bun install --frozen-lockfile")).toBe(false)
+    expect(isCiCovered('echo "done"')).toBe(false)
+    expect(isCiCovered("git fetch origin main")).toBe(false)
+  })
+})

packages/review/cli/src/ci.ts

@@ -0,0 +1,47 @@
+const COVERED =
+  /(?:^|[\s/])(test|tests|build|lint|typecheck|check-types|types:check|tsc|vitest|jest|pytest|eslint|prettier)(?::|\s|$)/
+
+export function isCiCovered(command: string): boolean {
+  return COVERED.test(command)
+}
+
+export function extractRunCommands(workflowYaml: string): string[] {
+  const commands: string[] = []
+  const lines = workflowYaml.split("\n")
+  for (let i = 0; i < lines.length; i++) {
+    const match = /^(\s*)(?:-\s+)?run:\s*(.*)$/.exec(lines[i]!)
+    if (!match) continue
+    const indent = match[1]!.length
+    const value = match[2]!.trim()
+    if (value && value !== "|" && value !== ">") {
+      commands.push(value)
+      continue
+    }
+    for (let j = i + 1; j < lines.length; j++) {
+      const line = lines[j]!
+      if (line.trim() === "") continue
+      const lineIndent = line.length - line.trimStart().length
+      if (lineIndent <= indent) break
+      const command = line.trim()
+      if (!command.startsWith("#")) commands.push(command)
+    }
+  }
+  return commands
+}
+
+/** Commands the repo's CI already runs that reviewers must not duplicate. */
+export async function ciCoveredCommands(root: string): Promise<string[]> {
+  const covered = new Set<string>()
+  const glob = new Bun.Glob(".github/workflows/*.{yml,yaml}")
+  try {
+    for await (const file of glob.scan({ cwd: root, dot: true })) {
+      const text = await Bun.file(`${root}/${file}`).text()
+      for (const command of extractRunCommands(text)) {
+        if (isCiCovered(command)) covered.add(command)
+      }
+    }
+  } catch {
+    return []
+  }
+  return [...covered].sort()
+}

packages/review/cli/src/engines.ts

@@ -17,6 +17,17 @@ export interface EngineDef {
   authHint: string
 }
 
+const CLAUDE_DENIED = [
+  "Write", "Edit", "MultiEdit", "NotebookEdit", "WebFetch", "WebSearch",
+  "Bash(bun test:*)", "Bash(bun run:*)", "Bash(bun install:*)", "Bash(bunx turbo:*)",
+  "Bash(npm test:*)", "Bash(npm run:*)", "Bash(npm install:*)", "Bash(npx turbo:*)",
+  "Bash(pnpm test:*)", "Bash(pnpm run:*)", "Bash(pnpm install:*)",
+  "Bash(yarn:*)", "Bash(turbo:*)", "Bash(tsc:*)", "Bash(next:*)",
+  "Bash(vitest:*)", "Bash(jest:*)", "Bash(pytest:*)", "Bash(go test:*)", "Bash(cargo test:*)",
+  "Bash(make:*)", "Bash(pip install:*)", "Bash(rm:*)", "Bash(mv:*)",
+  "Bash(git push:*)", "Bash(git commit:*)", "Bash(git checkout:*)", "Bash(git reset:*)", "Bash(git stash:*)",
+].join(",")
+
 const CLAUDE_ARGS = [
   "-p",
   "{prompt}",
@@ -25,7 +36,9 @@ const CLAUDE_ARGS = [
   "--max-turns",
   "40",
   "--allowedTools",
-  "Read,Grep,Glob,Bash(git diff:*),Bash(git log:*),Bash(git show:*),Bash(ls:*)",
+  "Read,Grep,Glob,Bash",
+  "--disallowedTools",
+  CLAUDE_DENIED,
 ]
 
 export const ENGINES: EngineDef[] = [

packages/review/cli/src/index.ts

@@ -1,4 +1,5 @@
 import { parseArgs } from "node:util"
+import { ciCoveredCommands } from "./ci"
 import { buildContext, repoRoot } from "./diff"
 import { ENGINES, engineById, engineStatus, runEngineRaw, type EngineDef } from "./engines"
 import { extractFindings, extractPayload } from "./extract"
@@ -131,7 +132,9 @@ async function review(flags: Flags): Promise<void> {
   }
   log(`reviewing ${ctx.changedFiles.length} changed file(s) with: ${selected.map((engine) => engine.id).join(", ")}`)
 
-  const prompt = reviewPrompt(ctx, config.maxFindingsPerEngine)
+  const ciCovered = await ciCoveredCommands(root)
+  if (ciCovered.length > 0) log(`execution policy: ${ciCovered.length} CI-covered command(s) off-limits to engines`)
+  const prompt = reviewPrompt(ctx, config.maxFindingsPerEngine, ciCovered)
   const runs: EngineRun[] = await Promise.all(
     selected.map(async (engine): Promise<EngineRun> => {
       log(`${engine.id}: starting`)

packages/review/cli/src/schema.ts

@@ -48,10 +48,17 @@ export const VERDICTS_SCHEMA = {
   },
 } as const
 
-export function reviewPrompt(ctx: ReviewContext, maxFindings: number): string {
+export function reviewPrompt(ctx: ReviewContext, maxFindings: number, ciCovered: string[]): string {
+  const ciSection =
+    ciCovered.length > 0
+      ? `\nCI for this repository already runs the following — never run them or their equivalents; if a finding depends on their outcome, state the expectation instead:\n${ciCovered.map((command) => `  ${command}`).join("\n")}\n`
+      : ""
   return `You are one reviewer on a multi-model code review panel. Different models review the same change independently; findings are cross-checked afterwards, so precision matters more than volume.
 
-You are inside the repository checkout (current working directory). The change under review is the diff below. Before reporting a finding, verify it against the actual code: read the surrounding file, check callers and tests with grep, confirm the claim holds. Never report an issue you could have disproven by reading the repo. Do not modify any files.
+You are inside the repository checkout (current working directory). The change under review is the diff below. Before reporting a finding, verify it against the actual code: read the surrounding file, check callers and tests with grep, confirm the claim holds. Never report an issue you could have disproven by reading the repo.
+
+EXECUTION POLICY: You may run small, targeted probes to verify a suspicion — evaluate a snippet (\`bun -e\`, \`node -e\`, \`python3 -c\`), exercise a single function against an edge case, inspect git history. You must NOT run the project's test suites, builds, linters, or type-checkers: CI runs those already, and re-running them wastes the panel's compute without adding signal. Never modify files, install packages, or access the network.
+${ciSection}
 
 Report only issues that matter: bugs, correctness, security, data loss, races, broken error handling, API misuse, significant performance problems. No style or formatting nits unless they hide a defect. Report at most ${maxFindings} findings — fewer, well-verified findings beat many speculative ones. If the change looks correct, return an empty findings array.
 
@@ -86,7 +93,7 @@ export function verifyPrompt(claims: VerifyClaim[]): string {
     .join("\n\n")
   return `You are an adversarial verifier on a code review panel. Each claim below was reported by only ONE reviewer, so it is suspect. You are inside the repository checkout: for each claim, read the actual code and decide whether the issue is real.
 
-Actively try to refute each claim. "confirmed" only if you can point at the code path that makes it true; if the claim is speculative, already handled elsewhere, or you cannot reproduce the reasoning from the code, return "refuted". Do not modify any files.
+Actively try to refute each claim. "confirmed" only if you can point at the code path that makes it true; if the claim is speculative, already handled elsewhere, or you cannot reproduce the reasoning from the code, return "refuted". You may run small targeted probes (snippet evaluation, single-function checks) to test a claim, but never the project's test suites or builds, and never modify files.
 
 OUTPUT: respond with ONLY a JSON object, no prose, one verdict per claim index:
 {"verdicts": [{"index": 0, "verdict": "confirmed", "reason": "..."}]}

packages/review/cli/tsconfig.json

@@ -2,7 +2,7 @@
   "extends": "@repo/typescript-config/base.json",
   "compilerOptions": {
     "noEmit": true,
-    "types": ["bun-types"],
+    "types": ["bun"],
     "module": "ESNext",
     "moduleResolution": "bundler"
   },

packages/state/db/tsconfig.json

@@ -3,7 +3,7 @@
   "compilerOptions": {
     "outDir": "dist",
     "rootDir": "src",
-    "types": ["bun-types"],
+    "types": ["bun"],
     "module": "ESNext",
     "moduleResolution": "bundler"
   },