Skip to content

Commit d6bfcc4

Browse files
Bordaclaude[bot]codex
committed
refine(codemap): anchor paths at git root, add guards
- Anchor every codemap index and log path at the git toplevel instead of the process working directory, across the provider and all four consumer plugins: queries from a repository subdirectory previously compared subdirectory-relative paths against root-relative index entries (permanent stale, self-heal every call) or derived a different index path than the writer (false no_index), and log shards split across two directories so session joins returned nothing. Provider-side this covers the staleness git anchoring, the memoized single git SHA query, the stale_undetermined verdict when git fails inside a repository, and the log-root resolver; consumer-side it rewrites cc_develop's resolver/scanner/gate and skill bash blocks, cc_foundry's six agent files, and cc_oss's review-mode context block. - Report the index file a query actually loaded as index.index_path, captured at load time rather than recomputed at emission; the codex-rig adapter records it per query, tolerates absence as null for older providers, and reports disagreement with the probe's resolver-derived path as evidence under index_path_divergence instead of reconciling it (structural-context artifact schema version 3). - Compose coexisting codex-rig caveats into stale+degraded instead of letting a stale index mask a coverage gap, and let a targetless standard batch omit target-requiring queries so analysis without --target can report an honest status. - Add fail-closed cache freshness to cc_oss's codemap cache via an index_stamp field (size plus mtime), and read the index under a shared rwgate lease in check-index-currency while raising the helper size ceilings from 50 MB to the engine's 512 MB — the old ceiling reported no_index for any index above 50 MB, measured against this repository's own 131 MB index. - Canonicalize module names from the index instead of sed-based path guessing: resolve_centrality.py gains --modules-only and ordered_modules(), the cc_oss review and dispatch modes query it, and consumer project-name sanitization is dropped for the provider's raw-basename rule. - Add guard infrastructure: new check_codemap_guard.py (MANIFEST-managed vs registry-declared vs provider-CLI taxonomy, wired into audit_static.py and a new check-codemap-guard pre-commit hook), a canonical codemap-context snippet for cc_foundry, and check_cli_flag_drift.py extended to validate flags in a script's own docstring Usage block against its argparse surface with origin tracking. - Align the shared codemap-gates contract with reality: the build route is codemap-py index (the scan-index alias leases in-engine and is a deprecated shim, not ungated as four prose sites claimed), consumer wrappers drop their now-redundant override clauses, and the inject-preamble hook's stale ungated-scan comment and model-facing directive are corrected. - Consolidate the five codemap-py hooks onto a shared _hookutil module for project_name and session_key so path derivation cannot silently diverge between the writer and reader of a session sentinel. - Port setup_scan_env.sh to stdlib-only setup_scan_env.py (Windows-safe, no python3-on-PATH dependency, shared format_scan_args() quoting) with the .sh kept as a deprecated exec shim, and convert claude-skills dispatcher invocations to bare PATH-literal codemap-py form. - Gate new codemap-py code on complexity limits (C901, PLR0911/0912/0915) from the root pyproject.toml via a negated per-file-ignore, refactor check_currency under the gate, and enumerate the six pre-existing offenders as accepted debt. - Correct the codex usage-aggregation docstring to what the README literally claims and record the audit of all 401 captured real turns (exactly one usage event per turn, so max() and sum() are indistinguishable on real data), and add benchmarks/conftest.py so doctest collection can import _bench_common. - Record per-skill codemap route selection in the codex-rig contract with a drift test, and bump plugin versions (codemap-py 0.30.0, codex-rig 0.7.5, cc_foundry 0.46.1, cc_develop 0.22.1, cc_oss 0.28.1, cc_research) with matching CHANGELOG and README updates. --- Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com> Co-authored-by: OpenAI Codex <codex@openai.com>
1 parent 67b5006 commit d6bfcc4

169 files changed

Lines changed: 8068 additions & 1474 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/codex-harness.sh

Lines changed: 72 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -28,17 +28,51 @@ case "${1:-}" in
2828
esac
2929

3030
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
31-
TMP_PARENT="${RUNNER_TEMP:-${TMPDIR:-/tmp}}"
31+
32+
# Git Bash on windows runners: bash keeps POSIX paths, but the child Python is a native
33+
# Windows build that cannot resolve them. Convert at every bash/native boundary.
34+
NATIVE_WINDOWS_PATHS=0
35+
case "$(uname -s)" in
36+
MINGW*|MSYS*|CYGWIN*)
37+
if command -v cygpath >/dev/null 2>&1; then
38+
NATIVE_WINDOWS_PATHS=1
39+
fi
40+
;;
41+
esac
42+
43+
to_child_path() {
44+
if [[ "$NATIVE_WINDOWS_PATHS" -eq 1 ]]; then
45+
cygpath -w "$1"
46+
else
47+
printf '%s\n' "$1"
48+
fi
49+
}
50+
51+
to_host_path() {
52+
if [[ "$NATIVE_WINDOWS_PATHS" -eq 1 ]]; then
53+
cygpath -u "$1"
54+
else
55+
printf '%s\n' "$1"
56+
fi
57+
}
58+
59+
TMP_PARENT="$(to_host_path "${RUNNER_TEMP:-${TMPDIR:-/tmp}}")"
3260
TMP_HOME="$(mktemp -d "$TMP_PARENT/codex-offline-home.XXXXXX")"
3361
TMP_BIN="$TMP_HOME/bin"
3462
RESULTS_DIR="${CODEX_HARNESS_RESULTS_DIR:-$ROOT/.github/codex-harness-results}"
3563
REAL_GIT="$(command -v git || true)"
64+
PYTHON="$(command -v python3 || command -v python || true)"
3665

3766
if [[ -z "$REAL_GIT" ]]; then
3867
echo "missing-command:git" >&2
3968
exit 2
4069
fi
4170

71+
if [[ -z "$PYTHON" ]]; then
72+
echo "missing-command:python3" >&2
73+
exit 2
74+
fi
75+
4276
cleanup() {
4377
rm -rf "$TMP_HOME"
4478
}
@@ -107,23 +141,54 @@ esac
107141
EOF
108142
chmod +x "$TMP_BIN/git"
109143

144+
# env -i also strips the Windows variables a native Python needs to seed hash randomization
145+
# (SystemRoot) and to spawn cmd/pwsh (ComSpec, PATHEXT); without them the child aborts with
146+
# _Py_HashRandomization_Init. Windows spells them mixed-case and bash is case-sensitive, so read
147+
# both spellings and emit both for SystemRoot. Empty on POSIX, where the env line is unchanged.
148+
CHILD_ENV=()
149+
if [[ "$(uname -s)" == MINGW* || "$(uname -s)" == MSYS* || "$(uname -s)" == CYGWIN* ]]; then
150+
WIN_SYSROOT="${SYSTEMROOT:-${SystemRoot:-}}"
151+
WIN_COMSPEC="${COMSPEC:-${ComSpec:-}}"
152+
if [[ -n "$WIN_SYSROOT" ]]; then
153+
CHILD_ENV+=("SystemRoot=$WIN_SYSROOT" "SYSTEMROOT=$WIN_SYSROOT")
154+
fi
155+
if [[ -n "$WIN_COMSPEC" ]]; then
156+
CHILD_ENV+=("COMSPEC=$WIN_COMSPEC")
157+
fi
158+
if [[ -n "${PATHEXT:-}" ]]; then
159+
CHILD_ENV+=("PATHEXT=$PATHEXT")
160+
fi
161+
# HOME alone isolates nothing here: ntpath.expanduser reads USERPROFILE, then HOMEDRIVE plus
162+
# HOMEPATH, and never HOME, so `~` would either escape to the runner's real profile or raise
163+
# RuntimeError once env -i drops all three. Point the Windows spelling at the same temp home.
164+
CHILD_ENV+=("USERPROFILE=$(to_child_path "$TMP_HOME")")
165+
fi
166+
110167
set +e
111168
CALIBRATION_OUTPUT="$(env -i \
112-
HOME="$TMP_HOME" \
113-
CODEX_HOME="$TMP_HOME/.codex" \
114-
TMPDIR="$TMP_PARENT" \
169+
${CHILD_ENV[@]+"${CHILD_ENV[@]}"} \
170+
HOME="$(to_child_path "$TMP_HOME")" \
171+
CODEX_HOME="$(to_child_path "$TMP_HOME/.codex")" \
172+
TMPDIR="$(to_child_path "$TMP_PARENT")" \
115173
PATH="$TMP_BIN:$PATH" \
116174
CI="true" \
117175
CODEX_OFFLINE_HARNESS="1" \
118-
"$ROOT/plugins/codex-rig/runtime/calibration/run.py" --layout plugin --root "$ROOT" 2>&1)"
176+
"$PYTHON" "$(to_child_path "$ROOT/plugins/codex-rig/runtime/calibration/run.py")" \
177+
--layout plugin --root "$(to_child_path "$ROOT")" 2>&1)"
119178
CALIBRATION_EXIT=$?
120179
set -e
121180

122181
if [[ -n "$CALIBRATION_OUTPUT" ]]; then
123182
printf '%s\n' "$CALIBRATION_OUTPUT"
124183
fi
125184

126-
RESULT_PATH="$(printf '%s\n' "$CALIBRATION_OUTPUT" | awk '/\/result\.json$/ { path = $0 } END { print path }')"
185+
# A native Windows Python prints a backslashed path, so accept either separator (the separator
186+
# itself stays required — it is what keeps prose lines from matching), then bring it back to a
187+
# POSIX path for the bash-side -f/dirname/cp below.
188+
RESULT_PATH="$(printf '%s\n' "$CALIBRATION_OUTPUT" | awk '$0 ~ "[/\\\\]result\\.json$" { path = $0 } END { print path }')"
189+
if [[ -n "$RESULT_PATH" ]]; then
190+
RESULT_PATH="$(to_host_path "$RESULT_PATH")"
191+
fi
127192
if [[ -z "$RESULT_PATH" || ! -f "$RESULT_PATH" ]]; then
128193
echo "missing-result-artifact: expected calibration run to print a result.json path" >&2
129194
exit 1
@@ -137,7 +202,7 @@ for artifact in result.json behavioral.json recommendations.md checks.txt leaks.
137202
done
138203

139204
STATUS="$(
140-
python3 - "$RESULT_PATH" "$RESULTS_DIR/summary.md" <<'PY'
205+
"$PYTHON" - "$(to_child_path "$RESULT_PATH")" "$(to_child_path "$RESULTS_DIR/summary.md")" <<'PY'
141206
import json
142207
import sys
143208
from pathlib import Path

.github/workflows/ci-harness.yml

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,9 +19,17 @@ on:
1919
permissions:
2020
contents: read
2121

22+
defaults:
23+
run:
24+
shell: bash
25+
2226
jobs:
2327
offline-harness:
24-
runs-on: ubuntu-latest
28+
runs-on: ${{ matrix.os }}
29+
strategy:
30+
fail-fast: false
31+
matrix:
32+
os: ["ubuntu-latest", "macos-latest", "windows-latest"]
2533
timeout-minutes: 10
2634
env:
2735
CODEX_HARNESS_RESULTS_DIR: ".github/codex-harness-results"

.github/workflows/ci-manifests.yml

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
name: CI manifests
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
paths:
8+
- "plugins/**"
9+
- "benchmarks/**"
10+
- "pyproject.toml"
11+
12+
defaults:
13+
run:
14+
shell: bash
15+
16+
jobs:
17+
manifests:
18+
name: manifest ${{ matrix.manifest }} (${{ matrix.os }})
19+
runs-on: ${{ matrix.os }}
20+
strategy:
21+
fail-fast: false
22+
matrix:
23+
os: ["ubuntu-latest", "macos-latest", "windows-latest"]
24+
manifest: ["provider-parity-methodology", "codex-integration", "codex-agentic"]
25+
timeout-minutes: 10
26+
env:
27+
UV_PYTHON: "3.12"
28+
steps:
29+
- name: 📥 Checkout
30+
uses: actions/checkout@v4
31+
32+
# The job-level UV_PYTHON is uv's own equivalent of --python, so no uv run below
33+
# repeats the version; the action input installs that interpreter.
34+
- name: 📦 Install uv and Python
35+
uses: astral-sh/setup-uv@v5
36+
with:
37+
python-version: ${{ env.UV_PYTHON }}
38+
39+
# Manifests are gitignored generated artifacts, so a fresh checkout has none to
40+
# compare against: build first, then --check that a rebuild is byte-identical.
41+
# The builders form a chain — codex-agentic hashes codex-integration.json, and both
42+
# Codex builders read provider-parity-methodology.json — so a job builds every
43+
# manifest its own target depends on before verifying that target alone.
44+
# Bytes are Python-version independent but embed an OS-resolved scan root, so the
45+
# matrix covers operating systems, not interpreter versions.
46+
- name: 🔨 Build provider-parity-methodology
47+
run: uv run --only-group test python benchmarks/build-provider-parity-methodology-manifest.py
48+
49+
- name: 🔨 Build codex-integration
50+
if: matrix.manifest != 'provider-parity-methodology'
51+
run: uv run --only-group test python benchmarks/build-codex-integration-manifest.py
52+
53+
- name: 🔨 Build codex-agentic
54+
if: matrix.manifest == 'codex-agentic'
55+
run: uv run --only-group test python benchmarks/build-codex-agentic-manifest.py
56+
57+
- name: 🔒 Verify ${{ matrix.manifest }} rebuilds byte-identically
58+
run: uv run --only-group test python "benchmarks/build-${{ matrix.manifest }}-manifest.py" --check

.github/workflows/ci-tests.yml

Lines changed: 47 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -10,51 +10,71 @@ on:
1010
- "pyproject.toml"
1111
- "sync.sh"
1212

13-
defaults:
14-
run:
15-
shell: bash
16-
1713
jobs:
18-
tests:
14+
codex-rig:
1915
runs-on: ${{ matrix.os }}
2016
strategy:
2117
fail-fast: false
2218
matrix:
2319
os: ["ubuntu-latest", "macos-latest", "windows-latest"]
24-
python-version: ["3.10", "3.11", "3.12", "3.13"]
2520
timeout-minutes: 10
26-
21+
env:
22+
UV_PYTHON: "3.12"
23+
# Bytecode written beside a packaged file changes what the manifest checks hash over.
24+
PYTHONDONTWRITEBYTECODE: "1"
2725
steps:
2826
- name: 📥 Checkout
2927
uses: actions/checkout@v4
3028

31-
- name: 🐍 Set up Python
32-
uses: actions/setup-python@v5
29+
# The job-level UV_PYTHON is uv's own equivalent of --python, so no uv run below
30+
# repeats the version; the action input installs that interpreter.
31+
- name: 📦 Install uv and Python
32+
uses: astral-sh/setup-uv@v5
3333
with:
34-
python-version: ${{ matrix.python-version }}
34+
python-version: ${{ env.UV_PYTHON }}
35+
36+
- name: 🐧 Verify Codex Rig entrypoints
37+
if: runner.os != 'Windows'
38+
shell: bash
39+
run: |
40+
uv run --only-group test python plugins/codex-rig/scripts/build_package.py --check
41+
uv run --only-group test python plugins/codex-rig/scripts/validate_package.py
42+
uv run --only-group test python plugins/codex-rig/shared/collect_pr.py --help
43+
44+
# Native pwsh, not the Git Bash the job default supplies: it is the shell a Windows user
45+
# actually invokes these entrypoints from, and the one that surfaced the path defects.
46+
- name: 🪟 Verify native Codex Rig entrypoints
47+
if: runner.os == 'Windows'
48+
shell: pwsh
49+
run: |
50+
uv run --only-group test python plugins/codex-rig/scripts/build_package.py --check
51+
uv run --only-group test python plugins/codex-rig/scripts/validate_package.py
52+
uv run --only-group test python plugins/codex-rig/shared/collect_pr.py --help
53+
54+
tests:
55+
runs-on: ${{ matrix.os }}
56+
strategy:
57+
fail-fast: false
58+
matrix:
59+
os: ["ubuntu-latest", "macos-latest", "windows-latest"]
60+
python-version: ["3.10", "3.11", "3.12", "3.13"]
61+
timeout-minutes: 25
62+
63+
steps:
64+
- name: 📥 Checkout
65+
uses: actions/checkout@v4
3566

3667
- name: 🟢 Set up Node
3768
uses: actions/setup-node@v4
3869
with:
3970
node-version: "20"
4071

41-
- name: 📦 Install uv
72+
# The action's python-version input sets UV_PYTHON, uv's equivalent of --python,
73+
# so the pytest run below does not repeat the matrix version.
74+
- name: 📦 Install uv and Python
4275
uses: astral-sh/setup-uv@v5
43-
44-
- name: 🔒 Verify deterministic benchmark manifests
45-
run: |
46-
uv run --only-group test --python ${{ matrix.python-version }} python benchmarks/build-provider-parity-methodology-manifest.py --check
47-
uv run --only-group test --python ${{ matrix.python-version }} python benchmarks/build-codex-integration-manifest.py --check
48-
uv run --only-group test --python ${{ matrix.python-version }} python benchmarks/build-codex-agentic-manifest.py --check
76+
with:
77+
python-version: ${{ matrix.python-version }}
4978

5079
- name: 🧪 Run plugin tests (Python bin/ + JS hooks)
51-
run: uv run --only-group test --python ${{ matrix.python-version }} pytest -W error::DeprecationWarning
52-
53-
- name: 🪟 Verify native Codex Rig entrypoints
54-
if: runner.os == 'Windows'
55-
shell: pwsh
56-
run: |
57-
$env:PYTHONDONTWRITEBYTECODE = "1"
58-
uv run --only-group test --python ${{ matrix.python-version }} python plugins/codex-rig/scripts/build_package.py --check
59-
uv run --only-group test --python ${{ matrix.python-version }} python plugins/codex-rig/scripts/validate_package.py
60-
uv run --only-group test --python ${{ matrix.python-version }} python plugins/codex-rig/shared/collect_pr.py --help
80+
run: uv run --only-group test pytest -W error::DeprecationWarning --durations=100

.pre-commit-config.yaml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -259,6 +259,13 @@ repos:
259259
pass_filenames: false
260260
always_run: true
261261

262+
- id: check-codemap-guard
263+
name: 🗺️ unmanaged codemap index-guard copy
264+
language: system
265+
entry: python3 plugins/cc_foundry/bin/check_codemap_guard.py
266+
pass_filenames: false
267+
always_run: true
268+
262269
- id: check-bash-persistence
263270
name: 🫙 shell var persistence across Bash blocks
264271
language: system

AGENTS.md

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,19 @@ Simplicity and reliability come first. Understand the affected flow and root cau
1818

1919
Verification is part of implementation. Work is not complete until relevant checks pass and failures, residual risks, and deliberately deferred scope are reported accurately.
2020

21+
## Multi-OS Executables
22+
23+
Scripts, hooks, `bin/` entry points, and CI steps all run on Linux, macOS, and native Windows. A POSIX-only assumption is a defect to fix at the source, never a reason to skip the platform.
24+
25+
- `pathlib`; `Path(p).is_absolute()` not a leading-slash check; `PurePath(p).as_posix()` before hashing, serializing, or comparing a path — native separators change the digest.
26+
- POSIX-absolute literals are not portable fixtures: `/host/x` resolves to `D:\host\x` on Windows.
27+
- Byte-asserted or hashed writes use `newline="\n"` or bytes; text mode emits CRLF on Windows.
28+
- Sanitized subprocess `env=` keeps `SystemRoot`, `SYSTEMROOT`, `COMSPEC`, `PATHEXT`, `TEMP`, `TMP` on win32, else the child Python aborts before running; temp dirs via `os.environ.get("TMPDIR") or tempfile.gettempdir()`, never `/tmp`.
29+
- A workflow `run:` step invoking `.sh` needs explicit `shell: bash` — the Windows default shell dot-sources it and exits zero, a false green.
30+
- Symlinks, file modes, and uid checks are capabilities: degrade in production code first.
31+
- Skips are the last resort: never a blanket `skipif(sys.platform == "win32")`, always a capability probe skipping on `OSError`, with each surviving skip documented and re-audited.
32+
- Green macOS is absence of regression, not Windows support: prove Windows semantics with `PureWindowsPath` or `ntpath`, since monkeypatching `os.name` does not change `pathlib`.
33+
2134
## Benchmark Isolation
2235

2336
Benchmark task IDs, target repositories, prompt wording, expected answers, and task-specific source or symbol examples are test evidence, not production content. Never copy them into shipped plugins, Skills, templates, or user-facing docs; use neutral generic examples and encode the generalized contract in a regression test instead.
@@ -39,6 +52,6 @@ Plugin-specific authoring, installability, cross-reference, versioning, and veri
3952
- Python minimum: 3.10. The repository root is an environment anchor, not an installable package.
4053
- Bootstrap test tooling with `uv sync --only-group test`; benchmark-only dependencies use `uv sync --only-group bench`.
4154
- Run focused tests with `.venv/bin/python -m pytest <paths>` and broaden to the affected suite before completion.
42-
- Lint/format Python edits via the pinned pre-commit hooks, never the bare tool: `pre-commit run ruff-check --files <changed-python-paths>` and `pre-commit run ruff-format --files <changed-python-paths>`; direct `ruff` invocation drifts from the version/config pinned in `.pre-commit-config.yaml`.
55+
- Lint/format edits via the pinned pre-commit hooks, never the bare tool: `pre-commit run ruff-check --files <changed-python-paths>`, `pre-commit run ruff-format --files <changed-python-paths>`, and `pre-commit run mdformat --files <changed-markdown-paths>`; direct `ruff` or `mdformat` invocation drifts from the version/config pinned in `.pre-commit-config.yaml`.
4356
- Use `pre-commit run --all-files` only when the task requires the repository-wide gate; preserve unrelated working-tree changes.
4457
- Release and build entry points are plugin-specific; follow `plugins/AGENTS.md` and the owning plugin's scripts and README. Remote publication remains human-owned.

CLAUDE.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,19 @@ Hook ids (from `.pre-commit-config.yaml`): `ruff-check`, `ruff-format`, `eslint`
4242
- Bootstrap test tooling with `uv sync --only-group test`; benchmark-only dependencies use `uv sync --only-group bench`.
4343
- Run tests with `.venv/bin/python -m pytest <paths>`**not** `uv run pytest` or a bare `pytest`; the project venv is the pinned environment. Start focused, broaden to the affected suite before completion.
4444

45+
## Multi-OS Executables — POSIX Assumption = Defect
46+
47+
Scripts, hooks, `bin/`, CI steps all run Linux + macOS + native Windows. Fix at source; skip never.
48+
49+
- `pathlib`; `Path(p).is_absolute()` not `startswith("/")`; `PurePath(p).as_posix()` before hash/serialize/compare — separators change digests
50+
- POSIX-absolute literals unportable as fixtures: `/host/x``D:\host\x` on Windows
51+
- Byte-asserted or hashed writes: `newline="\n"` or bytes — text mode emits CRLF
52+
- Sanitized subprocess `env=` keeps `SystemRoot`, `SYSTEMROOT`, `COMSPEC`, `PATHEXT`, `TEMP`, `TMP` on win32 — else child Python aborts: `_Py_HashRandomization_Init: failed to get random numbers`; temp dir via `os.environ.get("TMPDIR") or tempfile.gettempdir()`, never `/tmp`
53+
- CI `run:` calling `.sh` needs explicit `shell: bash` — Windows pwsh dot-sources it, exits 0, runs nothing (false green)
54+
- Symlink/mode/uid = capabilities: degrade in production code first
55+
- Skip last resort: never blanket `skipif(sys.platform == "win32")` — probe capability, skip on `OSError`; document + re-audit each surviving skip
56+
- Green macOS ≠ Windows support: prove with `PureWindowsPath`/`ntpath`; monkeypatching `os.name` does not change `pathlib`
57+
4558
## Benchmark Isolation
4659

4760
Benchmark task IDs, target repositories, prompt wording, expected answers, and task-specific source or symbol examples are test evidence, not production content. Never copy them into shipped plugins, skills, templates, or user-facing docs; use neutral generic examples and encode the generalized contract in a regression test instead.

benchmarks/AGENTS.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,12 @@
11
# Benchmark Instructions
22

3+
Root `AGENTS.md` applies here and is not restated: benchmark isolation, the test/lint workflow, multi-OS executables, and Markdown no-wrap. Below is benchmark-specific only.
4+
35
- Use Fire for every Python CLI. Add a study to an existing provider runner when it shares transport or isolation; keep stage-specific contracts and scorers in focused modules rather than creating a second launcher.
46
- Interactive A/B/C result rows are a CLI contract: persist plain rows to a stage run log when that stage has one, and always route terminal output through the existing shared Rich arm renderer. Do not add direct `print()` paths for arm rows; redirected output must remain ANSI-free. Add a focused renderer-forwarding regression for every new stage or rescore path.
57
- Never run paid models. Give the user the exact command emitted by a fresh dry run with its 16-character `--paid-approval` token; retain the complete SHA-256 in benchmark provenance, then analyze only the artifact they provide.
68
- Treat task suites, manifests, frozen repositories, indexes, and input snapshots as immutable benchmark coordinates. Regenerate generated manifests after contract or consumer changes; do not edit result artifacts.
7-
- Convert benchmark findings into generic production contracts; never copy benchmark task IDs, target repositories, prompt wording, expected answers, or task-specific source/symbol examples into shipped plugins, Skills, templates, or user-facing docs.
89
- Keep A/B/C arms symmetric except for the documented treatment supplement. State Codemap's static-graph boundary: use it for compact symbol/dependency/importer/caller facts, not runtime validation, test execution, or edits.
910
- Treat A_plain versus C_strict as the decision-grade comparison. B_auto is an optional-use canary: if it costs more than A_plain, recommend the installed integration rather than treating that as a failure of the strict treatment.
1011
- Executable tasks require benchmark-owned disposable worktrees, canonical diff capture, a second clean scoring worktree, ordinary patch application, independent behavior oracle, and verified cleanup. `--recount` is diagnostic-only.
11-
- Run focused `pytest` for behavior, then invoke the exact pre-commit hooks for changed files: `pre-commit run ruff-check --files <python-files>`, `pre-commit run ruff-format --files <python-files>`, and `pre-commit run mdformat --files <markdown-files>`. For a changed runner execute its relevant `--dry-run` and scope-resolution command; no-model checks may be run by Codex.
12+
- For a changed runner execute its relevant `--dry-run` and scope-resolution command; no-model checks may be run by Codex.

0 commit comments

Comments
 (0)