Agent Team Readiness scan core (atr-1)

[Gandy2025]

Agent Team Readiness scan core (atr-1)

Evidence-first, read-only scanner that assesses whether a repository gives multiple coding agents enough repo-level structure to work safely in parallel — not a generic "has AGENTS.md" check. Conclusions are limited to repo-observable capability; runtime, organization, and team-behavior questions stay explicit unknowns.

What's in this PR

• Six-dimension rubric (weights sum to 100): instruction convergence (18), task/workspace isolation (18), code & domain ownership (16), shared decisions & context (16), repeatable verification (20), handoff & definition of done (12). Per dimension: score/band, strongest evidence (file/line/command), must-fix blocker, minimum improvement, preserved unknowns.
• atr-1 result contract: executable validator (lib/validate.mjs), published JSON Schema (schemas/atr-1.schema.json), and prose reference (references/atr-1.md). Headline score is withheld below 60% evidence coverage instead of being guessed.
• Deterministic collectors first, interpretation second (lib/evidence.mjs): instruction files & conflict detection, isolation-policy polarity (shared-branch mandates are critical blockers, not credit), verification commands (package scripts / Make / Cargo / Go / pytest / CI) with no-op and hollow-surface rejection, CODEOWNERS rule counting, architecture/ADR discovery (md/rst/adoc/txt), handoff templates with placeholder rejection.
• Generated advisory drafts (never written into the scanned repo): tailored AGENTS.draft.md + context-tree-seed-map.md, both marked review-only.
• Safety boundary: never executes target-repo commands; disables fsmonitor/hooks/LFS smudge and repo-local Git config; skips symlinks, deleted-but-cached, unreadable, and out-of-root paths; refuses output inside the target and refuses to overwrite/follow pre-existing artifact symlinks; bounds all emitted evidence to 240 chars and redacts token families (GitHub, GitLab, Slack, AWS, Stripe, npm, PyPI, Google API, bearer/credential URLs, high-entropy strings).
• Fixtures + tests: 25 fixtures covering the required matrix (no instructions / conflicting / monorepo-scoped / missing & hollow & no-op verification / ownership clear & missing / handoff clear & missing) plus adversarial ones (fixture shadowing, untrusted content, unsafe shared-branch wording variants, placeholder/frontmatter-only evidence, prose-as-command, uses-only CI, long conflict evidence). 44 node tests.
• 12-repo pinned calibration (calibration/): first-tree, express, flask, requests, chi, ripgrep, axum, fastify, vue-core, github-cli, uv, p-map — human-reviewed bands per dimension, 72/72 band agreement, with per-dimension evidence/unknowns preserved and an honest false-positive / false-negative / unknown analysis (calibration/analysis.md, 20 discrepancies found and fixed across calibration and review rounds, residual risks listed).
• Real sample report (examples/first-tree/): all four artifacts from scanning agent-team-foundation/first-tree at a pinned commit.

Run it

node ./bin/atr-scan.mjs https://github.com/agent-team-foundation/first-tree \
  --ref 2d5bacb917c26400598563ef5500f9b4be7c5140 \
  --generated-at 2026-07-16T00:00:00.000Z --output /tmp/first-tree-atr

Output (real run):

{
  "schema_version": "atr-1",
  "repository": "agent-team-foundation/first-tree",
  "revision": "2d5bacb917c26400598563ef5500f9b4be7c5140",
  "headline_score": 84,
  "evidence_coverage": 99,
  "artifacts": ["atr-1.json", "evidence.json", "AGENTS.draft.md", "context-tree-seed-map.md"]
}

Validate and calibrate:

node ./.claude/skills/agent-team-readiness/scripts/validate-report.mjs /tmp/first-tree-atr/atr-1.json
npm run calibrate -- --manifest calibration/repos.json \
  --review calibration/human-review.json --output /tmp/atr-calibration
npm test   # 38/38
npm run check

Reproducibility: re-running calibration and the first-tree example against this branch reproduces calibration/baseline.json, calibration/REPORT.md, and examples/first-tree/* byte-for-byte (implementation + input digests are embedded and asserted).

Review history

• R1/R2 (independent adversarial passes during development) surfaced 5 boundary blockers — shared-branch wording misjudged, empty/no-op verification & CI miscounted, frontmatter-only pseudo-evidence, GitLab token redaction gap, unbounded conflict evidence. All fixed with regression tests (tests/scanner.test.mjs).
• R3 (pre-PR): two independent reviewers — adversarial security/correctness and contract/deliverable compliance. The security pass confirmed all five R2 fixes have non-vacuous killing tests, ruled out ReDoS, and validated the git-hardening/symlink/output/redaction defenses; the contract pass reproduced the calibration rows from pinned commits and mutation-fuzzed the schema against the executable validator. New findings, all fixed here with regression tests:
  • (blocker) English prose starting with a tool word ("Make sure you run the test…", "Go through the docs.") parsed as commands — fabricating false root-command conflicts and letting prose-only instruction files score actionable. Command candidates are now case-sensitive and reject sentences with English function words (fixtures/prose-instructions + prose-only test).
  • (major) uses:-only automation workflows (stale bots, labelers) counted as CI verification gates, suppressing the no-CI-gate blocker — found independently by both reviewers. Only shell-executing steps count now (fixtures/uses-only-ci); in calibration this replaced Flask's lock-bot and Fastify's backport-bot citations with their real check workflows, zero score drift.
  • (major) Makefile variable assignments (test := …) matched the target probe, fabricating make test and downgrading the critical no-test blocker (fixtures/make-assignment-verification).
  • (major) Unqualified bans on isolation mechanisms ("Do not use git worktrees.") earned safe strong-policy credit instead of the critical shared-state blocker; bare descriptive keyword mentions ("Git worktrees are neat.") earned +3 root-strong credit against the rubric's own rule. Both fixed with directive/polarity filters and sentence-matrix tests.
  • (minors/nits) top-3 prioritization formula documented in the contract references + schema; calibration test gate restructured from hard-pinned 72/0 to structural consistency with explained disagreements; verification caps documented in the rubric; README node version, calibrate fresh-output note, PR-template dead tool reference, --generated-at round-trip validation, broader npm run check coverage.
  • Post-fix: full 12-repo calibration re-run — 72/72 band agreement preserved, all 72 dimension scores unchanged, evidence citations improved; first-tree example artifacts byte-identical.

Provenance

Implementation started by gandy-coder; taken over, verified, and finished by gandy-s-assistant per gandy2025's direction. Campaign/landing wiring (First Tree catalog + website landing page) follows as separate small PRs from latest main.