Agentic engineering roadmap: contract-driven implementation, independent acceptance and gated delivery #442

Goal

Evolve this repository from agents following engineering instructions into a traceable delivery loop: requirement contract -> scoped implementation -> independent acceptance -> deterministic review/CI gates -> existing release delivery -> effectiveness review.

The user requested that the proposed direction be recorded in an issue and implementation move forward. This is authorization to improve repository engineering, not to install into a live profile, read credentials, perform OAuth, restart Sessions, modify Core, or activate new paid/scheduled agents.

Existing foundation

• AGENTS.md, agent-contract.json, and scripts/agent.mjs define ownership, plans, risks and permission boundaries.
• pnpm verify covers contract checks, types, tests, build and package imports.
• Windows/Linux CI verifies unchanged exact tagged Core source and published artifacts.
• release-ready and the reusable Release workflow gate main publication and verify GitHub/npm artifacts.
• The evidence ladder and evidence-driven iteration already distinguish synthetic, runtime, published, installed and loaded evidence.

Roadmap

• Phase 1: add bounded, machine-checkable task acceptance contracts and commit-bound evidence validation through existing tooling; prevent current-Core documentation drift; document precisely what the checker does and does not attest. Completed in feat(tooling): add commit-bound acceptance evidence pilot #445; implementation issue Add commit-bound acceptance evidence tooling and current-Core template drift checks #443.
• Phase 2: integrate independently executed acceptance and bounded repair loops using native sessions/jobs and existing Actions. Enforce evaluator/implementation separation through trusted workflow and permissions, not prompt text alone. Preserve required reviews and release ownership. Completed in feat(tooling): enforce explicit environments and exact-head CI evidence #452 (merge a7a59c56); exact-head CI and deterministic evidence guard passed. Evidence validation remains consistency-only and does not itself attest agent independence.
• Phase 3: curate isolated historical engineering tasks with held-out acceptance cases; compare task success, first-pass acceptance, repair rounds, human intervention, elapsed time and total cost including failed attempts. Keep samples and unknowns explicit.

Start with a small credential-free tooling pilot. These phases are separate increments, not permission to claim the entire roadmap complete after a schema check.

Acceptance design

Each task identifies its tracking issue, scope, non-goals, requirement IDs, verification methods and required evidence. Evidence identifies the exact source revision, environment, result and provenance. Results distinguish passed, failed, blocked and not-run; missing, stale, duplicate or unknown evidence must not pass. Later code changes invalidate earlier acceptance evidence.

Independent acceptance starts from original requirements and adverse cases, not only the implementation report. For bug fixes, demonstrate the relevant regression on the old version and success on the candidate when feasible. Browser interaction and persistence checks complement unit tests; screenshots alone are not behavioral acceptance.

Use deterministic checks for machine-verifiable outcomes. AI review is additional evidence, not a substitute for required human review, branch protection, real execution, or product judgment. A report written by an agent is not an attestation of execution. Protect acceptance policy, tests and workflow changes from self-approval in the trusted integration phase.

Concrete initial gap

.github/PULL_REQUEST_TEMPLATE.md names 0.2.1-alpha.5 while package metadata and current CI target 0.2.1-rc.3. Resolve this drift and mechanically prevent recurrence without rewriting historical evidence.

Boundaries

• Plugin-only; no Core changes, patched dependencies, second orchestration daemon or parallel publication path.
• Reuse existing contract/tooling and source-of-truth metadata.
• No production account/profile access, automatic diagnostic uploads, sign-out, installation or restart.
• Bounded work and repair attempts; do not rerun failures until green or weaken assertions.
• Keep repository-internal tooling separate from product release-bearing changes; existing release policy determines publication.
• Evidence validation cannot prove agent independence, live runtime behavior or branch-protection configuration by itself.

References

• OpenAI, Harness engineering: https://openai.com/index/harness-engineering/
• Anthropic, Effective harnesses for long-running agents: https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents
• Anthropic, Demystifying evals for AI agents: https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents
• GitHub Agentic Workflows (currently public preview): https://docs.github.com/en/copilot/concepts/agents/about-github-agentic-workflows

Progress reporting

Link implementation issues/PRs and actual evidence for each phase. Leave later phases open until their outcomes are demonstrated. Report merged/published/installed/loaded states separately.