Skip to content

README.en.md

Latest commit

 

History

History
676 lines (581 loc) · 27.6 KB

README.en.md

File metadata and controls

676 lines (581 loc) · 27.6 KB
Raw

简体中文 · English

RepoPilot — Evidence-first AgentTeam

An auditable repository-maintenance AgentTeam for open-source and engineering teams

Move GitHub Issues and failed CI runs safely toward Pull Requests backed by verifiable execution evidence, or review new Pull Request revisions and maintain one structured review comment — while preserving every Agent, Skill, tool call, approval, rollback point, and verified artifact.


RepoPilot dual loop: Issue to Verified PR, and Pull Request to Review Comment
6
Specialized Agents		 15
Dual-transport MCP Tools		 52/52
Control-plane Reliability Tests		 0
Unapproved Automatic Merges

RepoPilot is built on AgentTeams v1.2.2 and focuses on safe, auditable repository-maintenance automation. Its default policy is pull_request_only: Agents may create branches, commits, and Pull Requests, but they cannot automatically merge, delete branches, change permissions, or modify secrets.

Two Automation Loops

Trigger Automated Work Deliverable Safety Boundary
Issue / Failed CI Triage → root-cause localization → minimal patch → independent verification Pull Request with a Proof Bundle Stops at an open PR
Pull Request Pin head SHA → paginate Diff / Checks → independent review Create or update one managed review comment Comments only; no approval or code mutation

The PR review loop listens for opened, reopened, synchronize, and ready_for_review. Before publication, RepoPilot rechecks the current head SHA. If a new commit appears while the review is running, publication is rejected as stale. The comment carries a stable <!-- repopilot-review --> marker, so each PR has one managed RepoPilot review instead of repeated comment noise.

Workflow Demo

RepoPilot end-to-end demo: webhook event, Agent triage, root-cause localization, multi-file patch, parallel verification, and a safe Pull Request

Public Issue #3 → failing CI → Repo Lead triage → Locator proves the race → Fixer applies a five-file patch → Verifier passes 7/7 checks → PR #4

Open Issue → PR Demo ↗ · Open PR → Comment Demo ↗ · Inspect the Public Review Comment ↗ · Watch the HD MP4 · Inspect Public PR #4

The live demo requires neither model credentials nor an administrator account. By default it replays the verified delivery for repopilot-testbed#3: a failing baseline deterministically reproduces a concurrency race; RepoPilot repairs it with a five-file Types + Store + Processor + Tests + Docs patch; and PR #4 is created after GitHub Actions passes. The PR remains open, leaving merge authority with a human.

What does the automated PR review publish? 
<!-- repopilot-review -->

## RepoPilot PR Review

**Verdict:** NEEDS ATTENTION
**Reviewed revision:** `b504cec7c05cd2f3b84ee9c6ad7a3d3db6eead5c`

### Findings

| Severity | Finding                                              | Location                          |
| -------- | ---------------------------------------------------- | --------------------------------- |
| HIGH     | Current head SHA is not revalidated before publish   | `src/reviews/publisher.ts:19`     |
| MEDIUM   | Review comment lookup scans only the first 100 rows  | `src/reviews/github-client.ts:43` |
| HIGH     | Failed GitHub writes still record published evidence | `src/reviews/publisher.ts:31`     |

The public PR #5 contains five files and 373 added lines, and its GitHub Actions run passes. RepoPilot still identified two HIGH and one MEDIUM semantic issues from the real diff and published the public review comment. The repopilot-reviewer generated the review through the pull-request-review Skill and published it with repopilot_publish_review_comment. The tool validates the Run, repository, PR number, active Reviewer Step, and immutable head SHA before appending review_publication Evidence. The sanitized Run Proof JSON contains the Run ID, Step, eight Evidence entries, and the evidence-chain head hash.

Maintenance Loop

GitHub Issue / Failed CI
          │
          ▼
   Repo Lead triages and decomposes
          │
          ▼
 Locator finds the cause ──► Fixer creates a minimal patch ──► Verifier independently checks it
                                                                    │
                     Human approval ◄── high-risk gate ◄────────────┤
                                                                    │
                                                                    ▼
                                                        Archivist stores a Runbook

GitHub Pull Request
          │
          ▼
 Reviewer pins head SHA ──► paginates Diff / Checks ──► updates one managed Review Comment

Every critical stage records decisions, tool calls, Git references, CI results, and approval events in a PostgreSQL append-only SHA-256 evidence chain. OpenTelemetry traces and the evidence console make the complete execution replayable.

Proof-Carrying Pull Requests

RepoPilot does not treat “a generated PR” as the finish line. Every Pull Request carries a machine-verifiable Proof Bundle:

Run identity + AgentTeams Step timeline + Skill versions
             + tool and decision Evidence + Git / CI references
             + approval history + SHA-256 chain root + deterministic quality gates

curl http://127.0.0.1:3000/api/v1/runs/<run-id>/proof \
  --output artifacts/proof-bundle.json
pnpm build
pnpm evaluate artifacts/proof-bundle.json artifacts/evaluation-report.json

The Proof Score measures evidence completeness. It does not misrepresent control-plane tests as model-generated patch quality; patch correctness is still established by the public testbed, the independent Verifier, and GitHub Checks.

After Runbook archival, the Archivist calls repopilot_publish_proof_comment to publish a sanitized summary of evidence completeness, Agent and Skill outcomes, and the SHA-256 chain root to the PR. Repeated execution updates the same comment, so the Proof travels with the Pull Request rather than remaining accessible only through the control-plane API.

Why RepoPilot

Problem RepoPilot Approach Verifiable Evidence
Issue triage depends on manual effort Repo Lead performs classification, risk assessment, and DAG decomposition Task plan, Matrix events, Run state
Automated fixes tend to guess Locator and Fixer are separated; reproduction and root-cause evidence come first Reproduction command, code locations, impact scope, patch
Self-verification by the author is biased Verifier independently runs before/after tests and inspects GitHub Checks Test results, Check Runs, residual risks
High-risk actions lack control Merge and similar actions require a versioned, single-use human approval Approver, decision, version, consumption time
Maintenance experience is not reusable Archivist deduplicates, sanitizes, structures, and stores Runbooks Source Run, evidence chain, retrieval result

AgentTeam

Agent Responsibility Autonomy Boundary
Repo Lead Triage, risk assessment, DAG decomposition, and delegation Does not modify code; must request approval for high-risk actions
Locator Reproduction, code and symbol localization, impact analysis Read and experiment only; does not modify the repository
Fixer Minimal patch, regression tests, branch, commit, and Pull Request Stops at the Pull Request; never merges or force-pushes
Verifier Independent reproduction, tests, CI, and residual-risk verification Does not modify the patch or treat green CI as merge authorization
Archivist Runbook deduplication, sanitization, structuring, and retention Does not modify repository or GitHub state
Reviewer Reviews a PR at an immutable head SHA and publishes a structured comment Does not approve, modify code, or merge

Complete Agent identities live in deploy/agentteams/repopilot-team.yaml, and reusable Skill contracts live under skills/.

Core Capabilities

Native AgentTeams orchestration
Uses the official agentteams.io/v1beta1 Worker and Team CRDs, Team Leader, Matrix rooms, shared task state, and MinIO workspaces instead of a custom multi-agent simulator.
Production-oriented Skills and MCP tools
Ships six Apache-2.0 custom Skills with versioned manifests. The control plane exposes 15 MCP tools over Streamable HTTP and stdio for Agent Steps, Evidence, approvals, Runbooks, Issues, PRs, changed files, Checks, PR review comments, PR proof comments, and approval-gated merges.
Immutable execution evidence
Evidence uses canonical JSON and a SHA-256 hash chain. Database triggers reject updates and deletions, while the console revalidates the full chain and reports CHAIN VERIFIED.
Production safety boundaries
Includes a GitHub repository allowlist, Webhook HMAC verification, delivery-level concurrency idempotency, an explicit state machine, optimistic approval locking, and single-use approval consumption.
Retrieval and observability
PostgreSQL provides full-text Runbook retrieval with a pgvector-ready schema. An official Alibaba Cloud alibabacloud-agentloop-experience Skill can be enabled optionally. OpenTelemetry exports HTTP, Agent Skill, MCP, and end-to-end Run traces and metrics through OTLP.

MCP Registry

RepoPilot dynamic MCP server scorecard on Glama

The scorecard is generated dynamically by Glama from the public repository and hosted Release. Click it for the current results and checks.

RepoPilot provides reproducible inputs for MCP Registry inspection and hosted releases:

Check Verifiable Implementation
License The repository includes the complete Apache License 2.0 text
Ownership glama.json declares the GitHub maintainer
Buildability Dockerfile builds a non-root production image; the root stdio launcher loads a self-contained runtime
Tool definitions All 15 tools document call boundaries, parameter semantics, and standard MCP safety annotations
Regression protection Release verification runs real initialize and tools/list calls against the isolated runtime package

MCP tool discovery requires neither model credentials nor GitHub credentials. Database-backed and GitHub-backed tools require their corresponding runtime services and secrets only when invoked. See docs/deployment.md for the complete deployment contract.

Architecture

flowchart LR
    GH[GitHub Issue / Failed CI / Pull Request] --> CP[RepoPilot Control Plane]
    CP --> DB[(PostgreSQL + pgvector)]
    CP --> MX[Matrix Admin → Manager DM]
    MX --> M[AgentTeams Manager]
    M --> TL[Repo Lead]
    TL --> L[Locator]
    TL --> F[Fixer]
    TL --> V[Verifier]
    TL --> A[Archivist]
    M --> R[Reviewer]
    L & F & V & A --> MCP[RepoPilot MCP via Higress]
    R --> MCP
    MCP --> GHAPI[GitHub REST API]
    MCP --> DB
    DB --> UI[Evidence Console]
    CP --> OTEL[OTLP / AgentLoop / LoongSuite]
Loading

See docs/architecture.md for the full design.

Repository Layout

RepoPilot/
├── apps/
│   ├── control-plane/       # Fastify REST / Webhook / MCP / approvals / evidence ledger
│   └── console/             # Flight-recorder-style React evidence console
├── packages/contracts/      # Zod schemas, shared types, and explicit state machine
├── deploy/agentteams/       # AgentTeams v1.2.2 Worker / Team manifests
├── skills/                  # 6 reusable RepoPilot Skills
├── evaluation/              # Proof Bundle protocol and reliability benchmark
├── scripts/                 # Skill validation, reliability baseline, and offline evaluation
├── docs/                    # Architecture, security, deployment, and demo documentation
├── CONTRIBUTING.md          # Development workflow, quality gates, and contribution guide
├── Dockerfile               # Production Streamable HTTP Control Plane image
├── glama.json               # Glama MCP Registry maintainer declaration
├── API.md                   # REST / Webhook / MCP inputs and outputs
├── Method.md                # External SDK and HTTP method contracts
└── docker-compose.yml       # PostgreSQL 16 with pgvector

Quick Start

Requirements

  • Node.js 20+
  • pnpm 9+
  • Docker Desktop or Docker Engine

Model credentials are not required to build, test, or run the local control plane.

git clone https://github.com/wellkilo/RepoPilot.git
cd RepoPilot
cp .env.example .env
docker compose up -d postgres
pnpm install --registry=https://registry.npmjs.org
pnpm build
pnpm --filter @repopilot/control-plane start

Open the evidence console at:

http://127.0.0.1:3000

Alternatively, use the initialization script:

./init.sh

Production Container

The root Dockerfile builds the Console, shared contracts, and Control Plane, then runs the production process as a non-root user:

docker build -t repopilot:local .
docker run --rm -p 3000:3000 \
  -e DATABASE_URL=postgres://repopilot:repopilot@host.docker.internal:5432/repopilot \
  -e GITHUB_ALLOWED_REPOSITORIES=wellkilo/repopilot-testbed \
  repopilot:local

Connect MCP clients to http://127.0.0.1:3000/mcp. Inject GITHUB_TOKEN as a runtime secret only when GitHub reads or writes are required; credentials are never included in the image.

stdio MCP

For an MCP Registry or a local stdio client, build the isolated runtime package and start the stable root entry point:

pnpm glama:build
pnpm glama:verify
node mcp-server.mjs

glama:verify performs a real MCP initialization and validates all 15 tools without database or GitHub credentials. Discovery does not connect to external services; actual data or GitHub operations still require real backing services and credentials.

Create the First Run

curl -X POST http://127.0.0.1:3000/api/v1/runs \
  -H 'Content-Type: application/json' \
  -d '{
    "source": {
      "type": "github_issue",
      "repository": "wellkilo/repopilot-testbed",
      "issueNumber": 1
    },
    "executionPolicy": "pull_request_only"
  }'

Set GITHUB_TOKEN in the local .env file when RepoPilot needs to read a GitHub Issue. Without an AgentTeams Matrix configuration, the Run stops at awaiting_dispatch; RepoPilot does not fabricate execution through a mock Agent.

After the repository Webhook subscribes to Pull requests, creating, reopening, synchronizing, or marking a non-draft PR ready for review triggers an independent review Run. The Reviewer reads the PR, paginated diff, and Checks, then idempotently creates or updates one RepoPilot PR Review comment. If the PR receives another commit during review, the stale Run is rejected before publication.

Real AgentTeams Collaboration

Real Agent reasoning requires an OpenAI-compatible model endpoint. Supported choices include:

  • Alibaba Cloud Model Studio and other hosted endpoints;
  • another OpenAI-compatible API;
  • a compatible local endpoint such as Ollama.

Model credentials are provided only to AgentTeams or Higress. They never enter RepoPilot source code, its database, or deployment manifests.

Deployment documentation:

Reproducible Testbed

Repository wellkilo/repopilot-testbed
Issue #3 · Duplicate webhook retries dispatch multiple maintenance tasks
Failing baseline GitHub Actions Run 32444544920
Fix PR Pull Request #4 · 5 files · +75 / -22
Green CI GitHub Actions Run 32444690068 · 7/7 tests
Review PR Pull Request #5 · 5 files · +373 / -0
Review Comment 2 HIGH + 1 MEDIUM · revision b504cec
Review CI / Proof GitHub Actions Run 32557838055 · 8-entry evidence chain

The testbed contains a deterministic concurrency defect: two requests carrying the same GitHub delivery can pass through the find / save window concurrently, create two tasks, and dispatch twice. RepoPilot consolidates create-or-reuse behavior into DeliveryTaskStore.getOrCreate, shares one in-flight Promise for the same delivery, and adds sequential-retry and different-delivery negative controls. The PR remains open for inspection and has not been merged automatically.

The PR review demo uses a separate public PR #5. The code compiles, all five tests pass, and GitHub Actions is green, but the Reviewer does not equate green CI with semantic correctness. It paginates five changed files, finds stale-SHA, comment-pagination, and failed-Evidence defects at pinned revision b504cec, and publishes one managed comment through the real GitHub REST API. The sanitized docs/assets/demo/pr-review-run.json contains the review_publication record and its eight-entry hash-chained Evidence trail.

Validation

pnpm typecheck
pnpm benchmark:reliability
pnpm skills:validate
pnpm lint
pnpm format:check
pnpm build

The current control-plane reliability baseline is 52/52. Tests cover state transitions, Webhook signature verification, evidence hashing, immutable database triggers, delivery-level concurrency idempotency, approval versioning and single-use consumption, Agent Skill Step lifecycles, Proof Bundle integrity, PR review provenance and stale-SHA protection, HTTP conflict semantics, and console labels. CI produces a structured JSON report.

Safety Boundaries

Allowed by Default Explicit Human Approval Required
Read Issues / PRs / CI, create branches and commits, push non-protected branches, create Pull Requests, create or update managed review comments, read Checks, and append Evidence Merge Pull Requests, delete branches, perform destructive rollbacks, change permissions, modify secrets, or execute another high-risk tool

See docs/security.md for the threat model and production-hardening checklist.

Documentation

Document Contents
API.md REST, Webhook, and MCP schemas
Method.md AgentTeams, Matrix, GitHub, PostgreSQL, and OpenTelemetry method contracts
docs/architecture.md Architecture, state machine, and deployment profiles
docs/security.md Permissions, approvals, credentials, and Evidence integrity
docs/deployment.md Local, AgentTeams, Webhook, and observability deployment
docs/demo.md Public reproduction workflow, execution evidence, and failure branches
evaluation/README.md Reliability layers, Proof Bundle, and benchmark
docs/roadmap.md Engineering roadmap and planned capabilities
CONTRIBUTING.md Development setup, quality gates, and contribution workflow

Current Boundaries

  • Without a configured model service, RepoPilot cannot perform real AgentTeams reasoning. Build, tests, the control plane, and the public testbed remain fully usable.
  • Runbooks use PostgreSQL full-text search by default; vector(1536) is reserved for semantic retrieval.
  • AgentLoop Recall is optional and falls back to local Runbooks when credentials are unavailable.
  • The console currently receives approval identity from trusted reverse-proxy headers. Production deployments must integrate OIDC or SSO.

RepoPilot · Make repository automation observable, reviewable, and reversible.

Apache-2.0