examples: add skills-based sandboxed code review agent#164
Open
Eternally72 wants to merge 1 commit into
Open
Conversation
|
CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅ |
There was a problem hiding this comment.
Pull request overview
This PR adds a complete, end-to-end example “skills-based” automated code review agent to the repository, combining deterministic rule-based findings with governed sandbox execution (Docker), filter-based command governance, SQLite persistence, and JSON/Markdown reporting.
Changes:
- Introduces a
code-reviewSkill with bounded/paginated diff parsing, deterministic rule scripts, and controlled repository file inspection helpers. - Adds a hardened Docker sandbox runtime with strict resource/network constraints and output redaction/limits, plus a fake sandbox for dry-run testing.
- Implements SQLite-backed audit/persistence + reporting pipeline and provides fixtures + acceptance/integration tests for the required scenarios.
Reviewed changes
Copilot reviewed 67 out of 67 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| examples/skills_code_review_agent/tests/run_docker_tests.py | Docker-backed integration test that validates sandbox hardening, pagination/output limits, and governed execution. |
| examples/skills_code_review_agent/tests/fixtures/test-missing.diff | Fixture diff for “tests missing” signal. |
| examples/skills_code_review_agent/tests/fixtures/sensitive-redaction.diff | Fixture diff containing representative secrets to validate redaction and detection. |
| examples/skills_code_review_agent/tests/fixtures/security.diff | Fixture diff for command-execution security detection. |
| examples/skills_code_review_agent/tests/fixtures/sandbox-timeout.diff | Fixture diff to simulate sandbox timeout behavior in fake mode. |
| examples/skills_code_review_agent/tests/fixtures/sandbox-failure.diff | Fixture diff to simulate sandbox failure behavior in fake mode. |
| examples/skills_code_review_agent/tests/fixtures/duplicate-finding.diff | Fixture diff to validate deduplication behavior. |
| examples/skills_code_review_agent/tests/fixtures/database-lifecycle.diff | Fixture diff for DB connection lifecycle detection. |
| examples/skills_code_review_agent/tests/fixtures/clean.diff | Fixture diff to validate low false-positive behavior. |
| examples/skills_code_review_agent/tests/fixtures/async-resource-leak.diff | Fixture diff for async detached-task detection. |
| examples/skills_code_review_agent/tests/evaluate_fixtures.py | Public fixture evaluation harness that enforces recall/FP/redaction thresholds. |
| examples/skills_code_review_agent/storage/sqlite.py | SQLite persistence implementation with schema hardening, redaction, and normalized audit tables. |
| examples/skills_code_review_agent/storage/schema.sql | SQLite schema for tasks, inputs, runs, filter decisions, findings, monitoring, and full reports. |
| examples/skills_code_review_agent/storage/factory.py | Env/CLI-driven storage backend selection (currently SQLite only). |
| examples/skills_code_review_agent/storage/base.py | Storage interface contract for review persistence. |
| examples/skills_code_review_agent/storage/init.py | Storage package marker. |
| examples/skills_code_review_agent/skills/code-review/SKILL.md | Skill entry doc describing allowed workflows, pagination, and safety rules. |
| examples/skills_code_review_agent/skills/code-review/scripts/run_review_rules.py | Aggregate runner that paginates findings + changed-line evidence under output limits. |
| examples/skills_code_review_agent/skills/code-review/scripts/review_tests.py | Deterministic “test_missing” rule for source-only behavioral changes. |
| examples/skills_code_review_agent/skills/code-review/scripts/review_security.py | Deterministic security rule for execution/deserialization/dynamic SQL boundaries. |
| examples/skills_code_review_agent/skills/code-review/scripts/review_secrets.py | Deterministic rule that flags already-redacted secret lines without leaking plaintext. |
| examples/skills_code_review_agent/skills/code-review/scripts/review_resources.py | Deterministic resource lifecycle rule (files/sockets/processes/locks). |
| examples/skills_code_review_agent/skills/code-review/scripts/review_git_changes.py | Bounded git diff collector that emits paginated rule evidence + digest. |
| examples/skills_code_review_agent/skills/code-review/scripts/review_database.py | Deterministic DB lifecycle rule (connections/sessions/cursors/transactions). |
| examples/skills_code_review_agent/skills/code-review/scripts/review_common.py | Shared helpers: diff loading, evidence extraction, dedup, and rule CLI harness. |
| examples/skills_code_review_agent/skills/code-review/scripts/review_async.py | Deterministic async correctness rule (detached tasks, missing await, blocking calls). |
| examples/skills_code_review_agent/skills/code-review/scripts/parse_unified_diff.py | Unified diff parser with redaction, line mapping, and bounded parsing. |
| examples/skills_code_review_agent/skills/code-review/scripts/inspect_git_files.py | Bounded git file enumeration with pagination/digest for scope verification. |
| examples/skills_code_review_agent/skills/code-review/scripts/inspect_files.py | Controlled file reader with path safety checks, scope enforcement, and redaction. |
| examples/skills_code_review_agent/skills/code-review/scripts/inspect_file_list.py | File-list validator that rejects unsafe/secret paths and paginates output. |
| examples/skills_code_review_agent/skills/code-review/references/RULES.md | Rule taxonomy + confidence/severity guidance. |
| examples/skills_code_review_agent/skills/code-review/agents/openai.yaml | Skill UI metadata for “Code Review”. |
| examples/skills_code_review_agent/security.py | Central redaction utilities for report/store outputs and secret-path heuristics. |
| examples/skills_code_review_agent/sandbox/lazy.py | Lazy runtime wrapper to defer Docker startup until needed. |
| examples/skills_code_review_agent/sandbox/fake.py | Fake sandbox execution for deterministic dry-run/fake-model testing. |
| examples/skills_code_review_agent/sandbox/factory.py | Env/CLI-driven sandbox backend selection (currently Docker only). |
| examples/skills_code_review_agent/sandbox/Dockerfile | Minimal hardened image definition for the review sandbox. |
| examples/skills_code_review_agent/sandbox/docker.py | Hardened Docker runtime: read-only mounts, no network, resource limits, bounded output, redaction. |
| examples/skills_code_review_agent/sandbox/base.py | Sandbox provider interface contract. |
| examples/skills_code_review_agent/sandbox/.dockerignore | Restricts Docker build context to Dockerfile only. |
| examples/skills_code_review_agent/sandbox/init.py | Sandbox package marker. |
| examples/skills_code_review_agent/run_agent.py | CLI entrypoint wiring inputs, workflow, storage, sandbox selection, and fake/dry-run modes. |
| examples/skills_code_review_agent/reports/writers.py | Atomic JSON/Markdown report writer with Markdown-escaping to prevent injection. |
| examples/skills_code_review_agent/reports/models.py | Pydantic models defining report/task/run/finding schemas. |
| examples/skills_code_review_agent/reports/init.py | Reports package marker. |
| examples/skills_code_review_agent/README.md | Chinese README explaining architecture, usage, safety model, and validation commands. |
| examples/skills_code_review_agent/pyproject.toml | Example project metadata and Python version constraints. |
| examples/skills_code_review_agent/inputs/parser.py | Normalizes diff/file-list/worktree inputs and stages safe copies for sandbox mounting. |
| examples/skills_code_review_agent/inputs/models.py | Parsed input model used during workflow execution. |
| examples/skills_code_review_agent/inputs/init.py | Inputs package marker. |
| examples/skills_code_review_agent/filters/sdk_filter.py | Tool filter that enforces the command policy and records decisions in metadata. |
| examples/skills_code_review_agent/filters/policy.py | Deterministic governance policy restricting commands, args, environment, scope, and budgets. |
| examples/skills_code_review_agent/filters/init.py | Filters package marker. |
| examples/skills_code_review_agent/examples/review_report.md | Sample Markdown report output. |
| examples/skills_code_review_agent/examples/review_report.json | Sample JSON report output. |
| examples/skills_code_review_agent/docs/design.md | Short design document describing the architecture and safety boundaries. |
| examples/skills_code_review_agent/agent/tools.py | Skill tool wiring + filter attachment + safe tool exposure. |
| examples/skills_code_review_agent/agent/prompts.py | Prompt builder + caching evidence compaction for model requests. |
| examples/skills_code_review_agent/agent/normalization.py | Dedup/noise reduction, confidence routing, and scope validation for model outputs. |
| examples/skills_code_review_agent/agent/fake.py | Fake-model path that reuses deterministic rules without external model calls. |
| examples/skills_code_review_agent/agent/config.py | Model config validation + review budget limits from env. |
| examples/skills_code_review_agent/agent/agent.py | Constructs the LLM agent with toolset + output schema. |
| examples/skills_code_review_agent/agent/init.py | Agent package marker. |
| examples/skills_code_review_agent/.gitignore | Ignores local env, reports, sqlite db, and tool caches for the example. |
| examples/skills_code_review_agent/.env.example | Example env file documenting required/optional settings and safety ceilings. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+17
to
+23
| def _safe_path(data: bytes) -> tuple[str, bool, bool]: | ||
| text = data.decode("utf-8", errors="replace") | ||
| normalized = any(ord(character) < 32 for character in text) | ||
| text = "".join(character if ord(character) >= 32 else "�" for character in text) | ||
| if len(text) <= MAX_PATH_CHARS: | ||
| return text, False, normalized | ||
| return f"{text[: MAX_PATH_CHARS - 1]}…", True, normalized |
Comment on lines
+60
to
+64
| recommendation TEXT NOT NULL, | ||
| confidence REAL NOT NULL, | ||
| source TEXT NOT NULL, | ||
| UNIQUE(task_id, bucket, file, line, category) | ||
| ); |
Comment on lines
+306
to
+309
| backing_runtime = runtime._runtime.runtime | ||
| container_attributes = backing_runtime.container.container.attrs | ||
| host = container_attributes.get("HostConfig", {}) | ||
| config = container_attributes.get("Config", {}) |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #164 +/- ##
==========================================
Coverage ? 87.54280%
==========================================
Files ? 467
Lines ? 44103
Branches ? 0
==========================================
Hits ? 38609
Misses ? 5494
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Author
|
I have read the CLA Document and I hereby sign the CLA |
Comment on lines
+65
to
+69
| status = raw[:2].decode("ascii", errors="replace") | ||
| path_bytes = raw[3:] | ||
| # Porcelain -z adds the original path as the next NUL record. | ||
| if "R" in status or "C" in status: | ||
| index += 1 |
Comment on lines
+82
to
+84
| CREATE INDEX IF NOT EXISTS idx_findings_task_id ON findings(task_id); | ||
| CREATE UNIQUE INDEX IF NOT EXISTS idx_findings_unique_issue | ||
| ON findings(task_id, file, COALESCE(line, -1), category); |
Comment on lines
+306
to
+308
| backing_runtime = runtime._runtime.runtime | ||
| container_attributes = backing_runtime.container.container.attrs | ||
| host = container_attributes.get("HostConfig", {}) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a Skills-based automated code review Agent with sandbox execution,
Filter governance, SQLite persistence, structured findings, monitoring,
sensitive-data redaction, and dry-run support.
Fixes #92
Validation
Release notes
RELEASE NOTES: Added a sandboxed automated code review Agent example.