research: Greptile MCP vs codereviewbuddy — feature comparison & agent experience log

[ichoosetoaccept]

Goal

Compare Greptile's MCP server with codereviewbuddy's MCP server by running them in isolation and logging the agent experience. Identify what Greptile does better that we should adopt, and what codereviewbuddy does better that Greptile lacks.

Experiment

1. Disable codereviewbuddy MCP server in Windsurf
2. Enable only Greptile MCP server (https://api.greptile.com/mcp)
3. Use it for real PR review work and log observations below
4. Re-enable codereviewbuddy and compare

Greptile MCP tools (11 total)

Tool	What it does	codereviewbuddy equivalent
list_pull_requests	List PRs with stats	— (we use gh pr list)
get_merge_request	PR details + Greptile comments	—
list_merge_request_comments	PR comments, filterable by addressed	list_review_comments
list_code_reviews	Review history with status	—
get_code_review	Full review body	—
trigger_code_review	Programmatic re-review	request_rereview
search_greptile_comments	Cross-repo comment search	—
list_custom_context	List review rules/patterns	show_config (partial)
get_custom_context	Get rule details	—
search_custom_context	Search rules	—
create_custom_context	Create review rules	—

What Greptile MCP may do better

• addressed tracking — semantic "was this comment addressed?" vs our file-level staleness
• trigger_code_review — direct API call vs posting @greptileai review comment
• linkedMemory — each comment links to the rule that triggered it
• Cross-repo comment search — find patterns across all repos
• Custom context CRUD — manage review rules via MCP (we use a TOML file)

What codereviewbuddy does that Greptile MCP lacks

• Multi-reviewer aggregation — unified view across Greptile + Unblocked + any reviewer
• Severity triage — 🔴 bug / 🚩 flagged / 🟡 warning / 📝 info classification
• Stack auto-discovery — walk Graphite branch chain to find PR stack
• Config-based resolve policies — block resolving bugs without a reply
• Staleness detection — compare API diff-based staleness
• Batch resolve — resolve_stale_comments bulk operation
• Reply to comments — reply_to_comment with GraphQL mutation
• Issue creation — create_issue_from_comment for tracking suggestions

Agent experience log

Observations will be added as comments after using Greptile MCP in isolation.

[ichoosetoaccept]

Agent Experience Log — Entry 1

Setup

• Greptile MCP server added to Windsurf, codereviewbuddy MCP server disabled
• Greptile GitHub App was suspended since Jan 11 — unsuspended to fix

Greptile MCP Tools — First Impressions

What worked well:

• list_code_reviews returned historical reviews for ismar.ch ✅
• list_merge_request_comments with greptileGenerated=true filter returned the review with full structured data ✅
• Review output quality is good: summary, confidence score, file table, sequence diagram

What failed:

• list_merge_requests returned 0 results for codereviewbuddy despite the repo being enabled (may need time to index?)
• trigger_code_review returned 404 "Cannot POST /" — broken endpoint
• get_merge_request returned "Merge request not found" for PR fix: use MCP workspace roots for gh CLI cwd (#142) #155
• All failures were for codereviewbuddy; may be an indexing lag after unsuspension

Comparison notes:

• Greptile's PR summary is comparable to our review_pr_descriptions tool
• The addressed field on comments is interesting but untested yet (only 1 comment so far)
• No equivalent to our triage_review_comments severity classification
• No stack auto-discovery — Greptile reviews PRs individually
• The linkedMemory field was null on the comment — may need custom context rules configured first

Missing from Greptile MCP (things I needed but couldn't do):

1. Resolve/reply to comments — no tool for this, would need our server or gh CLI
2. Stack-aware operations — no concept of PR stacks
3. Batch operations — no bulk resolve, no triage across multiple PRs

[ichoosetoaccept]

Agent Experience Log — Entry 2

Using Greptile MCP while implementing #110

What worked well:

• list_merge_request_comments(greptileGenerated=true) cleanly fetches only Greptile comments ✅
• list_code_reviews shows real-time review status (REVIEWING_FILES for feat: block resolving threads without a reply (#110) #157) ✅
• Auto-trigger on push worked — feat: block resolving threads without a reply (#110) #157 got queued automatically without @greptileai review ✅
• All 4 PRs in stack reviewed: feat: expose show_config MCP tool (#108) #153 (5/5), docs: improve server instructions with severity-first workflow (#135) #154 (5/5), fix: use MCP workspace roots for gh CLI cwd (#142) #155 (4/5), feat: block resolving threads without a reply (#110) #157 (in progress)
• Summaries are high quality: accurate descriptions, useful confidence scores, sequence diagrams

What still fails:

• trigger_code_review — still 404 "Cannot POST /" even after unsuspension ❌
• This is a Greptile MCP server bug — their endpoint routing is broken

Observations:

• Greptile posts summary-only comments (no inline code comments) when it finds nothing to flag
• CommitGuard (commit hook) caught a valid architecture concern (first: 50 hardcoding) that Greptile missed
• Greptile reviewed 4 PRs in ~2 minutes each — fast turnaround
• The addressed field was false on all comments (makes sense, they are new)

Comparison scorecard update

Feature	Greptile MCP	codereviewbuddy	Winner
Auto-trigger reviews	✅ Push-based	❌ Manual only	Greptile
Review quality	✅ Summary + diagram	N/A (aggregator)	Greptile
Fetch review comments	✅ list_merge_request_comments	✅ list_review_comments	Tie
Review status tracking	✅ list_code_reviews with status	❌	Greptile
Programmatic re-review	❌ Broken (404)	✅ request_rereview	CRB
Resolve threads	❌ Not available	✅ resolve_comment	CRB
Reply to threads	❌ Not available	✅ reply_to_comment	CRB
Severity triage	❌	✅	CRB
Stack awareness	❌	✅	CRB
Batch operations	❌	✅	CRB

Verdict after entry 2: Greptile is an excellent reviewer but the MCP server is read-only + one broken write tool. For the full review workflow (triage → reply → resolve → re-review), codereviewbuddy is still essential. The ideal setup: Greptile as reviewer, codereviewbuddy as the orchestration/workflow layer.

[ichoosetoaccept]

Agent Experience Log — Entry 3 (Final)

Stack merge behavior & conclusion

New finding: Greptile does NOT support stacked PRs. When PRs #153→#154→#155→#157 merged bottom-up via Graphite, Greptile interpreted the base branch merge into the next PR as a "new commit" and triggered a spurious re-review with new inline comments during the merge cascade. This is disruptive noise at exactly the wrong time.

Greptile auto-resolve: Does not auto-resolve its own inline comments. My reply was posted as a PR-level comment instead of directly on the review thread — that may be why, but even so, the lack of auto-resolve is a gap.

Final feature coverage

Feature	Greptile MCP	codereviewbuddy	Winner
Auto-trigger reviews	✅ Push-based	❌ Manual only	Greptile
Review quality	✅ Summary + diagram + inline	N/A (aggregator)	Greptile
Stacked PR support	❌ Spurious re-reviews on merge	✅ Stack-aware	CRB
Fetch review comments	✅ list_merge_request_comments	✅ list_review_comments	Tie
Review status tracking	✅ list_code_reviews	❌	Greptile
Programmatic re-review	❌ Broken (404)	✅ request_rereview	CRB
Resolve threads	❌ Not available	✅ resolve_comment	CRB
Reply to threads	❌ Not available	✅ reply_to_comment	CRB
Auto-resolve own comments	❌	N/A	—
Custom context/rules	✅ create_custom_context (unexplored)	N/A	Greptile
Severity triage	❌	✅	CRB
Stack awareness	❌	✅	CRB
Batch operations	❌	✅	CRB
Cross-PR comment search	✅ search_greptile_comments (unexplored)	❌	Greptile

MCP tools explored

• ✅ list_merge_requests — works well
• ✅ list_code_reviews — real-time status tracking, great
• ✅ list_merge_request_comments — clean filtering by greptileGenerated
• ❌ trigger_code_review — 404 "Cannot POST /", broken
• 🔲 create_custom_context — not explored
• 🔲 search_greptile_comments — not explored
• 🔲 get_merge_request — not explored (addressed tracking)

Verdict

Greptile is an excellent reviewer — fast, accurate summaries, useful sequence diagrams, actionable inline comments. But the MCP server is essentially read-only (no resolve, no reply, broken trigger). Combined with no stacked PR support and no auto-resolve, it cannot serve as the sole review workflow tool.

Recommendation: Use Greptile as a reviewer alongside codereviewbuddy as the orchestration layer. Or switch to CodeRabbit which has native stack support + richer MCP/CLI integration.

Decision: Removing Greptile from this repo, re-enabling CodeRabbit for further testing.