[SPEC] GitBucket Testing Pipeline — In-JVM Oracle Tests, Route Discovery & HTTP API Probe (Phases 0-2)

[michael-conrad]

Problem

The MCP plugin needs to call GitBucket's internal services correctly, but there is no oracle defining what "correct" means. GitBucket has no official API documentation — the only source of truth is the source code itself. Existing test scripts (Python-based, in the parent repo) test a fixed set of known endpoints from outside via HTTP, cannot call internal services, and cannot discover new or changed routes.

This spec establishes a phased testing pipeline that:

1. Defines correct behavior via in-JVM service oracle tests
2. Discovers all Scalatra routes via reflection (not source parsing)
3. Probes every discovered route via HTTP against live GitBucket instances
4. Produces version-compatibility matrices for regression detection

Approach

Phase 0: In-JVM Service Oracle Tests

Create service-layer tests using GitBucket's own ServiceSpecBase pattern. These tests define what correct behavior IS for each service method.

Mechanism:

• Tests run in the Gradle test task using GitBucket's ServiceSpecBase.withTestDB
• Each service gets a dedicated spec file: IssuesServiceSpec, MilestonesServiceSpec, ReleaseServiceSpec, etc.
• Tests exercise every public method with known inputs and expected outputs
• H2 in-memory database, no external GitBucket instance needed

Coverage targets (from empirical analysis of GitBucket source):

Service	Public Methods	Current Coverage
IssuesService	38	2 INCIDENTAL
MilestonesService	7	0
ReleaseService	8	0
HandleCommentService	—	0
CommitsService	—	0
WikiService	—	0
RepositorySearchService	—	0
PrioritiesService	—	0

Test data seeding:

• Git data: git clone --mirror from 1-3 public GitHub repos (e.g., gitbucket/gitbucket), pushed to test instance
• Issue/PR data: Programmatic creation of canonical dataset (3 issues, 2 PRs with labels/comments)
• No private repos, no NewsRx data
• Network dependency only at setup time (clone); all test execution is local

Gradle configuration:

• test task runs Phase 0 tests via standard JUnit runner
• Compile-only dependency on gitbucket_2.13:4.46.0 already exists
• Add testImplementation for GitBucket test infrastructure classes

Phase 1: In-JVM Scalatra Route Discovery

Discover all API routes by scanning the GitBucket classpath via reflection at test time. This replaces static source-code parsing with canonical runtime discovery.

Mechanism:

• JUnit test that loads all *Controller classes from the compile-time classpath
• Reflects over Scalatra route definitions (get(), post(), put(), patch(), delete() mappings)
• Extracts: HTTP verb, path pattern, parameter names (:owner, :repository, :number), request body form classes, response JSON serializers
• Outputs a route manifest (JSON) — the canonical list of every API route that exists in the current GitBucket version

Route manifest schema:

{
  "gitbucket_version": "4.46.0",
  "discovered_at": "2026-04-18T...",
  "routes": [
    {
      "verb": "GET",
      "path": "/api/v3/repos/:owner/:repository/issues/:number",
      "params": ["owner", "repository", "number"],
      "controller": "IssuesController",
      "form_class": null,
      "response_format": "IssueJSON"
    }
  ]
}

Key advantage over source parsing: The route manifest is version-locked to the compiled classpath. No source download, no Scala parser, no stale results. The manifest serves as the input to Phase 2.

Phase 2: External HTTP API Probe

Test every discovered route against a live GitBucket instance via HTTP. Download .war from GitHub releases, start as subprocess, probe all endpoints.

Mechanism:

• Custom Gradle task (probe) using JavaExec to start GitBucket .war
• Downloads .war from github.com/gitbucket/gitbucket/releases (default: latest release)
• --version flag for specific version
• --sweep <range> flag for version range testing (e.g., --sweep 4.42.0..4.46.0)
• Starts GitBucket on randomized port with temp directory for instance data
• Polls GET /api/v3/swagger.json until 200 (startup detection)
• Consumes route manifest from Phase 1 (routes discovered for current version)
• Probes each route via HTTP client (Java HttpClient)
• Classifies each endpoint:

Classification	Meaning
✅ Working	Returns expected status codes and response shapes
⚠️ Partial	Returns data but with deviations from oracle expectations
❌ Broken	Returns unexpected errors or 404s for routes that exist in code
🔍 New	Routes in current manifest but not in previous probe results
📋 Regression	Was working in older version, now broken

Sweep testing:

• Route manifest is always from current version's classpath
• Probe each route against each version in the sweep range
• Regression = route returns 200 in older version, 404/500 in newer version
• Improvement = route returns 404/500 in older version, 200 in newer version

Sweep output — version matrix example:

Route	4.42.0	4.43.0	4.44.0	4.45.0	4.46.0
GET /contents/{path}	?	?	?	200 ✅	404 ❌ REGRESSION
PUT /contents/{path}	?	?	?	200 ✅	500 ❌ REGRESSION
GET /issues	?	?	?	200 ✅	200 ✅
PATCH /issues/{n}	?	?	?	404	404

Output formats:

• JSON (canonical): Full results with response bodies, headers, timing — machine-readable
• Markdown (derived): Human-readable version matrix, generated from JSON — can be committed as API capability reference

Cleanup:

• SIGTERM to Java process after probe completes
• Temp directory deleted (unless --keep flag)
• --keep preserves instance data for debugging

Test Data Strategy

Git Data (Clone + Mirror)

• Default: git clone --mirror https://github.com/gitbucket/gitbucket.git → push to test instance
• Optional: additional public repos via --seed-repo <owner/repo> flag
• Provides real commit history, branches, tags for git-specific endpoint testing
• No authentication needed for public GitHub clones

Programmatic Data (Create via API)

• 3 issues (varying states: open, closed; with labels, comments)
• 2 pull requests (one open, one merged)
• Labels, milestones as needed by service oracle tests
• Deterministic and repeatable — no network dependency after git mirror

Success Criteria

1. Phase 0: Every public method in IssuesService (38 methods) has at least one oracle test; service spec files exist for all 7+ zero-coverage services
2. Phase 1: Route manifest contains every Scalatra route defined in GitBucket's *Controller.scala files, with verb, path, params, form class, and response format
3. Phase 2: ./gradlew probe starts a local GitBucket instance, probes all discovered routes, and produces JSON + markdown reports — no Docker, no Python, no external dependencies beyond Java
4. Sweep: ./gradlew probe --sweep 4.42.0..4.46.0 produces a version matrix showing regressions and improvements across the range
5. Reproducible: Probe results can be reproduced by any developer with Java installed
6. No external dependencies: All Java/Gradle — no Python, no Docker, no Node.js
7. Temp directory isolation: All instance data in ./tmp/gb-probe-<version>/

Out of Scope

• MCP plugin verification (separate spec — Phase 3)
• REST API fixes (upstream GitBucket concern)
• CI/CD pipeline setup
• Publishing to plugin repository
• GitBucket versions requiring JDK <17 (v4.42.0+ requires JDK 17+)

Supersedes

• [SPEC] GitBucket API Local Test Probe — Source-Based Endpoint Discovery Against Local Java Instance Brothertown-Language/snea-shoebox-editor#1058 (Python-based test probe — replaced by this Java/Gradle pipeline)
• [SPEC] GitBucket Test Coverage Gaps — Service Layer & REST API #11 (Test coverage gaps — absorbed into Phase 0)
• Setup: Complete Gradle build configuration #6 (Gradle build setup — prerequisite, not part of pipeline)
• [Spec] Migrate Gradle build to Kotlin DSL and complete configuration #7 (Kotlin DSL migration — prerequisite, not part of pipeline)
• [Plan] Migrate Gradle build to Kotlin DSL #8 (Kotlin DSL plan — prerequisite, not part of pipeline)