feat: regression testing system (Phase 1) — 5 scenarios x 4 SQ versions

[joshua-quek-sonarsource]

Summary

• Matrix-based GitHub Actions workflow testing 5 PRIORITY bug scenarios across all 4 SonarQube versions (9.9, 10.0, 10.4, 2025.1) = 20 matrix jobs
• Ephemeral SonarQube Enterprise Docker containers per job (self-healing, no persistent test instances)
• Assertion scripts verify each fixed bug hasn't regressed by querying SonarCloud API post-migration
• Integrated into existing regression.yml orchestrator as a workflow_call child

Scenarios covered (Phase 1)

Scenario	Issues	What it tests
large-project-10k-plus	#53, #94, #98	>10K issues migrate with correct SCM date-bucket distribution
github-actions-language	#88	GitHub Actions language issues appear in SQC
external-analyzers	#56, #82	Pylint/Ruff external analyzer rules migrate
issue-sync-first-migration	#91	Issue sync triggers on first migration run
kill-and-continue	#15, #57	SIGTERM + resume produces no duplicates

New files (22)

• .github/workflows/regression-bug-fixes.yml — main workflow
• .github/actions/setup-sonarqube/action.yml — ephemeral SQ container setup
• test/regression/helpers/ — shared SQC client (retry/backoff), assertion utilities, config tools
• test/regression/enrichment/ — data enrichment scripts (comments, statuses, hotspots)
• test/regression/assert-*.js — 5 assertion scripts, one per scenario
• test/regression/sample-projects/ — minimal JS project for small scenarios

Prerequisites (before CI will pass)

1. Add GitHub secrets: SONARQUBE_9_9_LICENSE_KEY, SONARQUBE_10_0_LICENSE_KEY, SONARQUBE_10_4_LICENSE_KEY, SONARQUBE_2025_1_LICENSE_KEY
2. Verify SQ Docker image tag format matches Docker Hub (sonarqube:9.9-enterprise etc.)

Review process

Code reviewed by 5 parallel AI agents (helpers, enrichment, workflow YAML, assertions, meta-tests). 10 P1s, 8 P2s, 4 P3s found and fixed before commit.

Test plan

• Add SQ Enterprise license keys as GitHub secrets
• Trigger workflow on push to main (or manual dispatch for testing)
• Verify all 20 matrix jobs pass (5 scenarios x 4 SQ versions)
• Check assertion output for PASS/FAIL per issue number
• Verify log artifacts upload on failure

🤖 Generated with Claude Code

---

Note

Medium Risk
Adds a new matrix-based GitHub Actions regression workflow that spins up ephemeral SonarQube Enterprise containers and performs real migrations/assertions against SonarCloud, which could be flaky or resource-intensive and depends on new secrets and external APIs.

Overview
Adds a new regression-bug-fixes.yml workflow (wired into regression.yml) that runs 5 bug-fix scenarios × 4 SonarQube versions and asserts post-migration outcomes by querying SonarCloud.

Introduces a composite action setup-sonarqube that boots PostgreSQL + SonarQube Enterprise in Docker, waits for health, injects an Enterprise license, and generates an admin analysis token for the job.

Adds Node-based regression tooling under test/regression/ (SQC API client with retry/backoff, config patch/cleanup helpers, enrichment scripts to mutate SQ issues/hotspots, and per-scenario assertion scripts), plus updates docs to reflect the shift to single-analysis backdateChangesets() SCM date-bucket distribution terminology.

^{Reviewed by Cursor Bugbot for commit d751b0c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[sonar-review-alpha]

Summary

This PR adds a regression testing system (Phase 1) for 5 critical bug fix scenarios across all 4 supported SonarQube versions (9.9, 10.0, 10.4, 2025.1), creating 20 parallel test jobs.

What changed:

• New workflow regression-bug-fixes.yml: Matrix-based job that tests migrations don't regress on bugs #15, #53, #56, #57, #82, #88, #91, #94, #98
• New GitHub Action setup-sonarqube: Spins up ephemeral SQ Enterprise containers with PostgreSQL per job, self-healing and torn down after tests
• Test infrastructure: SonarCloud API client with retry/backoff, assertion utilities, data enrichment scripts (comments, issue statuses, hotspots)
• 5 assertion scripts: Each verifies one scenario didn't regress by querying SQC API post-migration
• Minimal JS test project for small scenarios

Why: Ensures critical fixes remain fixed across SQ version upgrades and migration runs. Integrated into existing regression.yml orchestrator as a parallel sibling to migrate/sync-metadata/verify jobs.

What reviewers should know

Where to start

1. Workflow structure: .github/workflows/regression-bug-fixes.yml — 5 scenarios × 4 SQ versions matrix (line 17-41 shows matrix definition). Note fail-fast: false (line 14) means all 20 jobs run even if one fails, and max-parallel: 4 (line 15) limits concurrent execution to avoid resource exhaustion.

2. Container lifecycle: .github/actions/setup-sonarqube/action.yml — Creates fresh Docker network + PostgreSQL + SQ Enterprise per job. Key: watches for UP status (line 78), injects license key (line 92-103), creates admin token for CLI (line 105-120). All containers torn down in "Teardown" step (regression-bug-fixes.yml line 126-131).

Key implementation details

Test flow per job:

1. Generate migration config → setup SQ container + license
2. Enrich test data: add comments, change issue statuses, add hotspots (lines 85-101)
3. Run migration (normal or SIGTERM-interrupted for kill-and-continue)
4. Assert results by querying SQC API
5. Upload logs if failed; always teardown containers

Test data preparation: Enrichment scripts (test/regression/enrichment/) add realistic data before migration runs, so assertions can verify the data migrated correctly. This tests real-world scenarios (comments on issues, changed statuses, hotspots).

Assertions: Each assert-*.js script queries SQC API for specific issue counts/metadata. Example: assert-large-project-10k-plus.js checks >10K issues spread across date buckets. Assertions use SQC_TARGET_KEY environment variable (line 46) to know which SQC project to query.

API resilience: helpers/sqc-client.js implements exponential backoff (base 1s, max 3 retries) for SQC API calls to handle transient failures.

Non-obvious decisions

• Each scenario needs its own assertion script (not parameterized) because test logic is scenario-specific (different API queries, different pass/fail criteria)
• Test data enrichment is intentional: it seeds realistic SQ state before migration so assertions verify both the bug fix AND data integrity during migration
• SQC_TARGET_KEY uses scenario name + SQ version (line 46) to isolate test projects, allowing all 20 jobs to run in parallel without conflicts

Prerequisites before CI passes

• Add GitHub secrets: SONARQUBE_9_9_LICENSE_KEY, SONARQUBE_10_0_LICENSE_KEY, SONARQUBE_10_4_LICENSE_KEY, SONARQUBE_2025_1_LICENSE_KEY
• Verify Docker Hub image tags (sonarqube:X.Y-enterprise) exist for all 4 versions

Review checklist

• Assertion logic is correct for each scenario (does it actually test the bug fix?)
• Container lifecycle is safe (no leaks if workflow aborts)
• API client retry logic handles transient failures
• Test data enrichment is realistic (comments/hotspots/status changes actually use the migration data)
• Matrix dimensions (scenarios × SQ versions) align with author's description

---

• Generate Walkthrough
• Generate Diagram

🗣️ Give feedback

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d751b0c. Configure here.}

[cursor]

Wrong token type blocks enrichment script API access

High Severity

The token is created with type=GLOBAL_ANALYSIS_TOKEN, which is scoped exclusively to scanner analysis operations (submitting reports via /api/ce/submit). However, the enrichment scripts and migration tool need full user-level API access — including /api/issues/search, /api/issues/add_comment, /api/issues/do_transition, /api/hotspots/search, and /api/hotspots/change_status. A GLOBAL_ANALYSIS_TOKEN cannot perform these operations, causing 403 errors. The type needs to be USER_TOKEN (or omitted entirely, which defaults to USER_TOKEN).

Additional Locations (1)

• .github/workflows/regression-bug-fixes.yml#L84-L101

 

^{Reviewed by Cursor Bugbot for commit d751b0c. Configure here.}

[cursor]

Workflow enrichment fails on empty SonarQube instance

High Severity

The workflow starts a fresh, empty SonarQube instance (no db-dump-artifact is provided and no sonar-scanner step exists), yet the enrichment steps immediately query for issues in project angular which doesn't exist. Both add-issue-comments.js and change-issue-status.js unconditionally throw when zero issues are found, failing the step. Since all subsequent steps (migration, assertions) use the default if: success() condition, they're skipped — making all 20 matrix jobs non-functional.

Additional Locations (2)

• test/regression/enrichment/add-issue-comments.js#L53-L56
• test/regression/enrichment/change-issue-status.js#L53-L56

 

^{Reviewed by Cursor Bugbot for commit d751b0c. Configure here.}