Feat/regression testing system

[joshua-quek-sonarsource]

Note

Medium Risk
Moderate risk because it changes CI behavior and secret handling: regression bug-fix tests are no longer run from this repo and SonarCloud scanning workflow is removed. Primary risk is reduced coverage/missed checks if the private repo workflow sync or secret configuration is incorrect.

Overview
CI secret handling is tightened by masking the SonarQube Enterprise license key in the setup-sonarqube composite action and switching license injection to HTTP-status-based success checks without logging response bodies.

Sensitive regression and scanning workflows are moved out of this public repo: regression-bug-fixes.yml is converted to manual-only (workflow_dispatch) and uses a single SONARQUBE_LICENSE_KEY secret, the bug-fixes job is removed from regression.yml, and the standalone sonarcloud.yml workflow is deleted.

Docs are updated to describe the Phase 1 regression testing system and note that these workflows now run in the private sonar-solutions/cloudvoyager-ci repo.

^{Reviewed by Cursor Bugbot for commit 5bb8a86. Bugbot is set up for automated code reviews on this repo. Configure here.}

[joshua-quek-sonarsource]

updated docs

[sonar-review-alpha]

Summary

What changed: Implemented a regression testing system with 5 test scenarios (large projects, GitHub Actions language, external analyzers, issue sync, kill-and-resume) tested across all 4 supported SonarQube versions (9.9, 10.0, 10.4, 2025.1) = 20 matrix jobs. Each test spins up ephemeral Docker containers (SonarQube Enterprise + PostgreSQL), seeds realistic test data, runs CloudVoyager, and validates outcomes.

Why: Prior bug fixes lacked automated verification that they remained correct across version upgrades and edge cases. This system catches regressions immediately.

Security decision: Sensitive workflows (SQ Enterprise license keys, SonarCloud tokens) moved to private repo sonar-solutions/cloudvoyager-ci; public repo CI has no access to secrets. Private repo syncs from public every 15 minutes and reports status back.

Code impact: Public repo changes are infrastructure-only — workflow restructuring, enhanced license injection with better error handling, and updated documentation. The actual test code (22 files: helpers, enrichment scripts, 5 assertion scripts, sample projects) was added in PR #127.

What reviewers should know

Where to start:

1. Review .github/workflows/regression-bug-fixes.yml — changed from workflow_call to workflow_dispatch and simplified secret references (now uses single SONARQUBE_LICENSE_KEY instead of per-version matrix)
2. Review .github/actions/setup-sonarqube/action.yml — license injection now checks HTTP status codes instead of response body parsing, and masks the license in logs
3. Check docs/technical-details.md (line ~750) and docs/architecture.md (line ~650) for container lifecycle and test matrix details

Non-obvious decisions:

• Deleted .github/workflows/sonarcloud.yml entirely (moved to private repo) — this removes SonarCloud scanning from the public repo CI. Check that this is intentional and that SonarCloud scanning still runs somewhere.
• License injection changed from parsing JSON error response to HTTP status code checking — more reliable, but verify this works with all SQ versions in the matrix
• Bug-fixes job removed from public regression.yml summary job dependency — the test still runs, just in the private repo and reports status back

Things to watch:

• Ensure the private repo (sonar-solutions/cloudvoyager-ci) actually exists and has the workflow files + secrets configured before merging
• Verify the 15-minute sync cadence from public to private is fast enough for your release cycle
• Check that committed test code in test/regression/ (assertion scripts, helpers, sample projects) matches the interfaces referenced in the docs

---

• Generate Walkthrough
• Generate Diagram

🗣️ Give feedback