diff --git a/docs/regression-testing.md b/docs/regression-testing.md index a4170cc..8c79dc7 100644 --- a/docs/regression-testing.md +++ b/docs/regression-testing.md @@ -1,42 +1,43 @@ # Regression Testing Protocol - + -A reusable protocol for verifying features and fixes against a live SonarQube → SonarCloud migration. Designed to be given as a prompt to an AI coding agent (Claude Code, etc.) after implementation is complete. +A reusable protocol for verifying features and fixes against a live SonarQube → SonarCloud migration across **any project**. Designed to be given as a prompt to an AI coding agent (Claude Code, etc.) after implementation is complete. --- ## The Prompt -> After implementing the feature/fix, test it end-to-end on the **Angular Framework** project (38K+ issues, 400+ hotspots). Follow this protocol exactly: -> +> After implementing the feature/fix, test it end-to-end on **SonarQube project(s)** to verify regression testing. Follow this protocol exactly: + > ### Phase 1 — Adversarial Code Review (before running anything) -> + > Re-read every changed file with fresh eyes. For each file, ask: -> + > 1. **Data flow**: Does the data survive serialization boundaries? (worker_threads structured clone, JSON.stringify round-trips, URL query parameter encoding, protobuf encoding) > 2. **Error handling**: Are failures tracked or silently swallowed? Every `catch` block must either increment a stat, log, or re-throw — never empty. > 3. **Race conditions**: Are shared counters mutated from concurrent async contexts? In single-threaded JS this is safe, but verify each case. > 4. **API contract match**: Do HTTP calls use the exact method, endpoint, parameter names, and encoding that the original client uses? Compare line-by-line against the existing client implementation. > 5. **Edge cases**: What happens with 0 items? 1 item? Exactly at the threshold? `null`/`undefined` fields? Empty arrays vs missing fields? > 6. **Stat integrity**: Will `Object.assign(stats, mergedStats)` preserve fields set earlier (like `filtered`)? Will counters add up to the expected totals? -> + > Write down every concern. Then verify or disprove each one before proceeding. -> + > ### Phase 2 — Environment Assessment > Before running the test, check the live state: -> -> 1. **SonarQube**: `curl` the SQ API to confirm the project exists, count issues, sample 5 issues to check their fields (status, tags, comments, assignee, updateDate vs creationDate). -> 2. **SonarCloud**: Check if the SC project exists. If it has stale data from a prior run, delete it (`/api/projects/delete`). Verify auth tokens are valid (`/api/system/status`). -> 3. **Pre-filter viability**: Will enough issues pass the `hasManualChanges` pre-filter to trigger the code path under test? If not, create test conditions (bulk-tag issues, add comments, transition statuses via SQ API). Document what you created and how many. -> 4. **Clean slate**: Delete `migration-output/`, `.cloudvoyager-state.json*`, and the SC project. Every test run must start from zero. -> + +> 1. **List SonarQube projects**: `curl` the SQ API to list available projects, identify the target project(s) for regression testing. +> 2. **SonarQube**: Confirm the target project exists, count issues, sample 5 issues to check their fields (status, tags, comments, assignee, updateDate vs creationDate). +> 3. **SonarCloud**: Check if the SC project exists. If it has stale data from a prior run, delete it (`/api/projects/delete`). Verify auth tokens are valid (`/api/system/status`). +> 4. **Pre-filter viability**: Will enough issues pass the `hasManualChanges` pre-filter to trigger the code path under test? If not, create test conditions (bulk-tag issues, add comments, transition statuses via SQ API). Document what you created and how many. +> 5. **Clean slate**: Delete `migration-output/`, `.cloudvoyager-state.json*`, and the SC project(s). Every test run must start from zero. + > ### Phase 3 — Build and Execute -> + > 1. `npm run package` — must succeed. If it fails, fix before proceeding. > 2. Create or verify the config file (`config.json` for `transfer`, `migrate-config.json` for `migrate`). > 3. Run the command: `./dist/bin/cloudvoyager-macos-arm64 transfer -c config.json --verbose` @@ -45,12 +46,12 @@ A reusable protocol for verifying features and fixes against a live SonarQube > - Progress stalls (no output for >3 minutes — could indicate rate limiting or hangs) > - Error/warn lines > - The final summary stats line -> + > ### Phase 4 — Analyze Results > When the command finishes (exit code 0 or non-zero): -> + > 1. **Parse the summary stats line.** For every counter: > - Does it match expectations? (e.g., if you tagged 700 issues, do ~700 show as tagged?) > - Are there unexpected zeros? (0 comments when you added 20 — investigate) @@ -63,37 +64,37 @@ A reusable protocol for verifying features and fixes against a live SonarQube > - Check `api-errors` count (if tracked) > - Check the error log: `migration-output/logs/*/cloudvoyager-transfer.error.log` > - Check the warn log for rate limiting or timeout messages -> + > ### Phase 5 — Investigate Anomalies -> + > For every unexpected result, trace the full code path: -> + > 1. **Start from the data source.** Query the SQ API with the exact same parameters the code uses. Verify the field you expect (e.g., `comments`) is present in the response. > 2. **Check serialization.** If data crosses a boundary (worker_threads, JSON), write a minimal reproduction: create a Worker, pass the data via workerData, verify it arrives intact. > 3. **Check the API call.** Make the exact same HTTP request manually with `curl`. Does it succeed? What status code? What response body? > 4. **Check the matching.** If an issue wasn't synced, verify it was matched: same rule + file + line on both SQ and SC. > 5. **Identify the root cause.** Don't guess — prove. "The comments field was empty because..." with evidence. -> + > ### Phase 6 — Fix and Re-test -> + > If any issue was found: -> + > 1. **Fix the code.** Apply the minimal change that addresses the root cause. > 2. **Verify syntax**: `node --check ` on every changed file. > 3. **Rebuild**: `npm run package` > 4. **Clean slate**: Delete SC project, delete `migration-output/`, delete state files. > 5. **Re-run**: Full transfer from scratch. Do NOT assume a partial re-run is sufficient. > 6. **Re-analyze**: Repeat Phase 4 and Phase 5. The fix may have introduced new issues. -> + > **Repeat Phase 6 until every stat matches expectations and spot-checks pass.** -> + > ### Phase 7 — Declare Clean Pass -> + > A clean pass requires ALL of: -> + > - [ ] Exit code 0 (no crashes) > - [ ] 0 `failed` in the summary stats > - [ ] All expected counters are non-zero and within tolerance (>95% of matched pairs for tags/source-links/metadata-sync) @@ -104,24 +105,68 @@ A reusable protocol for verifying features and fixes against a live SonarQube --- -## Reference: Angular Framework Test Environment +## Reference: Regression Testing Projects +For regression testing, use any SonarQube project. Below are common options: + +### Large Project (Angular Framework) + | Property | Value | |----------|-------| | SQ project key | `angular-framework` | -| SQ URL | `http://localhost:9000` | +| SQ URL | `http://localhost:8998` | | SC org | `open-digital-society` | | SC URL | `https://sc-staging.io` | | SQ issues | ~38,400 | | SC issues (after transfer) | ~31,600 | | Hotspots | ~409 | + +### Small Project (for fallback verification) + +| Property | Value | +|----------|-------| +| SQ project key | (any small project with <500 issues) | +| SQ URL | `http://localhost:8998` | +| SC org | `open-digital-society` | +| SC URL | `https://sc-staging.io` | + +### Selecting a Project for Regression Testing + +When regression testing an issue, select the project that best reproduces the problem: + +1. **Issue involves specific issue states/statuses**: Use a project with diverse issue statuses +2. **Issue involves manual changes**: Use a project with `hasManualChanges` issues, or create them via API +3. **Issue involves comments/tags/assignees**: Use a project where these fields are populated +4. **Issue involves hotspots**: Use a project with known hotspots (like Angular Framework with ~409) +5. **Issue involves file matching**: Use a project where SQ and SC have overlapping file paths + +### Listing Available Projects on SonarQube + +```bash +# List all projects on SonarQube +curl -s -u "${SONAR_TOKEN}:" "http://localhost:8998/api/projects/search" | jq '.components[] | {key: .key, name: .name}' + +# Count total issues for a project +curl -s -u "${SONAR_TOKEN}:" "http://localhost:8998/api/issues/search?projectKeys=${PROJECT_KEY}&ps=1" | jq '.total' + +# Sample issues to check field distribution +curl -s -u "${SONAR_TOKEN}:" "http://localhost:8998/api/issues/search?projectKeys=${PROJECT_KEY}&ps=5" | jq '.issues[] | {key: .key, status: .status, tags: .tags, hasComments: .hasComments}' +``` + +--- + +## Reference: Config Files + | Config (transfer) | `config.json` | +|-------------------|---------------| | Config (migrate) | `migrate-config.json` | | Delete SC projects script | `.debugging/delete-all-sonarcloud-projects.sh` | | Test migration script | `.debugging/test-migrate.sh` | -### Creating Test Data on SQ +--- + +## Creating Test Data on SonarQube To trigger code paths that require manual changes (pre-filter keeps only issues with human-authored changes): @@ -129,27 +174,29 @@ To trigger code paths that require manual changes (pre-filter keeps only issues ```bash # Bulk-tag issues (creates updateDate != creationDate) python3 -c " -import urllib.request, urllib.parse, json, base64, concurrent.futures +import urllib.request, urllib.parse, json, base64, concurrent.futures, sys + TOKEN = 'squ_...' +PROJECT_KEY = sys.argv[1] if len(sys.argv) > 1 else 'angular-framework' auth = base64.b64encode(f'{TOKEN}:'.encode()).decode() headers = {'Authorization': f'Basic {auth}', 'Content-Type': 'application/x-www-form-urlencoded'} # Fetch issue keys -url = 'http://localhost:9000/api/issues/search?projectKeys=angular-framework&ps=500' +url = f'http://localhost:8998/api/issues/search?projectKeys={PROJECT_KEY}&ps=500' req = urllib.request.Request(url, headers=headers) keys = [i['key'] for i in json.load(urllib.request.urlopen(req))['issues']] # Tag them def tag(key): params = urllib.parse.urlencode({'issue': key, 'tags': 'test-tag'}).encode() - req = urllib.request.Request('http://localhost:9000/api/issues/set_tags', data=params, headers=headers, method='POST') + req = urllib.request.Request('http://localhost:8998/api/issues/set_tags', data=params, headers=headers, method='POST') urllib.request.urlopen(req) return 'ok' with concurrent.futures.ThreadPoolExecutor(max_workers=20) as pool: results = list(pool.map(tag, keys)) print(f'Tagged: {sum(1 for r in results if r == \"ok\")}/{len(keys)}') -" +" ${PROJECT_KEY} ``` Adapt similarly for `do_transition` (confirm/reopen), `add_comment`, and `assign`. diff --git a/src/pipelines/sq-2025/protobuf/build-components/helpers/build-file-components.js b/src/pipelines/sq-2025/protobuf/build-components/helpers/build-file-components.js index 28159ee..a884b10 100644 --- a/src/pipelines/sq-2025/protobuf/build-components/helpers/build-file-components.js +++ b/src/pipelines/sq-2025/protobuf/build-components/helpers/build-file-components.js @@ -31,4 +31,17 @@ export function buildFileComponents(builder, componentsMap, sanitizeLang) { }); } }); + + // Ensure all FIL components are in componentsMap even without source code. + // This catches Text files and other FIL components whose source code could not be fetched. + builder.data.components.forEach(comp => { + if (comp.qualifier === 'FIL' && !componentsMap.has(comp.key)) { + const lineCount = Number.parseInt(comp.measures?.find(m => m.metric === 'lines')?.value) || 0; + componentsMap.set(comp.key, { + ref: builder.getComponentRef(comp.key), type: 4, + language: sanitizeLang(comp.language || ''), + lines: lineCount, status: 3, projectRelativePath: comp.path || comp.name, + }); + } + }); } diff --git a/src/pipelines/sq-2025/sonarcloud/uploader/helpers/add-protobuf-files.js b/src/pipelines/sq-2025/sonarcloud/uploader/helpers/add-protobuf-files.js index 430ce94..510fd63 100644 --- a/src/pipelines/sq-2025/sonarcloud/uploader/helpers/add-protobuf-files.js +++ b/src/pipelines/sq-2025/sonarcloud/uploader/helpers/add-protobuf-files.js @@ -31,4 +31,20 @@ export function addProtobufFiles(zip, encodedReport) { zip.addFile('activerules.pb', Buffer.from(encodedReport.activeRules)); logger.debug(`Added activerules.pb (${encodedReport.activeRules.length} bytes)`); } + + // External issues (externalissues-{ref}.pb) + if (encodedReport.externalIssues && encodedReport.externalIssues.size > 0) { + let extIssueFileCount = 0; + encodedReport.externalIssues.forEach((buf, ref) => { + zip.addFile(`externalissues-${ref}.pb`, Buffer.from(buf)); + extIssueFileCount++; + }); + logger.debug(`Added ${extIssueFileCount} external issue files`); + } + + // Ad-hoc rules (adhoerules.pb) + if (encodedReport.adHocRules && encodedReport.adHocRules.length > 0) { + zip.addFile('adhoerules.pb', Buffer.from(encodedReport.adHocRules)); + logger.debug(`Added adhoerules.pb (${encodedReport.adHocRules.length} bytes)`); + } } diff --git a/src/pipelines/sq-2025/sonarqube/api-client/helpers/source-methods.js b/src/pipelines/sq-2025/sonarqube/api-client/helpers/source-methods.js index dbf74d7..8c8d40b 100644 --- a/src/pipelines/sq-2025/sonarqube/api-client/helpers/source-methods.js +++ b/src/pipelines/sq-2025/sonarqube/api-client/helpers/source-methods.js @@ -14,7 +14,7 @@ export function attachSourceMethods(inst) { inst.getSourceFiles = async (branch = null) => { logger.info(`Fetching source files for project: ${inst.projectKey}`); - const params = { component: inst.projectKey, qualifiers: 'FIL' }; + const params = { component: inst.projectKey, qualifiers: 'FIL,UTS' }; if (branch) params.branch = branch; return await inst.getPaginated('/api/components/tree', params, 'components'); };