From 069dc1ada9e00d9c143595dca348e1d50eecdb77 Mon Sep 17 00:00:00 2001 From: Joshua Quek Date: Thu, 7 May 2026 02:15:23 +0800 Subject: [PATCH] docs: add timestamp annotations to all documentation headings Add standardized timestamp annotations to all section/subsection headings in docs folder and README.md using the format: Co-Authored-By: Claude Opus 4.7 --- README.md | 12 +- docs/Audit & Error Events Logging Strategy.md | 74 +++++- docs/Business Value.md | 84 ++++-- docs/CHANGELOG.md | 194 ++++++++++---- docs/CONTRIBUTING.md | 21 +- docs/Core Technologies.md | 56 +++- docs/Design Principles.md | 68 ++++- docs/architecture.md | 28 ++ docs/backward-compatibility.md | 25 ++ docs/bespoke-algorithms.md | 43 ++- docs/configuration.md | 39 +-- docs/design-review-regression-testing.md | 11 +- docs/desktop-app.md | 26 ++ docs/dry-run-csv-reference.md | 28 +- docs/key-capabilities.md | 247 +++++++++--------- docs/local-development.md | 76 +++--- docs/milestone-1.2-verification-report.md | 18 ++ docs/pseudocode-explanation.md | 20 ++ docs/regression-testing.md | 15 +- docs/scenario-multi-org.md | 73 +++--- docs/scenario-single-org.md | 72 +++-- docs/scenario-single-project.md | 68 +++-- docs/technical-details.md | 65 +++-- docs/troubleshooting.md | 84 ++++-- docs/verification.md | 51 +++- docs/versioning.md | 21 +- 26 files changed, 1024 insertions(+), 495 deletions(-) diff --git a/README.md b/README.md index f2c04170..52fe39c0 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,5 @@ # ☁️ πŸ‹ CloudVoyager - - + Migrate your data from self-hosted SonarQube to SonarCloud β€” no re-scanning needed. This was done by reverse-engineering SonarScanner (scan report protobuf files) & then fully rebuilding everything from the ground up on Node.js. @@ -8,6 +7,7 @@ CloudVoyager copies everything β€” projects, code issues, security hotspots, qua ## βœ… Quick Start (Recommended) + **Choose your scenario:** @@ -18,6 +18,7 @@ CloudVoyager copies everything β€” projects, code issues, security hotspots, qua | Migrate **everything** from SonarQube to **multiple** SonarCloud orgs | [Full Migration β€” Multiple Orgs](docs/scenario-multi-org.md) | ## πŸ–₯️ Desktop App (New!) + Prefer a visual interface? **CloudVoyager Desktop** provides a guided wizard UI β€” no terminal or config files needed. Includes a **Run History** sidebar for quick access to past migration results. @@ -59,6 +60,7 @@ See the [Desktop App Guide](docs/desktop-app.md) for setup instructions and a wa ## πŸ”₯ Single Command Full Migration (Slightly Dangerous) + 1. Download the latest release of CloudVoyager from the [releases page](https://github.com/sonar-solutions/cloudvoyager/releases). 2. Ensure that you have full admin access API tokens for your SonarQube server and your SonarCloud organization. @@ -75,6 +77,7 @@ See the [Desktop App Guide](docs/desktop-app.md) for setup instructions and a wa ## πŸ”„ Pause and Resume + All migrations support **automatic checkpointing**. Progress is saved after every phase (extract, build, encode, upload). If a migration is interrupted β€” whether by CTRL+C (graceful shutdown), a crash, or a network failure β€” simply re-run the same command to resume from where it left off. No data is lost or duplicated. @@ -101,6 +104,7 @@ See the [Configuration Reference](docs/configuration.md#checkpoint-settings) for ## πŸ”„ SonarQube Version Compatibility + CloudVoyager ships **four fully independent pipelines** β€” one per SonarQube version range. At runtime, `version-router.js` calls `/api/system/status`, detects the server version, and dynamically loads the matching pipeline. No special configuration is needed. @@ -117,6 +121,7 @@ See [Backward Compatibility](docs/backward-compatibility.md) for technical detai ## πŸ§ͺ CI / Regression Testing + Two independent workflows run on every push to `main`: @@ -131,11 +136,13 @@ The regression suite tests all `migrate`, `sync-metadata`, and `verify` commands ## πŸ› οΈ Local Development + Want to build and test CloudVoyager locally? See the [Local Development Guide](docs/local-development.md) for step-by-step instructions. ## πŸ“š Documentation + | Document | Description | |----------|-------------| @@ -155,6 +162,7 @@ Want to build and test CloudVoyager locally? See the [Local Development Guide](d ## πŸ“ License + MIT diff --git a/docs/Audit & Error Events Logging Strategy.md b/docs/Audit & Error Events Logging Strategy.md index 14946811..ab74800f 100644 --- a/docs/Audit & Error Events Logging Strategy.md +++ b/docs/Audit & Error Events Logging Strategy.md @@ -1,6 +1,6 @@ # Audit & Error Events Logging Strategy - + + CloudVoyager uses [Winston](https://github.com/winstonjs/winston) as its logging framework, providing a flexible and extensible architecture for handling logs across all pipeline operations. ### Core Components + + CloudVoyager uses the standard severity levels defined by Winston. Each level has a specific use case within the migration pipelines. @@ -59,6 +63,8 @@ CloudVoyager uses the standard severity levels defined by Winston. Each level ha ### Environment-Based Level Control + + CloudVoyager supports multiple log destinations, configurable based on environment and requirements. ### Console Transport + + ### Text Format + + CloudVoyager defines a hierarchy of error types in `src/shared/utils/errors/`, all inheriting from the base `CloudVoyagerError` class. ### Error Class Hierarchy + + CloudVoyager logs specific events for compliance auditing and operational visibility. ### Migration Lifecycle Events + + CloudVoyager creates new log directories for each migration run, providing natural separation for rotation. ### Directory Naming Convention + + ## 1. Time Savings β€” No Re-scanning Required - + ### The Hidden Cost of Re-Scanning + + Traditional migration approaches require re-running CI/CD scanners against every project. For a portfolio of 50 projects across multiple teams, this means: - **Days to weeks of pipeline execution** β€” Each scan must complete in sequence or with limited parallelism, constrained by CI/CD runner availability @@ -19,9 +21,11 @@ Traditional migration approaches require re-running CI/CD scanners against every - **Locked CI/CD resources** β€” Build agents are occupied running scans instead of their normal development work - **Extended validation cycles** β€” Each re-scan potentially surfaces new issues that did not exist in the original SonarQube, requiring investigation - + ### CloudVoyager Eliminates This Entirely + + CloudVoyager connects directly to your existing SonarQube server via API, extracts all project data, and uploads it to SonarCloud as a legitimate scanner submission β€” without touching your source code or CI/CD pipelines. **Real-world example from production use:** @@ -36,9 +40,11 @@ CloudVoyager connects directly to your existing SonarQube server via API, extrac This same migration via re-scanning would have taken days,占用 significant CI/CD resources and requiring coordination across multiple development teams. - + ### What This Means for Your Team + + - **No CI/CD disruption** β€” Migration runs in the background while your pipelines continue normal operations - **No coordination required** β€” Individual development teams do not need to take any action - **Predictable timeline** β€” A 50-project portfolio migrates in hours, not weeks @@ -46,12 +52,14 @@ This same migration via re-scanning would have taken days,占用 significant CI/ --- - + ## 2. Data Preservation β€” All Historical Issues, Hotspots, and Measures Preserved - + ### Complete Data Fidelity + + CloudVoyager migrates every category of data that SonarQube tracks: | Category | Data Preserved | @@ -63,9 +71,11 @@ CloudVoyager migrates every category of data that SonarQube tracks: | **SCM Data** | Changeset information (author, date, revision) per file | | **Branches** | All branches transferred, with main branch processed first | - + ### Issue Lifecycle Preservation + + Each issue carries its complete lifecycle history from SonarQube: - **Original creation date** β€” Preserved via SCM date backdating, so SonarCloud shows the same temporal distribution as SonarQube @@ -74,24 +84,30 @@ Each issue carries its complete lifecycle history from SonarQube: - **Comments** β€” Full comment history preserved with `[Migrated from SonarQube]` attribution - **Tags** β€” Custom categorization labels maintained - + ### Accurate Issue Creation Dates + + CloudVoyager preserves each issue's original SonarQube creation date by backdating SCM changeset blame dates in the protobuf report. Issues appear in SonarCloud with the same creation timestamp they had in SonarQube, maintaining the historical distribution in the creation date facet. For calendar days with more than 5,000 issues, CloudVoyager automatically splits into sub-groups with 1-day-spaced synthetic dates to prevent clustering, ensuring a realistic distribution across the project's history. --- - + ## 3. Risk Reduction β€” Verified Migration with Rollback Capability - + ### Three-Layer Safety System - + + + #### Layer 1: Dry-Run Preview + + Before any data touches SonarCloud, run the migration in dry-run mode: ```bash @@ -107,9 +123,11 @@ This extracts all data and generates 9 CSV mapping files for review. You can: Only after reviewing the generated CSVs do you proceed to the actual migration. - + #### Layer 2: Checkpoint Journal with Pause/Resume + + Every migration step is individually checkpointed. If migration is interrupted β€” by CTRL+C, network failure, or system crash β€” running the same command again resumes from the last completed step. No data is lost or duplicated. The checkpoint journal tracks: @@ -118,9 +136,11 @@ The checkpoint journal tracks: - Upload deduplication via CE task ID verification (prevents re-uploading completed analyses) - Session fingerprint validation (warns on SonarQube version changes between runs) - + #### Layer 3: Post-Migration Verification + + After migration completes, the verification pipeline runs **58+ automated checks** comparing SonarQube against SonarCloud: | Verification Area | Checks Performed | @@ -145,12 +165,14 @@ Reports generate in JSON, Markdown, and PDF formats, including collapsible detai --- - + ## 4. Cost Efficiency β€” No CI/CD Pipeline Re-Runs Needed - + ### The True Cost of Re-Scanning + + Re-running CI/CD pipelines for migration is not free. Consider the hidden costs: | Cost Factor | Impact | @@ -163,9 +185,11 @@ Re-running CI/CD pipelines for migration is not free. Consider the hidden costs: For a 50-project portfolio, re-scanning could consume hundreds of CI/CD minutes, tie up developer attention, and extend the migration timeline from hours to weeks. - + ### One Command, Full Migration + + CloudVoyager runs as a single command that orchestrates the entire migration: ```bash @@ -182,9 +206,11 @@ The `--auto-tune` flag automatically detects your hardware (CPU cores, RAM) and | Medium (10-50 projects, 1K-50K issues) | 4-8 GB | 15-60 minutes | | Large (50+ projects, 50K+ issues) | 8+ GB | 1-3 hours | - + ### Parallel Processing Architecture + + CloudVoyager uses a zero-dependency concurrency engine to maximize throughput: - **Source file extraction** β€” Parallel fetching across 10+ concurrent connections @@ -196,12 +222,14 @@ This parallelism means CloudVoyager fully utilizes available resources without r --- - + ## 5. Governance Continuity β€” Permissions, Quality Gates, Profiles Maintained - + ### Complete Governance Preservation + + Moving to SonarCloud should not mean rebuilding your entire quality governance framework from scratch. CloudVoyager migrates every governance artifact: | Governance Element | What Gets Migrated | @@ -214,9 +242,11 @@ Moving to SonarCloud should not mean rebuilding your entire quality governance f | **Permission Templates** | Reusable templates with group assignments, set as defaults where applicable | | **Portfolios** | Portfolio definitions with project associations preserved for executive-level views | - + ### Quality Gate Integrity + + Quality gates are recreated with their exact condition logic: - **Metric + operator + threshold** preserved for every condition @@ -226,9 +256,11 @@ Quality gates are recreated with their exact condition logic: This means your existing quality standards continue uninterrupted β€” projects that passed SonarQube's gate pass SonarCloud's gate with the same criteria. - + ### Quality Profile Fidelity + + Quality profiles use SonarQube's native backup/restore XML format, preserving: - **All rule activations and deactivations** @@ -244,9 +276,11 @@ Quality profiles use SonarQube's native backup/restore XML format, preserving: This enables governance teams to review rule parity before cutting over to SonarCloud. - + ### Project Settings and Bindings + + Per-project configuration that transfers includes: - **Project visibility and description** @@ -260,7 +294,7 @@ This ensures each project arrives in SonarCloud fully configured and ready for y --- - + ## Summary CloudVoyager's business value is straightforward: @@ -277,9 +311,11 @@ CloudVoyager is purpose-built for enterprises that need to migrate to SonarCloud --- - + ## Further Reading + + - [Key Capabilities](key-capabilities.md) β€” Comprehensive technical overview - [Architecture](architecture.md) β€” Project structure and data flow - [Scenario: Single Organization Migration](scenario-single-org.md) β€” Step-by-step migration guide diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index a148f6cb..aca1b951 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -1,13 +1,12 @@ # Changelog -All notable changes to CloudVoyager are documented in this file. Entries are ordered with the most recent changes first. + -Fixed project configuration settings (e.g., `sonar.exclusions`, `sonar.coverage.exclusions`) not being migrated from SonarQube Server to SonarQube Cloud. + -Added `worker_threads`-based parallel issue sync to dramatically reduce wall-clock time for large projects (e.g., Angular Framework with 31,622 issues β€” from ~90 minutes to ~5-10 minutes). + -Rewrote `backdateChangesets()` to preserve each issue's original SonarQube creation date in SonarCloud. Verified 99.92% accuracy (26/31,641 mismatches on Angular Framework β€” all from issues sharing the same startLine). + -Built and shipped the regression testing system for CloudVoyager. 5 bug-fix scenarios tested across all 4 SonarQube versions (9.9, 10.0, 10.4, 2025.1) = 20 matrix jobs. + -Fixed issue migration data loss and implemented SCM-based date-bucket distribution to work around SonarCloud's 10K Elasticsearch visualization cap. + -/plan-design-review completed on the CloudVoyager desktop Electron app. Initial score: 5/10, final: 7/10. Key decisions: (1) Collapse settings Step 3 to show only mode selection by default, addresses issues #61/#63/#64/#67/#68. (2) Per-item status for execution + connection test (partial failure visibility). (3) Completion summary card with migration stats (victory moment). (4) Bundle custom typeface (Inter or Geist). (5) Mandatory pre-flight validation before Start, addresses issue #72. (6) Fix color contrast on glassmorphic cards for WCAG compliance. Deferred: first-time user flow, results empty state, PREVIOUS RUN warning (#87), light theme polish. + -/plan-ceo-review completed on the regression testing design. Key decisions: (1) Ephemeral SQ Docker containers per test job replace persistent test instances, eliminating stale test data risk. (2) All 4 SQ versions tested (9.9, 10.0, 10.4, 2025.1) = 76 matrix jobs. (3) Workflow integrated into existing regression.yml orchestrator. (4) SQC target keys namespaced with cv-regression-* prefix. (5) Budget unlimited: largest runners, max JVM for sonar-scanner. (6) sqc-us-region moved to Phase 2 (requires new secrets). Outside voice (Claude subagent) ran, 8 findings, 5 actioned. + -Design document approved for comprehensive regression testing via GitHub Actions matrix strategy. Covers 19 matrix entries mapping to all fixed issues (PRIORITY bugs #53, #56, #70, #88, #89, #91, #94, #98 plus changelog fixes and non-PRIORITY issues). Phased rollout: 5 PRIORITY entries in Phase 1 (2 days), remaining 14 in Phase 2 (3 more days). Tests run against real SonarQube/SonarCloud instances with `max-parallel: 4`. Includes test data health check, cleanup policy, and rate limiting mitigation. Design doc: `~/.gstack/projects/sonar-solutions-cloudvoyager/joshua.quek-main-design-20260422-184500.md` + -Two additional bugs found by re-running migration after Round 1 fixes and analysing `migration-output/logs/2026-04-22T01-49-13-947Z/` logs. + -Three bugs found by analysing `migration-output/logs/2026-04-22T01-14-10-349Z/` logs. + -`enableFileLogging` was called in each command handler **before** `ensureHeapSize` in the action handler. When the process respawned for more heap memory, the child process called `enableFileLogging` again, creating a second per-run log folder. The initial process's folder contained only startup messages; all real logs went to the child's folder. + -Applied all fixes from the deep codebase audit (5 critical, 16 warnings, 9 info). Zero new test failures introduced. + -Full codebase review of 95 files across shared utilities, state management, pipelines, protobuf encoding, verification, and CLI commands. Findings documented in `REVIEW.md` at project root. + -Each migration run now creates a **timestamped subfolder** under `migration-output/logs/`, e.g. `migration-output/logs/2026-04-22T03-01-00-123Z/`. This makes it easy to find and compare logs across different runs. + -Fixed a bug where the "New code definitions" migration step failed with `SonarCloud API error (400): Either 'value', 'values' or 'fieldValues' must be provided`. + + -Fixed several missing import bugs that caused the `verify` command's report generation to fail, and a quoting bug in the build script. + + + + Each supported SonarQube version range gets its own **complete, independent pipeline directory** under `src/pipelines/`. There are no runtime version checks within any pipeline -- each pipeline has its behavior hardcoded for its target version range. @@ -26,6 +27,7 @@ The **version router** (`src/version-router.js`) detects the SonarQube server ve --- ## Directory Structure + ``` src/ @@ -111,8 +113,10 @@ New files must go in the appropriate directory. Shared, version-independent code --- ## Patterns + ### 1. Extractor Pattern (`src/pipelines/sq-{version}/sonarqube/extractors/`) + Each extractor is a standalone async function: @@ -133,6 +137,7 @@ export async function extractXxx(client, state = null, branch = null) { - The `DataExtractor` orchestrator in `extractors/index.js` coordinates all extractors ### 2. API Module Pattern (`src/pipelines/sq-{version}/sonarqube/api/`, `src/pipelines/sq-{version}/sonarcloud/api/`) + Stateless functions that wrap a single API endpoint: @@ -151,6 +156,7 @@ export async function getXxx(client, param) { - The `SonarQubeClient` delegates to these via thin wrappers ### 3. Migrator Pattern (`src/pipelines/sq-{version}/sonarcloud/migrators/`) + Transform extracted SonarQube data and apply it to SonarCloud: @@ -167,6 +173,7 @@ export async function migrateXxx(extractedData, scClient, options) { - Handle partial failures gracefully (log warning, continue) ### 4. Client Pattern (`src/pipelines/sq-{version}/sonarqube/api-client.js`) + Each pipeline has its own `SonarQubeClient` tailored to its version range: @@ -177,6 +184,7 @@ Each pipeline has its own `SonarQubeClient` tailored to its version range: - Version-specific behavior is hardcoded -- no runtime version checks needed ### 5. Error Handling + **Custom error hierarchy** (all extend `CloudVoyagerError` from `src/shared/utils/errors.js`): @@ -195,6 +203,7 @@ Each pipeline has its own `SonarQubeClient` tailored to its version range: - **Don't wrap**: Core endpoints where failure should stop execution ### 6. Pagination + All paginated SonarQube APIs use `getPaginated()`: @@ -209,6 +218,7 @@ return await this.getPaginated('/api/xxx/search', { ps: 500 }, 'items'); - Never implement manual pagination -- always use `getPaginated()` ### 7. Concurrency + Use `mapConcurrent()` from `src/shared/utils/concurrency.js` for parallel I/O: @@ -227,6 +237,7 @@ const results = await mapConcurrent(items, async (item) => { - Use `createProgressLogger()` for large batches ### 8. Issue Sync Pre-Filter Pattern (`src/shared/utils/issue-sync/`) + Issue sync across all four pipelines uses a **pre-filter optimization** (see [GitHub #90](https://github.com/sonar-solutions/cloudvoyager/issues/90)) modeled after the [`sonar-findings-sync` algorithm](https://github.com/okorach/sonar-tools/blob/0be16b23d1eb9a374fc3cbbcb1c10242df0631a3/sonar/syncer.py#L564). @@ -258,8 +269,10 @@ The pre-fetched changelogs are used only to evaluate `hasManualChanges` during p --- ## Conventions + ### Imports + - **ESM only** -- the project uses `"type": "module"` - Node builtins use `node:` prefix: `import { readFile } from 'node:fs/promises'` @@ -268,6 +281,7 @@ The pre-fetched changelogs are used only to evaluate `hasManualChanges` during p - Commands use dynamic `import()` to load the correct pipeline at runtime via the version router ### Logging + ```js import logger from '../../shared/utils/logger.js'; @@ -285,6 +299,7 @@ logger.error('Fatal error, stopping'); // Errors that halt execution - Never use `console.log` -- always use the logger ### Configuration + - JSON Schema validated with Ajv (`useDefaults: true`) - Env vars override config values: `SONARQUBE_TOKEN`, `SONARCLOUD_TOKEN`, `SONARQUBE_URL`, `SONARCLOUD_URL` @@ -293,6 +308,7 @@ logger.error('Fatal error, stopping'); // Errors that halt execution - Config schemas live in `src/shared/config/` ### State Management + - State files track incremental sync progress (last sync time, processed issue keys) - State is only written after successful upload -- never partially update state @@ -303,6 +319,7 @@ logger.error('Fatal error, stopping'); // Errors that halt execution --- ## Testing + **Framework:** Ava + Sinon + esmock @@ -336,6 +353,7 @@ npm run lint # ESLint --- ## Adding New SonarQube Version Support + When a new SonarQube version introduces API changes: @@ -360,6 +378,7 @@ Key version differences to watch for: --- ## Checklist Before Submitting + - [ ] `npm run lint` passes with no errors - [ ] `npm test` passes @@ -374,7 +393,7 @@ Key version differences to watch for: --- ## Regression Test Conventions - + - Every bug fix should have a corresponding regression test in `test/regression/assert-{scenario}.js` - Assertion scripts import from `test/regression/helpers/sqc-client.js` (SQC API client with retry) and `test/regression/helpers/assert-utils.js` (PASS/FAIL output) diff --git a/docs/Core Technologies.md b/docs/Core Technologies.md index 6bbdc355..7c1bdc15 100644 --- a/docs/Core Technologies.md +++ b/docs/Core Technologies.md @@ -1,15 +1,17 @@ # Core Technologies - CloudVoyager is built on a carefully selected stack of battle-tested open-source technologies. Each technology was chosen to address specific requirements of the SonarQube-to-SonarCloud migration workflow. ## 1. Node.js - ### Why Node.js + + Node.js provides the ideal foundation for a cross-platform CLI tool that interfaces with REST APIs. The choice delivers: - **Cross-platform consistency** β€” Same JavaScript runtime on Windows, macOS, and Linux eliminates platform-specific bug hunting @@ -19,6 +21,8 @@ Node.js provides the ideal foundation for a cross-platform CLI tool that interfa ### Runtime Version Requirements + + | Role | Minimum Version | Notes | |------|----------------|-------| | Development / CLI | Node 18+ | Required by `engines.node` in `package.json` | @@ -26,6 +30,8 @@ Node.js provides the ideal foundation for a cross-platform CLI tool that interfa ### Single Executable Application (SEA) + + CloudVoyager ships as a standalone binary using Node.js SEA (Single Executable Application). The build pipeline: 1. Bundles the CLI with `esbuild` into `dist/cli.cjs` @@ -37,10 +43,12 @@ This approach produces a distributable binary that does not require the end user ## 2. Protobuf - ### Why Protobuf + + SonarCloud accepts scanner reports only in Protocol Buffer format. Protobuf was chosen because: - **SonarCloud API requirement** β€” The scanner report upload endpoint expects a binary-encoded Protobuf payload, not JSON @@ -50,6 +58,8 @@ SonarCloud accepts scanner reports only in Protocol Buffer format. Protobuf was ### Protobuf Schema + + The scanner report schema lives at: ``` @@ -68,6 +78,8 @@ Key message types defined in the schema: ### Protobufjs Integration + + The project uses `protobufjs` (v7.2.0) for runtime schema loading and encoding. The encoder factory is in: ``` @@ -76,10 +88,12 @@ src/pipelines/sq-2025/protobuf/encoder/helpers/create-protobuf-encoder.js ## 3. Winston - ### Why Winston + + Structured logging is critical for a migration tool that may run unattended in CI environments. Winston (v3.11.0) was chosen because: - **Multiple transport targets** β€” Logs can be written to the console, to files, and to custom destinations simultaneously @@ -90,6 +104,8 @@ Structured logging is critical for a migration tool that may run unattended in C ### Logger Usage in CloudVoyager + + The logger is initialized per the migration context and exported from: ``` @@ -106,10 +122,12 @@ logger.info('Starting migration', { projectKey: 'my-project', branch: 'main' }); ## 4. Commander.js - ### Why Commander.js + + Commander.js (v12.0.0) is the de facto standard for Node.js CLI frameworks. The choice is driven by: - **Zero-configuration subcommands** β€” Each migration subcommand (`migrate`, `verify`, `sync-metadata`) is registered as a separate module @@ -120,6 +138,8 @@ Commander.js (v12.0.0) is the de facto standard for Node.js CLI frameworks. The ### Command Architecture + + The CLI entry point at `src/index.js` registers all subcommands: ``` @@ -136,10 +156,12 @@ src/commands/ ## 5. Ajv - ### Why Ajv + + Configuration errors should be caught before any migration work begins. Ajv (v8.12.0) was chosen because: - **JSON Schema draft-07 compliance** β€” Industry-standard schema language with broad tooling support @@ -150,6 +172,8 @@ Configuration errors should be caught before any migration work begins. Ajv (v8. ### Schema Hierarchy + + The configuration is validated against a layered schema structure: ``` @@ -163,6 +187,8 @@ Top-level schema references sub-schemas for `sonarqube`, `sonarcloud`, `transfer ### Validation Flow + + ``` loadConfig() β†’ validateConfig() β†’ throws ValidationError on failure ``` @@ -171,10 +197,12 @@ The validator is compiled once and reused across all config loads, avoiding repe ## 6. Electron - ### Why Electron + + While CloudVoyager is primarily a CLI tool, a desktop GUI is provided for users who prefer a visual interface. Electron (v33.4.11) was chosen because: - **Cross-platform desktop** β€” Same codebase runs on Windows, macOS, and Linux @@ -185,6 +213,8 @@ While CloudVoyager is primarily a CLI tool, a desktop GUI is provided for users ### Desktop App Architecture + + ``` desktop/src/ main/main.js β€” Electron main process (window management, IPC) @@ -196,6 +226,8 @@ desktop/src/ ### Building Desktop Binaries + + The desktop app is packaged with `electron-builder`. Each platform is built separately: ```bash @@ -208,10 +240,12 @@ npm run build:linux-arm64 # Linux ARM64 ## 7. GitHub Actions - ### Why GitHub Actions + + CI/CD is implemented entirely in GitHub Actions because: - **Native to the repository** β€” No external CI service configuration required @@ -222,6 +256,8 @@ CI/CD is implemented entirely in GitHub Actions because: ### Workflows + + | Workflow | Purpose | |----------|---------| | `unit-tests.yml` | Run AVA unit tests on every push | @@ -234,6 +270,8 @@ CI/CD is implemented entirely in GitHub Actions because: ### Build Strategy for SEA Binaries + + Each platform is built in a dedicated job: | Platform | Runner | Strategy | @@ -247,4 +285,6 @@ Each platform is built in a dedicated job: ### Regression Testing Strategy + + Regression workflows spin up real SonarQube and SonarCloud instances and run actual migration commands. The workflows use `actions/cache` to share state between jobs rather than artifact uploads, following the project's CI best practice of using cache for inter-job data transfer. diff --git a/docs/Design Principles.md b/docs/Design Principles.md index 9645cf99..acbd3746 100644 --- a/docs/Design Principles.md +++ b/docs/Design Principles.md @@ -10,7 +10,7 @@ CloudVoyager follows a set of design principles that prioritize code clarity, ma ## 1. Folder-Centric Architecture - + One folder per feature, one file per responsibility. @@ -18,6 +18,8 @@ The folder structure tells the story of the system. A well-named folder tree rev ### The Pattern + + Each logical unit (feature, endpoint, pipeline stage) gets its own folder. Inside, concerns are split into separate files: ``` @@ -72,6 +74,8 @@ src/ ### Backward-Compatible Re-exports + + To preserve import paths when decomposing a module, a re-export shim sits at the original path: ```javascript @@ -83,6 +87,8 @@ export { CheckpointJournal, createCheckpointJournal } from './checkpoint/index.j ### Example: State Management Structure + + The `src/shared/state/` directory demonstrates this principle: - `storage.js` β€” Re-export shim pointing to `storage/index.js` @@ -93,16 +99,20 @@ The `src/shared/state/` directory demonstrates this principle: ## 2. Micro Files Over Monoliths - + Every file should do one thing and do it well. No file exceeds 50 lines. ### The Rule + + When a file grows beyond ~30 lines, decompose it into a folder with an orchestrator (`index.js`) and helper files. Each helper exports exactly one function. ### Examples from the Codebase + + **Error Classes (one per file):** ``` @@ -184,6 +194,8 @@ export function createFailedPhase(existingPhase, error) { ### Why It Matters + + - **Navigable:** Developers find exactly what they need without scrolling - **Testable:** Each function can be unit tested in isolation - **Reviewable:** Pull requests are smaller and focused @@ -193,12 +205,14 @@ export function createFailedPhase(existingPhase, error) { ## 3. Flat Over Nested - + Avoid deeply nested logic. Prefer flat sequential flows and helper decomposition. ### The Problem with Nesting + + Nested callbacks, promise chains, and conditional pyramids are hard to read and harder to debug: ```javascript @@ -216,6 +230,8 @@ generateSalt(function(salt) { ### The CloudVoyager Approach + + Chain small, focused functions together. Each step does one job: ```javascript @@ -245,6 +261,8 @@ async function transferProject(options) { ### Factory Functions Over Classes + + Instead of deeply nested class hierarchies, use factory functions that return configured instances: ```javascript @@ -269,6 +287,8 @@ function createCheckpointJournal(journalPath) { ### Helper Decomposition + + When a function has multiple steps, extract each step into a named helper: ```javascript @@ -282,12 +302,14 @@ When a function has multiple steps, extract each step into a named helper: ## 4. Readable at a Glance - + Code should be self-documenting. Use whitespace, comments, and consistent formatting so a developer can understand a file in seconds. ### Section Header Comments + + Divide files into scannable zones: ```javascript @@ -299,6 +321,8 @@ Divide files into scannable zones: ### Descriptive Naming + + Use descriptive, full-word names that convey intent: ```javascript @@ -316,6 +340,8 @@ export function createFailedPhase(existingPhase, error) { ... } ### Whitespace as a Visual Tool + + Use blank lines to separate logical sections within a file: ```javascript @@ -356,6 +382,8 @@ function createCheckpointJournal(journalPath) { ### Inline Comments for Why, Not What + + ```javascript // WHY: Explain non-obvious decisions // Binary-split a date window to handle 10K+ issue retrieval efficiently @@ -370,12 +398,14 @@ export function createCompletedPhase(existingPhase, meta = {}) { ... } ## 5. Shared Libraries for Reuse - + Don't duplicate logic. Extract reusable code into shared libraries under `src/shared/`. ### Shared Modules + + | Module | Purpose | Example Contents | |--------|---------|------------------| | `config/` | Configuration loading and validation | JSON schema, Ajv validators | @@ -387,6 +417,8 @@ Don't duplicate logic. Extract reusable code into shared libraries under `src/sh ### Example: Concurrency Utilities + + Instead of duplicating limiter logic across pipelines, once in `src/shared/utils/concurrency/`: ```javascript @@ -405,6 +437,8 @@ import { createLimiter, mapConcurrent } from '../../../shared/utils/concurrency/ ### Example: Error Classes + + All errors extend a common base, defined once: ```javascript @@ -428,6 +462,8 @@ export class SonarQubeAPIError extends CloudVoyagerError { ... } ### Version-Specific with Shared Foundation + + Each `src/pipelines/sq-{version}/` contains version-specific logic, but all rely on shared utilities: ``` @@ -441,12 +477,14 @@ src/pipelines/sq-2025/ ## 6. Error Handling - + Graceful degradation with proper error classification and logging. ### Custom Error Hierarchy + + All errors extend `CloudVoyagerError`, providing context for debugging: ```javascript @@ -472,6 +510,8 @@ export class LockError extends CloudVoyagerError { } ### Graceful Shutdown Handling + + The `GracefulShutdownError` allows pipelines to respond to SIGINT/SIGTERM: ```javascript @@ -510,6 +550,8 @@ try { ### Lock Files for Concurrent Run Prevention + + Advisory locks prevent concurrent pipeline runs: ```javascript @@ -528,12 +570,14 @@ export function createWriteLock() { ## 7. State Management - + Checkpoint journal for resumability. State persists across failures. ### Checkpoint Journal Pattern + + The `CheckpointJournal` tracks phase completion, enabling pause/resume: ```javascript @@ -565,6 +609,8 @@ function createCheckpointJournal(journalPath) { ### State Persistence + + Atomic writes with backup rotation ensure no state loss: ```javascript @@ -580,6 +626,8 @@ export class StateStorage { ### Journal Structure + + ```javascript { sessionFingerprint: { @@ -604,6 +652,8 @@ export class StateStorage { ### Resume Flow + + ```javascript async function resumeTransfer(transferConfig, projectKey) { const journal = await loadJournal(transferConfig.journalPath); @@ -630,6 +680,8 @@ async function resumeTransfer(transferConfig, projectKey) { ### Lock Integration + + Locks prevent concurrent runs that could corrupt state: ```javascript @@ -644,6 +696,8 @@ registerShutdown(shutdownCoordinator, journal, stateTracker, lockFile); ## Summary + + | Principle | Practice | |-----------|----------| | **Folder-Centric** | One folder per feature, `index.js` + `helpers/` pattern | diff --git a/docs/architecture.md b/docs/architecture.md index f6fbfbb1..da16ac62 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,9 +1,11 @@ # πŸ—οΈ Architecture + ## πŸ“ Project Structure + CloudVoyager uses a **pipeline-per-version** architecture. Each supported SonarQube version range has its own self-contained pipeline, while shared (version-independent) code lives in `src/shared/`. @@ -115,6 +117,7 @@ src/ ``` ### Version-Specific Pipeline Structure + Each pipeline under `src/pipelines/sq-{version}/` uses a **folder-based module architecture** where every module over ~30 lines is decomposed into `module-name/index.js` + `module-name/helpers/*.js`. A re-export file at the original path (e.g., `transfer-pipeline.js`) preserves backward-compatible import paths. @@ -251,6 +254,7 @@ sq-{version}/ ### Shared Utilities β€” SCM Date Backdating + The **batch-distributor** (`src/shared/utils/batch-distributor/`) preserves each issue's original SonarQube creation date in SonarCloud by rewriting SCM changeset blame dates in the protobuf report. @@ -271,6 +275,7 @@ Legacy helpers (`computeBatchPlan`, `computeBatchDate`, `createBatchExtractedDat ## πŸ”„ Version Routing + `version-router.js` detects the SonarQube server version and dynamically imports the correct pipeline: @@ -289,6 +294,7 @@ No runtime version checks exist within any pipeline β€” each pipeline has its be | 2025.1+ | sq-2025 | Modern `issueStatuses` param, Web API V2 with fallbacks | ### Detailed Version Differences + | Behavior | sq-9.9 | sq-10.0 | sq-10.4 | sq-2025 | |----------|--------|---------|---------|---------| @@ -300,9 +306,11 @@ No runtime version checks exist within any pipeline β€” each pipeline has its be ## πŸ”„ Commands and Pipelines + ### `transfer` β€” Single Project + Uses `pipelines/sq-{version}/transfer-pipeline.js` (selected by version-router): @@ -324,6 +332,7 @@ Interrupted transfers resume from the last completed checkpoint phase, skipping ### `migrate` β€” Full Multi-Org Migration + Uses `pipelines/sq-{version}/migrate-pipeline.js` (selected by version-router): @@ -355,6 +364,7 @@ On resume, completed organizations and projects are skipped based on the migrati ### `verify` β€” Migration Verification + Uses `shared/verification/verify-pipeline.js`: @@ -381,6 +391,7 @@ Uses `shared/verification/verify-pipeline.js`: ## 🧩 Key Design Patterns + - **Extractor Pattern** β€” specialized modules for each data type with consistent interface - **Migrator Pattern** β€” specialized modules for each SonarCloud migration target @@ -401,6 +412,7 @@ Uses `shared/verification/verify-pipeline.js`: ## ⚑ Concurrency and Performance + CloudVoyager uses a zero-dependency concurrency layer (`src/shared/utils/concurrency.js`) for parallel I/O: @@ -415,11 +427,13 @@ Extractors and migrators use `mapConcurrent` to parallelize HTTP calls (source f ## πŸ“¦ Build and Packaging + CloudVoyager uses **esbuild + Node.js SEA** (Single Executable Applications) as the default, stable packaging pipeline. An experimental **Bun compile** pipeline is also available but may silently crash at runtime in some environments. ### Build Process (`scripts/build.js`) + **Default (Node.js SEA):** Two-step β€” esbuild bundles `src/index.js` into `dist/cli.cjs` (with `.proto` schemas inlined as text), then Node.js SEA packages it into a standalone binary with V8 code cache via postject. @@ -427,6 +441,7 @@ CloudVoyager uses **esbuild + Node.js SEA** (Single Executable Applications) as ### Output Structure + ``` dist/ @@ -444,6 +459,7 @@ dist/ ### Build Commands + ```bash npm run build # Bundle only via esbuild (dist/cli.cjs) @@ -459,14 +475,17 @@ All CLI flags (`--concurrency`, `--max-memory`, `--project-concurrency`) work id ## πŸ§ͺ CI Workflows + All workflows trigger **only on push to `main`** (merged PRs). No workflow runs on feature branches or pull requests. ### Unit Tests + A standalone `Unit Tests` workflow (`unit-tests.yml`) runs on every push to `main`. It installs dependencies, then runs `npm test`. ### Regression Tests + A separate `Regression Tests` workflow runs on every push to `main`. It does **not** block the release workflow. @@ -484,10 +503,12 @@ setup ───────── β”œβ”€ sync-metadata (4 pa - **Summary:** Gate job that only passes when all 30 integration tests pass ### SonarCloud Analysis + A standalone `SonarCloud Analysis` workflow (`sonarcloud.yml`) runs SAST and SCA scanning on every push to `main`. It does **not** run unit tests or ingest coverage β€” those are handled by the separate Unit Tests workflow. ### Auto Version Bump + The `Auto Version Bump` workflow (`version-bump.yml`) triggers when a PR is merged to `main` **and** the PR has a milestone assigned. It automatically bumps the version in `package.json` and `package-lock.json`: @@ -498,12 +519,14 @@ The `Auto Version Bump` workflow (`version-bump.yml`) triggers when a PR is merg **To use:** Assign a milestone (e.g., `1.2`) to your PR in the GitHub sidebar before merging. The version is derived automatically from the milestone title. ### Build and Release + The `Build and Release` workflow (`release.yml`) builds binaries for 6 platforms and creates a GitHub Release. The release body includes: - Auto-generated release notes from merged PRs - A link to the corresponding GitHub milestone (derived from `package.json` version) ### Workflow files + | File | Purpose | |---|---| @@ -528,6 +551,7 @@ The `Build and Release` workflow (`release.yml`) builds binaries for 6 platforms ## πŸ“„ Generated Report Structure + ``` scanner-report.zip: @@ -548,10 +572,12 @@ Measures are only generated for file components (no project-level `measures-1.pb ## πŸ–₯️ Desktop App Architecture + CloudVoyager Desktop is an Electron (v33) application in the `desktop/` directory that wraps the CLI binary with a guided wizard UI. ### Directory Structure + ``` desktop/ @@ -580,6 +606,7 @@ desktop/ ``` ### How It Works + 1. **Config Wizard** β€” User fills out forms in the renderer process 2. **Config Persistence** β€” Settings saved to `electron-store` (encrypted tokens at rest) @@ -593,6 +620,7 @@ The renderer uses vanilla HTML/CSS/JS with no build step. Security follows Elect > **New desktop components:** `progress-parser.js` parses CLI log output to compute real-time progress percentages and ETA for all three pipeline types (migrate, transfer, verify). `whale-animator.js` renders a pixel-art whale sprite animation with starfield, cloud parallax, and typewriter phase labels during execution. ## πŸ“š Further Reading + - [Configuration Reference](configuration.md) β€” all config options, environment variables, npm scripts - [Technical Details](technical-details.md) β€” protobuf encoding, measure types, concurrency model diff --git a/docs/backward-compatibility.md b/docs/backward-compatibility.md index fc8144c1..81a1bada 100644 --- a/docs/backward-compatibility.md +++ b/docs/backward-compatibility.md @@ -1,8 +1,10 @@ # Backward Compatibility: SonarQube Version Support + CloudVoyager supports migrating from **SonarQube 9.9 LTS** through **SonarQube 2025.1** (and newer) to SonarCloud. This document explains how the tool handles the differences between SonarQube versions. ## Background + Different SonarQube versions introduced significant API and taxonomy changes: @@ -26,6 +28,7 @@ Different SonarQube versions introduced significant API and taxonomy changes: SonarCloud (always the latest version) uses the new Clean Code taxonomy. CloudVoyager bridges these gaps automatically. ## Architecture: Pipeline-Per-Version + Instead of a single codebase with runtime version checks, CloudVoyager uses a **pipeline-per-version** architecture. Each supported SonarQube version range has its own self-contained pipeline under `src/pipelines/`: @@ -72,6 +75,7 @@ sq-{version}/ **No runtime version checks exist within any pipeline.** Each pipeline has its behavior hardcoded for its target SonarQube version range. This eliminates branching logic and makes each pipeline easier to understand and maintain. ## How Version Routing Works + 1. `version-router.js` makes a lightweight `GET /api/system/status` call to detect the SonarQube server version 2. `resolvePipelineId()` maps the parsed version to a pipeline folder: @@ -85,8 +89,10 @@ sq-{version}/ If version detection fails (e.g., network error), the router falls back to the `sq-9.9` pipeline. ## Version-Specific Differences + ### Summary Table + | Behavior | sq-9.9 | sq-10.0 | sq-10.4 | sq-2025 | |----------|--------|---------|---------|---------| @@ -97,6 +103,7 @@ If version detection fails (e.g., network error), the router falls back to the ` | Groups API | `/api/user_groups/search` | Same | Same | Standard (with V2 API fallback) | ### Issue Lifecycle Differences + The issue search API changed significantly across SonarQube versions: @@ -115,12 +122,14 @@ Key changes: Each pipeline uses the correct parameter directly β€” no conditional logic required. ### Metric Keys Batching + SonarQube 9.9 through 10.8 enforces a limit of **15 metric keys per request** to the measures API. Pipelines `sq-9.9`, `sq-10.0`, and `sq-10.4` batch metric key requests accordingly. SonarQube 2025.1+ removed this limit, so the `sq-2025` pipeline sends all metric keys in a single request. ### Clean Code Taxonomy Enrichment + The `sq-9.9` pipeline fetches the Clean Code taxonomy from SonarCloud because SonarQube 9.9 does not provide it: @@ -132,6 +141,7 @@ Rule enrichment map built: 1,247 rules with Clean Code data The `sq-10.0`, `sq-10.4`, and `sq-2025` pipelines read Clean Code data natively from SonarQube β€” no enrichment fetch needed. ### Rule Enrichment from SonarCloud (sq-9.9 only) + For SonarQube 9.9, the `rule-enrichment.js` module in the `sq-9.9` pipeline: @@ -142,6 +152,7 @@ For SonarQube 9.9, the `rule-enrichment.js` module in the `sq-9.9` pipeline: This means the migrated data in SonarCloud will have the **exact same** Clean Code classification as if it had been scanned natively. ### Fallback Chain (sq-9.9 only) + The enrichment follows a two-level fallback: @@ -168,6 +179,7 @@ The enrichment follows a two-level fallback: | `BLOCKER` | `BLOCKER` | ### Groups API + | Pipeline | API Used | |----------|----------| @@ -179,12 +191,14 @@ The enrichment follows a two-level fallback: SonarQube 2025.1+ began deprecating some legacy Web API endpoints. The `sq-2025` pipeline uses the standard groups API but includes fallback to the V2 API (`/api/v2/authorizations/groups`) if the legacy endpoint is unavailable. ### Performance Optimization + - **Transfer pipeline** (`transfer` command): Enrichment map is built once per project transfer (sq-9.9 only) - **Migrate pipeline** (`migrate` command): Enrichment map is built **once per SonarCloud organization** and reused across all projects in that org (sq-9.9 only) - **sq-10.0, sq-10.4, sq-2025**: Enrichment fetch is skipped entirely (no extra API calls) ## Common API Constraints (All Versions) + These constraints apply across all four pipelines: @@ -202,6 +216,7 @@ These constraints apply across all four pipelines: | Built-in quality gates/profiles | Permission APIs return HTTP 400 (expected, handled gracefully) | ## External Issues (All Versions) + All four pipelines support automatic detection and migration of external/plugin issues (e.g., MuleSoft, Checkstyle, PMD): @@ -233,6 +248,7 @@ Clean Code attribute enum values: The `impacts` and `defaultImpacts` (Impact message) fields are also required for external issues to be accepted by SonarCloud. ## Feature Support Matrix + All features are supported across all four pipelines: @@ -256,6 +272,7 @@ All features are supported across all four pipelines: | Fallback rule repositories | Yes | Yes | Yes | Yes | ## Search Slicing and Fallback Repos (All Versions) + Two resilience features apply identically across all four pipelines: @@ -263,6 +280,7 @@ Two resilience features apply identically across all four pipelines: - **Fallback rule repositories** β€” Each pipeline's `getRuleRepositories()` retries 3 times with exponential backoff and falls back to the shared `src/shared/utils/fallback-repos/index.js` (43 known SonarCloud repositories). The `isExternalIssue()` guard in each pipeline uses the fallback set when the live set is empty. ## What Works Identically Across Pipelines + These components are shared across all version-specific pipelines (via `src/shared/`): @@ -283,6 +301,7 @@ These components are shared across all version-specific pipelines (via `src/shar - Graceful shutdown handling ## Usage + No special configuration is needed. CloudVoyager detects the SonarQube version automatically and loads the correct pipeline: @@ -305,6 +324,7 @@ node src/index.js test -c config.json ``` ## Supported SonarQube Versions + | Version | Pipeline | Support Level | Notes | |---------|----------|--------------|-------| @@ -315,8 +335,10 @@ node src/index.js test -c config.json | < 9.9 | sq-9.9 (fallback) | Best effort | APIs may differ; not actively tested | ## Troubleshooting + ### "Failed to detect SonarQube version" + If version detection fails, CloudVoyager falls back to the `sq-9.9` pipeline. This is non-fatal β€” the migration will proceed. Check: @@ -325,6 +347,7 @@ If version detection fails, CloudVoyager falls back to the `sq-9.9` pipeline. Th - Network connectivity to the SonarQube server ### "Failed to build rule enrichment map" + If the enrichment fetch fails (sq-9.9 pipeline), CloudVoyager falls back to type-based inference. This is non-fatal β€” the migration will proceed, but Clean Code attributes may be less precise for active rules. Check: @@ -333,6 +356,7 @@ If the enrichment fetch fails (sq-9.9 pipeline), CloudVoyager falls back to type - Quality profiles exist in SonarCloud for the relevant languages ### Issues appear with generic Clean Code attributes + If migrated issues show `CONVENTIONAL` instead of more specific attributes (like `LOGICAL` or `TRUSTWORTHY`), it means: @@ -342,6 +366,7 @@ If migrated issues show `CONVENTIONAL` instead of more specific attributes (like This is expected for external/plugin rules that don't exist in SonarCloud. For native rules, verify that SonarCloud quality profiles are properly configured for the relevant languages. ### External issues not appearing in SonarCloud + If external issues are missing after migration: diff --git a/docs/bespoke-algorithms.md b/docs/bespoke-algorithms.md index 458c1202..21b8ae13 100644 --- a/docs/bespoke-algorithms.md +++ b/docs/bespoke-algorithms.md @@ -1,6 +1,6 @@ # Bespoke Algorithms - + This document describes custom, non-trivial algorithms implemented from scratch within CloudVoyager β€” logic that required deliberate design decisions rather than off-the-shelf solutions. @@ -8,7 +8,7 @@ This document describes custom, non-trivial algorithms implemented from scratch ## Table of Contents -1. [Date-Window Slicing (10K+ Issue Retrieval)](#1-date-window-slicing-10k-issue-retrieval) + + ## πŸ“‹ Single Project Config Used by: `transfer`, `test`, `validate`, `status`, `reset` @@ -51,7 +51,7 @@ Used by: `transfer`, `test`, `validate`, `status`, `reset` See `examples/config.example.json` for a complete example. - + ## πŸ“‹ Migration Config Used by: `migrate`, `sync-metadata`, `verify` @@ -115,10 +115,10 @@ See `examples/migrate-config.example.json` for a complete example. --- - + ## πŸ”§ Configuration Options - + ### SonarQube Settings | Option | Required | Description | @@ -127,7 +127,7 @@ See `examples/migrate-config.example.json` for a complete example. | `token` | Yes | SonarQube API token (or set via `SONARQUBE_TOKEN` env var) | | `projectKey` | For `transfer` only | Project key to export (not needed for `migrate`) | - + ### SonarCloud Settings (Single Org) Used by `transfer`, `test`, `validate`, `status`, `reset`. @@ -139,7 +139,7 @@ Used by `transfer`, `test`, `validate`, `status`, `reset`. | `organization` | Yes | SonarCloud organization key | | `projectKey` | For `transfer` only | Destination project key. The display name is automatically carried over from SonarQube | - + ### SonarCloud Settings (Multi-Org) Used by `migrate`, `sync-metadata`. Instead of a single org, you provide an array of target organizations. @@ -151,7 +151,7 @@ Used by `migrate`, `sync-metadata`. Instead of a single org, you provide an arra | `organizations[].url` | No | SonarCloud server URL. Use `https://sonarcloud.io` (EU, default) or `https://sonarqube.us` (US). In the Desktop app an EU/US radio button sets this automatically. | | `enterprise.key` | Optional | SonarCloud enterprise key. Required only for portfolio migration via V2 API. If absent, portfolio migration is gracefully skipped and the migration continues normally. | - + ### Transfer Settings | Option | Default | Description | @@ -174,7 +174,7 @@ The `transfer.checkpoint` block controls the pause/resume behavior. All settings | `cacheMaxAgeDays` | `7` | Maximum age of extraction cache files in days before auto-purge | | `strictResume` | `false` | Fail on SonarQube version mismatch when resuming (default: warn only) | - + ### Migrate Settings | Option | Default | Description | @@ -187,7 +187,7 @@ The `transfer.checkpoint` block controls the pause/resume behavior. All settings > **Project key behavior (migrate command):** By default, the `migrate` command uses the original SonarQube project key on SonarCloud. If the key is already taken by another SonarCloud organization, the tool falls back to a prefixed key (`{org}_{key}`) and logs a warning. Key conflicts are listed in the migration report. - + ### Rate Limit Settings Controls retry and throttling behavior for SonarCloud API requests. By default, retries are enabled (`maxRetries: 3`) but request throttling is off (`minRequestInterval: 0`). Add a `rateLimit` section to any config file to customize. @@ -220,7 +220,7 @@ Controls retry and throttling behavior for SonarCloud API requests. By default, } ``` - + ### Performance Settings Controls CPU, memory, and concurrency tuning. Add a `performance` section to any config file. All settings are optional β€” defaults are tuned for safe, moderate parallelism. @@ -330,7 +330,7 @@ Multiple components can be combined: `--only scan-data,quality-gates,permissions ./cloudvoyager migrate -c migrate-config.json --verbose --max-memory 8192 ``` - + ## 🌍 Environment Variables | Variable | Description | @@ -344,7 +344,7 @@ Multiple components can be combined: `--only scan-data,quality-gates,permissions | `MAX_SOURCE_FILES` | Limit number of source files to extract (0 = all) | | `SONAR_TOKEN` | SonarCloud token used by the `sonarcloud.yml` GitHub Actions workflow for SAST/SCA scanning of the CloudVoyager repository itself (not used by the CLI) | - + ## πŸ“œ npm Scripts vs Binary Commands CloudVoyager can be run in two ways: @@ -403,12 +403,12 @@ All CLI flags work identically in both modes. The table below shows every availa > **Note:** The npm scripts use hardcoded config file paths (`config.json` or `migrate-config.json`). When using the binary directly, you can specify any config file path with `-c `. - + ## πŸš€ Recommended Migration Workflow For multi-project migrations (`migrate` command), we recommend the following 3-step approach. This gives you the best combination of safety, speed, and reliability. - + ### Step 1: Dry run β€” verify everything Run a dry run first to extract all data, generate mapping CSVs, and validate your config without touching SonarCloud: @@ -423,7 +423,7 @@ npm run migrate:dry-run Check the generated files in `./migration-output/` (especially `mappings/organizations.csv`) to verify project-to-org assignments look correct. - + ### Step 2: Migrate without metadata + auto-tune Run the actual migration with metadata sync disabled and auto-tuned performance. This transfers all projects, quality gates, profiles, groups, permissions, and report data β€” but skips the slower issue/hotspot status transitions: @@ -438,7 +438,7 @@ npm run migrate:skip-all-metadata:auto-tune Skipping metadata during the main migration avoids SonarCloud rate limiting (503 errors) that can occur during high-volume issue/hotspot sync. - + ### Step 3: Sync metadata separately Once all projects are migrated, sync issue and hotspot metadata as a standalone step. This transitions issue statuses, copies comments, sets assignees, and syncs tags: @@ -453,7 +453,7 @@ npm run sync-metadata This step is safely retryable β€” if it hits rate limits, just run it again. Already-synced items are matched by rule+file+line and won't be duplicated. - + ### Why this approach? | Step | What it does | Why | @@ -463,7 +463,7 @@ This step is safely retryable β€” if it hits rate limits, just run it again. Alr | Sync metadata | Transitions issue/hotspot statuses | Retryable, isolated from main migration | | Verify | Compares SQ and SC data exhaustively | Confirms 1:1 migration completeness | - + ## πŸ”„ Incremental Transfers When using incremental mode, the tool: @@ -476,6 +476,7 @@ When using incremental mode, the tool: To force a full transfer, use the `reset` command to clear the state. ### Checkpoint Journal (Pause/Resume) + The `transfer` command automatically maintains a **checkpoint journal** that tracks progress at the phase level. If a transfer is interrupted (CTRL+C, crash, network failure), you can resume from where it left off by simply running the same command again. @@ -494,6 +495,7 @@ The journal records: Use `--force-restart` to discard the checkpoint journal and start from scratch. Use `--force-fresh-extract` to clear extraction caches while keeping the journal. ### State File + The state file (`.cloudvoyager-state.json` by default) contains: - Last sync timestamp @@ -508,6 +510,7 @@ Additional files created during transfer: - `cache/` directory β€” Extraction cache files (gzipped JSON) ## πŸ“š Further Reading + - [Architecture](architecture.md) β€” project structure, data flow, report format - [Technical Details](technical-details.md) β€” protobuf encoding, measure types, concurrency model diff --git a/docs/design-review-regression-testing.md b/docs/design-review-regression-testing.md index 4ff7f911..9338c9a6 100644 --- a/docs/design-review-regression-testing.md +++ b/docs/design-review-regression-testing.md @@ -1,5 +1,5 @@ # Design Review: Matrix-Based Regression Testing for All Fixed Issues - + > **Implementation Status: SHIPPED (Phase 1)** β€” PR [#127](https://github.com/sonar-solutions/cloudvoyager/pull/127). 22 files, 1,309 lines. 5 scenarios x 4 SQ versions. Sensitive CI moved to private repo `sonar-solutions/cloudvoyager-ci`. @@ -9,14 +9,15 @@ **Quality score:** 7/10 ## Summary - + 9 issues found across 4 of 5 dimensions. No blockers, but fixes needed before implementation to prevent surprises. Scope dimension passed cleanly. ## Issues by Dimension - + ### Completeness (3 issues) + | ID | Issue | Suggested Fix | |----|-------|---------------| @@ -25,6 +26,7 @@ | 1.3 | `issue-sync-first-migration` assertion is unmeasurable ("triggers correctly") | Define as: SQC API returns >0 issues within N seconds post-sync | ### Consistency (2 issues) + | ID | Issue | Suggested Fix | |----|-------|---------------| @@ -32,6 +34,7 @@ | 2.2 | Trigger is push-to-main only, but success criteria claims pre-merge PR gating | Either add PR trigger or reword to "post-merge detection" | ### Clarity (2 issues) + | ID | Issue | Suggested Fix | |----|-------|---------------| @@ -39,10 +42,12 @@ | 3.2 | Cleanup "deletes and recreates" doesn't specify API calls or whether recreation is implicit | Clarify: DELETE endpoint + implicit recreation via scanner report upload | ### Scope + PASS -- no YAGNI violations. ### Feasibility (2 issues) + | ID | Issue | Suggested Fix | |----|-------|---------------| diff --git a/docs/desktop-app.md b/docs/desktop-app.md index 21e903ba..808751dc 100644 --- a/docs/desktop-app.md +++ b/docs/desktop-app.md @@ -1,4 +1,5 @@ # Desktop App + @@ -9,6 +10,7 @@ CloudVoyager Desktop wraps the CLI binary in a guided wizard UI built with Elect Available for: **Linux x64**, **Linux ARM64**, **macOS ARM64**, **macOS x64**, **Windows x64**, **Windows ARM64**. ## Installation + Download the latest release from the [GitHub Releases](https://github.com/your-org/cloudvoyager/releases) page. Choose the installer for your platform: @@ -24,8 +26,10 @@ Download the latest release from the [GitHub Releases](https://github.com/your-o > The app bundles the CLI binary β€” no separate CLI install is needed. ## Getting Started + ### Welcome Screen + On launch, the Welcome screen presents the available workflows: @@ -37,6 +41,7 @@ On launch, the Welcome screen presents the available workflows: - **Clear Migration History** β€” Reset state and clear sync history ### Wizard Flow + Each workflow guides you through a series of screens: @@ -48,8 +53,10 @@ Each workflow guides you through a series of screens: The live log viewer shows migration progress in real-time with a timer, cancel button, and status badge. When complete, the results screen shows generated report files that you can browse and open. ## Wizard Screens + ### Transfer Config (4 steps) + | Step | Description | |------|-------------| @@ -59,6 +66,7 @@ The live log viewer shows migration progress in real-time with a timer, cancel b | 4. Review & Start | Review all settings and begin the transfer | ### Migrate Config (4 steps) + | Step | Description | |------|-------------| @@ -70,6 +78,7 @@ The live log viewer shows migration progress in real-time with a timer, cancel b ![Migration Settings Wizard](screenshots/full_migration.png) ### Verify Config (3 steps) + | Step | Description | |------|-------------| @@ -78,6 +87,7 @@ The live log viewer shows migration progress in real-time with a timer, cancel b | 3. Review & Start | Review settings and begin verification | ### Sync Metadata Config (3 steps) + | Step | Description | |------|-------------| @@ -86,6 +96,7 @@ The live log viewer shows migration progress in real-time with a timer, cancel b | 3. Review & Start | Review settings and begin metadata synchronization | ### Other Screens + - **Connection Test** β€” Runs the `test` command to verify connectivity to both SonarQube and SonarCloud - **Execution** β€” Live log viewer with elapsed timer, cancel button, and status badge (running/success/failed) @@ -100,6 +111,7 @@ The live log viewer shows migration progress in real-time with a timer, cancel b - **Status** β€” View migration progress, sync history, and reset state ### Progress Tracking and Animation + The execution screen includes real-time progress tracking with a whale animation: @@ -109,6 +121,7 @@ The execution screen includes real-time progress tracking with a whale animation The progress bar layout divides 0-100% across pipeline phases (e.g., for migrate: 0-10% setup, 10-15% org config, 15-95% per-project migration, 95-100% finalization). ## Configuration Persistence + - All settings are saved automatically as you navigate between wizard steps - Tokens are encrypted at rest using `electron-store` @@ -126,13 +139,16 @@ The progress bar layout divides 0-100% across pipeline phases (e.g., for migrate | Windows | `%APPDATA%\cloudvoyager-desktop\` | ## Building from Source + ### Prerequisites + - Node.js v20+ - The CLI binary placed in `desktop/resources/cli/` (use `node scripts/prepare-cli.js` to copy from `dist/bin/`) ### Development + ```bash cd desktop @@ -143,6 +159,7 @@ npm start # Run in dev mode (uses src/index.js as CLI fallback) Pass `--dev` to enable DevTools on startup. ### Distribution Builds + ```bash npm run build:linux-x64 @@ -156,6 +173,7 @@ npm run build:win-arm64 Output artifacts are placed in the `desktop/dist/` directory. ## Architecture + The desktop app lives in the `desktop/` directory at the repository root. It is built with **Electron 33.4.11** and vanilla HTML/CSS/JS β€” no frontend frameworks. @@ -207,6 +225,7 @@ desktop/ ``` ### Main Process + - **main.js**: Enforces single-instance lock so only one app window runs at a time. Persists window bounds across restarts. Registers `F6` as a global shortcut to capture screenshots. Passes `--dev` to open DevTools on startup. Detects OS theme for light/dark mode. - **ipc-handlers.js**: Registers 10 IPC channels under namespaces `config:*`, `cli:run`, `cli:cancel`, `dialog:*`, `reports:*`, `devtools:capture`, and `app:*`. @@ -214,10 +233,12 @@ desktop/ - **config-store.js**: Wraps `electron-store` with encryption enabled. Maintains two config schemas (`transferConfig` and `migrateConfig`), a 50-entry migration history, and theme preference. ### Preload Bridge + **preload.js** exposes the `window.cloudvoyager` namespace to the renderer via `contextBridge`. It provides 7 modules: `config`, `cli`, `dialog`, `reports`, `app`, `theme`, and `devtools`. All renderer-to-main communication goes through this bridge. ### Renderer + - **app.js**: Defines the `window.App` global, manages 9 screens, handles theme switching, and provides toast notifications (6-second auto-dismiss) and confirm dialogs with focus traps. - **Screens**: Each wizard screen manages its own step navigation and form state. Transfer and migrate configs use 4-step wizards; verify and sync-metadata use 3-step wizards. @@ -229,6 +250,7 @@ desktop/ - `sidebar-history.js` β€” Displays the last 50 migration/transfer runs with clickable entries to view reports ### Security + - **Context isolation** enabled; `nodeIntegration` disabled. Sandbox is disabled to allow CLI binary spawning. - **CSP headers** configured to restrict content sources. @@ -238,6 +260,7 @@ desktop/ - **Temporary config files** written with `0o600` permissions and auto-cleaned after CLI execution. ### Accessibility + - ARIA labels, roles, and live regions throughout the UI - Focus management and keyboard navigation support @@ -245,6 +268,7 @@ desktop/ - Focus traps in confirm dialogs ### Data Flow (Transfer Example) + 1. User fills the 4-step wizard β€” config is saved to `electron-store` at each step 2. `cli:run` IPC message sent to main process β€” main writes a temp config file, spawns the CLI binary @@ -254,6 +278,7 @@ desktop/ 6. History entry saved to `electron-store`; temp config file auto-cleaned ### Key Design Decisions + - **Electron 33.4.11** with vanilla HTML/CSS/JS β€” no framework dependency - **`contextBridge` / IPC** for secure renderer-to-main communication @@ -265,6 +290,7 @@ desktop/ - **Visual effects**: glassmorphism panels, scanline overlay, ambient gradients, typing animations, staggered card entrance, whale progress indicator ## CLI vs Desktop Feature Comparison + Both the CLI and Desktop app provide the same migration capabilities. The Desktop app adds a graphical wrapper with guided wizards and encrypted storage. diff --git a/docs/dry-run-csv-reference.md b/docs/dry-run-csv-reference.md index 281fbef0..2ea4c2a3 100644 --- a/docs/dry-run-csv-reference.md +++ b/docs/dry-run-csv-reference.md @@ -1,12 +1,15 @@ # Dry-Run CSV Reference + ## Overview + The `--dry-run` flag generates 9 exhaustive CSV files in `migration-output/mappings/`. Each CSV includes an **Include** column (first field) that defaults to `yes`. You can edit these CSVs to customize the migration before running the actual migration. ## Workflow + ``` 1. cloudvoyager migrate -c config.json --dry-run @@ -27,6 +30,7 @@ The `--dry-run` flag generates 9 exhaustive CSV files in `migration-output/mappi ``` ## Include Column + Every CSV has an `Include` column as the first field. Values are case-insensitive: @@ -37,6 +41,7 @@ Every CSV has an `Include` column as the first field. Values are case-insensitiv | *(empty)* | Include (treated as default) | ## CSV Files Summary + | # | File | Purpose | Editable columns | |---|------|---------|-----------------| @@ -53,6 +58,7 @@ Every CSV has an `Include` column as the first field. Values are case-insensitiv --- ### projects.csv + One row per branch per project. The main branch is listed first for each project. @@ -82,6 +88,7 @@ In this example, `main` and `develop` branches will be migrated. The `feature/ol **Backward compatibility:** If your CSV was generated by an older version and does not have a Branch column, all branches will be included (same as before). ### organizations.csv + One row per organization binding group. @@ -94,6 +101,7 @@ One row per organization binding group. | Projects Count | Read-only | Number of projects in this group | ### group-mappings.csv + One row per group. @@ -107,6 +115,7 @@ One row per group. | Target Organization | Read-only | Target SonarCloud organization | ### profile-mappings.csv + One row per quality profile. @@ -122,6 +131,7 @@ One row per quality profile. | Target Organization | Read-only | Target SonarCloud organization | ### gate-mappings.csv + One row per quality gate. Conditions are always migrated as-is from SonarQube β€” they cannot be modified via the CSV. @@ -145,6 +155,7 @@ no,Old Gate,false,false,1,my-org In this example, `Old Gate` is excluded from migration. `Sonar way` and `Custom Gate` will be migrated with all their original conditions intact. ### portfolio-mappings.csv (Parent/Child Pattern) + Uses a **parent/child row pattern**. Portfolio header rows have empty member fields; member rows have the project details. @@ -160,6 +171,7 @@ Uses a **parent/child row pattern**. Portfolio header rows have empty member fie | Target Organization | Read-only | Target SonarCloud organization | ### template-mappings.csv (Parent/Child Pattern) + Uses a **parent/child row pattern**. Template header rows have empty permission fields; permission rows have the assignment details. @@ -174,6 +186,7 @@ Uses a **parent/child row pattern**. Template header rows have empty permission | Target Organization | Read-only | Target SonarCloud organization | ### global-permissions.csv + One row per group+permission combination. @@ -184,6 +197,7 @@ One row per group+permission combination. | Permission | Read-only | Permission key (e.g., `admin`, `scan`, `provisioning`) | ### user-mappings.csv + One row per unique SonarQube issue assignee, sorted by issue count (descending). This CSV enables mapping SonarQube usernames to SonarCloud logins, which typically differ because SonarCloud uses SSO/GitHub authentication. @@ -216,36 +230,47 @@ In this example: --- ## Common Edit Scenarios + ### Exclude a project from migration + In `projects.csv`, set `Include` to `no` for **all** branch rows of the project (or at minimum the main branch row). ### Exclude specific branches from migration + In `projects.csv`, set `Include` to `no` for individual branch rows you want to skip. The main branch must remain `yes` for the project to be migrated. ### Exclude a quality gate + In `gate-mappings.csv`, set `Include` to `no` for the gate row. All its conditions will be excluded along with it. ### Exclude a group + In `group-mappings.csv`, set `Include` to `no` for the group row. ### Remove a specific permission from a template + In `template-mappings.csv`, set `Include` to `no` on the specific permission row (not the template header). ### Map a SonarQube user to a SonarCloud user + In `user-mappings.csv`, fill in the `SonarCloud Login` column with the user's SonarCloud login (e.g., their GitHub username). ### Skip assignment for a service account + In `user-mappings.csv`, set `Include` to `no` for the service account row. ### Remove a project from a portfolio + In `portfolio-mappings.csv`, set `Include` to `no` on the specific member row (not the portfolio header). ### CSV Entity Filtering Module + The shared `csv-entity-filters.js` module (`src/shared/mapping/csv-entity-filters.js`) implements the filtering logic that applies CSV overrides during migration. It supports 7 entity types: quality gates, quality profiles, groups, global permissions, permission templates, portfolios, and user mappings. Each function reads the `Include` column and filters the extracted data accordingly. ## Notes + - CSVs are RFC 4180 compliant. Fields containing commas, quotes, or newlines are properly quoted. - Running `--dry-run` again will regenerate fresh CSVs, overwriting any edits. @@ -253,7 +278,8 @@ The shared `csv-entity-filters.js` module (`src/shared/mapping/csv-entity-filter - Gate names, project keys, and group names are matched **case-sensitively**. - If a migration is interrupted mid-run, re-running the same command resumes from the last checkpoint. Already-migrated projects and resources are skipped. -## πŸ“š Further Reading +## Further Reading + - [Technical Details](technical-details.md) β€” protobuf encoding, dry-run workflow, concurrency model - [Configuration Reference](configuration.md) β€” all config options, environment variables, npm scripts diff --git a/docs/key-capabilities.md b/docs/key-capabilities.md index 7026835e..077e7c22 100644 --- a/docs/key-capabilities.md +++ b/docs/key-capabilities.md @@ -1,12 +1,10 @@ # CloudVoyager β€” Key Capabilities - - -A comprehensive overview of CloudVoyager's engineering, architecture, and capabilities for techno-functional leadership review. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + -For large projects (β‰₯500 matched issue pairs), issue sync is parallelized across `worker_threads` to overcome the single event loop bottleneck: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + ## βœ… Prerequisites 1. **Node.js** v20 LTS (required for `npm run package` β€” see warning below) @@ -33,6 +33,7 @@ npm install ``` ### Key Dependencies + **Production (25 total):** protobufjs 7.2.0, commander 12.0.0, axios 1.6.0, ajv 8.12.0, winston 3.11.0, pdfmake 0.3.4, form-data 4.0.0, adm-zip 0.5.16, and others. @@ -42,12 +43,12 @@ npm install --- - + ## πŸ“¦ Building the Binary CloudVoyager can be compiled into a standalone binary using two packaging backends. Both produce a self-contained binary that does not require Node.js or Bun to be installed on the target machine. - + ### Node.js SEA (Default) ```bash @@ -60,6 +61,7 @@ Uses esbuild for bundling + Node.js [Single Executable Applications (SEA)](https > **WARNING: Requires Node.js v20 LTS.** Node.js v22+ causes a `postject` injection failure ("Multiple occurrences of sentinel"). Run `nvm use 20` before packaging. See [Prerequisites](#-prerequisites) for details. #### Build Pipeline Steps + The Node.js SEA build runs three stages internally: @@ -68,7 +70,7 @@ The Node.js SEA build runs three stages internally: 3. **postject injection** β€” injects the SEA blob into a copy of the Node.js binary β†’ `dist/bin/cloudvoyager-{platform}-{arch}` 4. **macOS only** β€” code signature removal and re-signing is required after injection - + ### Bun Compile (Experimental) ```bash @@ -78,7 +80,7 @@ npm run package:bun:cross # Cross-compile 5 platform binaries Uses Bun's single-step compile β€” source goes directly to a native binary with no intermediate bundle. Bun is installed as an optional dependency β€” no global install required. While faster to build, Bun binaries may silently crash at runtime in some environments, so this is considered experimental. - + ### Output | Platform | Output Binary | Build Method | @@ -102,17 +104,19 @@ This creates `dist/cli.cjs`, which can be run with `node dist/cli.cjs --- - + ## πŸ–₯️ Desktop App CloudVoyager Desktop is an Electron-based GUI that wraps the CLI binary. It provides a wizard interface for configuring and running migrations without using the terminal. Past successful runs are saved in a sidebar history for quick access to reports. ### Prerequisites + - Node.js v20+ and npm - The CLI binary built and available in `dist/bin/` (run `npm run package` first) ### Running in Development + ```bash cd desktop @@ -128,6 +132,7 @@ npm start In development mode, if no CLI binary is found in `resources/cli/`, the app falls back to running `node src/index.js` directly. ### Building for Distribution + Each command builds a platform-specific installer: @@ -158,7 +163,7 @@ See the [Desktop App Guide](desktop-app.md) for the full user guide. --- - + ## πŸƒ Running the Binary After building, make it executable (macOS/Linux) and run it directly: @@ -171,7 +176,7 @@ chmod +x dist/bin/cloudvoyager-macos-arm64 ./dist/bin/cloudvoyager-macos-arm64 migrate -c migrate-config.json --verbose ``` - + ### Quick Start Examples ```bash @@ -220,7 +225,7 @@ See the [CLI Reference](#-cli-reference) section below for all available flags a This section documents every command and flag available in CloudVoyager. The examples use `./cloudvoyager` as shorthand β€” substitute with your actual binary path (e.g. `./dist/bin/cloudvoyager-macos-arm64`). - + ### Global Flag | Flag | Short | Description | @@ -230,14 +235,14 @@ This section documents every command and flag available in CloudVoyager. The exa --- - + ### `validate` β€” Validate configuration file | Flag | Short | Required | Argument | Description | |------|-------|----------|----------|-------------| | `--config ` | `-c` | Yes | File path | Path to the configuration file to validate | - + #### Examples ```bash @@ -250,7 +255,7 @@ This section documents every command and flag available in CloudVoyager. The exa --- - + ### `test` β€” Test connections to SonarQube and SonarCloud | Flag | Short | Required | Argument | Description | @@ -258,7 +263,7 @@ This section documents every command and flag available in CloudVoyager. The exa | `--config ` | `-c` | Yes | File path | Path to the configuration file containing connection details | | `--verbose` | `-v` | No | β€” | Enable debug-level logging for detailed output | - + #### Examples ```bash @@ -271,14 +276,14 @@ This section documents every command and flag available in CloudVoyager. The exa --- - + ### `status` β€” Show current synchronization status | Flag | Short | Required | Argument | Description | |------|-------|----------|----------|-------------| | `--config ` | `-c` | Yes | File path | Path to the configuration file (reads the state file path from it) | - + #### Examples ```bash @@ -288,7 +293,7 @@ This section documents every command and flag available in CloudVoyager. The exa --- - + ### `reset` β€” Reset state and clear sync history | Flag | Short | Required | Argument | Description | @@ -296,7 +301,7 @@ This section documents every command and flag available in CloudVoyager. The exa | `--config ` | `-c` | Yes | File path | Path to the configuration file (reads the state file path from it) | | `--yes` | `-y` | No | β€” | Skip the confirmation prompt and reset immediately | - + #### Examples ```bash @@ -312,7 +317,7 @@ This section documents every command and flag available in CloudVoyager. The exa --- - + ### `transfer` β€” Transfer a single project from SonarQube to SonarCloud | Flag | Short | Required | Argument | Description | @@ -329,7 +334,7 @@ This section documents every command and flag available in CloudVoyager. The exa | `--force-unlock` | β€” | No | β€” | Force release a stale lock file from a previous run | | `--show-progress` | β€” | No | β€” | Display checkpoint progress status table and exit | - + #### Examples ```bash @@ -404,7 +409,7 @@ This section documents every command and flag available in CloudVoyager. The exa --- - + ### `migrate` β€” Full migration from SonarQube to one or more SonarCloud organizations | Flag | Short | Required | Argument | Description | @@ -582,7 +587,7 @@ This section documents every command and flag available in CloudVoyager. The exa --- - + ### `verify` β€” Verify migration completeness by comparing SonarQube and SonarCloud data | Flag | Short | Required | Argument | Description | @@ -644,7 +649,7 @@ This section documents every command and flag available in CloudVoyager. The exa --- - + ### `sync-metadata` β€” Sync issue and hotspot metadata for already-migrated projects | Flag | Short | Required | Argument | Description | @@ -659,7 +664,7 @@ This section documents every command and flag available in CloudVoyager. The exa | `--auto-tune` | β€” | No | β€” | Auto-detect hardware and set optimal concurrency and memory values | | `--skip-all-branch-sync` | β€” | No | β€” | Only sync the main branch of each project (skip non-main branches). Equivalent to setting `transfer.syncAllBranches: false` in config | - + #### Examples ```bash @@ -726,7 +731,7 @@ This section documents every command and flag available in CloudVoyager. The exa --- - + ## 🧹 Linting ```bash @@ -741,7 +746,7 @@ npm run lint:fix --- - + ## πŸ§ͺ Running Tests ```bash @@ -753,6 +758,7 @@ npm run test:fast ``` ### Test Configuration + - **Test runner:** [AVA](https://github.com/avajs/ava) v6.4.1 - **Test glob:** `test/**/*.test.js` (50+ test files) @@ -765,7 +771,7 @@ npm run test:fast --- - + ## πŸ—οΈ CI/CD (GitHub Actions) The project uses a multi-stage GitHub Actions pipeline: @@ -780,10 +786,12 @@ The project uses a multi-stage GitHub Actions pipeline: | `gh-release.yml` | GitHub Releases | Creates releases with milestone links derived from version tags | ### SonarCloud Scanning + The repository includes a `sonarcloud.yml` workflow and a `sonar-project.properties` file at the project root for automatic SAST/SCA scanning via SonarCloud. The workflow runs on every push to `main` and on pull requests. It requires a `SONAR_TOKEN` secret configured in the GitHub repository settings. ### Platform Build Matrix (6 targets) + | Platform | Runner | |----------|--------| @@ -796,7 +804,7 @@ The repository includes a `sonarcloud.yml` workflow and a `sonar-project.propert --- - + ## 🌍 Environment Variables | Variable | Description | @@ -809,7 +817,7 @@ The repository includes a `sonarcloud.yml` workflow and a `sonar-project.propert | `SONARCLOUD_URL` | Override SonarCloud URL from config | | `MAX_SOURCE_FILES` | Limit number of source files to extract (`0` = all) | - + ### Examples with environment variables ```bash @@ -825,7 +833,7 @@ MAX_SOURCE_FILES=10 ./cloudvoyager transfer -c config.json --verbose --- - + ## ⚑ npm Scripts The following npm scripts are available for building, testing, and linting: @@ -889,7 +897,7 @@ The following npm scripts are available for building, testing, and linting: --- - + ## πŸ› Debugging Scripts The `.debugging/` folder contains convenience scripts for local testing: @@ -907,7 +915,7 @@ These scripts expect the binary at `./dist/bin/cloudvoyager-macos-arm64` and con --- - + ## πŸ“š Further Reading - [Configuration Reference](configuration.md) β€” all config options, environment variables, npm scripts @@ -920,17 +928,19 @@ These scripts expect the binary at `./dist/bin/cloudvoyager-macos-arm64` and con --- - + ## πŸ§ͺ Regression Testing The regression testing system validates that bug fixes and behavioral changes in CloudVoyager remain correct across SonarQube versions. ### Where the Code Lives + - **Assertion scripts and helpers:** `test/regression/` in this repository contains the assertion scripts (`assert-*.js`), shared helpers, and enrichment scripts used by regression tests. - **CI workflow:** The actual CI pipeline that orchestrates regression runs lives in the **private** repo `sonar-solutions/cloudvoyager-ci` (not this public repo). The workflow file is `regression-bug-fixes.yml`. ### Running Regression Tests Locally + Full regression tests require an environment that mirrors CI: @@ -941,6 +951,7 @@ Full regression tests require an environment that mirrors CI: > **Note:** These prerequisites make local runs heavyweight. Most day-to-day development does not require running the full regression suite locally β€” CI handles it on every PR. ### Meta-Tests (No Docker Required) + The helper utilities in `test/regression/helpers/` have their own unit tests (`test/regression/helpers/*.test.js`). These **meta-tests** run with the regular test command and do not require Docker or a running SonarQube instance: @@ -949,6 +960,7 @@ npm test ``` ### Adding a New Regression Test + 1. **Create an assertion script:** Add `test/regression/assert-{scenario-name}.js` in this repository. Follow the conventions of existing assertion scripts in the same directory. 2. **Add a matrix entry:** In the private repo `sonar-solutions/cloudvoyager-ci`, add a new entry to the matrix in `regression-bug-fixes.yml` that references your assertion script and the SonarQube version(s) it should run against. diff --git a/docs/milestone-1.2-verification-report.md b/docs/milestone-1.2-verification-report.md index 8a137a11..20ea7672 100644 --- a/docs/milestone-1.2-verification-report.md +++ b/docs/milestone-1.2-verification-report.md @@ -1,4 +1,5 @@ # Milestone 1.2 β€” Verification Report + **Branch:** `contributors/joshuaquek` **Date:** 2026-03-26 @@ -7,6 +8,7 @@ --- ## Summary + | Issue | Title | Status | |-------|-------|--------| @@ -18,12 +20,14 @@ --- ## Issue #53 β€” 10K+ Issues Search Slicing + **Problem:** SonarQube's `/api/issues/search` endpoint has a hard 10,000-result limit. Projects exceeding this silently lost data during migration. **Solution:** Implemented a date-window search-slicing algorithm that automatically activates when results hit the 10K limit. ### Files Created + | File | Purpose | |------|---------| @@ -38,6 +42,7 @@ | `src/pipelines/sq-2025/sonarqube/api-client/helpers/probe-total.js` | (identical) | ### Files Modified + | File | Change | |------|--------| @@ -46,6 +51,7 @@ | `src/pipelines/sq-2025/sonarqube/api-client/helpers/issue-methods.js` | Additional delegation file unique to sq-2025; also wired with `fetchWithSlicing` | ### Verification + | Check | Result | |-------|--------| @@ -60,18 +66,21 @@ --- ## Issue #56 β€” Third-Party Issue Migration Bug + **Problem:** When `getRuleRepositories()` failed to fetch the SonarCloud rules API, it returned an empty Set. This caused `isExternalIssue()` to return `false` for ALL issues, silently dropping third-party analyzer issues (ruff, pylint, Trivy, etc.) during migration. **Solution:** Added a fallback set of 43 known SonarCloud rule repositories, retry logic with exponential backoff on `getRuleRepositories()`, and edge-case guards in `isExternalIssue()`. ### Files Created + | File | Purpose | |------|---------| | `src/shared/utils/fallback-repos/index.js` | Exports `FALLBACK_SONARCLOUD_REPOS` β€” Set of 43 known built-in SonarCloud rule repository keys | ### Files Modified + | File | Change | |------|--------| @@ -86,6 +95,7 @@ | `src/pipelines/sq-2025/.../query-methods-4.js` | (same change) | ### Verification + | Check | Result | |-------|--------| @@ -102,12 +112,14 @@ --- ## Issue #66 β€” SonarCloud Public Scanning + **Problem:** No automated SAST, unit test coverage, or SCA scanning was configured for the repository. **Solution:** Added a standalone GitHub Actions workflow that runs SonarCloud analysis on every push to `main` and on pull requests. ### Files Created + | File | Purpose | |------|---------| @@ -115,6 +127,7 @@ | `sonar-project.properties` | SonarCloud project configuration | ### Verification + | Check | Result | |-------|--------| @@ -135,24 +148,28 @@ | `sonar-project.properties` has appropriate exclusions | PASS | ### Note + Requires the `SONAR_TOKEN` secret to be configured in the GitHub repository settings before the workflow will succeed. --- ## Issue #75 β€” Release Milestone References + **Problem:** GitHub Releases did not reference the milestone (1.2, 1.3, etc.), making it hard to associate a release with its planned scope. **Solution:** Modified the `gh-release.yml` workflow to extract the version from `package.json`, derive the milestone, and include it in the release description with a clickable link. ### Files Modified + | File | Change | |------|--------| | `.github/workflows/gh-release.yml` | Added checkout step, milestone extraction step, and `body:` parameter with milestone link | ### Verification + | Check | Result | |-------|--------| @@ -168,6 +185,7 @@ Requires the `SONAR_TOKEN` secret to be configured in the GitHub repository sett --- ## Overall Result + **All 4 milestone 1.2 issues are fully resolved.** diff --git a/docs/pseudocode-explanation.md b/docs/pseudocode-explanation.md index 6083c613..35850259 100644 --- a/docs/pseudocode-explanation.md +++ b/docs/pseudocode-explanation.md @@ -5,6 +5,7 @@ This document describes each feature of the CloudVoyager migration tool in pseud --- ## Table of Contents + 1. [CLI Commands](#1-cli-commands) 2. [Transfer Pipeline (Single-Project)](#2-transfer-pipeline) @@ -29,6 +30,7 @@ This document describes each feature of the CloudVoyager migration tool in pseud --- ## 1. CLI Commands + ``` COMMAND: validate @@ -86,6 +88,7 @@ COMMAND: verify --- ## 2. Transfer Pipeline + Transfers a single SonarQube project to SonarCloud. @@ -193,6 +196,7 @@ FUNCTION transferBranchBatched(extractedData, branch, scConfig, ...): --- ## 3. Migration Pipeline + Migrates all projects across multiple SonarCloud organizations. @@ -309,6 +313,7 @@ FUNCTION migrateOneOrganization(org, projects, extractedData, ...): --- ## 4. Metadata Sync + Re-syncs issue and hotspot metadata for already-migrated projects. @@ -328,6 +333,7 @@ COMMAND sync-metadata: --- ## 5. Verification Pipeline + Compares SonarQube and SonarCloud data to verify migration completeness. @@ -390,6 +396,7 @@ FUNCTION verifyAll(sonarqubeConfig, sonarcloudOrgs, outputDir, onlyComponents, . --- ## 6. Data Extraction + Extracts all data from SonarQube for a single project. @@ -467,6 +474,7 @@ FUNCTION DataExtractor.extractBranch(branchName, mainData): --- ## 7. Protobuf Building & Encoding + Transforms extracted data into SonarCloud's scanner report protobuf format. @@ -614,6 +622,7 @@ CLASS ProtobufEncoder: --- ## 8. Report Upload to SonarCloud + Packages the encoded protobuf report and submits it to SonarCloud's Compute Engine. @@ -682,6 +691,7 @@ CLASS ReportUploader: --- ## 9. External Issues & Plugin Migration + Automatically detects issues from unsupported rule repositories (e.g., MuleSoft) and migrates them as external issues. @@ -772,6 +782,7 @@ FUNCTION buildRuleEnrichmentMap(scClient, scProfiles): --- ## 10. Issue & Hotspot Metadata Sync + Syncs issue/hotspot statuses, assignments, comments, and tags from SonarQube to SonarCloud after scanner report upload. @@ -857,6 +868,7 @@ FUNCTION syncHotspots(projectKey, sqHotspots, scClient, options): --- ## 11. Quality Gates Migration + ``` FUNCTION migrateQualityGates(extractedGates, scClient): @@ -895,6 +907,7 @@ FUNCTION assignQualityGatesToProjects(gateMapping, projectAssignments, scClient) --- ## 12. Quality Profiles Migration + ``` FUNCTION migrateQualityProfiles(extractedProfiles, scClient): @@ -957,6 +970,7 @@ FUNCTION buildInheritanceChains(profiles): --- ## 13. Permissions & Groups Migration + ``` FUNCTION migrateGroups(extractedGroups, scClient): @@ -997,6 +1011,7 @@ FUNCTION migrateProjectPermissions(project, scClient): --- ## 14. Organization Mapping & CSV Generation + Maps SonarQube projects to SonarCloud organizations and generates editable CSV files. @@ -1071,6 +1086,7 @@ FUNCTION applyCsvOverrides(csvData, extractedData, resourceMappings, orgAssignme --- ## 15. Version Router and Pipeline Selection + Detects SonarQube version at runtime and loads the correct version-specific pipeline. @@ -1106,6 +1122,7 @@ FUNCTION detectAndRoute(sonarqubeConfig): --- ## 16. State Management + Tracks migration progress for incremental transfers. @@ -1157,6 +1174,7 @@ CLASS StateTracker: --- ## 17. Configuration & Validation + ``` FUNCTION loadConfig(configPath): @@ -1226,6 +1244,7 @@ CONFIG SCHEMA: --- ## 18. Performance Tuning + ``` PERFORMANCE FEATURES: @@ -1267,6 +1286,7 @@ PERFORMANCE FEATURES: --- ## 19. Checkpoint Journal and Pause/Resume + ``` CHECKPOINT JOURNAL: diff --git a/docs/regression-testing.md b/docs/regression-testing.md index 604f584c..a4170cc4 100644 --- a/docs/regression-testing.md +++ b/docs/regression-testing.md @@ -6,10 +6,12 @@ A reusable protocol for verifying features and fixes against a live SonarQube --- ## The Prompt + > After implementing the feature/fix, test it end-to-end on the **Angular Framework** project (38K+ issues, 400+ hotspots). Follow this protocol exactly: > > ### Phase 1 β€” Adversarial Code Review (before running anything) + > > Re-read every changed file with fresh eyes. For each file, ask: > @@ -23,7 +25,8 @@ A reusable protocol for verifying features and fixes against a live SonarQube > Write down every concern. Then verify or disprove each one before proceeding. > > ### Phase 2 β€” Environment Assessment -> + + > Before running the test, check the live state: > > 1. **SonarQube**: `curl` the SQ API to confirm the project exists, count issues, sample 5 issues to check their fields (status, tags, comments, assignee, updateDate vs creationDate). @@ -32,6 +35,7 @@ A reusable protocol for verifying features and fixes against a live SonarQube > 4. **Clean slate**: Delete `migration-output/`, `.cloudvoyager-state.json*`, and the SC project. Every test run must start from zero. > > ### Phase 3 β€” Build and Execute + > > 1. `npm run package` β€” must succeed. If it fails, fix before proceeding. > 2. Create or verify the config file (`config.json` for `transfer`, `migrate-config.json` for `migrate`). @@ -43,7 +47,8 @@ A reusable protocol for verifying features and fixes against a live SonarQube > - The final summary stats line > > ### Phase 4 β€” Analyze Results -> + + > When the command finishes (exit code 0 or non-zero): > > 1. **Parse the summary stats line.** For every counter: @@ -60,6 +65,7 @@ A reusable protocol for verifying features and fixes against a live SonarQube > - Check the warn log for rate limiting or timeout messages > > ### Phase 5 β€” Investigate Anomalies + > > For every unexpected result, trace the full code path: > @@ -70,6 +76,7 @@ A reusable protocol for verifying features and fixes against a live SonarQube > 5. **Identify the root cause.** Don't guess β€” prove. "The comments field was empty because..." with evidence. > > ### Phase 6 β€” Fix and Re-test + > > If any issue was found: > @@ -83,6 +90,7 @@ A reusable protocol for verifying features and fixes against a live SonarQube > **Repeat Phase 6 until every stat matches expectations and spot-checks pass.** > > ### Phase 7 β€” Declare Clean Pass + > > A clean pass requires ALL of: > @@ -97,6 +105,7 @@ A reusable protocol for verifying features and fixes against a live SonarQube --- ## Reference: Angular Framework Test Environment + | Property | Value | |----------|-------| @@ -113,6 +122,7 @@ A reusable protocol for verifying features and fixes against a live SonarQube | Test migration script | `.debugging/test-migrate.sh` | ### Creating Test Data on SQ + To trigger code paths that require manual changes (pre-filter keeps only issues with human-authored changes): @@ -147,6 +157,7 @@ Adapt similarly for `do_transition` (confirm/reopen), `add_comment`, and `assign --- ## Example: Parallel Issue Sync Test (2026-04-28) + Applied this protocol to verify `worker_threads`-based parallel issue sync. diff --git a/docs/scenario-multi-org.md b/docs/scenario-multi-org.md index f6beab06..6f472f13 100644 --- a/docs/scenario-multi-org.md +++ b/docs/scenario-multi-org.md @@ -1,4 +1,5 @@ -# 🏒 Migrate Everything to Multiple SonarCloud Organizations +# Migrate Everything to Multiple SonarCloud Organizations + @@ -6,8 +7,8 @@ Use this when you want to migrate **all projects and configuration** from your S --- - -## πŸ“¦ What Gets Migrated +## What Gets Migrated + | Category | Details | |----------|---------| @@ -20,15 +21,15 @@ Use this when you want to migrate **all projects and configuration** from your S | **New Code Definitions** | Per-project and per-branch new code period settings | | **Server Info** | Server version, plugins, settings, and webhooks (saved as reference files) | - ### How projects are assigned to orgs + CloudVoyager uses DevOps platform bindings (GitHub, GitLab, etc.) to automatically decide which projects go to which organization. Projects without bindings are assigned to the first organization in the list. **You can review and adjust the project-to-org assignments in the generated `mappings/organizations.csv` file before running the actual migration.** --- - -## βœ… Prerequisites +## Prerequisites + 1. **Admin access** to your SonarQube server 2. **Admin access** to **each** target SonarCloud organization @@ -40,8 +41,8 @@ CloudVoyager uses DevOps platform bindings (GitHub, GitLab, etc.) to automatical --- - -## πŸ“₯ Step 1: Download +## Step 1: Download + Download the latest binary for your platform from the [Releases](https://github.com/sonar-solutions/cloudvoyager/releases) page: @@ -60,8 +61,8 @@ On macOS/Linux, make the binary executable: chmod +x cloudvoyager-* ``` - -## πŸ“ Step 2: Create a config file +## Step 2: Create a config file + Create a file called `migrate-config.json` with an entry for **each** target organization: @@ -98,8 +99,8 @@ Create a file called `migrate-config.json` with an entry for **each** target org See [`examples/migrate-config.example.json`](../examples/migrate-config.example.json) for a ready-to-use template with all optional fields (rate limiting, performance tuning, etc.). - ### Config fields + | Field | Required | Description | |-------|----------|-------------| @@ -130,13 +131,13 @@ See [`examples/migrate-config.example.json`](../examples/migrate-config.example. } ``` - -## πŸš€ Step 3: Run the migration (recommended 3-step approach) +## Step 3: Run the migration (recommended 3-step approach) + We recommend a 3-step migration: dry run, migrate without metadata, then sync metadata separately. This gives you the best combination of safety, speed, and reliability. - ### Step 3a: Dry run β€” verify everything + A dry run extracts all data and generates mapping CSVs so you can review **which projects go to which org**, without changing anything in SonarCloud: @@ -146,8 +147,8 @@ A dry run extracts all data and generates mapping CSVs so you can review **which Check `./migration-output/mappings/organizations.csv` to verify the project-to-org assignments look correct before proceeding. - ### Step 3b: Migrate without metadata + auto-tune + Run the actual migration with metadata sync disabled and auto-tuned performance. This transfers all projects, quality gates, profiles, groups, permissions, and report data β€” but skips the slower issue/hotspot status transitions: @@ -159,8 +160,8 @@ Skipping metadata during the main migration avoids SonarCloud rate limiting (503 > **Note:** By default, the tool does not wait for each project's analysis to complete on SonarCloud before moving on to the next project. This speeds up large migrations significantly. Add `--wait` if you need to block until each analysis finishes. - ### Step 3c: Sync metadata separately + Once all projects are migrated, sync issue and hotspot metadata as a standalone step: @@ -178,8 +179,8 @@ This step is safely retryable β€” if it hits rate limits, just run it again. You ./cloudvoyager sync-metadata -c migrate-config.json --skip-issue-metadata-sync --verbose ``` - ### Step 3d: Verify migration completeness (recommended) + After syncing metadata, run the verification command to confirm everything was migrated correctly across all organizations: @@ -201,7 +202,8 @@ This performs read-only checks comparing SonarQube and SonarCloud data and gener --- -## πŸ—οΈ Per-organization resource migration +## Per-organization resource migration + For each target organization, the tool runs these steps before migrating individual projects: @@ -217,17 +219,19 @@ For each target organization, the tool runs these steps before migrating individ After all organizations are fully migrated, **portfolios** are created at the enterprise level (requires `sonarcloud.enterprise.key`). ### Per-project steps (11 steps per project) + Each project within an organization goes through 11 individually-checkpointed steps β€” see the [single-org scenario](scenario-single-org.md#-what-happens-per-project-11-steps) for the full breakdown. ### Parallel project extraction + During the extraction phase, the tool fetches branches for all projects concurrently using bounded parallelism (`mapConcurrent`). The concurrency level is controlled by `performance.sourceExtraction.concurrency` in the config or by `--concurrency` on the CLI. This significantly speeds up the data-gathering phase for large SonarQube instances. --- - -## ⚑ Performance tuning (optional) +## Performance tuning (optional) + The `--auto-tune` flag (used in Step 3b) detects your hardware (CPU cores and RAM) and sets optimal values automatically. You can also manually set specific values: @@ -269,8 +273,8 @@ Keep `hotspotSync.concurrency` low (3–5) to avoid SonarCloud rate limits. See --- - -## πŸ“„ Generated Output Files +## Generated Output Files + | File | What's in it | |------|-------------| @@ -292,8 +296,8 @@ Keep `hotspotSync.concurrency` low (3–5) to avoid SonarCloud rate limits. See Server info (version, plugins, settings, webhooks) is saved to `{outputDir}/server-info/` as JSON files. Per-project state files are saved to `{outputDir}/state/` for incremental transfer tracking. - ### Verification Output Files + When you run `./cloudvoyager verify`, reports are written to `./verification-output/` (or the path specified by `--output-dir`): @@ -305,8 +309,8 @@ When you run `./cloudvoyager verify`, reports are written to `./verification-out --- - -## 🚩 All CLI Flags +## All CLI Flags + | Flag | What it does | |------|-------------| @@ -327,15 +331,15 @@ When you run `./cloudvoyager verify`, reports are written to `./verification-out --- - -## ⚠️ Limitations +## Limitations + - Historical metrics (the charts in each project's **Activity** tab in SonarQube) cannot be migrated. All actual issues and hotspots are migrated β€” only the historical trend data is lost. --- - -## πŸ“š Further Reading +## Further Reading + - [Configuration Reference](configuration.md) β€” all config options, environment variables, npm scripts - [Architecture](architecture.md) β€” project structure, data flow, report format @@ -344,14 +348,3 @@ When you run `./cloudvoyager verify`, reports are written to `./verification-out - [Key Capabilities](key-capabilities.md) β€” comprehensive overview of engineering and capabilities - [Dry-Run CSV Reference](dry-run-csv-reference.md) β€” CSV schema documentation for the dry-run workflow - [Changelog](CHANGELOG.md) β€” release history and notable changes - - diff --git a/docs/scenario-single-org.md b/docs/scenario-single-org.md index 41a38b52..d7e7bef3 100644 --- a/docs/scenario-single-org.md +++ b/docs/scenario-single-org.md @@ -1,4 +1,5 @@ -# 🏒 Migrate Everything to One SonarCloud Organization +# Migrate Everything to One SonarCloud Organization + @@ -6,8 +7,8 @@ Use this when you want to migrate **all projects and configuration** from your S --- - -## πŸ“¦ What Gets Migrated +## What Gets Migrated + | Category | Details | |----------|---------| @@ -22,8 +23,8 @@ Use this when you want to migrate **all projects and configuration** from your S --- - -## βœ… Prerequisites +## Prerequisites + 1. **Admin access** to your SonarQube server 2. **Admin access** to your SonarCloud organization @@ -35,8 +36,8 @@ Use this when you want to migrate **all projects and configuration** from your S --- - -## πŸ“₯ Step 1: Download +## Step 1: Download + Download the latest binary for your platform from the [Releases](https://github.com/sonar-solutions/cloudvoyager/releases) page: @@ -55,8 +56,8 @@ On macOS/Linux, make the binary executable: chmod +x cloudvoyager-* ``` - -## πŸ“ Step 2: Create a config file +## Step 2: Create a config file + Create a file called `migrate-config.json`: @@ -85,8 +86,8 @@ See [`examples/migrate-config.example.json`](../examples/migrate-config.example. > **Project keys and names:** Each project's display name is automatically carried over from SonarQube. By default, the tool uses the **original SonarQube project key** on SonarCloud. If the key is already taken by another SonarCloud organization, the tool falls back to a prefixed key (`{org-key}_{sonarqube-project-key}`) and logs a warning. Any key conflicts are listed in the migration report. - ### Config fields + | Field | Required | Description | |-------|----------|-------------| @@ -99,13 +100,13 @@ See [`examples/migrate-config.example.json`](../examples/migrate-config.example. > **Tip:** You can set tokens via environment variables (`SONARQUBE_TOKEN` and `SONARCLOUD_TOKEN`) instead of putting them in the config file. - -## πŸš€ Step 3: Run the migration (recommended 3-step approach) +## Step 3: Run the migration (recommended 3-step approach) + We recommend a 3-step migration: dry run, migrate without metadata, then sync metadata separately. This gives you the best combination of safety, speed, and reliability. - ### Step 3a: Dry run β€” verify everything + A dry run extracts all data and generates mapping CSVs so you can review what will be migrated, without changing anything in SonarCloud: @@ -115,8 +116,8 @@ A dry run extracts all data and generates mapping CSVs so you can review what wi Check the generated files in `./migration-output/` to make sure everything looks right. - ### Step 3b: Migrate without metadata + auto-tune + Run the actual migration with metadata sync disabled and auto-tuned performance. This transfers all projects, quality gates, profiles, groups, permissions, and report data β€” but skips the slower issue/hotspot status transitions: @@ -128,8 +129,8 @@ Skipping metadata during the main migration avoids SonarCloud rate limiting (503 > **Note:** By default, the tool does not wait for each project's analysis to complete on SonarCloud before moving on to the next project. This speeds up large migrations significantly. Add `--wait` if you need to block until each analysis finishes. - ### Step 3c: Sync metadata separately + Once all projects are migrated, sync issue and hotspot metadata as a standalone step: @@ -147,8 +148,8 @@ This step is safely retryable β€” if it hits rate limits, just run it again. You ./cloudvoyager sync-metadata -c migrate-config.json --skip-issue-metadata-sync --verbose ``` - ### Step 3d: Verify migration completeness (recommended) + After syncing metadata, run the verification command to confirm everything was migrated correctly: @@ -170,7 +171,8 @@ This performs read-only checks comparing SonarQube and SonarCloud data and gener --- -## πŸ”§ What happens per project (11 steps) +## What happens per project (11 steps) + For each project in the organization, the migration pipeline runs these steps in order. Each step is individually checkpointed β€” on resume, completed steps are skipped. @@ -192,7 +194,8 @@ Use `--only` to run a subset of these steps. For example, `--only scan-data,qual --- -## πŸ“‹ Dry-run CSV files (9 types) +## Dry-run CSV files (9 types) + The `--dry-run` flag generates 9 CSV files in `{outputDir}/mappings/`. Each CSV has an **Include** column you can edit to exclude specific resources before the real migration. @@ -212,8 +215,8 @@ See the [Dry-Run CSV Reference](dry-run-csv-reference.md) for full column schema --- - -## ⚑ Performance tuning (optional) +## Performance tuning (optional) + The `--auto-tune` flag (used in Step 3b) detects your hardware (CPU cores and RAM) and sets optimal values automatically. You can also manually set specific values: @@ -255,8 +258,8 @@ Keep `hotspotSync.concurrency` low (3–5) to avoid SonarCloud rate limits. See --- - -## πŸ“„ Generated Output Files +## Generated Output Files + | File | What's in it | |------|-------------| @@ -278,8 +281,8 @@ Keep `hotspotSync.concurrency` low (3–5) to avoid SonarCloud rate limits. See Server info (version, plugins, settings, webhooks) is saved to `{outputDir}/server-info/` as JSON files. Per-project state files are saved to `{outputDir}/state/` for incremental transfer tracking. - ### Verification Output Files + When you run `./cloudvoyager verify`, reports are written to `./verification-output/` (or the path specified by `--output-dir`): @@ -291,8 +294,8 @@ When you run `./cloudvoyager verify`, reports are written to `./verification-out --- - -## 🚩 All CLI Flags +## All CLI Flags + | Flag | What it does | |------|-------------| @@ -313,15 +316,15 @@ When you run `./cloudvoyager verify`, reports are written to `./verification-out --- - -## ⚠️ Limitations +## Limitations + - Historical metrics (the charts in each project's **Activity** tab in SonarQube) cannot be migrated. All actual issues and hotspots are migrated β€” only the historical trend data is lost. --- - -## πŸ“š Further Reading +## Further Reading + - [Configuration Reference](configuration.md) β€” all config options, environment variables, npm scripts - [Architecture](architecture.md) β€” project structure, data flow, report format @@ -330,14 +333,3 @@ When you run `./cloudvoyager verify`, reports are written to `./verification-out - [Key Capabilities](key-capabilities.md) β€” comprehensive overview of engineering and capabilities - [Dry-Run CSV Reference](dry-run-csv-reference.md) β€” CSV schema documentation for the dry-run workflow - [Changelog](CHANGELOG.md) β€” release history and notable changes - - diff --git a/docs/scenario-single-project.md b/docs/scenario-single-project.md index b6243ec7..fda78e49 100644 --- a/docs/scenario-single-project.md +++ b/docs/scenario-single-project.md @@ -1,4 +1,5 @@ -# πŸ“¦ Migrate a Single Project +# Migrate a Single Project + @@ -8,8 +9,8 @@ This does **not** migrate org-level settings like quality gates, quality profile --- - -## πŸ“¦ What Gets Migrated +## What Gets Migrated + | Category | Details | |----------|---------| @@ -27,8 +28,8 @@ This does **not** migrate org-level settings like quality gates, quality profile --- - -## βœ… Prerequisites +## Prerequisites + 1. **Admin access** to your SonarQube server 2. **Admin access** to your SonarCloud organization @@ -40,8 +41,8 @@ This does **not** migrate org-level settings like quality gates, quality profile --- - -## πŸ“₯ Step 1: Download +## Step 1: Download + Download the latest binary for your platform from the [Releases](https://github.com/sonar-solutions/cloudvoyager/releases) page: @@ -60,8 +61,8 @@ On macOS/Linux, make the binary executable: chmod +x cloudvoyager-* ``` - -## πŸ“ Step 2: Create a config file +## Step 2: Create a config file + Create a file called `config.json`: @@ -85,8 +86,8 @@ Create a file called `config.json`: See [`examples/config.example.json`](../examples/config.example.json) for a ready-to-use template with all optional fields (rate limiting, performance tuning, etc.). - ### Config fields + | Field | Required | Description | |-------|----------|-------------| @@ -100,8 +101,8 @@ See [`examples/config.example.json`](../examples/config.example.json) for a read > **Tip:** You can set tokens via environment variables (`SONARQUBE_TOKEN` and `SONARCLOUD_TOKEN`) instead of putting them in the config file. - ### Optional: Transfer settings + Add a `transfer` section to control incremental mode, batch size, and checkpoint behavior: @@ -134,8 +135,8 @@ Add a `transfer` section to control incremental mode, batch size, and checkpoint | `checkpoint.cacheMaxAgeDays` | `7` | Discard extraction caches older than this many days | | `checkpoint.strictResume` | `false` | When `true`, abort if the session fingerprint (SQ version, URL, project key) has changed since the checkpoint was created | - -## πŸ§ͺ Step 3: Test your connections +## Step 3: Test your connections + ```bash ./cloudvoyager test -c config.json @@ -143,8 +144,8 @@ Add a `transfer` section to control incremental mode, batch size, and checkpoint You should see a success message for both SonarQube and SonarCloud. If not, double-check your URLs and tokens. - -## πŸš€ Step 4: Run the transfer +## Step 4: Run the transfer + ```bash ./cloudvoyager transfer -c config.json --verbose @@ -156,8 +157,8 @@ That's it! The tool uploads the report and returns immediately β€” it does not w --- - -## πŸ”§ Other useful commands +## Other useful commands + ```bash # Validate your config file @@ -172,8 +173,8 @@ That's it! The tool uploads the report and returns immediately β€” it does not w --- - -## ⚑ Speed up the transfer (optional) +## Speed up the transfer (optional) + For large projects with many source files or issues, use `--auto-tune` to automatically detect your hardware and set optimal performance values: @@ -191,7 +192,8 @@ See the [Configuration Reference](configuration.md#performance-settings) for all --- -## πŸ”„ Pause and Resume +## Pause and Resume + Transfers are automatically checkpointed. If a transfer is interrupted (CTRL+C, crash, network failure), simply re-run the same command to resume from where it left off: @@ -213,17 +215,19 @@ To discard progress and start fresh, use `--force-restart`: ``` ### Upload deduplication + On resume after a crash, the tool checks whether a Compute Engine (CE) task already exists for the current session before uploading. This prevents duplicate scanner reports from being submitted to SonarCloud. The check uses the `scm_revision_id` (git commit hash) included in each report. ### Lock file handling + An advisory lock file prevents multiple CloudVoyager instances from running against the same project simultaneously. The lock includes the process ID, hostname, and start time. Stale locks from crashed processes are auto-released after 6 hours. If a lock is held by another host or a non-stale process, the tool exits with an error. Use `--force-unlock` to manually release a lock you know is stale. --- - -## 🚩 All CLI Flags +## All CLI Flags + | Flag | What it does | |------|-------------| @@ -240,15 +244,15 @@ An advisory lock file prevents multiple CloudVoyager instances from running agai --- - -## ⚠️ Limitations +## Limitations + - Historical metrics (the charts in each project's **Activity** tab in SonarQube) cannot be migrated. All actual issues and hotspots are migrated β€” only the historical trend data is lost. --- - -## πŸ“š Further Reading +## Further Reading + - [Configuration Reference](configuration.md) β€” all config options, environment variables, npm scripts - [Architecture](architecture.md) β€” project structure, data flow, report format @@ -256,15 +260,3 @@ An advisory lock file prevents multiple CloudVoyager instances from running agai - [Troubleshooting](troubleshooting.md) β€” common errors and how to fix them - [Key Capabilities](key-capabilities.md) β€” comprehensive overview of engineering and capabilities - [Changelog](CHANGELOG.md) β€” release history and notable changes - - diff --git a/docs/technical-details.md b/docs/technical-details.md index fafcf90b..b78d4ef5 100644 --- a/docs/technical-details.md +++ b/docs/technical-details.md @@ -2,7 +2,7 @@ - + ## πŸ—ΊοΈ Main Flow Sequence Diagram The diagram below shows the high-level flows for the four main commands. The **βž•** markers reference the drill-down prompts in the table that follows β€” paste any prompt into a new chat (with this codebase in context) to generate a detailed diagram for that subsystem. @@ -141,12 +141,13 @@ sequenceDiagram | βž•F | Quality profiles | `Generate a mermaid sequence diagram for CloudVoyager quality profile migration including backup XML, rename of built-in profiles, and diff report` | | βž•G | Batch distribution | `Generate a mermaid sequence diagram showing how CloudVoyager batch-distributor splits large issue sets into ≀5K batches with backdated analysis dates and sequential CE uploads` | - + ## πŸ“‘ Protobuf Encoding The scanner report uses `scanner-report.proto` and `constants.proto` (in each pipeline's `protobuf/schema/` directory). Key protobuf messages: `Metadata`, `Component`, `Issue`, `ExternalIssue`, `AdHocRule`, `Measure`, `ActiveRule`, `Duplication`, `Changesets`, `Symbols`, `SyntaxHighlighting`, `LineCoverage`. ### Encoding Styles + The scanner report ZIP uses two encoding styles: - **Single message** (no length delimiter): `metadata.pb`, `component-{ref}.pb`, `changesets-{ref}.pb` @@ -155,6 +156,7 @@ The scanner report ZIP uses two encoding styles: - **Empty sentinel**: `context-props.pb` (always empty, matches real scanner behavior) ### Report ZIP Structure + ``` metadata.pb # Single Metadata message (NOT length-delimited) @@ -170,7 +172,7 @@ duplications-{ref}.pb # Length-delimited Duplication messages per co context-props.pb # Empty (matches real scanner) ``` - + ## πŸ”„ CE Submission Retry Mechanism Report submission to SonarCloud's Compute Engine (`/api/ce/submit`) uses a robust retry strategy (implemented in `ce-submitter.js` within each pipeline): @@ -184,6 +186,7 @@ Report submission to SonarCloud's Compute Engine (`/api/ce/submit`) uses a robus The form data is buffered before sending (not streamed) to avoid runtime-specific issues with Bun's HTTP client. Branch characteristics (`branch=`, `branchType=LONG`) are included for non-main branches. ### Key Enum Values + | Enum | Values | |------|--------| @@ -196,6 +199,7 @@ The form data is buffered before sending (not streamed) to avoid runtime-specifi **CRITICAL**: `cleanCodeAttribute` in `ExternalIssue` and `AdHocRule` must be encoded as a protobuf enum (varint), NOT a string. Despite the `.proto` file showing `optional string`, the real scanner uses enum encoding. SonarCloud CE silently ignores external issues if `cleanCodeAttribute` is string-encoded. ### Field Name Convention + protobufjs automatically converts snake_case field names to camelCase in JavaScript: - `analysis_date` becomes `analysisDate` @@ -204,7 +208,7 @@ protobufjs automatically converts snake_case field names to camelCase in JavaScr All field names in the codebase use camelCase to match this convention. - + ## πŸ“ Measure Type Mapping Measures use typed value fields based on metric type: @@ -212,26 +216,26 @@ Measures use typed value fields based on metric type: - **String metrics** (`stringValue`): `executable_lines_data`, `ncloc_data`, `alert_status` - **Float/percentage metrics** (`doubleValue`): `coverage`, `line_coverage`, `branch_coverage`, `duplicated_lines_density`, ratings - + ## πŸ“‹ Active Rules - Active rules are filtered by languages actually used in the project, resulting in ~84% reduction in payload size - Rule keys are stripped of the repository prefix (e.g., `S7788` not `jsarchitecture:S7788`) - Quality profile keys are mapped to SonarCloud profile keys (not SonarQube keys) - + ## 🧱 Component Structure Components use a flat structure - all files are direct children of the project component (no directory components). Line counts are derived from actual source file content rather than SonarQube measures API values. - + ## πŸ”– SCM Revision Tracking The tool includes `scm_revision_id` (git commit hash) in metadata. SonarCloud uses this to detect and reject duplicate reports, enabling proper analysis history tracking. **Batch uploads**: When issue batching is active (>5,000 issues), each batch receives a unique `scmRevisionId` generated via `randomBytes(20).toString('hex')` instead of the original git commit hash. This prevents the SonarCloud CE from deduplicating successive batch uploads as identical reports. See [Issue Batching for Upload](#-issue-batching-for-upload-5k-per-date-bucket) for details. - + ## 🌿 Branch Sync By default, every branch discovered in SonarQube is transferred to SonarCloud (main branch first, then non-main branches). Each branch produces its own scanner report with branch-specific issues, measures, sources, and SCM data. @@ -242,7 +246,7 @@ By default, every branch discovered in SonarQube is transferred to SonarCloud (m **Configuration:** Set `transfer.syncAllBranches` to `false` to only sync the main branch. Use `transfer.excludeBranches` to skip specific branch names (e.g., `["feature/old", "release/v1"]`). - + ## πŸ“„ API Pagination SonarQube client handles pagination automatically via `getPaginated` method with a default page size of 500 items. All paginated results are concatenated into single arrays. @@ -259,7 +263,7 @@ The extractors handle these lower limits automatically. **metricKeys batching**: SQ 9.9 through 10.8 limits `metricKeys` to 15 per request (must batch). SQ 2025.1+ has no batching limit. - + ## πŸ“¦ Accurate Issue Creation Date Backdating SonarCloud's CE assigns creation dates to NEW issues from SCM blame data. `backdateChangesets()` rewrites each file's changeset protobuf so that the CE assigns each issue its original SonarQube creation date. @@ -267,12 +271,14 @@ SonarCloud's CE assigns creation dates to NEW issues from SCM blame data. `backd **How it works:** The function reads each issue's `creationDate` field (preserved from SonarQube extraction) and maps it to the issue's `textRange` lines in the file's changeset data. The CE takes MAX(date) across an issue's line range, so using "oldest wins" for overlapping lines ensures accurate dates. A safety split handles days with >5K issues. ### Algorithm (3 Phases) + 1. **Phase 0 β€” Safety split:** Group issues by calendar day. For any day exceeding 5,000 issues, sort by component and sub-group into ≀5K batches with 1-day-spaced synthetic dates (no file splitting). 2. **Phase 1 β€” Per-line date map:** For each issue, map its `textRange.startLine..endLine` to its effective creation date. Oldest date wins when lines overlap. File-level issues (line=0/null) are skipped. 3. **Phase 2 β€” Rebuild changesets:** For each file with issues, create one changeset entry per unique date. `changesetIndexByLine[i]` points to the date for line `i+1`, or to the oldest date (index 0) for non-issue lines. Files with no issues keep their original stub. ### Design Decisions + | Decision | Rationale | |----------|-----------| @@ -284,6 +290,7 @@ SonarCloud's CE assigns creation dates to NEW issues from SCM blame data. `backd | All projects backdated | No early return for small projects β€” every project gets accurate dates | ### Helper Files (`src/shared/utils/batch-distributor/helpers/`) + | File | Role | |------|------| @@ -293,10 +300,11 @@ SonarCloud's CE assigns creation dates to NEW issues from SCM blame data. `backd Legacy files (unchanged, no longer used by `backdateChangesets`): `compute-batch-plan.js`, `compute-batch-date.js`, `create-batch-extracted-data.js`. ### Pipeline Integration + All 6 pipeline `transfer-branch` entry points call `backdateChangesets(extractedData)` before the protobuf build step. The function mutates `extractedData.changesets` in place β€” no signature change. - + ## πŸ” Search Slicing for 10K+ Issues SonarQube's `/api/issues/search` endpoint returns a maximum of 10,000 results regardless of pagination. Projects with more than 10,000 issues would silently lose data during extraction. @@ -311,6 +319,7 @@ SonarQube's `/api/issues/search` endpoint returns a maximum of 10,000 results re 6. **Merge & deduplicate** β€” `deduplicate-results.js` merges results from all windows and removes duplicates by `item.key || item.id` to handle boundary overlaps where an issue's creation timestamp falls on a window edge. **Helper files** (`src/shared/utils/search-slicer/helpers/`): + | File | Role | |------|------| @@ -329,7 +338,7 @@ The slicing is transparent to callers β€” `issues-hotspots.js` in each pipeline - `transfer` β€” calls `getIssuesWithComments()` via `fetch-and-sync-issues.js`, which invokes `fetchWithSlicing` under the hood. - `migrate` β€” calls `getIssuesWithComments()` via `sync-issue-metadata.js` β†’ `sync-project-issues.js`, using the same `fetchWithSlicing` function. - + ## πŸ”„ Fallback Rule Repositories External-issue detection depends on knowing which rule repositories exist in SonarCloud. If the `/api/rules/repositories` call fails, the tool falls back to a built-in set of 44 known SonarCloud repositories (`src/shared/utils/fallback-repos/index.js`). @@ -346,7 +355,7 @@ The fallback set includes the `githubactions` IaC analyzer (previously omitted b - Handles rules without a colon separator (treated as non-external). - Handles empty repository prefixes gracefully. - + ## 🚦 Rate Limit Handling The SonarCloud API client supports a configurable two-layer strategy for rate limiting. Customize it via the `rateLimit` section in your config file. @@ -355,7 +364,7 @@ The SonarCloud API client supports a configurable two-layer strategy for rate li 2. **Write request throttling** (`minRequestInterval`) β€” POST requests are spaced at least `minRequestInterval` ms apart via a request interceptor. This proactively reduces the chance of triggering SonarCloud's rate limits during high-volume operations like issue sync and hotspot sync. Default: `0` (no throttling). - + ## πŸ”Œ External Issues (Plugin Migration) Issues from SonarQube plugins that are not available in SonarCloud (e.g., MuleSoft, ABAP) are automatically migrated as **external issues** using the `ExternalIssue` and `AdHocRule` protobuf messages. @@ -377,7 +386,7 @@ Issues from SonarQube plugins that are not available in SonarCloud (e.g., MuleSo - `impacts` array β€” `Impact` messages with `softwareQuality` and `severity` fields - `defaultImpacts` β€” on `AdHocRule` messages - + ## πŸ”„ Issue Sync The `migrate` command syncs issue metadata after the scanner report is uploaded. The sync pipeline runs in several stages: @@ -391,6 +400,7 @@ The `migrate` command syncs issue metadata after the scanner report is uploaded. 7. **Tags** β€” Sets tags. ### Pre-filter: `hasManualChanges` (`src/shared/utils/issue-sync/`) + Before touching SonarCloud, the syncer applies a pre-filter that only keeps issues with human-authored changes: @@ -408,6 +418,7 @@ The pre-filter is implemented across three shared utilities: - `apply-pre-filter.js` β€” orchestrates the above two and sets `stats.filtered` with the skipped count ### Parallel Issue Sync (β‰₯500 matched pairs) + When the matched pair count reaches 500+, the syncer switches from single-process `mapConcurrent` to `worker_threads`-based parallelism (`src/shared/utils/concurrency/helpers/parallel-issue-sync.js`): @@ -420,6 +431,7 @@ When the matched pair count reaches 500+, the syncer switches from single-proces Total concurrent API calls: 20 workers Γ— 5 internal = **100** (vs 20 with single-process). Each worker includes exponential backoff retry for 429/transient errors. ### Wait for SC Indexing: `waitForScIndexing` + `src/shared/utils/issue-sync/wait-for-sc-indexing.js` wraps any SC fetch call with retry logic: - **Trigger**: SC returns 0 results but there are SQ items that need syncing @@ -434,7 +446,7 @@ Total concurrent API calls: 20 workers Γ— 5 internal = **100** (vs 20 with singl The `verify` command validates this by fetching changelogs from both sides (`/api/issues/changelog`) and comparing the transition sequences. - + ## πŸ”₯ Hotspot Sync Similar to issue sync, hotspot metadata is matched and synced: @@ -446,7 +458,7 @@ Hotspots are converted to `Issue` format for the scanner report (with `type=SECU **API gotcha**: The hotspot details API returns `comment` (singular, containing a list), not `comments`. - + ## πŸ”„ Issue Status Transition Mapping Each pipeline includes an `issue-status-mapper.js` that maps SonarQube issue changelog entries to SonarCloud transitions: @@ -466,7 +478,7 @@ The mapper handles both SQ < 10.4 (where `FALSE-POSITIVE` and `WONTFIX` appear a - **Changelog replay**: Extracts ordered transitions from the full issue changelog and replays them in sequence - **Fallback**: Maps the current SQ status/resolution to a single transition when no changelog is available - + ## πŸ”‘ Project Key Resolution SonarCloud requires globally unique project keys across all organizations. When migrating projects, the tool uses the following strategy: @@ -478,12 +490,12 @@ SonarCloud requires globally unique project keys across all organizations. When Key conflicts are reported in the migration summary and in the `reports/migration-report.txt` / `reports/migration-report.json` output files. - + ## πŸ—ΊοΈ Organization Mapping The `migrate` command maps projects to target SonarCloud organizations based on their DevOps platform bindings. Projects with the same ALM binding are grouped together. Mapping CSVs are generated for review before execution (via `--dry-run`). - + ## πŸ” Dry-Run & Editable CSV Workflow The `--dry-run` flag generates 8 exhaustive CSV files covering projects, organizations, groups, quality profiles, quality gates, portfolios, permission templates, and global permissions. Each CSV includes an `Include` column (defaulting to `yes`) that users can edit to filter what gets migrated. @@ -497,7 +509,7 @@ Quality gate CSVs use a flat one-row-per-gate pattern β€” users can include or e See [dry-run-csv-reference.md](dry-run-csv-reference.md) for full CSV schema documentation. - + ## πŸ“‹ Quality Profile Migration Quality profiles are migrated using SonarQube's backup/restore XML format, which preserves all rule configurations, severity overrides, and parameter values. Profile permissions (user and group access) are migrated separately via the permissions API. @@ -515,7 +527,7 @@ After profile migration, a **quality profile diff report** (`quality-profiles/qu - **Missing rules** β€” rules active in SonarQube but not available in SonarCloud (may cause fewer issues) - **Added rules** β€” rules available in SonarCloud but not in SonarQube (may create new issues) - + ## 🚧 Quality Gate Migration Quality gates are created with their full condition definitions (metric, operator, threshold). The SonarQube API uses gate `name` (not `id`) for all operations. Built-in gates are skipped since they already exist in SonarCloud. @@ -525,7 +537,7 @@ Quality gates are created with their full condition definitions (metric, operato - `/api/qualitygates/show` requires `name` param, not `id` - Built-in gates: permission APIs return 400 (expected, handle gracefully) - + ## ⚑ Concurrency Model CloudVoyager uses a custom concurrency layer (`src/shared/utils/concurrency.js`) with zero external dependencies. Key primitives: @@ -544,17 +556,18 @@ Performance config is resolved at startup by `resolvePerformanceConfig()`, which - `hotspotSync` = min(max(CPU cores / 2, 3), 5) - `projectMigration` = max(1, CPU cores / 3) - + ## πŸ’Ύ Memory Management When `maxMemoryMB` is set (via config or `--max-memory` flag), the tool automatically re-spawns itself with `NODE_OPTIONS="--max-old-space-size="` if the current heap limit is insufficient. This is transparent to the user β€” output streams seamlessly through the respawned process. - + ## πŸ”„ Checkpoint Journal and Pause/Resume CloudVoyager uses a write-ahead checkpoint journal to track progress at every major phase. If a transfer or migration is interrupted (CTRL+C, crash, network failure), re-running the same command resumes from the last completed checkpoint. ### Journal Structure + Each transfer creates a checkpoint journal file alongside the state file: @@ -563,6 +576,7 @@ Each transfer creates a checkpoint journal file alongside the state file: - **Session fingerprint**: SonarQube version, URL, and project key are recorded to detect environment changes between runs ### State Management Components + - **StateTracker**: Issue-level incremental sync tracking - **StateStorage**: Atomic write-to-temp + fsync + rename, backup rotation, 10MB disk space check @@ -572,6 +586,7 @@ Each transfer creates a checkpoint journal file alongside the state file: - **MigrationJournal**: Org/project/step-level tracking for multi-org resume, using a `completedSteps` array (supports parallel step completion; backward-compatible with old `lastCompletedStep` format on read) ### Concurrent Safety + Both `CheckpointJournal` and `MigrationJournal` use an in-process async mutex (`_withLock()`) to serialize all mutating operations. This prevents read-modify-write races when multiple branches or projects are transferred in parallel. All methods that modify journal state (start/complete/fail phase, record upload, mark interrupted, etc.) acquire the lock before reading and release it after writing. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 6a780552..7aa031c9 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -2,7 +2,7 @@ - + ## πŸ› Debugging a Migration Run After every `migrate` run (whether it succeeds, partially succeeds, or crashes), CloudVoyager writes report files to the `reports/` subdirectory of your output directory: @@ -12,7 +12,7 @@ After every `migrate` run (whether it succeeds, partially succeeds, or crashes), | `reports/migration-report.txt` | Human-readable report β€” open this first | | `reports/migration-report.json` | Machine-readable structured data for scripting | - + ### Where to start 1. **Open `reports/migration-report.txt`** β€” it's structured top-down so you can quickly find problems: @@ -27,7 +27,7 @@ After every `migrate` run (whether it succeeds, partially succeeds, or crashes), 3. **Check `reports/migration-report.json`** if you need to script post-migration analysis (e.g., count how many projects failed at "Sync hotspots" vs "Upload scanner report"). - + ### Example report output ``` @@ -66,7 +66,7 @@ From this you can see: - `my-legacy-project` failed at the scanner report upload (likely a protobuf/format issue) β€” but settings, tags, links, etc. still succeeded - `big-project` succeeded except for hotspot sync (rate limited) β€” you can re-run with only hotspot sync later - + ### Project statuses | Status | Meaning | @@ -75,7 +75,7 @@ From this you can see: | **partial** | Some steps succeeded, some failed β€” check the report for which ones | | **failed** | All non-skipped steps failed | - + ### Per-project migration steps Each project goes through these steps (in order). If the scanner report upload fails, issue/hotspot sync are automatically skipped (they depend on uploaded data), but remaining steps still run: @@ -94,7 +94,7 @@ Each project goes through these steps (in order). If the scanner report upload f | Assign quality profiles | Assigns migrated built-in quality profiles per language | No | | Project permissions | Sets group-level project permissions | No | - + ### Using log files for deeper investigation For detailed debugging beyond the report, save the full log to a file: @@ -122,7 +122,7 @@ grep "Rate limited" migration.log grep "SKIP\|skipping" migration.log ``` - + ### Re-running after failures The migration can be re-run safely. Projects that already exist in SonarCloud will be updated (not duplicated). To fix specific failures: @@ -134,6 +134,7 @@ The migration can be re-run safely. Projects that already exist in SonarCloud wi --- ## πŸ”„ Checkpoint and Resume Issues + ### Stale Lock File ("Another instance is running") @@ -146,6 +147,7 @@ If a previous run crashed without releasing the lock file, you may see an error The tool automatically detects stale locks from dead processes on the same machine. If the lock was created by a different machine (e.g., NFS-shared state file), manual intervention with `--force-unlock` is required. ### Corrupt Checkpoint Journal + If the checkpoint journal becomes corrupt (e.g., due to a system crash during a write), the tool falls back to the `.journal.backup` file. If both are corrupt: @@ -155,6 +157,7 @@ If the checkpoint journal becomes corrupt (e.g., due to a system crash during a ``` ### SonarQube Version Mismatch on Resume + If you upgrade SonarQube between pause and resume, the tool warns about a version mismatch in the session fingerprint. By default, this is a warning only β€” the transfer continues. To enforce strict version matching: @@ -171,6 +174,7 @@ If you upgrade SonarQube between pause and resume, the tool warns about a versio With `strictResume: true`, a version mismatch will fail the transfer and require `--force-restart` to proceed. ### Source Code Changed Between Pause and Resume + If source code in SonarQube changes between pause and resume (e.g., new analysis uploaded), already-cached extraction phases will use stale data. Use `--force-fresh-extract` to re-extract all data while keeping the checkpoint journal: @@ -179,6 +183,7 @@ If source code in SonarQube changes between pause and resume (e.g., new analysis ``` ### Clearing All Checkpoint State + The `reset` command now clears checkpoint journals, lock files, and extraction caches in addition to the state file: @@ -194,7 +199,7 @@ This removes: --- - + ## πŸ—οΈ Build Failures ### `npm run package` β€” "Multiple occurrences of sentinel found in the binary" @@ -217,27 +222,27 @@ This only affects building the binary. Tests, linting, and `npm run build` work --- - + ## πŸ” Authentication Errors - Verify your tokens have the correct permissions - Check that tokens haven't expired - Ensure the project key exists in SonarQube - Verify the organization key is correct in SonarCloud - + ## ⚠️ Generic "Issue whilst processing" Error This vague SonarCloud error can be caused by: - **Branch name mismatch** - SonarQube and SonarCloud have different main branch names. The tool handles this automatically via `getMainBranchName()`, but verify your SonarCloud project's branch configuration - **Line count mismatch** - Source file line counts don't match component metadata. The tool uses actual source content line counts to avoid this - + ## ❌ Report Rejected by SonarCloud - **Empty ScmInfo** - Ensure `changesetIndexByLine` is populated for ADDED files (array of zeros, one per line) - **Issue gap field** - The `gap` field should not be included in issues (it's scanner-computed, not from SonarQube) - **Duplicate report** - SonarCloud rejects reports with the same `scm_revision_id`. Use a different commit or update the source project - + ## πŸ”‘ Project Key Conflicts SonarCloud requires globally unique project keys across all organizations. By default, CloudVoyager uses the **original SonarQube project key** when creating projects on SonarCloud. If the key is already taken by another SonarCloud organization, the tool automatically falls back to a prefixed key (`{org}_{key}`) and logs a warning. @@ -249,13 +254,13 @@ Key conflicts are reported in three places: If you see key conflicts, the affected projects were still migrated successfully β€” they just use a different key than the original SonarQube key. You can rename them later via the SonarCloud API (`/api/projects/update_key`) if the conflicting key becomes available. - + ## 🌐 Connection Timeouts - Check network connectivity to both servers - Verify firewall rules allow access - Use `--verbose` flag for detailed connection logs - + ## 🚦 Rate Limiting (503 / 429 Errors) SonarCloud may return 503 or 429 errors when too many API requests are made in a short period, especially during issue and hotspot sync on large projects. @@ -281,19 +286,19 @@ If you still encounter rate limit errors after all retries are exhausted, consid - Running the migration during off-peak hours - Using `--skip-hotspot-metadata-sync` to skip the most rate-limit-prone operation - + ## 🏷️ Project Names Showing as Project Keys If projects in SonarCloud show the project key as the display name instead of the original human-readable name from SonarQube, the project was likely created by an older version of CloudVoyager. The current version automatically carries over the original project name from SonarQube when creating projects in SonarCloud. To fix already-migrated projects, you can rename them manually in SonarCloud via **Project Settings > General Settings > Project Name**, or delete and re-migrate the project. - + ## 🚧 Quality Gate / Profile Permission Errors (400) When migrating quality gates or profiles, permission APIs may return 400 errors for built-in gates/profiles. This is expected β€” built-in resources don't support custom permissions. The migrators handle this gracefully and skip these entries. - + ## πŸ“Š Issue Counts Differ Between SonarQube and SonarCloud If you see different issue counts (Security, Reliability, Maintainability) after migration, this is usually caused by **different active rules** between the SonarQube and SonarCloud quality profiles. @@ -302,7 +307,7 @@ The migrator now restores built-in profiles as custom profiles (e.g., "Sonar way If you'd prefer to skip quality profile migration entirely and use each language's default SonarCloud profile instead, use `--skip-quality-profile-sync`. - + ## πŸ“„ SonarQube API Pagination Limits Some SonarQube APIs enforce a maximum page size of 100 (not 500): @@ -313,12 +318,12 @@ Some SonarQube APIs enforce a maximum page size of 100 (not 500): The extractors handle this automatically, but if you see pagination-related errors, this is likely the cause. - + ## πŸ”€ Quality Gates Use Names, Not IDs The SonarQube quality gates API uses `name` for all operations (`/api/qualitygates/show`, `/api/qualitygates/select`), not `id`. If you see "not found" errors related to quality gates, check that you're using the gate name. - + ## πŸ’Ύ Out of Memory / Heap Allocation Errors If you see `FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory`, increase the heap size via `--max-memory` or the `maxMemoryMB` config option. The tool automatically restarts itself with the increased heap: @@ -343,7 +348,7 @@ Or use the `migrate:auto-tune` npm script, which detects your hardware and sets npm run migrate:auto-tune ``` - + ## 🐒 Slow Migration Performance If migrations are taking too long, the easiest fix is to use `--auto-tune` which detects your hardware and sets optimal values: @@ -363,7 +368,7 @@ For persistent config, add a `performance` section to your config file. See the Keep `hotspotSync.concurrency` low (3–5) to avoid SonarCloud rate limits. - + ## πŸ“¦ Large Reports Limit source file extraction for testing: @@ -373,10 +378,10 @@ export MAX_SOURCE_FILES=10 ./cloudvoyager transfer -c config.json ``` - + ## πŸ”„ Migration-Specific Issues - + ### Partial Migration Failures The `migrate` command continues to the next project (and the next step within each project) if one fails. After the run completes, check `reports/migration-report.txt` in your output directory for a detailed breakdown of what succeeded and what failed per project, per step. @@ -385,7 +390,7 @@ Projects with the status **partial** had some steps succeed and others fail. Pro You can re-run the migration β€” it will re-process all projects. - + ### Dry Run for Planning Always run with `--dry-run` first to generate mapping CSVs and verify organization assignments before executing the full migration: @@ -394,7 +399,7 @@ Always run with `--dry-run` first to generate mapping CSVs and verify organizati ./cloudvoyager migrate -c migrate-config.json --dry-run ``` - + ### Skipping Issue/Hotspot Metadata Sync If issue or hotspot metadata sync is causing rate limit errors on large projects, you can skip them during migration and sync them separately afterward: @@ -417,7 +422,7 @@ You can also sync just one type of metadata at a time: ./cloudvoyager sync-metadata -c migrate-config.json --skip-issue-metadata-sync --verbose ``` - + ## βœ… Verification Reports > **Import bug fixes (v1.3):** Earlier versions had missing imports in `issue-details.js`, `hotspot-details.js`, and `format-pdf/index.js` that could cause report generation to fail with `ReferenceError` or produce incomplete output. These have been fixed in v1.3. If you encountered report generation errors on a prior version, upgrade and re-run `verify`. @@ -449,6 +454,7 @@ The console also prints a summary with per-project breakdowns and overall pass/f | **error** | Check failed due to an API or connectivity error | ### Issue assignment failures + If issue assignments are failing during migration, the most likely cause is a **login mismatch** between SonarQube and SonarCloud. SonarQube uses local logins (e.g., `john.doe`) while SonarCloud typically uses SSO/GitHub logins (e.g., `john-doe-github`). @@ -464,6 +470,7 @@ If you've already migrated and need to fix assignments, re-run with `--only issu See [Dry-Run CSV Reference β€” user-mappings.csv](dry-run-csv-reference.md#user-mappingscsv) for the full schema and examples. ### Unsyncable items (expected differences) + Some differences are expected because the SonarCloud API does not support syncing them: @@ -476,6 +483,7 @@ Some differences are expected because the SonarCloud API does not support syncin These are reported as **warnings**, not failures. If the only differences are unsyncable items, the verification is considered successful. ### Selective verification + You can verify specific components to save time: @@ -491,6 +499,7 @@ You can verify specific components to save time: ``` ## πŸ›‘ Error Classes Reference + CloudVoyager uses a hierarchy of custom error classes (defined in `src/shared/utils/errors.js`). Understanding which error you're seeing helps narrow down the root cause: @@ -511,6 +520,7 @@ CloudVoyager uses a hierarchy of custom error classes (defined in `src/shared/ut --- ## πŸ”’ Lock File Details + Lock files prevent concurrent runs on the same state file. Key behaviors: @@ -522,6 +532,7 @@ Lock files prevent concurrent runs on the same state file. Key behaviors: --- ## 🧬 Checkpoint Fingerprint Validation + When resuming from a checkpoint journal, the tool validates a session fingerprint against the stored journal. The behavior depends on what changed: @@ -537,18 +548,21 @@ With `strictResume: true` in config, any fingerprint warning becomes a hard fail --- ## πŸ—ƒοΈ Extraction Cache TTL + Extraction caches are stored as **gzipped JSON** files in the `cache/` directory. Files older than **7 days** (configurable via `cacheMaxAgeDays`) are automatically purged. Use `--force-fresh-extract` to discard all caches and re-extract from SonarQube. --- ## πŸ”„ Upload Deduplication on Resume + The checkpoint journal records successful CE task uploads per branch (task ID + timestamp). On resume, if a branch was already uploaded successfully, the upload is skipped. This prevents duplicate CE tasks in SonarCloud after a crash between upload and journal save. --- ## ⏹️ Graceful Shutdown (SIGINT / SIGTERM) + CloudVoyager uses a `ShutdownCoordinator` that handles process signals: @@ -560,12 +574,14 @@ Between pipeline phases, the tool checks the shutdown flag and throws a `Gracefu --- ## πŸ”€ External Linter Issues Missing After Migration (SQ 2025+) + SonarQube 2025+ stores external linter rules (Ruff, Pylint, ESLint, Checkstyle, etc.) with an `external_` prefix in the rule key (e.g., `external_ruff:D200`). If you see zero Ruff/Pylint/etc. issues in SonarCloud after migration, ensure you are running CloudVoyager v1.2.0+ which correctly handles this prefix. Older versions misclassify these as native issues, causing SonarCloud to silently drop them. --- ## πŸ”€ External Issue `cleanCodeAttribute` Must Be Enum + When migrating issues from SonarQube plugins not available in SonarCloud (e.g., MuleSoft), the tool creates external issues. A critical encoding detail: @@ -576,6 +592,7 @@ When migrating issues from SonarQube plugins not available in SonarCloud (e.g., --- ## πŸ•˜ SonarQube 9.9 Issue Statuses + SQ 9.9 LTS uses the legacy issue status model with only 5 statuses: @@ -585,7 +602,7 @@ The modern statuses (`FALSE_POSITIVE`, `ACCEPTED`, `FIXED`) do **not** exist in --- - + ## πŸ“Š Projects with 10,000+ Issues SonarQube's `/api/issues/search` endpoint caps results at 10,000 due to an Elasticsearch hard limit. @@ -593,12 +610,14 @@ SonarQube's `/api/issues/search` endpoint caps results at 10,000 due to an Elast > **Note (v1.3+):** Large-project issue handling now has two complementary mechanisms. The **search slicer** handles retrieval of >10K issues from SonarQube by splitting the date range into windows that each stay under the 10K API limit. The **SCM date backdating** (`backdateChangesets`) preserves each issue's original SonarQube creation date in SonarCloud by writing per-line blame dates into the changeset protobuf. A safety split ensures no single calendar day exceeds 5K issues (50% margin under the 10K ES visualization cap). Together, these two features ensure that projects of any size are fully extracted from SonarQube and fully visible in SonarCloud with accurate historical creation dates. ### Error: `10001th result asked` + **Symptom:** Transfer or migration fails with `SonarQube API error (400): Can return only the first 10000 results. 10001th result asked.` **Fix (v1.2.1+):** The date-range probe that triggered this error has been replaced with a fixed epoch (`2006-01-01` β†’ now) range. No configuration needed β€” re-run the transfer or migration. ### Error: `Date cannot be parsed as either a date or date+time` + **Symptom:** Transfer fails with `SonarQube API error (400): Date '2007-09-08T21:21:02.125Z' cannot be parsed as either a date or date+time` when slicing activates. @@ -607,6 +626,7 @@ SonarQube's `/api/issues/search` endpoint caps results at 10,000 due to an Elast **Fix (v1.2.1+):** Date window boundaries now use `+0000` format. Re-run the transfer or migration. ### General info + The search slicer algorithm: 1. Detects when total issues exceed 10K @@ -618,7 +638,7 @@ The search slicer algorithm: --- - + ## πŸ“¦ Issue Creation Date Accuracy CloudVoyager preserves each issue's original SonarQube creation date in SonarCloud by rewriting SCM changeset blame dates in the protobuf report. This is automatic and transparent β€” no user configuration needed. @@ -626,24 +646,29 @@ CloudVoyager preserves each issue's original SonarQube creation date in SonarClo > **How it works (v1.3.1+):** `backdateChangesets()` reads each issue's `creationDate` field from SonarQube and maps it to the issue's `textRange` lines in the file's changeset data. The CE takes MAX(date) across an issue's line range to determine its creation date, so per-line dating with "oldest wins" for overlapping lines preserves accurate dates. A safety split handles calendar days with >5K issues. ### Issue creation dates don't match SonarQube + Check that the migration was run with v1.3.1+. Earlier versions used arbitrary 30-day-spaced batch dates instead of original creation dates. Re-transfer affected projects to get accurate dates. ### Issue counts look correct in the API but not in the UI + SonarCloud's Elasticsearch caps visualization at 10K per date bucket. With accurate creation dates, issues are naturally distributed across their original dates. The safety split ensures no single day exceeds 5K issues. If you see this problem, verify the migration used v1.3.1+. ### Warning: "N issues on DATE exceed 5K cap" + **Expected behavior.** This means a single calendar day had more than 5,000 issues with the same creation date. The safety split automatically sub-groups them into ≀5K batches with 1-day-spaced synthetic dates. The sub-groups will appear as separate adjacent dates in SonarCloud's creation date facet. ### Can I change the safety split threshold? + The threshold is hardcoded at 5,000 (50% safety margin under the 10K ES visualization limit). It is not configurable. --- ## πŸ”Œ Third-Party Issues Not Appearing in SonarCloud + External (third-party) issues from SonarQube plugins not available in SonarCloud (e.g., MuleSoft, ABAP) may be silently dropped if the rule-repository detection fails. @@ -661,6 +686,7 @@ External (third-party) issues from SonarQube plugins not available in SonarCloud --- ## πŸ“š Further Reading + - [Configuration Reference](configuration.md) β€” all config options, environment variables, npm scripts - [Architecture](architecture.md) β€” project structure, data flow, report format diff --git a/docs/verification.md b/docs/verification.md index 43c5e578..9291a3ea 100644 --- a/docs/verification.md +++ b/docs/verification.md @@ -1,4 +1,5 @@ # Verification + @@ -25,6 +26,7 @@ cloudvoyager verify -c migrate-config.json --concurrency 20 --max-memory 4096 ``` ### CLI Flags + | Flag | Description | |------|-------------| @@ -38,6 +40,7 @@ cloudvoyager verify -c migrate-config.json --concurrency 20 --max-memory 4096 > **Tip:** Verification is read-only and does not modify any data. If a verification run is interrupted, simply re-run it β€” there is no checkpoint state to manage for verification. ## Output + Verification produces three report files in `./verification-output/`: @@ -48,14 +51,17 @@ Verification produces three report files in `./verification-output/`: | `verification-report.pdf` | PDF | Shareable report for stakeholders | ## What Gets Verified + The verification system runs **58+ checks** across two levels: organization-wide and per-project. ### Organization-Level Checks + These run once per organization in your migration config. #### Quality Gates + Compares quality gate **definitions** between SQ and SC. @@ -69,6 +75,7 @@ Compares quality gate **definitions** between SQ and SC. **Fail** if a custom gate is missing or has condition mismatches. #### Quality Profiles + Compares quality profile **definitions** (not assignments β€” those are per-project). @@ -84,6 +91,7 @@ Compares quality profile **definitions** (not assignments β€” those are per-proj **Fail** if a profile is missing (for a supported language) or a custom profile's rule count differs. #### Groups + Compares custom user groups. @@ -95,6 +103,7 @@ Compares custom user groups. **Pass** if all custom SQ groups exist in SC. #### Global Permissions + Compares group-level permissions at the organization level. @@ -107,6 +116,7 @@ Compares group-level permissions at the organization level. **Pass** if all SQ group permissions (minus unsupported ones) exist in SC. #### Permission Templates + Compares permission template existence. @@ -119,16 +129,19 @@ Compares permission template existence. --- ### Per-Project Checks + These run for every project mapping in your migration config. #### Project Existence + Confirms the SC project exists. **Pass** if SC project is found. #### Branches + Compares branches between SQ and SC. @@ -142,6 +155,7 @@ Compares branches between SQ and SC. **Pass** if all SQ branches exist in SC (with default branch equivalence). #### Issues + The most detailed check. Compares all issues between SQ and SC. @@ -167,6 +181,7 @@ The most detailed check. Compares all issues between SQ and SC. - Up to 200 unmatched issue details (rule, file, line, type, severity, message) #### Hotspots + Compares security hotspots between SQ and SC. @@ -181,6 +196,7 @@ Compares security hotspots between SQ and SC. **Pass** if all genuine hotspots match and no status/comment mismatches. #### Measures + Compares project-level metric values. @@ -197,12 +213,14 @@ Compares project-level metric values. **Pass** if all remaining metrics match (after filtering and tolerance). #### Quality Gate (project-level) + Checks which quality gate is assigned to the project. **Pass** if SQ and SC have the same gate assigned (by name). #### Quality Profiles (project-level) + Checks which quality profile is assigned per language. @@ -215,6 +233,7 @@ Checks which quality profile is assigned per language. **Pass** if all SC-supported language profiles match. #### Settings + Compares project-level settings (non-inherited only). @@ -226,30 +245,35 @@ Compares project-level settings (non-inherited only). **Pass** if all shared settings have matching values. #### Tags + Compares project tags. **Pass** if all SQ tags exist in SC. Extra SC tags are allowed. #### Links + Compares project links (homepage, CI, issue tracker, etc.). **Pass** if all SQ links (matched by `name + url`) exist in SC. #### New Code Periods + Compares new code period definitions. **Pass** if the new code period type matches (e.g., `PREVIOUS_VERSION`, `NUMBER_OF_DAYS`). #### DevOps Binding + Compares ALM/DevOps platform bindings. **Pass** if both have the same binding (or neither has one). Fails if SQ has a binding that SC doesn't. #### Permissions (project-level) + Same logic as global permissions, but scoped to the project. @@ -258,12 +282,14 @@ Same logic as global permissions, but scoped to the project. --- ### Portfolios + Portfolio verification is **always skipped** β€” it requires Enterprise API access. SQ portfolios are listed in the report for reference. --- -### Report Modularization +## Report Modularization + Verification report generation has been modularized into separate section modules: - `markdown-sections/detail-sections.js` β€” Per-check detail sections (issues, hotspots, branches, measures, etc.) @@ -276,6 +302,7 @@ This enables easier maintenance and extension of verification reports. --- ## `--only` Component Filters + You can run verification for specific components only: @@ -294,6 +321,7 @@ You can run verification for specific components only: --- ## Result Statuses + Each check produces one of these statuses: @@ -308,10 +336,12 @@ Each check produces one of these statuses: --- ## Platform Differences (Gotchas) + These are known differences between SonarQube and SonarCloud that the verification system handles automatically. They are **not** migration failures. ### Rule Availability + SQ and SC have different rule sets. Some rules exist in SQ but not SC (or vice versa): @@ -322,6 +352,7 @@ SQ and SC have different rule sets. Some rules exist in SQ but not SC (or vice v The verifier handles this by checking if a rule has **zero presence** in the SC side. If so, unmatched items for that rule are excluded from the failure count. ### External Issues (Plugin Migration) + When SQ has issues from plugins not available in SC (e.g., MuleSoft), the migration creates **external issues** in SC. These have a different rule key format: @@ -337,10 +368,12 @@ The verifier normalizes rule keys by stripping the `external_` prefix before mat **Improved in v1.2:** External issue detection is now more reliable. If the SonarCloud `/api/rules/repositories` API is unreachable during migration, the tool falls back to a built-in set of 43 known repositories (with 3 retries and exponential backoff). Rules without a colon separator and empty repository prefixes are also handled gracefully. If verification shows missing external issues from a pre-v1.2 migration, re-run the migration for affected projects. ### Branch Naming + SQ and SC may have different default branch names (e.g., SQ uses `main`, SC uses `master`). The verifier detects each side's default branch (via the `isMain` flag) and treats them as equivalent. ### Quality Profile Naming + The migration creates profiles with a `(SonarQube Migrated)` suffix to avoid colliding with SC's built-in profiles: @@ -351,10 +384,12 @@ The migration creates profiles with a `(SonarQube Migrated)` suffix to avoid col The verifier strips this suffix before comparing. ### Built-In Quality Gates and Profiles + Built-in quality gates ("Sonar way", "Sonar way for AI Code") and built-in profiles ("Sonar way" for each language) are managed by the platform. Their conditions and rule counts naturally differ between SQ and SC versions. The verifier skips detailed comparison for these. ### SonarCloud-Unsupported Permissions + These SonarQube Enterprise permissions don't exist in SonarCloud and are excluded from comparison: @@ -362,10 +397,12 @@ These SonarQube Enterprise permissions don't exist in SonarCloud and are exclude - `portfoliocreator` β€” create Portfolios (Enterprise feature) ### Unsupported Languages + SQ may have plugins that add languages not available in SC (e.g., MuleSoft's `mulesoft` language). Quality profiles for these languages cannot be migrated and are excluded from the profile assignment check. ### Measure Differences + | Metric category | Behavior | |----------------|----------| @@ -375,7 +412,7 @@ SQ may have plugins that add languages not available in SC (e.g., MuleSoft's `mu | SQ-only metrics (`statements`, `functions`, `classes`, `coverage`, `line_coverage`) | Informational only β€” SC may not report these | ### Batch-Distributed Issues and Multiple Analysis Dates - + When a branch has more than 5,000 issues, the migration uses **batch distribution** to split those issues into chunks of up to 5,000 and assigns each chunk a separate `analysis_date` (computed by stepping backwards one day per batch from the original analysis date). This means a single branch that had one analysis in SonarQube may appear in SonarCloud with issues spread across multiple `analysis_date` values. @@ -393,20 +430,24 @@ For reference, the batching logic lives in `src/shared/utils/batch-distributor/` | **Batch plan** | `computeBatchPlan(totalIssues)` returns an array of `{ startIndex, endIndex, batchIndex, isLast }` descriptors | ### Tag Differences on Issues + SC may add its own tags to issues that SQ doesn't have (e.g., `type-dependent` on `typescript:S7755`). The verifier only flags tags that are **missing** from SC (SQ tags not found in SC), not extra SC tags. ### Hotspot Assignments + Hotspot assignments (who the hotspot is assigned to) cannot be synced via SC's API. Assignment differences on hotspots are reported as **unsyncable warnings**, not failures. ### Type and Severity Reclassification + SQ and SC may classify the same rule with different types (BUG vs CODE_SMELL) or severities (MAJOR vs CRITICAL). These are platform-level decisions that can't be overridden by the migration. They're reported as **unsyncable warnings**. --- ## SonarQube API Gotchas + These API quirks affect both migration and verification: @@ -425,24 +466,29 @@ These API quirks affect both migration and verification: --- ## Troubleshooting Verification Failures + ### "X unmatched issues" + 1. Check the `unmatchedSqIssues` array in the JSON report for details (rule, file, line). 2. If all unmatched issues share the same rule, that rule likely doesn't exist in SC β€” this is a platform difference and should already be handled. 3. If issues are on different rules, check if the files exist in SC and were transferred correctly. ### "X unmatched hotspots" + Same as issues β€” check if the rules exist in SC. Common causes: - Rule reclassified from hotspot to issue (or vice versa) - Rule not available in SC's analyzer version ### "Quality profile rule count mismatch" + For built-in profiles, this is expected and should be skipped automatically. For custom profiles, check if SC has additional rules activated or if some SQ rules aren't available in SC. ### "Measures mismatch" + Check which metric is mismatched: - `lines`/`ncloc` with small differences β€” scanner implementation difference (should be within 1% tolerance) @@ -450,6 +496,7 @@ Check which metric is mismatched: - `duplicated_*` β€” SC recalculates these, should be skipped automatically ### Report shows 19 failures but source code fixes pass + If you're running the compiled binary (`dist/bin/cloudvoyager-*`) and see failures that `node src/index.js verify` doesn't, you need to rebuild: diff --git a/docs/versioning.md b/docs/versioning.md index 3d71b19e..2a219716 100644 --- a/docs/versioning.md +++ b/docs/versioning.md @@ -1,12 +1,12 @@ # Versioning - + How CloudVoyager versions are managed, bumped, and released. --- - ## Source of Truth + The **single source of truth** for the application version is: @@ -22,8 +22,8 @@ The CLI (`src/index.js`) reads the version dynamically rather than hardcoding it --- - ## Where the Version Appears + | Location | File | Updated By | |----------|------|------------| @@ -37,12 +37,13 @@ The package.json files are updated automatically by the version-bump workflow. T --- - ## How the Version Gets Bumped + The version bump is triggered by **merging a pull request to `main` that has a GitHub milestone attached**. ### Trigger Conditions + Both conditions must be true: 1. The PR is **merged** (not just closed) @@ -51,6 +52,7 @@ Both conditions must be true: If either condition is missing, no version bump occurs. ### Bump Logic + The workflow (`.github/workflows/version-bump.yml`) compares the milestone title to the current version: @@ -60,6 +62,7 @@ The workflow (`.github/workflows/version-bump.yml`) compares the milestone title | New major.minor | `1.2.5` | `1.3` | `1.3.0` (reset patch to 0) | ### What the Workflow Does + 1. Reads the milestone title from the merged PR 2. Reads the current version from `package.json` @@ -70,8 +73,8 @@ The workflow (`.github/workflows/version-bump.yml`) compares the milestone title --- - ## Release Pipeline + Every push to `main` triggers the full release pipeline (`.github/workflows/release.yml`): @@ -85,6 +88,7 @@ push to main ``` ### Tag Format + ``` v1.2.0-20260328120445 @@ -95,6 +99,7 @@ v1.2.0-20260328120445 ``` ### Release Body + The release includes: - Auto-generated release notes (from commit history) @@ -103,8 +108,8 @@ The release includes: --- - ## Desktop App Versioning + The desktop app version is read from `desktop/package.json` by `electron-builder`. This determines the version shown in: - Installer filenames (e.g., `CloudVoyager.Desktop-1.2.0-macos-arm64.dmg`) @@ -115,8 +120,8 @@ The `electron-builder.yml` config uses `${version}` which resolves to `desktop/p --- - ## Milestones + GitHub milestones serve two purposes: @@ -127,8 +132,8 @@ Milestone titles must be in `MAJOR.MINOR` format (e.g., `1.2`, `1.3`, `2.0`). --- - ## Workflow Files + | Workflow | File | Purpose | |----------|------|---------|