diff --git a/README.md b/README.md index be662985..be6bbdab 100644 --- a/README.md +++ b/README.md @@ -1,676 +1,196 @@ -# SonarQube Migration Tool +# SonarQube to SonarCloud Migration Tool -A CLI tool for migrating SonarQube Server configurations to SonarQube Cloud. Extract data from one or more SonarQube Server instances, map organizations, and migrate quality profiles, quality gates, groups, permissions, and projects to your cloud environment. +Migrate your SonarQube Server configurations to SonarQube Cloud — quality gates, quality profiles, groups, permissions, projects, and portfolios. -## Migration Workflow +## What Gets Migrated -``` -┌─────────────┐ ┌─────────────┐ ┌──────────────┐ ┌─────────────┐ ┌─────────────┐ -│ EXTRACT │───►│ STRUCTURE │───►│ MAPPINGS │───►│ MIGRATE │───►│ PIPELINES │ -│ Phase 1 │ │ Phase 2 │ │ Phase 3 │ │ Phase 4 │ │ Phase 5 │ -└─────────────┘ └─────────────┘ └──────────────┘ └─────────────┘ └─────────────┘ - │ │ │ │ │ - ▼ ▼ ▼ ▼ ▼ - Pull data Generate Create Push configs Update CI/CD - from SQS org/project mapping to SonarQube pipelines - via API CSV files CSV files Cloud (optional) -``` - -**Phase 1 - Extract**: Pull all configuration data from your SonarQube Server instance(s)\ -**Phase 2 - Structure**: Generate organization and project mappings based on DevOps bindings\ -**Phase 3 - Mappings**: Create detailed mapping files for gates, profiles, groups, and templates\ -**Phase 4 - Migrate**: Push all configurations to SonarQube Cloud\ -**Phase 5 - Pipelines**: Automatically update CI/CD pipeline files to use SonarQube Cloud (optional) - ---- - -## Prerequisites - -### Required Software - -- **Docker** installed and running on your machine - -### SonarQube Server Requirements - -- Admin user token with the following permissions: - - Administer System - - Administer Quality Gates - - Administer Quality Profiles - - Browse all projects - -### SonarQube Cloud Requirements - -- Enterprise license with organizations already created -- Organizations added to the enterprise -- Admin user token with permissions at the enterprise level and all target organizations - -### Token Permissions Summary - -| Environment | Token Type | Required Permissions | -|-------------|------------|---------------------| -| SonarQube Server | Admin Token | System admin, Quality Gates admin, Quality Profiles admin | -| SonarQube Cloud | User Token | Enterprise admin + Organization admin for all target orgs | - ---- - -## Getting Started - -Choose your approach: -- **[Interactive Wizard (Recommended)](#interactive-wizard-recommended)** — guided, step-by-step (recommended for most users) -- **[Manual CLI Method](#manual-cli-method)** — direct commands for scripting or advanced use - ---- - -## Interactive Wizard (Recommended) - -For a guided migration experience, use the interactive wizard command. The wizard walks you through each phase, prompting for credentials and providing progress feedback. - -### Command - -```bash -docker run -it -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest wizard -``` - -> **Note**: Use `-it` flag for interactive mode to enable prompts. - -### Options - -| Option | Description | Default | -|--------|-------------|---------| -| `--export_directory` | Root directory for export files and wizard state | `/app/files/` | - -### What the Wizard Does - -The wizard guides you through all migration phases: - -1. **Extract** - Prompts for SonarQube Server URL and admin token, optional client certificate -2. **Structure** - Automatically generates organization and project mappings -3. **Organization Mapping** - Prompts you to map each SonarQube Server org to a SonarQube Cloud org key -4. **Mappings** - Generates entity mappings (gates, profiles, groups, templates) -5. **Validate** - Runs pre-flight validation checks -6. **Migrate** - Confirms before pushing configurations to SonarQube Cloud -7. **Pipelines** - Optional CI/CD pipeline updates (if secrets.json exists) - -### Features - -- **Resume support**: If interrupted, the wizard saves state and resumes from the last completed phase -- **Client certificate support**: Prompts for mTLS certificate details when needed -- **Progress display**: Shows current phase and overall progress -- **Validation**: Checks that all organizations are mapped before migration - -### Example Session - -``` -$ docker run -it -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest wizard - -╔════════════════════════════════════════════════════════════════╗ -║ SonarQube Migration Wizard ║ -║ Migrate from SonarQube Server to SonarQube Cloud ║ -╚════════════════════════════════════════════════════════════════╝ - -Phase 1 of 7: Extract -───────────────────── -SonarQube Server URL: https://sonar.example.com -SonarQube Server Admin Token: ******** -Do you need to use a client certificate? [y/N]: n - -Extracting data from SonarQube Server... -✓ Extract complete: 1706745600 - -Phase 2 of 7: Structure -─────────────────────── -... -``` - ---- - -## Manual CLI Method - -Most users should use the wizard. For scripting or automation, use the manual commands below. - -### Step 1: Create a working directory - -```bash -mkdir sonar-migration -cd sonar-migration -``` - -### Step 2: Extract data from SonarQube Server - -```bash -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest \ - extract https://your-sonarqube-server.com YOUR_SQS_ADMIN_TOKEN -``` - -### Step 3: Generate organization structure - -```bash -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest structure -``` - -### Step 4: Map your SonarQube Cloud organizations - -Open `./files/organizations.csv` and fill in the `sonarcloud_org_key` column with your SonarQube Cloud organization keys: - -```csv -alm,server_url,org_name,sonarcloud_org_key -github,https://your-sonarqube-server.com,my-github-org,my-cloud-org-key -``` - -### Step 5: Generate entity mappings - -```bash -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest mappings -``` - -### Step 6: Run the migration - -```bash -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest \ - migrate YOUR_SQC_ENTERPRISE_TOKEN your-enterprise-key -``` - -### Step 7: Verify migration - -Log in to your SonarQube Cloud organization and verify: -- Quality Gates were created with correct conditions -- Quality Profiles were restored with rules -- Groups exist with proper permissions -- Projects are linked to repositories - ---- - -### Phase 1: Extract - -Extract configuration data from your SonarQube Server instance via the REST API. - -#### Command - -```bash -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest \ - extract [OPTIONS] -``` - -#### Arguments - -| Argument | Description | +| Migrated | NOT Migrated | |----------|-------------| -| `URL` | Full URL of your SonarQube Server (e.g., `https://sonar.example.com`) | -| `TOKEN` | Admin user token with system administration permissions | - -#### Options - -| Option | Description | Default | -|--------|-------------|---------| -| `--export_directory` | Output directory for extracted data | `/app/files/` | -| `--concurrency` | Maximum concurrent API requests | 25 | -| `--timeout` | Request timeout in seconds | 60 | -| `--extract_id` | Resume a previous extraction | (new extraction) | -| `--extract_type` | Type of extraction to run | (full) | -| `--target_task` | Run only a specific task and its dependencies | (all tasks) | -| `--pem_file_path` | Client certificate PEM file (for mTLS) | - | -| `--key_file_path` | Client certificate key file (for mTLS) | - | -| `--cert_password` | Password for client certificate | - | - -#### Resuming a Failed Extraction - -If extraction fails partway through, resume using the extract ID: - -```bash -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest \ - extract https://your-server.com YOUR_TOKEN --extract_id 1706745600 -``` - ---- - -### Phase 2: Structure - -Generate organization and project mappings based on DevOps platform bindings. - -#### Command - -```bash -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest \ - structure [OPTIONS] -``` - -#### Options - -| Option | Description | Default | -|--------|-------------|---------| -| `--export_directory` | Directory containing extracted data | `/app/files/` | - -#### Required Manual Step - -After running `structure`, you must edit `organizations.csv` and fill in the `sonarcloud_org_key` column with your SonarQube Cloud organization keys before proceeding to the next phase. - ---- - -### Phase 3: Mappings - -Create detailed mapping files for all entities that will be migrated. - -#### Command - -```bash -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest \ - mappings [OPTIONS] -``` - -#### Options - -| Option | Description | Default | -|--------|-------------|---------| -| `--export_directory` | Directory containing extracted data | `/app/files/` | - -Outputs `gates.csv`, `profiles.csv`, `groups.csv`, `templates.csv`, and `portfolios.csv` in the export directory. See [Output Files Reference](#output-files-reference) for column details. +| Quality Gates (with conditions) | Historical analysis data | +| Quality Profiles (with rules) | Issues and their history | +| Groups and Permissions | Code coverage history | +| Permission Templates | Security hotspots | +| Projects (settings, tags) | Source code (you need to re-scan) | +| Portfolios | | --- -### Phase 4: Migrate +## Quick Start -Execute the migration by pushing configurations to SonarQube Cloud. - -#### Command - -```bash -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest \ - migrate [OPTIONS] -``` - -#### Arguments - -| Argument | Description | -|----------|-------------| -| `TOKEN` | SonarQube Cloud token with enterprise and organization admin permissions | -| `ENTERPRISE_KEY` | Key of your SonarQube Cloud enterprise | +The fastest way to migrate: download the executable, create a config file, run one command. -#### Options +### 1. Download the Executable -| Option | Description | Default | -|--------|-------------|---------| -| `--url` | SonarQube Cloud URL | `https://sonarcloud.io/` | -| `--edition` | SonarQube Cloud license edition | - | -| `--export_directory` | Directory containing mapping files | `/app/files/` | -| `--concurrency` | Maximum concurrent API requests | 25 | -| `--run_id` | Resume a previous migration | (new migration) | -| `--target_task` | Run only a specific task and its dependencies | (all tasks) | -| `--skip_profiles` | Skip migration of quality profiles | `false` | +Grab the pre-built binary for your platform from the [Releases page](https://github.com/sonar-solutions/sonar-reports/releases): -#### Migration Tasks +| Platform | Download | +|----------|----------| +| macOS (Apple Silicon) | `sonar-reports-macos-arm64` | +| macOS (Intel) | `sonar-reports-macos-x86_64` | +| Linux (x86_64) | `sonar-reports-linux-x86_64` | +| Linux (ARM64) | `sonar-reports-linux-arm64` | +| Windows (x86_64) | `sonar-reports-windows-x86_64.exe` | +| Windows (ARM64) | `sonar-reports-windows-arm64.exe` | -The migration executes 44 tasks in dependency order: - -| Category | What Gets Created | -|----------|-------------------| -| **Groups** | User groups with descriptions | -| **Permissions** | Organization-level group permissions | -| **Templates** | Permission templates with user/group assignments | -| **Quality Gates** | Gates with conditions, default assignments | -| **Quality Profiles** | Profiles restored from backup, inheritance set, defaults configured | -| **Projects** | Projects created/linked, gates assigned, profiles assigned, tags set | -| **Portfolios** | Portfolios created with project assignments | - -#### Resuming a Failed Migration - -If migration fails partway through, resume using the run ID: +Make it runnable (macOS/Linux): ```bash -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest \ - migrate YOUR_TOKEN your-enterprise-key --run_id 1706745600 +chmod +x sonar-reports-* ``` -The tool tracks completed tasks and resumes from the last completed task. - ---- - -### Phase 5: Pipelines (Optional) - -Automatically update CI/CD pipeline configurations to use SonarQube Cloud. - -#### Command +### 2. Create a Config File ```bash -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest \ - pipelines [OPTIONS] +cp examples/migration-config.example.json migration-config.json ``` -#### Arguments - -| Argument | Description | -|----------|-------------| -| `SECRETS_FILE` | Path to JSON file containing org-key to DevOps-token mappings (relative to `--input_directory`) | -| `SONAR_TOKEN` | SonarQube Cloud token to set as organization secret | -| `SONAR_URL` | SonarQube Cloud URL to set as organization secret | - -#### Options - -| Option | Description | Default | -|--------|-------------|---------| -| `--input_directory` | Directory containing migration files (must
include `generateOrganizationMappings` task output) | `/app/files/` | -| `--output_directory` | Directory to place pipeline update output files | Same as
`--input_directory` | - -#### Supported Platforms - -- GitHub (primary) -- GitLab (extensible) -- Azure DevOps (extensible) -- Bitbucket (extensible) - -#### Supported Scanners - -- SonarQube CLI Scanner -- Maven (`pom.xml`) -- Gradle (`build.gradle`) -- .NET Scanner - ---- - -## Additional Commands - -### Full Migrate - -Run a complete end-to-end migration from a single JSON config file. Automatically runs extract, structure, org mapping, mappings, and migrate phases in sequence. - -```bash -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest \ - full_migrate -``` - -#### Arguments - -| Argument | Description | -|----------|-------------| -| `CONFIG_FILE` | Path to a JSON configuration file (see below) | - -#### Config File Format +Edit `migration-config.json` with your details: ```json { "sonarqube": { "url": "https://your-sonarqube-server.com", - "token": "YOUR_SQS_ADMIN_TOKEN" + "token": "YOUR_SONARQUBE_ADMIN_TOKEN" }, "sonarcloud": { "url": "https://sonarcloud.io/", - "token": "YOUR_SQC_TOKEN", - "enterprise_key": "your-enterprise", - "org_key": "your-org-key" + "token": "YOUR_SONARCLOUD_ADMIN_TOKEN", + "enterprise_key": "YOUR_ENTERPRISE_KEY", + "org_key": "YOUR_TARGET_ORG_KEY" }, "settings": { - "export_directory": "/app/files", + "export_directory": "./files", "concurrency": 10, "timeout": 60 } } ``` -### Analysis Report - -Parse a migration run's `requests.log` and generate a CSV summary of all API calls with success/failure outcomes. - -```bash -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest \ - analysis_report [OPTIONS] -``` - -| Argument/Option | Description | Default | -|-----------------|-------------|---------| -| `RUN_ID` | ID of the migration run to analyze | — | -| `--export_directory` | Directory containing migration run folders | `/app/files/` | - -Outputs `final_analysis_report.csv` inside the run directory. - -### Report - -Generate markdown reports based on extracted data. - -```bash -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest \ - report [OPTIONS] -``` - -#### Options - -| Option | Description | Default | -|--------|-------------|---------| -| `--export_directory` | Directory containing extracted data | `/app/files/` | -| `--report_type` | Type of report to generate
(`migration` or `maturity`) | `migration` | -| `--filename` | Custom filename for the generated
report (without extension) | Same as `--report_type` | +**You will need:** +- A SonarQube Server admin token (with system admin, quality gate admin, and quality profile admin permissions) +- A SonarCloud admin token (with enterprise-level and organization-level admin permissions) +- Your SonarCloud enterprise key and target organization key -Report types include migration readiness and maturity assessments. +See [docs/SECURITY.md](docs/SECURITY.md) for token handling best practices. -### Reset - -Reset a SonarQube Cloud enterprise to its original state. +### 3. Run the Migration ```bash -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest \ - reset [OPTIONS] +./sonar-reports-macos-arm64 full-migrate migration-config.json ``` -**Warning**: This deletes everything in every organization within the enterprise. +Replace the binary name with the one you downloaded. On Windows, use `sonar-reports-windows-x86_64.exe` instead. -#### Arguments +The tool will automatically extract data, generate mappings, and migrate everything to SonarCloud. -| Argument | Description | -|----------|-------------| -| `TOKEN` | SonarQube Cloud token with enterprise and organization admin permissions | -| `ENTERPRISE_KEY` | Key of the SonarQube Cloud enterprise that will be reset | - -#### Options - -| Option | Description | Default | -|--------|-------------|---------| -| `--url` | SonarQube Cloud URL | `https://sonarcloud.io/` | -| `--edition` | SonarQube Cloud license edition | `enterprise` | -| `--concurrency` | Maximum concurrent API requests | `25` | -| `--export_directory` | Directory for interim files | `/app/files/` | +> **Migrating to multiple organizations?** The `full-migrate` command maps all projects to a single org. For multi-org migrations, use the [step-by-step method](docs/MANUAL-MIGRATION.md) where you can edit `organizations.csv` to assign different target orgs. --- -## Output Files Reference - -### Extraction Output +## Other Ways to Run -| File | Purpose | -|------|---------| -| `/extract.json` | Extraction metadata (version, edition, URL) | -| `/requests.log` | HTTP request/response log for debugging | -| `//results.*.jsonl` | Extracted data in NDJSON format | - -### Structure Output - -| File | Purpose | Requires Editing | -|------|---------|------------------| -| `organizations.csv` | Organization mappings | Yes - fill in `sonarcloud_org_key` | -| `projects.csv` | Project mappings | No | - -### Mappings Output - -| File | Purpose | Requires Editing | -|------|---------|------------------| -| `gates.csv` | Quality Gate mappings | No | -| `profiles.csv` | Quality Profile mappings | No | -| `groups.csv` | Group mappings | No | -| `templates.csv` | Permission Template mappings | No | -| `portfolios.csv` | Portfolio mappings | No | +| Method | Best For | Guide | +|--------|----------|-------| +| **Docker** | No local installs needed | [docs/DOCKER.md](docs/DOCKER.md) | +| **Python** | `pip install -r requirements.txt && python src/main.py ` | [docs/MANUAL-MIGRATION.md](docs/MANUAL-MIGRATION.md) | +| **Interactive Wizard** | Guided step-by-step prompts | See below | +| **Shell Scripts** | Automated/scripted migrations | `scripts/execute_full_migration.sh` or `scripts/execute_migration_with_binary.sh` | +| **Build from Source** | Custom builds | [docs/BUILD.md](docs/BUILD.md) | --- -## Troubleshooting - -### Common Errors - -#### "Token does not have sufficient permissions" +## Interactive Wizard -**Cause**: The admin token lacks required permissions. +For a guided experience with prompts at each step: -**Solution**: Ensure your SonarQube Server token has: -- Administer System permission -- Administer Quality Gates permission -- Administer Quality Profiles permission - -#### "Organization not found" - -**Cause**: The `sonarcloud_org_key` in `organizations.csv` doesn't match an existing organization. - -**Solution**: -1. Verify the organization exists in SonarQube Cloud -2. Check for typos in the organization key -3. Ensure the organization is part of your enterprise - -#### "Request timeout" - -**Cause**: Large datasets or slow network connection. - -**Solution**: Increase the timeout value: ```bash -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest \ - extract https://your-server.com YOUR_TOKEN --timeout 120 +./sonar-reports-macos-arm64 wizard ``` -#### "Connection refused" or SSL errors - -**Cause**: Network issues or certificate problems. +Or with Docker: -**Solutions**: -- Verify the URL is accessible from your machine -- For self-signed certificates, use client certificate options: - ```bash - docker run -v ./files:/app/files -v ./certs:/app/certs ghcr.io/sonar-solutions/sonar-reports:latest \ - extract https://your-server.com YOUR_TOKEN \ - --pem_file_path /app/certs/client.pem \ - --key_file_path /app/certs/client.key - ``` - -#### Migration task fails midway - -**Cause**: API error, rate limiting, or network interruption. - -**Solution**: Resume the migration using the run ID shown in the output: ```bash -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest \ - migrate YOUR_TOKEN your-enterprise-key --run_id +docker run -it -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest wizard ``` -### Finding Logs - -- **Request logs**: `./files//requests.log` contains all HTTP requests and responses -- **Docker logs**: `docker logs ` shows CLI output - -### Reducing Memory Usage - -For large SonarQube instances (50,000+ projects), reduce concurrency: - -```bash -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest \ - extract https://your-server.com YOUR_TOKEN --concurrency 10 -``` +The wizard walks you through extraction, organization mapping, validation, migration, and optional pipeline updates. It saves progress so you can resume if interrupted. --- -## Docker Deployment +## Step-by-Step Manual Migration -### Basic Usage +If you need more control (e.g., migrating to multiple orgs, extracting from multiple servers, or running specific tasks), see the full manual guide: -Mount a local directory to persist data between commands: - -```bash -docker run -v /path/to/local/files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest -``` +**[docs/MANUAL-MIGRATION.md](docs/MANUAL-MIGRATION.md)** -### With Client Certificates +The manual process follows these phases: -Mount your certificate files: - -```bash -docker run \ - -v ./files:/app/files \ - -v ./certs:/app/certs \ - ghcr.io/sonar-solutions/sonar-reports:latest \ - extract https://your-server.com YOUR_TOKEN \ - --pem_file_path /app/certs/client.pem \ - --key_file_path /app/certs/client.key \ - --cert_password YOUR_CERT_PASSWORD -``` - -### Docker Compose - -```yaml -services: - sonar-migration: - image: ghcr.io/sonar-solutions/sonar-reports:latest - volumes: - - ./files:/app/files - - ./certs:/app/certs # Optional: for client certificates ``` - -Run commands with: - -```bash -docker compose run sonar-migration extract https://your-server.com YOUR_TOKEN -docker compose run sonar-migration structure -docker compose run sonar-migration mappings -docker compose run sonar-migration migrate YOUR_SQC_TOKEN your-enterprise-key +Extract → Structure → Edit Mappings → Generate Mappings → Migrate → Pipelines (optional) ``` --- -## Multi-Server Migration +## After Migration -To migrate from multiple SonarQube Server instances: +1. **Verify** in SonarCloud — check quality gates, profiles, groups, and projects +2. **Re-scan your projects** — historical data is not migrated, so run new scans +3. **Set up CI/CD integration** — configure your DevOps platform (GitHub, GitLab, Azure DevOps, Bitbucket) -### Step 1: Extract from each server +--- -```bash -# Server 1 -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest \ - extract https://sonar1.example.com TOKEN1 +## Troubleshooting -# Server 2 -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest \ - extract https://sonar2.example.com TOKEN2 -``` +**Migration failed?** Here are the most common issues: -### Step 2: Run structure (aggregates all extractions) +| Problem | Solution | +|---------|----------| +| Token permission errors | Ensure your SonarQube token has system admin permissions and your SonarCloud token has enterprise admin permissions | +| Organization not found | Check that `sonarcloud_org_key` in `organizations.csv` matches an existing org in your enterprise | +| Timeout errors | Increase `--timeout` (e.g., `120`) or reduce `--concurrency` (e.g., `5`) | +| Docker can't reach localhost | Use `host.docker.internal` instead of `localhost` | +| Migration failed midway | Resume with `--run_id` — the tool tracks completed tasks | -```bash -docker run -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest structure -``` - -### Step 3: Edit organizations.csv +For the full troubleshooting guide, see **[docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md)**. -Map organizations from all servers to your cloud organizations. +--- -### Step 4: Continue with mappings and migrate +## All Available Commands -The remaining steps work the same as a single-server migration. +| Command | Description | +|---------|-------------| +| `full-migrate` | End-to-end migration from a single config file | +| `wizard` | Interactive guided migration | +| `extract` | Pull data from SonarQube Server | +| `structure` | Generate organization/project mappings | +| `mappings` | Generate entity mapping CSVs | +| `migrate` | Push configurations to SonarCloud | +| `pipelines` | Update CI/CD pipeline files (optional) | +| `report` | Generate migration or maturity reports | +| `analysis_report` | Generate CSV summary of API calls from a migration run | +| `reset` | Reset a SonarCloud enterprise to original state (**destructive**) | --- -## Best Practices +## Documentation -1. **Create a dedicated migration user** in SonarQube Cloud with enterprise admin permissions -2. **Test with a subset first** using `--target_task` to migrate specific entities -3. **Review CSV mappings** before running migrate to ensure accuracy -4. **Monitor the requests.log** for API errors during extraction and migration -5. **Use lower concurrency** for large instances to avoid overwhelming the server -6. **Back up existing configurations** before running reset on your cloud enterprise +| Guide | Description | +|-------|-------------| +| [docs/MANUAL-MIGRATION.md](docs/MANUAL-MIGRATION.md) | Step-by-step migration with binary, Docker, or Python | +| [docs/DOCKER.md](docs/DOCKER.md) | Docker-specific usage guide | +| [docs/CONFIG.md](docs/CONFIG.md) | Configuration file reference | +| [docs/BUILD.md](docs/BUILD.md) | Building executables from source | +| [docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) | Common errors and solutions | +| [docs/SECURITY.md](docs/SECURITY.md) | Token handling and security best practices | +| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Internal architecture for contributors | --- ## Version Support -The tool automatically detects your SonarQube Server version and uses the appropriate API endpoints: - -- Supports SonarQube Server 6.3+ -- Authentication adapts based on version: - - Server < 10: Basic authentication - - Server >= 10: Bearer token authentication -- Edition-aware: Handles Community, Developer, Enterprise, and Data Center editions - ---- +- SonarQube Server 6.3+ +- Supports Community, Developer, Enterprise, and Data Center editions +- SonarQube Cloud Enterprise and Teams editions ## License diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 00000000..2741e6b8 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,154 @@ +# Architecture + +## Overview + +sonar-reports is a Python CLI application built with Click. It can run as a standalone PyInstaller executable, Docker container, or directly with Python. Its primary purpose is to facilitate migration from SonarQube Server to SonarQube Cloud (formerly SonarCloud). + +## Project Structure + +``` +src/ +├── main.py # CLI entry point (Click commands) +├── config.py # JSON config file loader +├── plan.py # Task plan generation (DAG) +├── execute.py # Task execution engine +├── dependencies.py # Dependency loading strategies +├── constants.py # Built-in repos, task lists +├── validate.py # Pre-migration validation +├── utils.py # CSV/JSON utilities +├── parser.py # Field/value parsing +├── analysis_report.py # Migration analysis report +├── operations/ # Data transformation operations +│ ├── http_request/ # HTTP request operations +│ ├── apply_filter.py +│ ├── expand_list.py +│ ├── extend_list.py +│ ├── get_date.py +│ ├── join_string.py +│ ├── match_devops_platform.py +│ ├── process_array.py +│ └── set_key.py +├── structure/ # Org/project structure mapping +├── wizard/ # Interactive wizard +├── pipelines/ # CI/CD pipeline updater +├── report/ # Report generators +└── tasks/ # JSON task definitions + ├── export/ # 72+ extraction tasks + ├── migrate/ # 46+ migration tasks + └── delete/ # Cleanup/reset tasks +``` + +## Task Engine + +The core of the tool is a JSON-driven task execution engine. Tasks are declarative definitions that describe how to extract data from SonarQube Server, transform it, and load it into SonarQube Cloud. + +### Task Definitions + +Tasks are JSON files in `src/tasks/` organized by SonarQube version (e.g., `4.3.json`, `10.4.json`, `cloud.json`). Each task specifies: + +- **`dependencies`** -- prerequisite tasks with loading strategies (`each`, `chunk`, `all`, `map`, `none`) +- **`editions`** -- which SonarQube editions support this task (`community`, `developer`, `enterprise`, `datacenter`) +- **`operations`** -- an ordered pipeline of data transformations + +### Dependency Loading Strategies + +The dependency system controls how prerequisite task output is fed into downstream tasks: + +- **`each`** -- yield each object individually. The task executes once per dependency object. +- **`chunk`** -- group objects into fixed-size chunks. Useful for batch API calls. +- **`all`** -- load all dependency data into memory at once. Suitable for lookups and joins. +- **`map`** -- group objects by a key field. Enables efficient key-based access. +- **`none`** -- no dependency data needed. The task runs independently. + +### Operations + +Operations are pluggable transformations that form the processing pipeline within each task: + +- **`http_request`** -- make API calls with payload mapping and result extraction +- **`apply_filter`** -- filter objects based on conditions +- **`set_key`** -- add or modify fields on the current object +- **`process_array`** -- array transformations +- **`expand_list`** / **`extend_list`** -- list expansion and concatenation operations +- **`join_string`** -- concatenate fields into a single string +- **`match_devops_platform`** -- detect the DevOps platform (GitHub, GitLab, Azure DevOps, Bitbucket) +- **`get_date`** -- date parsing and formatting operations + +### Execution Flow + +1. **`get_available_task_configs()`** loads task configurations based on the detected SonarQube version and edition. +2. **`generate_task_plan()`** builds a directed acyclic graph (DAG) and resolves all dependencies. +3. **`plan_tasks()`** organizes tasks into sequential execution phases, respecting dependency order. +4. **`execute_plan()`** runs phases in order. +5. **`execute_task()`** loads dependencies, runs the operation pipeline, and exports results as NDJSON. + +## Data Flow + +The migration proceeds through a well-defined sequence of stages: + +``` +SonarQube Server API + | extract (get* tasks) + v +NDJSON files in files// + | structure + v +organizations.csv + projects.csv + | (user fills in sonarcloud_org_key) + | mappings + v +gates.csv, profiles.csv, groups.csv, templates.csv, portfolios.csv + | migrate (create*/set*/add* tasks) + v +SonarQube Cloud API + | pipelines (optional) + v +CI/CD pipeline PRs +``` + +**Extract phase:** The tool queries SonarQube Server APIs and writes each entity (projects, quality gates, quality profiles, users, groups, permissions, etc.) to NDJSON files on disk. + +**Structure phase:** Extracted data is organized into CSV mapping files that define how SonarQube Server entities correspond to SonarQube Cloud organizations and projects. Users review and fill in required fields such as `sonarcloud_org_key`. + +**Migrate phase:** Using the completed mapping files, the tool creates corresponding entities in SonarQube Cloud via its API. + +**Pipelines phase (optional):** The tool can generate pull requests to update CI/CD pipeline configurations (e.g., GitHub Actions, GitLab CI, Azure Pipelines) to point to the new SonarQube Cloud instance. + +## Configuration System + +- `config.py` loads JSON configuration files that define connection details and migration parameters. +- CLI arguments override config file values, allowing per-invocation customization. +- `export_directory` is restricted to the working directory or `/tmp` for security -- this prevents path traversal attacks. + +## Version Detection + +The tool auto-detects SonarQube Server version and selects appropriate API endpoints and authentication methods: + +- **Server < 10:** Basic authentication (username:token) +- **Server >= 10:** Bearer token authentication +- **Edition-aware:** The tool detects whether the source instance is Community, Developer, Enterprise, or Data Center edition. Tasks are filtered based on edition support -- for example, portfolio-related tasks only run against Enterprise and Data Center editions. + +## Runtime Dependencies + +| Dependency | Purpose | +|---------------|----------------------------------| +| httpx | Async HTTP client for API calls | +| click | CLI framework | +| markdown | Markdown rendering | +| tenacity | Retry logic for transient errors | +| PyNaCl | Cryptographic operations | +| ruamel.yaml | YAML parsing (pipeline updates) | +| bashlex | Bash script parsing | +| lxml | XML parsing | + +## Build System + +- **PyInstaller** creates standalone executables from the spec file `sonar-reports.spec`. +- Builds target **6 platforms**: macOS (x86_64, ARM64), Linux (x86_64, ARM64), Windows (x86_64, ARM64). +- **Docker image** is based on Python 3.14-slim for minimal footprint. +- **CI/CD** uses GitHub Actions workflows for automated testing, Docker image publishing, and release builds. + +## Testing + +- **pytest** is the test runner, with **pytest-asyncio** for async test support, **pytest-mock** for mocking, and **respx** for HTTP request mocking. +- Coverage is configured via `.coveragerc`. +- SonarQube Cloud analysis is configured through `sonar-project.properties` for continuous code quality tracking of the tool itself. diff --git a/docs/DOCKER.md b/docs/DOCKER.md new file mode 100644 index 00000000..815918ac --- /dev/null +++ b/docs/DOCKER.md @@ -0,0 +1,173 @@ +# Docker Guide + +## Overview + +The SonarQube-to-SonarCloud migration tool is published as a ready-to-use Docker image at: + +``` +ghcr.io/sonar-solutions/sonar-reports:latest +``` + +You do not need Python, pip, or any other dependencies installed on your machine — just Docker. Everything the tool needs is already baked into the image. + +## Basic Usage + +Mount a local directory so that generated files (CSVs, configs, logs) persist between commands: + +```bash +docker run -v /path/to/local/files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest +``` + +Replace `` with any of the available migration commands described below. + +## Quick Migration with Docker + +If you already have a `migration-config.json` file prepared, you can run the entire migration in one shot using `full_migrate`: + +```bash +docker run --rm -v "$(pwd)/files:/app/files" ghcr.io/sonar-solutions/sonar-reports:latest full_migrate /app/files/migration-config.json +``` + +The `--rm` flag automatically removes the container when it finishes, keeping things tidy. + +## Using the Interactive Wizard + +The wizard walks you through the migration step by step with prompts. Because it is interactive, you need the `-it` flags so Docker connects your terminal to the container: + +```bash +docker run -it -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest wizard +``` + +> **Note:** The `-it` flag is required for interactive mode. Without it the wizard will not be able to read your input and will exit immediately. + +## Step-by-Step Commands + +If you prefer to run each migration phase individually, here is the full sequence: + +### 1. Extract + +Pull project data from your SonarQube Server: + +```bash +docker run --rm -v "$(pwd)/files:/app/files" ghcr.io/sonar-solutions/sonar-reports:latest extract https://your-sonarqube-server.com YOUR_SONARQUBE_TOKEN +``` + +### 2. Structure + +Generate the organizational structure files: + +```bash +docker run --rm -v "$(pwd)/files:/app/files" ghcr.io/sonar-solutions/sonar-reports:latest structure +``` + +### 3. Edit organizations.csv manually + +Open `files/organizations.csv` on your host machine in any spreadsheet editor or text editor. Map your SonarQube projects to the desired SonarCloud organizations. Save the file when you are done — because the directory is mounted, the container will see your changes on the next run. + +### 4. Mappings + +Generate the mapping files based on your edited organizations: + +```bash +docker run --rm -v "$(pwd)/files:/app/files" ghcr.io/sonar-solutions/sonar-reports:latest mappings +``` + +### 5. Migrate + +Run the actual migration to SonarCloud: + +```bash +docker run --rm -v "$(pwd)/files:/app/files" ghcr.io/sonar-solutions/sonar-reports:latest migrate +``` + +## Docker Compose + +For repeated use, a `docker-compose.yaml` file can save you from typing long commands. Here is an example: + +```yaml +version: "3.8" + +services: + sonar-migration: + image: ghcr.io/sonar-solutions/sonar-reports:latest + volumes: + - ./files:/app/files + # Uncomment the line below if you need client certificates (mTLS) + # - ./certs:/app/certs +``` + +Run commands through Compose like this: + +```bash +docker compose run sonar-migration extract https://your-sonarqube-server.com YOUR_SONARQUBE_TOKEN +docker compose run sonar-migration structure +docker compose run sonar-migration mappings +docker compose run sonar-migration migrate +``` + +Or use the wizard interactively: + +```bash +docker compose run sonar-migration wizard +``` + +## Client Certificates (mTLS) + +If your SonarQube Server requires mutual TLS (client certificates), mount your certificate files into the container and pass the paths as arguments: + +```bash +docker run -v ./files:/app/files -v ./certs:/app/certs ghcr.io/sonar-solutions/sonar-reports:latest extract https://your-server.com YOUR_TOKEN --pem_file_path /app/certs/client.pem --key_file_path /app/certs/client.key --cert_password YOUR_PASSWORD +``` + +Make sure the paths inside the container (`/app/certs/...`) match the volume mount target. + +## Important: Accessing localhost + +If your SonarQube Server is running on `localhost` on your host machine, the container cannot reach it using `localhost` (that would refer to the container itself). Instead, use Docker's special hostname: + +``` +host.docker.internal +``` + +For example: + +```bash +docker run --rm -v "$(pwd)/files:/app/files" ghcr.io/sonar-solutions/sonar-reports:latest extract http://host.docker.internal:9000 YOUR_TOKEN +``` + +This works on Docker Desktop for macOS and Windows. On Linux you may also need to add `--add-host=host.docker.internal:host-gateway` to the `docker run` command. + +## Building the Docker Image Locally + +If you want to build the image from source (for example, to test local changes), run: + +```bash +docker build -t sonar-reports:local . +``` + +Then use `sonar-reports:local` in place of `ghcr.io/sonar-solutions/sonar-reports:latest` in any of the commands above. + +## Building Linux Binaries with Docker + +If you need to cross-compile a standalone Linux binary (for example, from a macOS or Windows host), you can use the dedicated build Dockerfile: + +```bash +docker buildx build --platform linux/amd64 -f docker/Dockerfile.linux-build -t sonar-reports-linux-builder --load . +docker run --rm -v "$(pwd)/dist:/output" sonar-reports-linux-builder cp /app/dist/sonar-reports-linux-x86_64 /output/ +``` + +The compiled binary will appear in the `dist/` directory on your host. + +## Migration Script (Automated) + +The repository includes `scripts/execute_full_migration.sh`, a shell script that wraps the Docker commands for a fully automated end-to-end migration. To use it: + +1. Open the script and fill in your SonarQube and SonarCloud credentials. +2. Make it executable and run it: + +```bash +chmod +x scripts/execute_full_migration.sh +./scripts/execute_full_migration.sh +``` + +This is a good option when you want a repeatable, hands-off migration process. diff --git a/docs/MANUAL-MIGRATION.md b/docs/MANUAL-MIGRATION.md new file mode 100644 index 00000000..668b8087 --- /dev/null +++ b/docs/MANUAL-MIGRATION.md @@ -0,0 +1,352 @@ +# Manual Migration Guide + +## Overview + +This guide walks you through migrating from SonarQube Server to SonarQube Cloud using the manual (step-by-step) approach. The manual method gives you more control over the process by letting you run each phase separately, inspect intermediate files, and make adjustments along the way. + +If you prefer a more guided experience, there is also a `wizard` command that walks you through everything interactively. See the [Interactive Wizard Alternative](#interactive-wizard-alternative) section at the bottom of this guide. + +--- + +## Prerequisites + +Before you begin, make sure you have the following: + +- **A SonarQube Server admin token** -- This token needs the following permissions: + - Administer System + - Quality Gates (read/write) + - Quality Profiles (read/write) + - Browse on all projects you want to migrate + +- **A SonarQube Cloud enterprise account with an admin token** -- This token must have: + - Enterprise-level access + - Admin access to all target organizations in SonarQube Cloud + +### Token Permissions Summary + +| Token | Required Permissions | +|--------------------|-----------------------------------------------------------------------------| +| SonarQube Server | Administer System, Quality Gates, Quality Profiles, Browse (all projects) | +| SonarQube Cloud | Enterprise-level access, Admin on all target organizations | + +--- + +## Migration Workflow Diagram + +The migration follows this sequence of phases: + +``` +EXTRACT --> STRUCTURE --> MAPPINGS --> MIGRATE --> PIPELINES (optional) +``` + +1. **Extract** -- Pull data out of your SonarQube Server instance. +2. **Structure** -- Generate an organization structure file from the extracted data. +3. **Mappings** -- Create entity mapping files (gates, profiles, groups, etc.). +4. **Migrate** -- Push everything into SonarQube Cloud. +5. **Pipelines** -- Optionally update your CI/CD pipeline configurations. + +--- + +## Step-by-Step Guide + +### Step 1: Create a Working Directory + +Start by creating a fresh directory to hold all migration files: + +```bash +mkdir sonar-migration && cd sonar-migration +``` + +All subsequent commands assume you are running them from inside this directory. + +--- + +### Step 2: Extract Data from SonarQube Server + +This step connects to your SonarQube Server and exports all the data needed for migration. + +**Using Binary:** + +```bash +./sonar-reports- extract http://localhost:9000 YOUR_TOKEN --export_directory=./files +``` + +**Using Docker:** + +```bash +docker run --rm -v "$(pwd)/files:/app/files" ghcr.io/sonar-solutions/sonar-reports:latest extract http://your-server:9000 YOUR_TOKEN +``` + +> **Note:** If your SonarQube Server is running on `localhost`, use `host.docker.internal` instead of `localhost` when running via Docker. For example: `http://host.docker.internal:9000`. + +**Using Python:** + +```bash +python src/main.py extract http://localhost:9000 YOUR_TOKEN --export_directory=./files +``` + +#### Extract Options + +| Option | Description | Default | +|---------------------|----------------------------------------------------------|---------| +| `--concurrency` | Number of concurrent API requests | 25 | +| `--timeout` | Request timeout in seconds | 60 | +| `--extract_id` | Resume a previously started extraction by its ID | -- | +| `--extract_type` | Type of extraction to perform | -- | +| `--target_task` | Run a specific extraction task only | -- | +| `--pem_file_path` | Path to a PEM client certificate file | -- | +| `--key_file_path` | Path to a private key file for client certificate auth | -- | +| `--cert_password` | Password for the client certificate | -- | + +--- + +### Step 3: Generate Organization Structure + +This step reads the extracted data and generates an `organizations.csv` file that maps your SonarQube Server projects to SonarQube Cloud organizations. + +**Using Binary:** + +```bash +./sonar-reports- structure --export_directory=./files +``` + +**Using Docker:** + +```bash +docker run --rm -v "$(pwd)/files:/app/files" ghcr.io/sonar-solutions/sonar-reports:latest structure +``` + +**Using Python:** + +```bash +python src/main.py structure --export_directory=./files +``` + +The `--export_directory` option tells the tool where to find the extracted data and where to write the output. + +--- + +### Step 4: Edit organizations.csv + +Open the generated file at `files/organizations.csv` in any spreadsheet editor or text editor. You need to fill in the `sonarcloud_org_key` column with the key of the SonarQube Cloud organization where each group of projects should be migrated. + +Here is an example of what the file looks like: + +```csv +server_url,sonarcloud_org_key +http://localhost:9000,my-cloud-org-key +``` + +Each row represents a SonarQube Server instance. Fill in the `sonarcloud_org_key` value with the organization key from SonarQube Cloud where you want projects from that server to land. + +Save the file when you are done. + +--- + +### Step 5: Generate Entity Mappings + +This step generates mapping files that control how quality gates, quality profiles, groups, permission templates, and portfolios are migrated. + +**Using Binary:** + +```bash +./sonar-reports- mappings --export_directory=./files +``` + +**Using Docker:** + +```bash +docker run --rm -v "$(pwd)/files:/app/files" ghcr.io/sonar-solutions/sonar-reports:latest mappings +``` + +**Using Python:** + +```bash +python src/main.py mappings --export_directory=./files +``` + +This produces the following files in your `files/` directory: + +- `gates.csv` -- Quality Gate mappings +- `profiles.csv` -- Quality Profile mappings +- `groups.csv` -- Group mappings +- `templates.csv` -- Permission Template mappings +- `portfolios.csv` -- Portfolio mappings + +You can open and review (or edit) any of these files before proceeding to the migration step. + +--- + +### Step 6: Run the Migration + +Now it is time to push everything to SonarQube Cloud. You will need your SonarQube Cloud admin token and your enterprise key. + +**Using Binary:** + +```bash +./sonar-reports- migrate YOUR_CLOUD_TOKEN YOUR_ENTERPRISE_KEY --export_directory=./files +``` + +**Using Docker:** + +```bash +docker run --rm -v "$(pwd)/files:/app/files" ghcr.io/sonar-solutions/sonar-reports:latest migrate YOUR_CLOUD_TOKEN YOUR_ENTERPRISE_KEY +``` + +**Using Python:** + +```bash +python src/main.py migrate YOUR_CLOUD_TOKEN YOUR_ENTERPRISE_KEY --export_directory=./files +``` + +#### Migrate Options + +| Option | Description | Default | +|---------------------|-----------------------------------------------------------|---------------------------| +| `--url` | SonarQube Cloud URL | https://sonarcloud.io/ | +| `--edition` | Target edition | -- | +| `--concurrency` | Number of concurrent API requests | 25 | +| `--run_id` | Resume a previously started migration by its run ID | -- | +| `--target_task` | Run a specific migration task only | -- | +| `--skip_profiles` | Skip migrating quality profiles | -- | + +--- + +### Step 7: Update CI/CD Pipelines (Optional) + +If you want to automatically update your CI/CD pipeline configurations to point to SonarQube Cloud, you can use the `pipelines` command. This step is optional. + +You will need a secrets file (containing credentials for your CI/CD platform), your SonarQube Cloud token, and the SonarQube Cloud URL. + +**Using Binary:** + +```bash +./sonar-reports- pipelines SECRETS_FILE SONAR_TOKEN SONAR_URL --export_directory=./files +``` + +**Using Docker:** + +```bash +docker run --rm -v "$(pwd)/files:/app/files" ghcr.io/sonar-solutions/sonar-reports:latest pipelines SECRETS_FILE SONAR_TOKEN SONAR_URL +``` + +**Using Python:** + +```bash +python src/main.py pipelines SECRETS_FILE SONAR_TOKEN SONAR_URL --export_directory=./files +``` + +Supported CI/CD platforms: + +- GitHub Actions +- GitLab CI/CD +- Azure DevOps +- Bitbucket Pipelines + +--- + +### Step 8: Verify + +Once the migration is complete, take a few minutes to verify everything landed correctly: + +1. **Log in to SonarQube Cloud** and check: + - Quality Gates -- Are they all present and configured correctly? + - Quality Profiles -- Do they match what you had on SonarQube Server? + - Groups -- Are all user groups created with the right memberships? + - Projects -- Are all projects visible and assigned to the correct organization? + +2. **Generate a migration report** to get a summary of what was migrated: + + ```bash + ./sonar-reports- report --export_directory=./files --report_type=migration + ``` + +3. **Generate an analysis report** for a specific migration run (use the run ID from Step 6): + + ```bash + ./sonar-reports- analysis_report + ``` + +--- + +## Multi-Server Migration + +If you are migrating from multiple SonarQube Server instances, follow this approach: + +1. **Extract from each server separately** -- Run the `extract` command once for each SonarQube Server instance, each time pointing to a different server URL and token. + +2. **Run the `structure` command** -- This step automatically aggregates data from all extractions into a single `organizations.csv` file. + +3. **Edit `organizations.csv`** -- Fill in the `sonarcloud_org_key` for each server row. + +4. **Continue with `mappings` and `migrate`** as described above. The tool will handle all servers in one pass. + +--- + +## Resuming Failed Operations + +Things do not always go perfectly. If a step fails partway through, you can pick up where you left off instead of starting over: + +- **Resuming an extraction:** Use the `--extract_id` flag with the ID from the failed extraction run. + + ```bash + ./sonar-reports- extract http://localhost:9000 YOUR_TOKEN --extract_id= + ``` + +- **Resuming a migration:** Use the `--run_id` flag with the run ID from the failed migration. + + ```bash + ./sonar-reports- migrate YOUR_CLOUD_TOKEN YOUR_ENTERPRISE_KEY --run_id= + ``` + +--- + +## Output Files Reference + +Here is a summary of all the files generated during the migration process: + +| File | Description | +|--------------------------|----------------------------------------------------------------| +| `extract.json` | Metadata about the extraction (timestamps, server info, etc.) | +| `requests.log` | Log of all API requests made during extraction | +| `results.*.jsonl` | Raw extracted data in JSON Lines format (one file per entity) | +| `organizations.csv` | Server-to-organization mapping (you edit this) | +| `projects.csv` | List of all extracted projects | +| `gates.csv` | Quality Gate mappings | +| `profiles.csv` | Quality Profile mappings | +| `groups.csv` | Group mappings | +| `templates.csv` | Permission Template mappings | +| `portfolios.csv` | Portfolio mappings | + +--- + +## Interactive Wizard Alternative + +If you would rather not run each command separately, the `wizard` command provides an interactive, guided experience that walks you through the entire migration process. + +**Using Binary:** + +```bash +./sonar-reports- wizard +``` + +**Using Docker:** + +```bash +docker run -it -v ./files:/app/files ghcr.io/sonar-solutions/sonar-reports:latest wizard +``` + +**Using Python:** + +```bash +python src/main.py wizard +``` + +The wizard includes the following features: + +- **Resume support** -- If a previous wizard session was interrupted, it can pick up where it left off. +- **Client certificate prompts** -- If your SonarQube Server requires client certificate authentication, the wizard will prompt you for the necessary file paths and passwords. +- **Progress display** -- See real-time progress as each phase runs. +- **Validation** -- The wizard validates your inputs (tokens, URLs, file paths) before proceeding to each step. + +This is a great option if you are running a migration for the first time and want a bit of hand-holding through the process. diff --git a/docs/SECURITY.md b/docs/SECURITY.md new file mode 100644 index 00000000..f62c8c90 --- /dev/null +++ b/docs/SECURITY.md @@ -0,0 +1,68 @@ +# Security Best Practices + +## Overview + +This tool handles sensitive credentials (admin tokens for both SonarQube Server and SonarCloud). Follow these practices to keep your tokens safe. + +## Never Commit Secrets + +- Never commit `migration-config.json` or any config file with tokens to version control. +- Config files with secrets are already in `.gitignore`. +- Only `.example.json` files (in the `examples/` folder) should be committed. +- If you accidentally commit a token, rotate it immediately. + +## Protect Your Tokens + +- Store tokens in a secure location (password manager, secret manager). +- Tokens have full admin access — treat them as highly sensitive. +- Consider creating temporary tokens that you revoke after migration. +- Create a dedicated migration user in SonarQube Cloud with only the necessary permissions. + +## Token Permissions Reference + +| Environment | Token Type | Required Permissions | +|-------------|------------|---------------------| +| SonarQube Server | Admin Token | Administer System, Administer Quality Gates, Administer Quality Profiles, Browse all projects | +| SonarQube Cloud | User Token | Enterprise admin + Organization admin for all target orgs | + +## File Permissions + +Restrict access to config files containing tokens: + +```bash +chmod 600 migration-config.json +``` + +## Environment Variables in CI/CD + +For automated pipelines, use environment variables instead of hardcoded tokens: + +```bash +export SONAR_TOKEN="your-token" +cat > migration-config.json </requests.log` -- This file contains a detailed log of every HTTP request and response made during the migration. It is the first place to look when diagnosing issues. +- **Docker logs**: Run `docker logs ` to view the output from the Docker container. +- **Analysis report**: Run the `analysis_report ` command to generate a CSV summary of everything that happened during a migration run. This is a great way to get a high-level view of what succeeded and what failed. + +--- + +## Common Errors + +### "Token does not have sufficient permissions" + +**Cause**: The admin token you provided does not have the required permissions. + +**Solution**: + +- For your **SonarQube Server** token, ensure it has the following permissions: + - Administer System + - Quality Gates + - Quality Profiles +- For your **SonarCloud** token, ensure the user has: + - Enterprise admin privileges + - Organization admin privileges for all target organizations + +--- + +### "Organization not found" + +**Cause**: The `sonarcloud_org_key` value in your `organizations.csv` file does not match any organization in SonarCloud. + +**Solution**: + +1. Log in to SonarCloud and verify that the organization exists. +2. Double-check for typos in the org key -- even a single character difference will cause this error. +3. Make sure the organization is part of your SonarCloud enterprise. + +--- + +### "Request timeout" + +**Cause**: Large datasets or a slow network connection can cause requests to take longer than the default timeout allows. + +**Solution**: Increase the timeout value using the `--timeout` flag. For example: + +```bash +sonar-reports extract --timeout 120 +``` + +A value of 120 seconds is a good starting point for larger instances. + +--- + +### "Connection refused" or SSL errors + +**Cause**: Network connectivity issues or certificate problems, especially with self-signed certificates. + +**Solution**: + +1. Verify that the SonarQube URL is accessible from the machine running the migration tool. Try opening it in a browser or using `curl`. +2. For self-signed certificates, use the `--pem_file_path` and `--key_file_path` options to provide your certificate files. +3. If you are running the tool via Docker, make sure to mount the certificate files into the container: + +```bash +docker run -v /path/to/certs:/certs \ + sonar-reports extract \ + --pem_file_path /certs/my-cert.pem \ + --key_file_path /certs/my-key.key +``` + +--- + +### Migration task fails midway + +**Cause**: This can happen due to an API error, rate limiting, or a network interruption. + +**Solution**: Resume the migration using the `--run_id` flag with the ID from the failed run: + +```bash +sonar-reports extract --run_id +``` + +The tool keeps track of which tasks have already been completed and will skip them automatically. You do not need to start over from scratch. + +--- + +### No Projects Extracted + +If the tool completes without extracting any projects, check the following: + +- Verify that your token has admin-level permissions on the SonarQube instance. +- Confirm that projects actually exist in your SonarQube instance. +- Review the `requests.log` file for any API errors that might explain why projects were not returned. + +--- + +### Authentication Errors + +If you are seeing authentication failures: + +- Verify that your tokens are valid and have not expired. +- Be aware that SonarQube handles authentication differently depending on the version: + - **SonarQube Server < 10**: Uses Basic authentication. + - **SonarQube Server >= 10**: Uses Bearer token authentication. +- For SonarCloud, the token must belong to a user with enterprise-level admin permissions. + +--- + +### API Rate Limiting + +If you are hitting API rate limits, try reducing the concurrency and increasing the timeout: + +```bash +sonar-reports extract --concurrency 5 --timeout 120 +``` + +This slows down the number of parallel requests, which helps avoid triggering rate limits. + +--- + +### Docker: "localhost" Not Working + +If your SonarQube instance is running on your local machine and you are using Docker to run the migration tool, `localhost` will not work. Docker containers have their own network namespace, so `localhost` inside the container refers to the container itself, not your host machine. + +**Solution**: Use `host.docker.internal` instead of `localhost` in your SonarQube URL: + +```bash +docker run sonar-reports extract --sq_url http://host.docker.internal:9000 +``` + +--- + +## Reducing Memory Usage + +For large SonarQube instances with 50,000 or more projects, the migration tool can consume a significant amount of memory. To reduce memory usage, lower the concurrency setting to 10 or fewer: + +```bash +sonar-reports extract --concurrency 10 +``` + +This limits the number of operations being processed in parallel, which reduces the amount of data held in memory at any given time. + +--- + +## Resetting After a Bad Migration + +If a migration went wrong and you need to start fresh, use the `reset` command to wipe everything in the SonarCloud enterprise and start over: + +```bash +sonar-reports reset +``` + +**Warning: This is a destructive operation.** The reset command deletes all migrated data in your SonarCloud enterprise, including projects, quality profiles, quality gates, and organization configurations. Use this only when you are certain you want to start from scratch. + +--- + +## Getting Help + +If you are still stuck after reviewing this guide, here is how to get help: + +1. Check the log files located in `files/*/requests.log` for detailed error information. +2. Generate an analysis report using the `analysis_report` command to get a summary of what happened. +3. Search the repository for similar issues -- someone else may have encountered the same problem. +4. If none of the above helps, open a new issue in the repository and include the following: + - The command you ran (make sure to redact any tokens or sensitive credentials). + - The full error message you received. + - Relevant excerpts from the log files.