diff --git a/.github/actions/spelling/allow.txt b/.github/actions/spelling/allow.txt index 50ba6139..c4a31b67 100644 --- a/.github/actions/spelling/allow.txt +++ b/.github/actions/spelling/allow.txt @@ -786,4 +786,11 @@ WSS executability NTZ snowflakecomputing +aat +cicd +lpt +unconfigured +unpublishes + + diff --git a/docs/images/cicd_config_deployment_flow.png b/docs/images/cicd_config_deployment_flow.png new file mode 100644 index 00000000..ee9ac99f Binary files /dev/null and b/docs/images/cicd_config_deployment_flow.png differ diff --git a/docs/images/create_project.png b/docs/images/create_project.png new file mode 100644 index 00000000..7b26ecf0 Binary files /dev/null and b/docs/images/create_project.png differ diff --git a/docs/images/export_project.png b/docs/images/export_project.png new file mode 100644 index 00000000..eb005aae Binary files /dev/null and b/docs/images/export_project.png differ diff --git a/docs/images/git_config.png b/docs/images/git_config.png new file mode 100644 index 00000000..6344580b Binary files /dev/null and b/docs/images/git_config.png differ diff --git a/docs/images/github_actions_secrets_variables.png b/docs/images/github_actions_secrets_variables.png new file mode 100644 index 00000000..bda3ffe5 Binary files /dev/null and b/docs/images/github_actions_secrets_variables.png differ diff --git a/docs/images/github_environments_required_reviewers.png b/docs/images/github_environments_required_reviewers.png new file mode 100644 index 00000000..117bfcf8 Binary files /dev/null and b/docs/images/github_environments_required_reviewers.png differ diff --git a/docs/images/github_pat_permissions.png b/docs/images/github_pat_permissions.png new file mode 100644 index 00000000..a6c9cc13 Binary files /dev/null and b/docs/images/github_pat_permissions.png differ diff --git a/docs/images/github_runner_group_access.png b/docs/images/github_runner_group_access.png new file mode 100644 index 00000000..1fc78a44 Binary files /dev/null and b/docs/images/github_runner_group_access.png differ diff --git a/docs/images/tapdata_import_project_preview.png b/docs/images/tapdata_import_project_preview.png new file mode 100644 index 00000000..e7031092 Binary files /dev/null and b/docs/images/tapdata_import_project_preview.png differ diff --git a/docs/platform-ops/manage-project/README.md b/docs/platform-ops/manage-project/README.md new file mode 100644 index 00000000..ad0994ea --- /dev/null +++ b/docs/platform-ops/manage-project/README.md @@ -0,0 +1,9 @@ +# Manage projects + +Use TapData projects to package tasks, APIs, and their dependent connections into versioned deployment units. With GitHub and automated pipelines, teams can promote the same configuration across development, test, and production environments with traceable changes and fewer manual steps. + +This section explains how project management works, how to set up the deployment pipeline, and how to create and deploy a project. If you only need to export an archive and import it manually in another environment, see [Create and deploy a project](deploy-project.md). + +import DocCardList from '@theme/DocCardList'; + + diff --git a/docs/platform-ops/manage-project/deploy-project.md b/docs/platform-ops/manage-project/deploy-project.md new file mode 100644 index 00000000..5bf7d78d --- /dev/null +++ b/docs/platform-ops/manage-project/deploy-project.md @@ -0,0 +1,199 @@ +# Create and deploy a project + +After engineers configure TapData connections, tasks, and APIs, they can package those resources as a project, export the configuration, and deploy it to another environment. Deployment can run automatically through GitHub Actions or manually through file export and import. + +:::tip +This guide covers both automated deployment and manual import/export. If you plan to use GitHub and GitHub Actions, first complete [Set up an automated deployment pipeline](setup-pipeline.md). If you only need manual import/export, follow this guide and see [Appendix: Manually import configuration](#appendix-manually-import-configuration). +::: + + +## Example scenario + +This guide uses a common real-time data warehouse scenario. A team synchronizes data from an Oracle source database to a Doris warehouse. The team has already verified wide-table synchronization tasks and an external API in the development environment. The next step is to promote the same configuration to a testing or acceptance environment, and then to production. + +The following workflow shows how to create a project, export configuration, deploy automatically, and publish manually when needed. + +:::tip +Use the same connection names across environments. Use names that contain only letters, numbers, and underscores, such as `oracle_source`. During automated deployment, TapData matches connection names to GitHub Secrets and Variables, then injects the real address, username, and password for the target environment. +::: + +## Step 1: Create a project and select resources + +Package the team's tasks and API as one project. The project becomes the unit that you export, review, deploy, and roll back. + +1. Log in to the TapData console. In the left navigation pane, choose **Advanced Settings > Project Management**. +2. At the top of the left panel, click **+** to create a project. Enter a project name, such as `dw-pipeline`. Use the same name as the GitHub tenant repository when possible. +3. In the middle panel, switch between **Migrate Task**, **Transformation Task**, and **API**. Select `CRM_TO_DW`, `ORDER_TO_DW`, and `customer-api`, then click **Add selected >** to move them to the selected list. + + ![Create a project and select resources](../../images/create_project.png) + + :::tip + When you select tasks or APIs, TapData automatically includes dependent connections. In this example, `oracle_source` and `doris_target` are included automatically. + ::: + +4. Click **Save**. + +## Step 2: Connect the Git repository + +:::tip +If you do not want to integrate with GitHub, skip this step. In [Step 3](#step-3-export-the-configuration), choose file export and import the archive manually. If you use Git export, make sure the repository URL is writable and the personal access token has permission to write contents and create Pull Requests in the tenant repository. +::: + +Connect the TapData project to the GitHub tenant repository. After the connection is configured, exports can be pushed to the repository and opened as Pull Requests without downloading and uploading files manually. + +1. In the upper-right corner, click **Git Configuration**. +2. In the dialog box, enter the GitHub tenant repository URL and personal access token. + + ![Configure the Git repository](../../images/git_config.png) + +3. Click **Save**. + +## Step 3: Export the configuration + +Export the project configuration from the development environment and submit it to GitHub for later deployment. + +1. On the **Project Management** page, click **Export** in the upper-right corner. In the export dialog box, select the project to export. +2. Select an **Export type**. + + ![Export a project](../../images/export_project.png) + + - **Git Export**: Available after a Git repository is connected. TapData pushes the configuration files to GitHub and creates a Pull Request. Enter the following information: + + | Field | Description | + | --- | --- | + | Branch name | The system generates a branch name that starts with `feat_` and includes a timestamp. You can edit the branch name. | + | PR title | A short summary of the change for review. | + | PR description | Optional. Describe why the change is needed and what it affects. | + + - **File Export**: Downloads the configuration as a compressed archive. Use this option when Git integration is not configured. For the import steps, see [Appendix: Manually import configuration](#appendix-manually-import-configuration). + +3. In the resource list, review the tasks and APIs to be exported. If the list is correct, click **Confirm Export**. + + :::tip + Connection credentials are masked during export. Database passwords and other sensitive fields are not written to the configuration files. Enable **Rerun** only when the target environment needs the task to run a full synchronization again, for example after adding source tables or changing primary keys. For routine changes, keep the default setting so tasks continue from the last checkpoint. + ::: + +
+Exported file structure + + +Exported configuration is organized as a directory. Git export commits this directory to the repository. File export packages it as an archive. + +```text +{project-name}_tapdata_export/ +├── GroupInfo.json # Project metadata: project name, Git repository, and resource list +├── Connection/ # Connection configuration, including dependencies of tasks and APIs +│ ├── {id}_Connection_Config.json # Connection parameters with sensitive fields masked +│ └── {id}_Connection_Metadata.json # Table metadata for the connection +├── Task/ # Task configuration +│ ├── {id}_MigrateTask.json # Data replication task +│ └── {id}_SyncTask.json # Data transformation task +├── API/ # API configuration +│ ├── {id}_Module.json # API definition, including path, fields, and query logic +│ └── MetadataDefinition.json +└── User/ # User and role information for restoring operator context + ├── Users.json + ├── Roles.json + ├── RoleMappings.json + └── UserIdEmailMap.json +``` + +Notes: + +- **Connections**: TapData detects and exports connections based on task and API dependencies. You do not need to select them manually. +- **Sensitive information**: Usernames, passwords, and other credential fields are cleared during export. In automated deployment, TapData injects real values from GitHub Secrets and Variables. In manual import, update the connection values after import. +- **User data**: The `User` directory contains operator account and role information so the target environment can restore the user context. Passwords are stored as hashes and do not include plaintext values. + +
+ +## Step 4: Optional: Merge the Pull Request to deploy to development validation + +If the `dev` environment is configured, merge the Pull Request in GitHub to deploy the exported configuration to the development validation environment. This step verifies that the configuration files can be imported before you promote them further. If your process only uses testing and production, skip this stage and adjust the tenant deployment workflow accordingly. + +1. In the GitHub tenant repository, open the Pull Request created by TapData. Review the exported configuration, then click **Merge**. +2. The merge triggers the GitHub Actions `TapData Deploy` workflow and deploys the configuration to the development validation environment. +3. If the preview shows changes to connections, tasks, or APIs, approve the `deploy` gate on the **Actions** page. +4. After deployment finishes, sign in to the development validation TapData environment. Verify that `CRM_TO_DW`, `ORDER_TO_DW`, and `customer-api` were imported correctly and that the connections pass the connection test. + +## Step 5: Create a tag to deploy to testing or acceptance + +After the configuration is ready for the next validation environment, create and push a Git tag. The official template deploys tags to the testing or acceptance environment by default. + +```bash +git tag v1.0.0 +git push origin v1.0.0 +``` + +After the tag is pushed, GitHub Actions starts deployment to the corresponding validation environment. If the preview shows changes to connections, tasks, or APIs, approve the `deploy` gate before the import continues. + +When deployment finishes, complete business validation in the testing or acceptance environment. Check functional correctness, data volume, synchronization latency, and other acceptance criteria. If validation passes, continue to production deployment. + +## Step 6: Trigger production deployment manually + +After testing or acceptance validation passes, deploy the same tag to production. Using the same tag ensures that production receives the exact configuration that passed validation. + +The official tenant template does not include `prod` in the manual deployment options by default. To deploy to production, add `prod` to the tenant workflow options first. + +1. In the GitHub tenant repository, go to **Actions**, and select `TapData Deploy`. +2. Click **Run workflow**. For **Branch**, select the tag, such as `v1.0.0`. For **Target environment**, select `prod`. +3. Click **Run workflow**. If the preview shows changes to connections, tasks, or APIs, approve the `deploy` gate on the **Actions** page. +4. After deployment finishes, sign in to the target TapData environment. Verify task status and API availability, then complete the production release. + +## Roll back a deployment + +If a deployment does not behave as expected, such as when a task enters an abnormal state, roll the environment back to a previous stable tag. + +1. In the GitHub tenant repository, go to **Actions**, and select `TapData Rollback`. +2. Click **Run workflow**. Enter the target environment, such as `prod`, and the tag to roll back to, such as `v0.9.0`. +3. The rollback workflow stops the current tasks, unpublishes active APIs, clears the existing configuration, imports the configuration from the selected tag, and restores the runtime state for tasks and APIs that were active before rollback. +4. After rollback finishes, sign in to the target environment and verify the task status and API availability. + +Rollback affects only the selected target environment. Other environments are not changed. + +## FAQ + +**Q: How does project import handle existing resources?** + +Both automated deployment and manual import update existing connections, tasks, and APIs in the target environment based on the imported configuration. Unchanged resources remain unchanged. Resources that do not exist in the target environment are created. + +With GitHub-based deployment, real connection addresses, usernames, and passwords for new connections are injected from the target Environment Secrets and Variables based on connection names. With manual import, credentials are not injected automatically. Update or complete the connection values in the target environment after import. + +**Q: Why does GitHub Actions report `Could not find reusable workflow`?** + +- Check whether the Worker repository visibility is **Internal**. +- Check whether the Worker repository path in the tenant workflow has been replaced with the real value. + +**Q: The deployment succeeded, but the database password was not injected. What should I check?** + +- Check whether connection credentials are configured in the target Environment Secrets or Variables, not repository-level Secrets. +- Check whether the variable names exactly match the connection names in TapData after applying the naming rules. When credentials are configured under an Environment, do not add the environment prefix to the credential names again. +- Check whether the TapData connection name contains only letters, numbers, and underscores, and starts with a letter or underscore. If the connection name contains hyphens, spaces, or Chinese characters, GitHub Secrets and Variables might not be created with matching names. Rename the connection and export the project again. + +**Q: Git export reports `git-receive-pack not permitted` or cannot push. What should I check?** + +- Check whether the personal access token in Git configuration has read and write permissions for `Contents` and `Pull requests` in the tenant repository. +- For a fine-grained PAT, check whether **Resource owner** and **Repository access** include the current tenant repository. +- If this change includes files under `.github/workflows/`, also make sure the token has write permission for Actions or Workflows. + +**Q: Import fails and reports `tag does not exist`. What should I do?** + +The target environment might be missing a tag, Agent, or another runtime resource referenced by a task in the source environment. Create a resource with the same name in the target environment, or remove bindings that are not suitable for cross-environment deployment from the source environment and export again. + +**Q: What should I do if the import script fails?** + +- Check whether `{ENV}_TAPDATA_ACCESS_CODE` is configured correctly and is still valid. +- Open the GitHub Actions logs, find the error returned by the TapData API, and troubleshoot based on that response. + +## Appendix: Manually import configuration + +Use manual import when GitHub integration is not configured or when you need to import a specific exported version directly into a target environment. + +1. On the **Advanced Settings > Export/Import** page, click **Import**. +2. Upload the archive exported from the development environment. +3. Select a conflict handling policy, such as **Skip** or **Update existing configuration**. +4. Click **OK**. TapData validates the file format and shows an import preview, including the affected connections, tasks, and APIs. + The following image shows the import preview page, which is used to confirm the resources to be added or updated. + + ![View project import preview](../../images/tapdata_import_project_preview.png) +5. Review the preview and run the import. +6. After import finishes, sign in to the target TapData environment. Update the real connection addresses, usernames, and passwords, test the database connections, verify task status and API availability, and then start the tasks. diff --git a/docs/platform-ops/manage-project/introduction.md b/docs/platform-ops/manage-project/introduction.md new file mode 100644 index 00000000..c073d298 --- /dev/null +++ b/docs/platform-ops/manage-project/introduction.md @@ -0,0 +1,49 @@ +# Overview + +Project management packages TapData connections, tasks, APIs, and related configuration resources into versioned deployment units. With GitHub and GitHub Actions, teams can promote tested configurations across development, test, and production environments while keeping deployments auditable and repeatable. + +## Why use projects + +Enterprise TapData deployments usually include multiple environments, such as development, testing, performance validation, and production. Each environment has its own source databases, target databases, service endpoints, accounts, and network settings. + +Without a release process, engineers must recreate or update connections, tasks, and APIs in each environment by hand. As the number of resources grows, teams are more likely to miss dependencies, use the wrong parameters, or discover configuration drift only after a production incident. Manual promotion also makes it difficult to answer who changed what and when. + +TapData project management standardizes this process. It treats configuration as code, so teams can version, review, deploy, audit, and roll back data integration configurations through a controlled workflow. + +
+ +What is a project? + +In TapData, a **project** is a group of tasks, APIs, and dependent connections that share the same business goal. A project is the basic unit for unified management, export, deployment, and version tracking. + +Projects let you move related resources across environments as one package. This reduces the risk of missing dependent connections and keeps exported configuration and change history aligned with a business scope. + +
+ +## Deployment flow + +![Project deployment flow](../../images/cicd_config_deployment_flow.png) + +When a data integration requirement is ready, developers configure and verify tasks, APIs, and connections in a source environment. They then export the configuration as a TapData project and commit it to a GitHub repository for version control. + +When the configuration needs to move to the next environment, a GitHub Actions workflow starts the deployment. The workflow selects the target environment, such as testing, performance validation, or production, and loads the corresponding connection details from GitHub Environment Secrets and Variables. + +Before importing resources, the pipeline previews the differences. After approval, it imports the configuration into the target TapData environment. Engineers no longer need to sign in to each environment and recreate resources one by one. Configuration files stay versioned, credentials stay isolated, and deployment history stays traceable. + +## Key capabilities + +- **Project-based packaging**: Group tasks, APIs, and connections into one managed project. TapData includes dependencies automatically so the exported package stays complete. +- **Configuration as code**: Store exported configuration in GitHub. Each change has a Git history, supports review, and can be compared or rolled back. +- **Automated releases**: Use GitHub Actions for cross-environment deployment, with conditional triggers, difference previews, and manual approval gates. +- **Environment-specific credentials**: Reuse the same project configuration across environments while injecting the real connection values during deployment. +- **Credential isolation**: Exported configuration has sensitive fields masked. Passwords and other secrets are not committed to the repository and are managed through GitHub Secrets and Variables. +- **Version rollback**: Roll back an environment to a known stable Git tag when a release does not behave as expected. + +## Use cases + +Project management and automated deployment are useful in the following scenarios: + +- **Multi-environment promotion**: Promote data integration configuration from development to test and production without recreating resources manually. +- **Team-based delivery**: Let multiple engineers work on data integration changes through Git branches, Pull Requests, review, and workflow checks. +- **Security and audit requirements**: Keep configuration changes traceable in Git while separating credentials from exported project files. +- **Recoverable production changes**: Validate configuration in lower environments and roll back to a previous tagged version if a production release has issues. diff --git a/docs/platform-ops/manage-project/setup-pipeline.md b/docs/platform-ops/manage-project/setup-pipeline.md new file mode 100644 index 00000000..b095fcf5 --- /dev/null +++ b/docs/platform-ops/manage-project/setup-pipeline.md @@ -0,0 +1,233 @@ +# Set up an automated deployment pipeline + +Before you deploy TapData projects across environments with GitHub and GitHub Actions, prepare the repositories, environments, credentials, and self-hosted runner that the pipeline requires. This guide is intended for operations and implementation teams. + +## Prerequisites + +Prepare the following resources and information before you configure the pipeline: + +| Resource | Requirement | +| --- | --- | +| **GitHub organization** | You have administrator access to at least one GitHub organization. The Worker repository and tenant repositories can be in the same organization. | +| **TapData environments** | We recommend preparing development, testing, and production TapData environments. If you do not need a development environment, prepare at least testing and production environments. | +| **Internal runner host** | You have at least one Linux server, Ubuntu 20.04 or later recommended, that can access GitHub and the TapData service ports for all target environments. Install `git`, `bash`, `jq`, and `curl`, and register the runner with the `tapdata` label. For more information, see [Adding self-hosted runners](https://docs.github.com/en/actions/how-tos/manage-runners/self-hosted-runners/add-runners). | +| **Database connection details** | You have obtained the connection addresses, usernames, and passwords for each environment from the database administrator. | +| **Deployment approver** | You have designated at least one GitHub account to approve resource imports. | + +## Plan the pipeline + +### Repository model + +Automated deployment uses two types of GitHub repositories: + +| Repository type | Purpose | Visibility | +| --- | --- | --- | +| **Worker repository** | Stores the shared deployment scripts and reusable workflows. It is maintained by the operations team and called by tenant repositories. | Internal | +| **Tenant repository** | Stores TapData project configuration exported by a team or business domain. Use one tenant repository per team or business domain. | Internal or Private | + +### Environment model + +Plan business environments around development, testing, and production. In customer-facing communication, use the business environment names. In GitHub Environments and workflows, use the environment codes that the pipeline recognizes. + +| Business phase | Default environment code | Default trigger | Description | +| --- | --- | --- | --- | +| Development | `dev` | Automatically triggered after a merge to the `main` branch | Optional. If you do not need automated deployment to development, adjust the tenant repository workflow. | +| Testing or acceptance | `sit` | Automatically triggered when a Git tag is pushed | Optional. | +| Production | `prod` | Manually triggered | Recommended after testing or acceptance passes. Operations teams usually trigger this release manually. | +| Resource import approval gate | `deploy` | Entered automatically by the deployment workflow | Required. This is not a business environment and does not store TapData URLs or connection credentials. | + +:::tip + +This article uses "testing or acceptance environment" for the business phase. `sit` is only the default environment code in the official template. To change it to `test`, `uat`, or another custom code, update the tenant repository workflow, Worker script validation logic, Secrets and Variables prefixes, and rollback options together. + +::: + +**Adjust the deployment workflow for customer environments** + +Environment counts and naming conventions can vary by customer. When you adjust the workflow, update the GitHub Environment, URL and Access Code, tenant repository workflow, and release steps together. Do not change only one part. + +| Customer environment flow | Recommended configuration | Adjustment notes | +| --- | --- | --- | +| Development -> testing -> production | Keep the three business environments and the `deploy` approval gate. | Merges to `main` deploy to development. Tags deploy to testing. Production is released manually. | +| Development -> UAT -> production | Use the testing or acceptance environment for the customer's UAT stage. | You do not need an extra environment. Configure `DEV_*`, testing environment prefixes, and `PROD_*` variables based on the default environment code. | +| Testing -> production | Keep only testing, production, and the `deploy` approval gate. | If automated development deployment is not required, delete the `push.branches` and `push.paths` triggers from the tenant repository workflow. Keep tag and manual release triggers. | + +The following example shows where to configure `workflow_dispatch.target_env.options` in the tenant repository workflow. To release to production, add `prod` to this option. + +```yaml +# Other configuration omitted + workflow_dispatch: + inputs: + target_env: + description: 'Target environment' + required: true + type: choice + options: + - dev + - sit + - prod +# Other configuration omitted +``` + +:::tip + +Check the rollback workflow as well. In `tapdata-rollback.yml`, keep only the environments that can actually be rolled back in `workflow_dispatch.target_env.options`. + +::: + +### Permissions and security + +TapData masks sensitive fields in exported project configuration. Repositories store only business configuration. Real connection values, such as database URLs, usernames, and passwords, are stored in the corresponding GitHub Environment Secrets and Variables and are injected during deployment. + +- Protect the `main` branch in each tenant repository. Disallow direct pushes, require Pull Requests, and require code review and workflow checks before merge. +- Use an independent operations approver for the `deploy` approval gate. Developers should not approve their own deployment changes. +- Store shared values, such as `GH_DEPLOY_TOKEN`, TapData URLs, and TapData access codes, as organization-level Secrets or Variables. +- Store environment-specific connection values as Environment Secrets or Variables. + +### GitHub permissions and credentials + +Automated deployment requires access to GitHub repositories, TapData environments, and database connection credentials. We recommend splitting these settings instead of putting all information at the same level. + +| Configuration item | Location | Purpose | Recommended setting | +| --- | --- | --- | --- | +| `GH_DEPLOY_TOKEN` | GitHub organization-level or tenant repository-level Secret | Lets the runner pull Worker repository scripts, read tenant repository configuration, and push branches and create Pull Requests when TapData exports to Git. | A fine-grained PAT must have at least read access to the Worker repository and read/write access to `Contents` and `Pull requests` in the tenant repository. If it needs to write `.github/workflows/`, also grant Workflows write permission. A classic PAT can use the `repo` and `workflow` scopes. | +| `{ENV}_TAPDATA_ACCESS_CODE` | GitHub organization-level or tenant repository-level Secret | Gets the access token for the specified TapData environment. | Configure one value for each business environment, such as testing and production. | +| `{ENV}_TAPDATA_URL` | GitHub organization-level or tenant repository-level Variable | Specifies the target TapData environment URL. | Configure one value for each business environment, such as testing and production. | +| Database connection credentials | Secrets and Variables in the corresponding business Environment of the tenant repository | Injects real connection addresses, usernames, and passwords during deployment. | Configure these values under development, testing, production, and other business Environments. Do not configure them under `deploy`. | +| Runner Group access | GitHub organization **Settings > Actions > Runner groups** | Allows tenant repositories to use self-hosted runners. | Grant the Runner Group to the tenant repositories that execute deployment. Allow public repositories only when you explicitly accept the risk. | + +:::tip + +If Secrets and Variables are configured at the organization level, make sure the tenant repository is authorized to use them. If the values serve only one tenant repository, you can configure them directly in that repository. + +::: + +### Connection credential naming rules + +Secrets and Variables for TapData connections use the exported TapData connection name as the lookup key after converting letters to uppercase. GitHub Secret and Variable names support only letters, numbers, and underscores, and must start with a letter or underscore. We recommend using the same rule for TapData connection names. For example, the connection name `oracle_source` maps to the prefix `ORACLE_SOURCE`. Do not use spaces, hyphens (`-`), or Chinese characters in connection names, or deployment might not find the corresponding credentials. + +## Initialize the pipeline + +Follow these steps to turn the plan into an executable GitHub deployment pipeline. + +### Step 1: Create the GitHub repositories + +Use a two-repository model to separate deployment logic from business configuration. The **Worker repository** stores the shared deployment logic. Each **tenant repository** stores project configuration for one team or business domain and calls the Worker repository workflows. + +1. Create a copy of the official Worker repository, [tapdata/tapdata-cicd-worker](https://github.com/tapdata/tapdata-cicd-worker/tree/main), in your GitHub organization. You can use **Use this template** or clone the repository and push it to a new repository. + + Name the repository `tapdata-cicd-worker` and set its visibility to **Internal**. + + The Worker repository contains deployment orchestration, rollback orchestration, and scripts that call TapData APIs: + + ```text + tapdata-cicd-worker/ + ├── .github/workflows/ + │ ├── tapdata-deploy.yml # Core deployment workflow + │ └── tapdata-rollback.yml # Core rollback workflow + ├── conf/ + │ └── Task_Run_Order.json # Task startup order configuration + ├── scripts/ # TapData API scripts + └── tenant-template/.github/workflows/ + ├── tapdata-deploy.yml # Tenant deployment workflow template + └── tapdata-rollback.yml # Tenant rollback workflow template + ``` + +2. Create a tenant repository for the business team. Use a name that matches the TapData project name when possible, for example `user-center-sync`. Confirm that the default branch is `main`. If the default branch is still `master`, change it to `main` in GitHub repository settings, or update the watched branch in the workflow. + +3. In the tenant repository, add two lightweight workflow route files copied from `tenant-template/.github/workflows/` in the Worker repository: + + - **`tapdata-deploy.yml`**: Listens for exported configuration changes, such as changes under `*_tapdata_export/**` on the `main` branch, pushed tags, and manual dispatch events. By default, it uses the tenant repository name as the TapData project name. + - **`tapdata-rollback.yml`**: Accepts manual rollback requests by target environment and rollback tag. + + :::tip + In both copied workflow files, replace the `{WORKER_REPO}` placeholder with the Worker repository path you created, for example `your-org/tapdata-cicd-worker`. If the TapData project name differs from the tenant repository name, update the `project` input. + ::: + +4. Commit the workflow files and push them to the `main` branch of the tenant repository. + +### Step 2: Configure GitHub Secrets and Variables + +Configure repository access credentials, TapData access credentials, and service URLs so GitHub Actions can connect to and operate different TapData environments. The following example uses organization-level configuration. If the values serve only one tenant repository, you can configure the same Secrets and Variables in that repository. + +1. Sign in to GitHub with an account that has repository permissions, and go to **Settings > Developer settings > Personal access tokens**. + +2. Generate a token named `tapdata-deploy`. Use an expiration of 90 days or less, grant only the minimum permissions listed in the permissions table above, and copy the token immediately after it is created. + + :::tip + If the Worker repository and tenant repository are in the same GitHub organization, use a fine-grained PAT when possible. If you cannot assign different permissions to different repositories, use **Only select repositories** to limit the token scope to the Worker repository and the tenant repository that will actually run deployment. + ::: + +3. Go to **Organization settings > Secrets and variables > Actions**, or go to the tenant repository's **Settings > Secrets and variables > Actions**. + +4. On the **Secrets** tab, add the following encrypted values: + + ![Configure GitHub Actions Secrets and Variables](../../images/github_actions_secrets_variables.png) + + | Secret name | Description | + | --- | --- | + | `GH_DEPLOY_TOKEN` | The PAT created in the previous step. | + | `{ENV}_TAPDATA_ACCESS_CODE` for the testing environment | Access code for the testing TapData instance. If you use the official template, the name is `SIT_TAPDATA_ACCESS_CODE`. | + | `PROD_TAPDATA_ACCESS_CODE` | Access code for the production TapData instance. | + | `{ENV}_TAPDATA_ACCESS_CODE` | Optional. If the development environment is enabled, add the access code for that environment, such as `DEV_TAPDATA_ACCESS_CODE`. | + | `VAULT_ENCRYPTION_KEY` | Optional. Encrypts the `vault.json` credential file generated by the pipeline. | + +5. On the **Variables** tab, add the following plain-text values: + + | Variable name | Example value | + | --- | --- | + | `{ENV}_TAPDATA_URL` for the testing environment | Testing environment URL, such as `http://10.0.0.2:3030`. If you use the official template, the name is `SIT_TAPDATA_URL`. | + | `PROD_TAPDATA_URL` | Production environment URL. | + | `{ENV}_TAPDATA_URL` | Optional. If the development environment is enabled, add the URL for that environment, such as `DEV_TAPDATA_URL`. | + + :::tip + To get a TapData access code, sign in to the corresponding TapData environment as an administrator and go to **System Settings > User Management** to view the user information. In some environments, the user can also copy the access code from **Personal Settings** in the upper-right corner. + ::: + +### Step 3: Create Environments and configure connection values + +1. In the tenant repository, go to **Settings > Environments**. +2. Create the business Environments that are actually used, such as development, testing, and production, and create the fixed approval gate `deploy`. +3. For `deploy`, configure **Required reviewers**. This Environment acts as the resource import approval gate. We recommend adding an operations or release-owner team and enabling **Prevent self-review** so the person who triggers deployment cannot approve their own change. + + The following image shows where to configure required reviewers for the `deploy` Environment. + + ![Configure GitHub Environment required reviewers](../../images/github_environments_required_reviewers.png) + +4. `deploy` is only an approval gate. Do not configure TapData URLs, access codes, or connection credentials under it. Configure those values in testing, production, and other business Environments. +5. If production release itself requires environment-level approval, configure **Required reviewers** separately in the `prod` Environment. +6. Configure real connection values under each active environment, such as development, testing, and production. Do not add the environment prefix to connection credential names under an Environment. Use one of the following formats: + + - **URI format**: Use this for databases such as MongoDB where the connection string includes the username and password. Store it as a Secret named `{PREFIX}_URI`, such as `FDM_URI`. + - **Host and port format**: Use this for PostgreSQL, Oracle, MySQL, and similar databases. Store the address and username as Variables, and store the password as a Secret. Use names such as `{PREFIX}_URL`, `{PREFIX}_USER`, and `{PREFIX}_PASSWORD`. For a TapData connection named `oracle_source`, configure `ORACLE_SOURCE_URL`, `ORACLE_SOURCE_USER`, and `ORACLE_SOURCE_PASSWORD`. + + If multiple connections can share the same fallback values, configure `DEFAULT_URL`, `DEFAULT_USER`, and `DEFAULT_PASSWORD`. + +### Step 4: Install a self-hosted runner + +GitHub-hosted runners usually cannot access TapData services and databases in an internal network. Deploy at least one self-hosted runner on an internal Linux server. Register it at the organization level if you want multiple repositories to share it. + +1. In the GitHub organization, go to **Settings > Actions > Runners**, and click **New self-hosted runner**. +2. On the prepared Linux server, follow the GitHub setup commands to download, register, and start the runner. Add the `tapdata` label during registration. +3. If you use a Runner Group, go to **Settings > Actions > Runner groups** and confirm that the Runner Group is authorized for the tenant repository that will run deployment. Allow public repositories only when you explicitly accept the security risk. + + The following figure shows the Repository access area for Runner Group configuration. + + ![Configure Runner Group access](../../images/github_runner_group_access.png) + +4. Return to the **Runners** page and confirm that the runner status is `Idle`. Verify that it has the `tapdata` label and can access the TapData service ports for all target environments. + +## Validate the setup + +Before the first automated deployment, check the following items: + +- [ ] The Worker repository is **Internal** and contains the deployment workflow, rollback workflow, and core scripts. +- [ ] The `{WORKER_REPO}` placeholder in tenant repository workflows has been replaced with the real Worker repository path. +- [ ] The tenant repository default branch is `main`, or the watched branch in the workflow has been adjusted. +- [ ] GitHub Secrets and Variables include `GH_DEPLOY_TOKEN`, access codes, and TapData URLs for the active business environments, and the tenant repository is authorized to use them. +- [ ] The tenant repository contains the active business Environments and the fixed approval gate `deploy`. +- [ ] If the customer does not need automated deployment to development, the tenant repository workflow triggers have been adjusted. +- [ ] Connection values are configured for active environments according to the naming rules. +- [ ] At least one self-hosted runner is `Idle`, has the `tapdata` label, the Runner Group is authorized for the tenant repository, and the runner can access all target TapData environments. + +After the checklist is complete, continue with [Create and deploy a project](deploy-project.md) to package TapData configuration and release it to a target environment. diff --git a/sidebars.js b/sidebars.js index d5d7c580..31a24c64 100644 --- a/sidebars.js +++ b/sidebars.js @@ -404,6 +404,16 @@ const sidebars = { label: 'Platform Operations', link: {type: 'doc', id: 'platform-ops/README'}, items: [ + { + type: 'category', + label: 'Project Management', + link: {type: 'doc', id: 'platform-ops/manage-project/README'}, + items:[ + 'platform-ops/manage-project/introduction', + 'platform-ops/manage-project/setup-pipeline', + 'platform-ops/manage-project/deploy-project', + ] + }, { type: 'category', label: 'Production Deployment',