feat(TAP-8714): mysql support auto adjust stream read size

.github/workflows/mr-ci.yaml

@@ -44,7 +44,7 @@ jobs:
       TAG_NAME: ${{ steps.set-output.outputs.TAG_NAME }}
     steps:
       - name: Checkout Tapdata Opensource
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
         with:
           repository: 'tapdata/tapdata'
           token: ${{ secrets.TAPDATA_ENT_CICD_TOKEN }}
@@ -73,33 +73,12 @@ jobs:
           echo "::set-output name=TAG_NAME::${TAG_NAME}"
 
   Sync-Code-to-Office:
-    runs-on: ubuntu-latest
-    needs:
-      - Get-Stable-Branch
-    steps:
-      - name: Checkout Tapdata Connectors Code
-        uses: actions/checkout@v3
-        with:
-          repository: 'tapdata/tapdata-connectors'
-          ref: ${{ needs.Get-Stable-Branch.outputs.CONNECTORS_BRANCH }}
-          token: ${{ secrets.TAPDATA_ENT_CICD_TOKEN }}
-          path: tapdata-connectors
-          fetch-depth: 0
-      - name: Checkout Tapdata Application
-        uses: actions/checkout@v2
-        with:
-          repository: 'tapdata/tapdata-application'
-          ref: main
-          token: ${{ secrets.TAPDATA_ENT_CICD_TOKEN }}
-          path: tapdata-application
-          fetch-depth: 0
-      - name: Push Tapdata Connectors to Gogs
-        uses: nick-fields/retry@v2
-        with:
-          timeout_minutes: 10
-          max_attempts: 3
-          command: |
-            bash tapdata-application/build/upgrade.sh --upgrade-code=true --upgrade-code-path=tapdata-connectors --gogs-project-name=tapdata-connectors
+    needs: Get-Stable-Branch
+    uses: tapdata/tapdata-application/.github/workflows/sync-code-to-office.yaml@main
+    secrets: inherit
+    with:
+      tapdata-connectors: ${{ needs.Get-Current-Branch.outputs.branch }}
+      gitee-token-user: "${{ vars.GITEE_TOKEN_USER }}"
 
   Scan-Connectors:
     runs-on: ubuntu-latest
@@ -124,7 +103,7 @@ jobs:
           wait_workflow: true
       - name: Checkout Tapdata-Application
         if: ${{ always() && steps.sonar.outcome == 'failure' }}
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
         with:
           repository: 'tapdata/tapdata-application'
           ref: "main"

README.md

@@ -1,39 +1,209 @@
-# tapdata-connectors
-Connectors For Tapdata
+# TapData Connectors
 
-## Register Connector Help
+[![JDK](https://img.shields.io/badge/JDK-17+-green.svg)](https://openjdk.org/)
+[![Maven](https://img.shields.io/badge/Maven-3.6+-blue.svg)](https://maven.apache.org/)
 
-### Use **pdk** command
+TapData Connectors is an open-source project for developing and building custom data connectors, designed for the [TapData data synchronization platform](https://github.com/tapdata/tapdata). Through this project, developers can easily build custom data connectors and register them with the TapData platform, enabling rapid integration of various heterogeneous data sources.
 
-```shell
-    java -jar pdk.jar register -a ${access_code} -t ${tm_url} [-ak ${accessKey} [-sk ${secretKey}]] [-r ${oem_type}] [-f ${filter_type}] [-h] [-X] [-l]
+## Why Choose TapData Connectors?
+
+- **Rich Connector Ecosystem**: Supports mainstream databases, cloud services, SaaS applications, and more
+- **Rapid Development**: Based on the PDK framework, new connectors can be developed in just a few hours
+- **Enterprise Features**: Supports incremental synchronization, resume from breakpoint, and more
+- **Cross-Platform Support**: Compatible with Windows, Linux, macOS, and other mainstream operating systems
+- **High Performance**: Optimized data transmission engine supporting real-time synchronization of large data volumes
+
+## 🚀 Quick Start
+
+```bash
+# 1️⃣ Clone the project
+git clone https://github.com/tapdata/tapdata-connectors.git && cd tapdata-connectors
+
+# 2️⃣ Build MySQL connector (example)
+mvn clean install -pl connectors/mysql-connector -am -DskipTests
+
+# 3️⃣ Register with TapData platform
+java -jar pdk-deploy.jar register \
+  -a YOUR_ACCESS_CODE \
+  -t http://localhost:3030 \
+  connectors/mysql-connector/target/mysql-connector-*.jar
+```
+
+✅ **Done!** Now you can use this connector in the TapData platform.
+
+## Project Structure
+
+```bash
+tapdata-connectors/
+├── connectors                # Various Java connectors
+├── connectors-common         # Common dependencies and Debezium aggregation modules
+├── connectors-javascript     # JavaScript connectors and core
+├── connectors-unpackage      # Unpackaged special connectors
+├── connectors-tdd            # Development/test-driven demonstrations
+├── file-storages             # File storage connectors
+├── tapdata-cli               # CLI tools
+```
+
+## Development Environment Setup
+
+### System Requirements
+- **JDK**: 17 or higher
+- **Maven**: 3.6+
+- **Operating System**: Windows / Linux / macOS
+
+### Environment Configuration
+
+#### Linux
+```bash
+# Ubuntu/Debian
+sudo apt-get install openjdk-17-jdk maven
+
+# CentOS/RHEL
+sudo yum install java-17-openjdk-devel maven
+```
+
+#### macOS
+```bash
+# Auto-detect and set JDK 17
+export JAVA_HOME=$(/usr/libexec/java_home -v17 2>/dev/null || echo "/usr/lib/jvm/java-17-openjdk")
+export PATH=$JAVA_HOME/bin:$PATH
+java -version
+```
+
+#### Windows
+1. Download and install [JDK 17](https://adoptium.net/)
+2. Download and extract [Maven](https://maven.apache.org/download.cgi)
+3. Configure environment variables `JAVA_HOME` and `MAVEN_HOME`
+
+## Build Connectors
+
+```bash
+mvn clean install -DskipTests \
+  -pl 'module-name'
+  -am
+```
+
+**Optional Parameters**:
+
+- `-DskipTests`: Skip tests, recommended during compilation to improve build speed
+- `-pl`: Build only selected modules; to exclude certain modules, use exclusion syntax in `-pl`, e.g., `!module-name1,!module-name2`
+- `-am`: Automatically build dependencies of selected modules
+
+After compilation, you can find the compiled connector JAR files in the corresponding module's `target/*.jar`. Then follow the instructions below to register them with the TapData platform.
+
+
+## Register Connector to TapData Platform
+
+```bash
+java -jar pdk-deploy.jar register -a ${access_code} -t ${tm_url} \
+  [-ak ${accessKey} [-sk ${secretKey}]] [-r ${oem_type}] [-f ${filter_type}] [-l] [-h] [-X] \
+  /path/to/your-connector.jar
 ```
 
-Arguments：
-  - -l  (--latest): Whether replace the latest version
-  - -a (--auth): Provide auth token to register
-  - -ak (--accessKey): Provide auth accessKey when register connector to cloud 
-  - -sk (--secretKey): Provide auth secretKey when register connector to cloud 
-  - -t (--tm): Tapdata TM url, where you want register connector to
-  - -r (--replace): Replace Config file name, value is oem type
-  - -f (--filter): The list of the Authentication types should not be skipped. If value is empty, all connectors will register all connectors. if it contains multiple, please separate them with commas
-  - -h (--help): TapData cli help
-
-A. Tip: 
-
-You must save pdk file in file system, if pdk file path is: /build/pdk
-if the connector which will be register, path is /connectors/dist/demo-connector.jar
+**Parameter Description**:
 
-B. Full command such as:
+- `-a` (`--auth`): The `access_code` in TapData. Log in to the TapData management platform, click your username in the upper right corner, select **Personal Settings** to view it
+- `-t` (`--tm`): TapData management platform login address, e.g., `http://localhost:3030`
+- `-f` (`--filter`): Register only connectors of specified authentication types, multiple values separated by commas
+- `-l` (`--latest`): Replace with the latest version
+- `-h` (`--help`): View command help information
 
- (1) when you execute command in a java run environment(linux/windows/macOS)
+The `pdk-deploy.jar` tool can be found in your TapData deployment environment's `tapdata/apps/lib` directory.
 
-```shell
-java -jar /build/pdk register -a ${access_code} -ak ${access-key} -sk ${secret-key} -r ${oem-type} -f ${need-register-connector-type} -t http://${tm-server}  /connectors/dist/demo-connector.jar
+**Execution example**:
+
+```bash
+java -jar pdk-deploy.jar register \
+  -a 3324***********8d4562f \
+  -t http://127.0.0.1:3030 \
+  connectors/starrocks-connector/target/starrocks-connector-1.0-SNAPSHOT.jar
 ```
 
-(2) when you execute command in simple environment（linux/macOS）
+When the connector is successfully registered, it will prompt "Completed". Then you can log in to the TapData management page and use this connector when creating connections.
+
+
+## FAQ
+
+Q: Getting protoc compilation errors when building on macOS?
+
+A: For Apple Silicon (macOS ARM), it's recommended to exclude `debezium-connector-postgres` and `debezium-connector-highgo` to avoid local binary compatibility issues (see module list in `connectors-common/debezium-bucket/pom.xml:71-81`).
+
+Build command example:
+
+```bash
+mvn clean install -DskipTests \
+  -pl '!connectors-common/debezium-bucket/debezium-connector-postgres,!connectors-common/debezium-bucket/debezium-connector-highgo'
+  -am
+```
+
+## How to Contribute
+
+We warmly welcome more contributors to help improve and expand connectors! **Every contribution matters**, no matter how big or small!
+
+### Contribution Methods
+
+| Contribution Type | Description | Difficulty |
+|---------|------|------|
+| **Report Bugs** | Found a bug? Feel free to submit an [Issue](https://github.com/tapdata/tapdata-connectors/issues) | ⭐ |
+| **Improve Documentation** | Fix typos, add examples, optimize descriptions | ⭐ |
+| **New Features** | Develop new connectors or enhance existing functionality | ⭐⭐⭐ |
+| **Code Optimization** | Performance optimization, code refactoring | ⭐⭐ |
+| **Test Cases** | Add test cases to improve code quality | ⭐⭐ |
+
+### PR Template Suggestions
+- **Title**: Clear and concise, e.g., `Add MongoDB connector support`
+- **Description**: Explain the background, scope of changes, and test results
+- **Checklist**:
+  - [ ] Code passes local testing
+  - [ ] Documentation has been updated
+  - [ ] Follows code standards
+
+
+## Contact Us
+
+Encountered issues? Want to connect? We provide multiple support channels!
+
+### Instant Communication
+<table>
+<tr>
+<td>
+<a href="https://join.slack.com/t/tapdatacommunity/shared_invite/zt-1biraoxpf-NRTsap0YLlAp99PHIVC9eA">
+<img src="https://img.shields.io/badge/Slack-Join%20Channel-4A154B?style=flat&logo=slack&logoColor=white" alt="Slack"/>
+</a>
+</td>
+<td>
+<a href="https://twitter.com/tapdata_daas">
+<img src="https://img.shields.io/badge/Twitter-Follow%20Updates-1DA1F2?style=flat&logo=twitter&logoColor=white" alt="Twitter"/>
+</a>
+</td>
+<td>
+<a href="https://github.com/tapdata/tapdata-connectors/issues">
+<img src="https://img.shields.io/badge/GitHub-Submit%20Issue-181717?style=flat&logo=github&logoColor=white" alt="GitHub Issues"/>
+</a>
+</td>
+</tr>
+</table>
+
+### Technical Community
+<p align="left">
+<a href="https://20778419.s21i.faiusr.com/4/2/ABUIABAEGAAg-JPfhwYonMrzlwEwZDhk.png" rel="nofollow">
+<img src="https://20778419.s21i.faiusr.com/4/2/ABUIABAEGAAg-JPfhwYonMrzlwEwZDhk.png" width="200" alt="WeChat Group"/>
+</a>
+</p>
+
+## Reference Documentation
+
+- [TapData Community Edition Open Source Repository](https://github.com/tapdata/tapdata)
+- [TapData Online Documentation](https://docs.tapdata.io/)
+
+## 🎉 Acknowledgments
+
+Thank you to all developers, users, and supporters who have contributed to TapData Connectors. It's with your support that this project continues to grow and thrive.
+
+If this project has been helpful to you, please give us a Star!
+
+**Contributors**
 
-```shell
-/build/pdk register -a ${access_code} -ak ${access-key} -sk ${secret-key} -r ${oem-type} -f ${need-register-connector-type} -t http://${tm-server}  /connectors/dist/demo-connector.jar
-```
+<a href="https://github.com/tapdata/tapdata-connectors/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=tapdata/tapdata-connectors" alt="Contributors"/>
+</a>