Skip to content

README.md

Latest commit

 

History

History
76 lines (56 loc) · 1.85 KB

README.md

File metadata and controls

76 lines (56 loc) · 1.85 KB
Raw

ch-sink-tools

PostgreSQL/MySQL to ClickHouse CDC verification and snapshot tools.

Installation

# PostgreSQL-only (primary use case):
pip install ch_sink_tools-0.2.0-py3-none-any.whl

# With MySQL support:
pip install "ch_sink_tools-0.2.0-py3-none-any.whl[mysql]"

# Everything (MySQL + pandas):
pip install "ch_sink_tools-0.2.0-py3-none-any.whl[all]"

CLI Commands

Command Description
ch-checksum Full PG→CH checksum orchestrator (top-level)
ch-pg-checksum Single-table PostgreSQL checksum
ch-pg-count PostgreSQL row count comparison
ch-ch-checksum ClickHouse-side checksum
ch-ch-count ClickHouse row count
ch-pg-dump PostgreSQL → ClickHouse snapshot loader
ch-mysql-checksum MySQL checksum (requires [mysql] extra)
ch-mysql-dump MySQL dump (requires [mysql] extra)
ch-mysql-load MySQL → ClickHouse loader (requires [mysql] extra)

Quick Start

# Run checksum verification
ch-checksum --config /path/to/config.yml

# Single-table checksum with debug
ch-checksum --config config.yml --table my_table --debug

# Count-only check
ch-pg-count --config config.yml

# Snapshot load from PostgreSQL
ch-pg-dump --pg-host pgserver --ch-host chserver --pg-database mydb

Building

pip install build
python -m build --wheel
# Produces: dist/ch_sink_tools-0.2.0-py3-none-any.whl

Deploying

# Build + deploy in one step:
./build_wheel.sh --deploy

# Or manually:
scp dist/ch_sink_tools-*.whl user@ch-server:/tmp/
ssh user@ch-server '/opt/python-dump/.venv/bin/pip install --force-reinstall /tmp/ch_sink_tools-*.whl'

Dependencies

Core (always installed):

  • clickhouse-driver>=0.2.9
  • psycopg2-binary
  • pyyaml

Optional:

  • [mysql]: pymysql, sqlalchemy>=1.4, antlr4-python3-runtime==4.11.1
  • [dataframe]: pandas