Bug Description
db_replicator can crash-loop during realtime replication when clickhouse-connect receives a Python int for a ClickHouse String column while sizing an insert block.
Steps to Reproduce
Observed on a live realtime replicator:
db_replicator resumes realtime replication from a saved binlog position.
- The transaction is found in the local binlog spool.
- A pending insert batch is flushed to ClickHouse.
- The child process exits with
TypeError, and the runner restarts it at the same position.
- The same event/batch is retried and the crash repeats.
Relevant log excerpt:
[dbrepl 442hz_prod 2026-05-12 12:37:09,664 INFO] found transaction ('mysql-bin.000095', 69985728) inside /var/lib/mysql_ch_replicator/binlog/442hz_prod/5.bin
[dbrepl 442hz_prod 2026-05-12 12:37:09,667 ERROR] unhandled exception
Traceback (most recent call last):
File "/opt/mysql_ch_replicator/app/mysql_ch_replicator/db_replicator.py", line 227, in run
self.run_realtime_replication()
File "/opt/mysql_ch_replicator/app/mysql_ch_replicator/db_replicator.py", line 285, in run_realtime_replication
self.realtime_replicator.run_realtime_replication()
File "/opt/mysql_ch_replicator/app/mysql_ch_replicator/db_replicator_realtime.py", line 73, in run_realtime_replication
self.handle_event(event)
File "/opt/mysql_ch_replicator/app/mysql_ch_replicator/db_replicator_realtime.py", line 100, in handle_event
self.upload_records_if_required(table_name=event.table_name)
File "/opt/mysql_ch_replicator/app/mysql_ch_replicator/db_replicator_realtime.py", line 339, in upload_records_if_required
self.upload_records()
File "/opt/mysql_ch_replicator/app/mysql_ch_replicator/db_replicator_realtime.py", line 355, in upload_records
self.replicator.clickhouse_api.insert(target_table_name, records, table_structure=ch_table_structure)
File "/opt/mysql_ch_replicator/app/mysql_ch_replicator/clickhouse_api.py", line 349, in insert
self.client.insert(table=full_table_name, data=records_to_insert)
File "/opt/mysql_ch_replicator/app/.venv/lib/python3.11/site-packages/clickhouse_connect/driver/client.py", line 628, in insert
context.data = data
File "/opt/mysql_ch_replicator/app/.venv/lib/python3.11/site-packages/clickhouse_connect/driver/insert.py", line 96, in data
self.block_row_count = self._calc_block_size()
File "/opt/mysql_ch_replicator/app/.venv/lib/python3.11/site-packages/clickhouse_connect/driver/insert.py", line 118, in _calc_block_size
d_size = d_type.data_size(sample)
File "/opt/mysql_ch_replicator/app/.venv/lib/python3.11/site-packages/clickhouse_connect/datatypes/base.py", line 106, in data_size
d_size = self._data_size(sample)
File "/opt/mysql_ch_replicator/app/.venv/lib/python3.11/site-packages/clickhouse_connect/datatypes/string.py", line 30, in _data_size
total += len(x)
TypeError: object of type 'int' has no len()
Expected Behavior
Realtime replication should either insert the row according to the target ClickHouse schema or fail with enough context to identify the offending table, column, row shape, and binlog position. It should not crash-loop indefinitely on one insert batch.
Actual Behavior
The db_replicator child exits with code 1, the runner restarts it, and the same binlog position is retried.
Environment
- Deployed mysql_ch_replicator Git commit:
1e556caae3dce181ae8b40b323127cc295e0657f
- Upstream tag at deployed commit:
v0.1.15 (latest upstream tag at time of report)
- Package metadata in deployed
pyproject.toml: 0.0.71
- OS: Debian GNU/Linux 12 (bookworm)
- Kernel:
Linux debDB01 6.9.10+bpo-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.9.10-1~bpo12+1 (2024-07-26) x86_64
- Python version:
3.11.2
- ClickHouse server:
25.10.2.65
- clickhouse-connect:
0.8.11
- PyMySQL:
1.1.1
- mysql-connector-python:
9.1.0
Replicator Configuration
mysql:
host: 'localhost'
port: 3306
user: '<redacted>'
password: '<redacted>'
clickhouse:
host: 'localhost'
port: 8123
user: '<redacted>'
password: '<redacted>'
binlog_replicator:
data_dir: '/var/lib/mysql_ch_replicator/binlog/'
records_per_file: 100000
databases:
- '<redacted>'
- '<redacted>'
tables: '*'
exclude_tables:
- '<redacted>'
log_level: 'info'
optimize_interval: 86400
auto_restart_interval: 3600
http_host: '0.0.0.0'
http_port: 9128
Additional Information
The crash occurred in realtime replication, not during initial replication. The parent runner process stayed alive while the per-database db_replicator child crash-looped.
Binlogs were unfortunately deleted, so I can't provide a better reproduction with the exact table / column names / DDL etc.
Bug Description
db_replicatorcan crash-loop during realtime replication whenclickhouse-connectreceives a Pythonintfor a ClickHouseStringcolumn while sizing an insert block.Steps to Reproduce
Observed on a live realtime replicator:
db_replicatorresumes realtime replication from a saved binlog position.TypeError, and the runner restarts it at the same position.Relevant log excerpt:
Expected Behavior
Realtime replication should either insert the row according to the target ClickHouse schema or fail with enough context to identify the offending table, column, row shape, and binlog position. It should not crash-loop indefinitely on one insert batch.
Actual Behavior
The
db_replicatorchild exits with code 1, the runner restarts it, and the same binlog position is retried.Environment
1e556caae3dce181ae8b40b323127cc295e0657fv0.1.15(latest upstream tag at time of report)pyproject.toml:0.0.71Linux debDB01 6.9.10+bpo-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.9.10-1~bpo12+1 (2024-07-26) x86_643.11.225.10.2.650.8.111.1.19.1.0Replicator Configuration
Additional Information
The crash occurred in realtime replication, not during initial replication. The parent runner process stayed alive while the per-database
db_replicatorchild crash-looped.Binlogs were unfortunately deleted, so I can't provide a better reproduction with the exact table / column names / DDL etc.