Skip to content

[BUG] Realtime replication crash-loop: int value inserted into ClickHouse String column #261

Description

@ldf442

Bug Description

db_replicator can crash-loop during realtime replication when clickhouse-connect receives a Python int for a ClickHouse String column while sizing an insert block.

Steps to Reproduce

Observed on a live realtime replicator:

  1. db_replicator resumes realtime replication from a saved binlog position.
  2. The transaction is found in the local binlog spool.
  3. A pending insert batch is flushed to ClickHouse.
  4. The child process exits with TypeError, and the runner restarts it at the same position.
  5. The same event/batch is retried and the crash repeats.

Relevant log excerpt:

[dbrepl 442hz_prod 2026-05-12 12:37:09,664 INFO] found transaction ('mysql-bin.000095', 69985728) inside /var/lib/mysql_ch_replicator/binlog/442hz_prod/5.bin
[dbrepl 442hz_prod 2026-05-12 12:37:09,667 ERROR] unhandled exception
Traceback (most recent call last):
  File "/opt/mysql_ch_replicator/app/mysql_ch_replicator/db_replicator.py", line 227, in run
    self.run_realtime_replication()
  File "/opt/mysql_ch_replicator/app/mysql_ch_replicator/db_replicator.py", line 285, in run_realtime_replication
    self.realtime_replicator.run_realtime_replication()
  File "/opt/mysql_ch_replicator/app/mysql_ch_replicator/db_replicator_realtime.py", line 73, in run_realtime_replication
    self.handle_event(event)
  File "/opt/mysql_ch_replicator/app/mysql_ch_replicator/db_replicator_realtime.py", line 100, in handle_event
    self.upload_records_if_required(table_name=event.table_name)
  File "/opt/mysql_ch_replicator/app/mysql_ch_replicator/db_replicator_realtime.py", line 339, in upload_records_if_required
    self.upload_records()
  File "/opt/mysql_ch_replicator/app/mysql_ch_replicator/db_replicator_realtime.py", line 355, in upload_records
    self.replicator.clickhouse_api.insert(target_table_name, records, table_structure=ch_table_structure)
  File "/opt/mysql_ch_replicator/app/mysql_ch_replicator/clickhouse_api.py", line 349, in insert
    self.client.insert(table=full_table_name, data=records_to_insert)
  File "/opt/mysql_ch_replicator/app/.venv/lib/python3.11/site-packages/clickhouse_connect/driver/client.py", line 628, in insert
    context.data = data
  File "/opt/mysql_ch_replicator/app/.venv/lib/python3.11/site-packages/clickhouse_connect/driver/insert.py", line 96, in data
    self.block_row_count = self._calc_block_size()
  File "/opt/mysql_ch_replicator/app/.venv/lib/python3.11/site-packages/clickhouse_connect/driver/insert.py", line 118, in _calc_block_size
    d_size = d_type.data_size(sample)
  File "/opt/mysql_ch_replicator/app/.venv/lib/python3.11/site-packages/clickhouse_connect/datatypes/base.py", line 106, in data_size
    d_size = self._data_size(sample)
  File "/opt/mysql_ch_replicator/app/.venv/lib/python3.11/site-packages/clickhouse_connect/datatypes/string.py", line 30, in _data_size
    total += len(x)
TypeError: object of type 'int' has no len()

Expected Behavior

Realtime replication should either insert the row according to the target ClickHouse schema or fail with enough context to identify the offending table, column, row shape, and binlog position. It should not crash-loop indefinitely on one insert batch.

Actual Behavior

The db_replicator child exits with code 1, the runner restarts it, and the same binlog position is retried.

Environment

  • Deployed mysql_ch_replicator Git commit: 1e556caae3dce181ae8b40b323127cc295e0657f
  • Upstream tag at deployed commit: v0.1.15 (latest upstream tag at time of report)
  • Package metadata in deployed pyproject.toml: 0.0.71
  • OS: Debian GNU/Linux 12 (bookworm)
  • Kernel: Linux debDB01 6.9.10+bpo-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.9.10-1~bpo12+1 (2024-07-26) x86_64
  • Python version: 3.11.2
  • ClickHouse server: 25.10.2.65
  • clickhouse-connect: 0.8.11
  • PyMySQL: 1.1.1
  • mysql-connector-python: 9.1.0

Replicator Configuration

mysql:
  host: 'localhost'
  port: 3306
  user: '<redacted>'
  password: '<redacted>'

clickhouse:
  host: 'localhost'
  port: 8123
  user: '<redacted>'
  password: '<redacted>'

binlog_replicator:
  data_dir: '/var/lib/mysql_ch_replicator/binlog/'
  records_per_file: 100000

databases:
  - '<redacted>'
  - '<redacted>'

tables: '*'

exclude_tables:
  - '<redacted>'

log_level: 'info'
optimize_interval: 86400
auto_restart_interval: 3600
http_host: '0.0.0.0'
http_port: 9128

Additional Information

The crash occurred in realtime replication, not during initial replication. The parent runner process stayed alive while the per-database db_replicator child crash-looped.

Binlogs were unfortunately deleted, so I can't provide a better reproduction with the exact table / column names / DDL etc.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions