We have 28.53 million rows in a table called cdr_q_caller and when records are updated the new row in clickhouse takes _version value near to 5000 or something probably something to do with this:
|
limit=self.replicator.config.initial_replication_batch_size, |
|
DEFAULT_INITIAL_REPLICATION_BATCH_SIZE = 50000 |
the state file:
>>> d['tables_last_record_version']
{'cdr_a_leg': 50000, 'cdr_b_leg': 50000, 'cdr_q_agent': 2, 'cdr_q_caller': 50000, 'cdr_b_leg_distributed': 50003, 'cdr
_a_leg_distributed': 50004, 'cdr_q_caller_distributed': 50002}
the cdr_q_caller table should have this version for newer rows otherwise updates are lost when table is optimized and clickhouse merges & replaces rows based on _version column.
SELECT max(_version)
FROM freeswitch_cdr.cdr_q_caller
Query id: 38745fd8-2531-4081-bedb-c92222b7b34e
┌─max(_version)─┐
1. │ 50695 │
└───────────────┘
Proposed Solution:
We should do current_version = select max(_version) from {table_name} in clickhouse_api.insert to avoid situations where state file has got polluted and normal current_version+=1 breaks.
mysql_ch_replicator/mysql_ch_replicator/clickhouse_api.py
Line 335 in 05aa15f
We have 28.53 million rows in a table called
cdr_q_callerand when records are updated the new row in clickhouse takes _version value near to 5000 or something probably something to do with this:mysql_ch_replicator/mysql_ch_replicator/db_replicator_initial.py
Line 223 in 05aa15f
mysql_ch_replicator/mysql_ch_replicator/config.py
Line 138 in 05aa15f
the state file:
the cdr_q_caller table should have this version for newer rows otherwise updates are lost when table is optimized and clickhouse merges & replaces rows based on _version column.
Proposed Solution:
We should do
current_version = select max(_version) from {table_name}in clickhouse_api.insert to avoid situations where state file has got polluted and normal current_version+=1 breaks.