| Configuration | Description |
|---|---|
| name | Unique name for the connector. Attempting to register again with the same name will fail. |
| topic.prefix | Primary key used for state storage. If multiple connectors are writing to the same ClickHouse instance, this value needs to be unique for a connector. |
| database.hostname | IP address or hostname of the source database server. |
| database.port | Integer port number of the source database server listening for client connections. |
| database.user | Name of the source database user. User needs to have replication permission. For MySQL: GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'user' IDENTIFIED BY 'password'; Refer https://debezium.io/documentation/reference/stable/connectors/mysql.html |
| database.password | Password of the source database user. |
| database.server.id | Optional, MySQL specific: Unique numeric ID for this database client. Must be unique across all replicas in the MySQL replication topology. |
| database.server.name | Optional, Logical name for the MySQL database server. |
| database.include.list | Optional list of regular expressions that match database names to be monitored. Any database name not included will be excluded from monitoring. By default all databases will be monitored. |
| table.include.list | Optional list of regular expressions that match fully-qualified table identifiers for tables to be monitored. MySQL(db_name.table_name), PostgreSQL(schema_name.table_name) |
| clickhouse.server.url | ClickHouse URL, Specify only the hostname. For TLS(use https and set port to 8443) |
| clickhouse.server.user | ClickHouse username |
| clickhouse.server.password | ClickHouse password |
| clickhouse.server.port | ClickHouse port, For TLS(use the correct port 8443 or 443) |
| snapshot.mode | "initial" -> Data that already exists in source database will be replicated. "no_data" -> Replicate data that is added/modified after the connector is started. MySQL: https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-snapshot-mode PostgreSQL: https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-property-snapshot-mode MongoDB: initial, never. https://debezium.io/documentation/reference/stable/connectors/mongodb.html |
| snapshot.locking.mode | Optional, Required for Debezium 2.7.0 and later. Specifies the mode that the connector uses to lock tables during snapshotting. Example: "minimal" |
| offset.flush.interval.ms | Optional, The number of milliseconds to wait before flushing recent offsets. Default: 5000 |
| connector.class | The Java class for the connector. MySQL -> io.debezium.connector.mysql.MySqlConnector PostgreSQL -> io.debezium.connector.postgresql.PostgresConnector MongoDB -> io.debezium.connector.mongodb.MongoDbConnector |
| offset.storage | Optional, The Java class that implements the offset storage strategy. Default: io.debezium.storage.jdbc.offset.JdbcOffsetBackingStore for JDBC-based storage. |
| offset.storage.file.filename | Optional, Offset storage file (This stores the offsets of the source database). For MySQL: mysql binlog file and position, gtid set. Make sure this file is durable and not persisted in temp directories. (Used for file-based offset storage) |
| offset.storage.jdbc.url | Optional, The JDBC URL for the database where connector offsets are to be stored. Example: jdbc:clickhouse://clickhouse:8123/altinity_sink_connector |
| offset.storage.jdbc.user | Optional, The name of the database user to be used when connecting to the database where connector offsets are stored. |
| offset.storage.jdbc.password | Optional, The password of the database user to be used when connecting to the database where connector offsets are stored. Cannot be empty - use a valid password or omit this parameter if password is not required. |
| offset.storage.jdbc.table.name | Optional, The name of the database table where connector offsets are to be stored. Example: altinity_sink_connector.replica_source_info |
| offset.storage.jdbc.table.ddl | Optional, Advanced: The DDL statement used to create the database table where connector offsets are to be stored. |
| offset.storage.jdbc.table.delete | Optional, Advanced: The DML statement used to delete from the database table where connector offsets are stored. |
| offset.storage.jdbc.table.select | Optional, Advanced: The SELECT statement used to retrieve offsets. Example: SELECT id, offset_key, offset_val FROM %s FINAL ORDER BY record_insert_ts, record_insert_seq |
| database.history.file.filename | Optional, Database History file. Make sure this file is durable and not persisted in temp directories. (Used for file-based history storage) |
| schema.history.internal | Optional, The Java class that implements the schema history strategy. Default: io.debezium.storage.jdbc.history.JdbcSchemaHistory for JDBC-based storage. |
| schema.history.internal.file.filename | Optional, Schema History file. Make sure this file is durable and not persisted in temp directories. (Used for file-based history storage) |
| schema.history.internal.jdbc.url | Optional, The JDBC URL for the database where connector schema history is to be stored. Example: jdbc:clickhouse://clickhouse:8123/altinity_sink_connector |
| schema.history.internal.jdbc.user | Optional, The name of the database user to be used when connecting to the database where connector schema history is stored. |
| schema.history.internal.jdbc.password | Optional, The password of the database user to be used when connecting to the database where connector schema history is stored. Cannot be empty - use a valid password or omit this parameter if password is not required. |
| schema.history.internal.jdbc.table.name | Optional, The name of the database table where connector schema history is to be stored. Example: altinity_sink_connector.replicate_schema_history |
| schema.history.internal.jdbc.table.ddl | Optional, Advanced: The DDL statement used to create the database table where connector schema history is to be stored. |
| schema.history.internal.store.only.captured.tables.ddl | Optional, Works with table.include.list/table.exclude.list, set this to true to avoid debezium capturing schemas of all tables. Reduces slow startup when replicating databases with large number of tables. |
| schema.history.internal.store.only.captured.databases.ddl | Optional, Works with table.include.list/table.exclude.list, set this to true to avoid debezium capturing schemas of all databases. Reduces slow startup when replicating databases with large number of tables. |
| disable.ddl | Optional, Default: false. If set to true, DDL execution will be disabled. |
| enable.ddl.snapshot | Optional, Default: false. If set to true, the DDL that is passed as part of snapshot process will be executed. Default behavior is DROP/TRUNCATE as part of snapshot is disabled. (Deprecated - use enable.snapshot.ddl) |
| enable.snapshot.ddl | Optional, Default: false. If set to true, the connector will parse the DDL statements from the initial load. Warning: This might run DROP TABLE commands. |
| ignore.ddl.regex | Optional, Regex pattern to ignore specific DDL events. Example: (?i)(ANALYZE PARTITION).* |
| disable.drop.truncate | Optional, Default: false. If set to true, the connector will ignore drop and truncate events. |
| database.allowPublicKeyRetrieval | Optional, MySQL specific: true/false. https://rmoff.net/2019/10/23/debezium-mysql-v8-public-key-retrieval-is-not-allowed/ |
| database.connectionTimeZone | Optional, Example: "US/Samoa" or "America/Chicago". Specify source database timezone for DATETIME conversions. https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-temporal-types |
| database.ssl.mode | Optional, MySQL specific: SSL mode for database connection. Example: "disabled", "required", "verify_ca", "verify_identity" |
| database.keep.alive.interval.ms | Optional, Send keepalive to database at specified interval in milliseconds. Example: 30000 (30 seconds) |
| database.connection.reconnect.backoff.ms | Optional, Initial backoff time in milliseconds for database reconnection attempts. Example: 1000 |
| database.connection.reconnect.backoff.max.ms | Optional, Maximum backoff time in milliseconds for database reconnection attempts. Example: 10000 |
| auto.create.tables | Optional, Default: false. When True, connector will create tables in ClickHouse (transformed DDL from source). |
| auto.create.tables.replicated | Optional, Default: false. If set to true, the connector will create tables with Engine set to ReplicatedReplacingMergeTree. |
| persist.raw.bytes | Optional, Default: false. Debezium.BYTES data (usually UUID) is persisted as raw bytes (CH String) if set to true. |
| binary.handling.mode | Optional, The mode for handling binary values. Possible values: bytes, base64, decode. Default: bytes |
| ignore_delete | Optional, Default: false. If set to true, the connector will ignore delete events. |
| clickhouse.datetime.timezone | Optional, Override timezone for DateTime columns in ClickHouse server. Example: "America/Chicago" |
| clickhouse.database.override.map | Optional, Configuration to override the ClickHouse database name for a given source database name. Format: "source_db:clickhouse_db". If not provided, the source database name will be used. |
| clickhouse.jdbc.params | Optional, ClickHouse JDBC configuration parameters as a comma-separated list of key-value pairs. Example: keepalive.timeout=3,max_buffer_size=1000000,socket_timeout=30000,connection_timeout=30000 |
| clickhouse.jdbc.settings | Optional, ClickHouse JDBC configuration settings as a comma-separated list of key-value pairs. Example: input_format_null_as_default=0,allow_experimental_object_type=1,insert_allow_materialized_columns=1 |
| skip_replica_start | Optional, Default: false. If set to true, replication is not started automatically. The user is expected to start replication with the sink-connector-client program. |
| restart.event.loop | Optional, Default: false. This will restart the CDC event loop if there are no messages received after timeout specified in restart.event.loop.timeout.period.secs. Workaround to restart debezium loop (in case of freeze). |
| restart.event.loop.timeout.period.secs | Optional, If the last change record (CDC) received from source database exceeds this threshold period defined in seconds, then replication is restarted. |
| buffer.flush.time.ms | Optional, Flush time of the buffer in milliseconds. The buffer is stored in memory before being flushed to ClickHouse. Example: 1000 |
| buffer.max.records | Optional, Maximum number of records for the flush buffer. Example: 10000 |
| batch.max.records | Optional, Size of the batch that is persisted to ClickHouse. Default: 100000 |
| thread.pool.size | Optional, Maximum number of threads in the thread pool for processing CDC records. Example: 10 |
| sink.connector.max.queue.size | Optional, Size of the Queue (in Memory) that holds the CDC records. Use a lower number for Out of Memory exceptions. Example: 100000 |
| single.threaded | Optional, Default: false. This mode skips the entire sink connector thread pool/queue and inserts records in batches on a single thread. Refer doc/production_setup.md |
| errors.max.retries | Optional, Default: 10. Maximum number of retry attempts for all ClickHouse operations (database creation, DDL execution, metadata queries, database connections, etc.). Example: 20 |
| metrics.enable | Optional, Default: false. Enable metrics (Prometheus target), required for Grafana Dashboard. |
| connection.pool.disable | Optional, Default: false. Set to true to disable connection pooling. |
| connection.pool.max.size | Optional, Maximum number of connections in the pool. Example: 500 |
| connection.pool.timeout | Optional, Timeout for acquiring a connection from the pool. |
| connection.pool.min.idle | Optional, Minimum number of idle connections in the pool. |
| connection.pool.max.lifetime | Optional, Maximum lifetime of a connection in the pool. |
| use.nongraceful.disconnect | Optional, Default: false. Required for fixing bug with freeze in debezium loop. Set to true for non-graceful disconnects. |
| non.default.value | Optional, Default: false. If set to true, a non-default value will be set. For example, if NULL is passed from source, the value will be NULL. |