Skip to content

feat: composite primary key support in mysql_replicator (#7) - #55

Merged
y-ken merged 1 commit into
masterfrom
feature/composite-primary-key
Jun 16, 2026
Merged

feat: composite primary key support in mysql_replicator (#7)#55
y-ken merged 1 commit into
masterfrom
feature/composite-primary-key

Conversation

@y-ken

@y-ken y-ken commented Jun 16, 2026

Copy link
Copy Markdown
Owner

Summary

Implements composite primary key support for mysql_replicator, requested in #7.

primary_key now accepts a comma-separated list of columns:

<source>
  @type        mysql_replicator
  query        SELECT tenant_id, id, name FROM items
  primary_key  tenant_id,id
</source>

Change detection (insert/update/delete) then keys on the combination of
those columns, and the Elasticsearch document _id becomes their values joined
by , (e.g. 10,7). A single-column primary_key (the default id) behaves
exactly as before.

Changes

  • in_mysql_replicator: primary_key parsed as an array; a row's id is
    extract_id(row) (the array of key-column values), used as the change-
    detection hash key; missing-key check covers every component; delete records
    are emitted as {col => val} for each key column.
  • out_mysql_replicator_elasticsearch: builds the _id from one or more
    key columns via join_id. Backward compatible with single keys and with
    mysql_replicator_multi (which still uses a single-column key).
  • README documents the feature.

Relationship to #7

This reimplements the idea from #7 (by @k4200) on the current codebase
(which has diverged substantially since 2015) and fixes a bug in the
original: its update branch read table_hash[row[id]] (always nil) instead of
table_hash[id]. #7 can be closed as superseded by this PR.

Tests

  • input: primary_key defaults to ['id'], parses tenant_id,id, and
    extract_id returns single/composite values.
  • ES output: a composite-key tag produces a joined _id (10,7).

Verified locally on Ruby 3.4 (input 18 tests, ES output 19 tests, 0 failures).

Scope note: mysql_replicator_multi is unchanged (its integer-gap delete
detection assumes a single numeric key); composite support there could be a
follow-up.

🤖 Generated with Claude Code

`primary_key` now accepts a comma-separated list of columns. A row's id
becomes the array of those column values, used for change detection and as
the Elasticsearch document _id (values joined by ","). A single-column key
(the default `id`) behaves exactly as before.

- in_mysql_replicator: parse `primary_key` as an array, key the change-
  detection hash on the composite id via `extract_id`, check every component
  for nil, and emit delete records as {col => val} for each key column.
- out_mysql_replicator_elasticsearch: build the document _id from one or more
  key columns (`join_id`); backward compatible with single keys and with
  mysql_replicator_multi (which keeps a single-column primary key).

Reimplements the idea from #7 (by @k4200) on the current codebase and fixes
its `table_hash[row[id]]` lookup bug. Adds unit tests for both plugins.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@y-ken
y-ken merged commit 7d66777 into master Jun 16, 2026
9 checks passed
@y-ken
y-ken deleted the feature/composite-primary-key branch June 16, 2026 11:49
@y-ken y-ken mentioned this pull request Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant