Skip to content

Add JSON column support for nested object indexing - #48

Merged
y-ken merged 2 commits into
masterfrom
json-column-support
Jun 16, 2026
Merged

Add JSON column support for nested object indexing#48
y-ken merged 2 commits into
masterfrom
json-column-support

Conversation

@y-ken

@y-ken y-ken commented Jun 15, 2026

Copy link
Copy Markdown
Owner

Summary

MySQL JSON columns (MySQL 5.7.8+ / 8.x) are returned by the driver as plain
strings, so they reached Elasticsearch as escaped strings instead of nested
objects (e.g. "geometry":"{\"type\":\"Polygon\",...}"). This made them
unusable for indexing.

This PR adds an opt-in json_columns option that parses the listed columns'
values into nested objects before they are emitted, so they are indexed as real
nested JSON.

This implements the request from #18 in a type-safe, opt-in way (instead of the
unconditional string replacement proposed there). It supersedes #18.

Changes

  • mysql_replicator (single): new json_columns config_param (comma-separated).
  • mysql_replicator_multi: new json_columns column on the settings
    management table (varchar(255)). Existing installs can migrate with:
    ALTER TABLE settings ADD COLUMN `json_columns` varchar(255) DEFAULT NULL AFTER `primary_key`;
  • Malformed JSON, non-string values, and unlisted columns are left untouched, so
    the default behavior is unchanged (no side effects).
  • Documented in the README that this targets Elasticsearch — leave it empty for
    Solr or destinations that cannot store JSON objects.

Notes / scope

  • Only top-level columns are parsed (columns inside nested documents are not).
  • Diff detection is unaffected: the single plugin hashes the raw string, the
    multi plugin hashes the parsed value, and MySQL returns JSON in a normalized
    form so both are stable across scans.

Testing

  • Unit tests (Ruby 3.4): config parsing + parse_json_columns! (valid
    object/array, malformed JSON, non-string, unlisted, empty). All plugin tests
    pass, no regressions.
  • E2E (MySQL 8.4 + Elasticsearch 8.18): both single and multi pipelines
    assert a JSON column is replicated as a nested object
    (_source.profile.city == "Tokyo", _source.profile.tags == ["a","b"])
    alongside the existing insert/update/delete checks.

🤖 Generated with Claude Code

y-ken and others added 2 commits June 16, 2026 19:29
MySQL JSON columns are returned by the driver as plain strings, so they
reached Elasticsearch as escaped strings instead of nested objects. Add a
`json_columns` option that lists the columns whose values should be parsed
into nested objects before they are emitted.

- in_mysql_replicator: new `json_columns` config_param
- in_mysql_replicator_multi: new `json_columns` settings column (varchar)
- Malformed JSON, non-string and unlisted columns are left untouched, so
  the default behavior is unchanged (opt-in, no side effects)
- Document that this targets Elasticsearch; leave it empty for Solr or
  destinations that cannot store JSON objects
- Unit tests for parsing/config and single & multi E2E that assert the
  column is indexed as a nested object

Refs #18

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Align the dev container base image with the Ruby version used for unit
test verification and present in the CI matrix and Fluentd.
@y-ken
y-ken force-pushed the json-column-support branch from 9ed47fe to 7895853 Compare June 16, 2026 10:31
@y-ken
y-ken merged commit c38824d into master Jun 16, 2026
9 checks passed
@y-ken
y-ken deleted the json-column-support branch June 16, 2026 11:07
This was referenced Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant