Add JSON column support for nested object indexing - #48
Merged
Conversation
MySQL JSON columns are returned by the driver as plain strings, so they reached Elasticsearch as escaped strings instead of nested objects. Add a `json_columns` option that lists the columns whose values should be parsed into nested objects before they are emitted. - in_mysql_replicator: new `json_columns` config_param - in_mysql_replicator_multi: new `json_columns` settings column (varchar) - Malformed JSON, non-string and unlisted columns are left untouched, so the default behavior is unchanged (opt-in, no side effects) - Document that this targets Elasticsearch; leave it empty for Solr or destinations that cannot store JSON objects - Unit tests for parsing/config and single & multi E2E that assert the column is indexed as a nested object Refs #18 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Align the dev container base image with the Ruby version used for unit test verification and present in the CI matrix and Fluentd.
y-ken
force-pushed
the
json-column-support
branch
from
June 16, 2026 10:31
9ed47fe to
7895853
Compare
This was referenced Jun 16, 2026
Merged
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MySQL
JSONcolumns (MySQL 5.7.8+ / 8.x) are returned by the driver as plainstrings, so they reached Elasticsearch as escaped strings instead of nested
objects (e.g.
"geometry":"{\"type\":\"Polygon\",...}"). This made themunusable for indexing.
This PR adds an opt-in
json_columnsoption that parses the listed columns'values into nested objects before they are emitted, so they are indexed as real
nested JSON.
This implements the request from #18 in a type-safe, opt-in way (instead of the
unconditional string replacement proposed there). It supersedes #18.
Changes
mysql_replicator(single): newjson_columnsconfig_param (comma-separated).mysql_replicator_multi: newjson_columnscolumn on thesettingsmanagement table (
varchar(255)). Existing installs can migrate with:the default behavior is unchanged (no side effects).
Solr or destinations that cannot store JSON objects.
Notes / scope
multi plugin hashes the parsed value, and MySQL returns JSON in a normalized
form so both are stable across scans.
Testing
parse_json_columns!(validobject/array, malformed JSON, non-string, unlisted, empty). All plugin tests
pass, no regressions.
assert a JSON column is replicated as a nested object
(
_source.profile.city == "Tokyo",_source.profile.tags == ["a","b"])alongside the existing insert/update/delete checks.
🤖 Generated with Claude Code