feat: plugin-managed Elasticsearch index templates (#20, #36 geo_point) - #57
Merged
Conversation
Add template_name / template_file / template_overwrite / use_legacy_template options (names and defaults mirror fluent-plugin-elasticsearch) to install an index template on the first write. The template is a server-side rule, so Elasticsearch applies it to every new index matching its index_patterns, including future date-rolled indices, with no per-write work. This lets users control mappings that dynamic mapping cannot fix after the fact: keyword / not_analyzed fields (#20) and geo_point (#36). - use_legacy_template true (default) -> PUT /_template (ES 6.x+) - use_legacy_template false -> PUT /_index_template (ES >= 7.8) Template-install failures are logged but never abort indexing. Version detection is refactored to capture [major, minor] (needed for the composable 7.8 check) and shared with the existing _type-suppression logic. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jun 16, 2026
y-ken
added a commit
that referenced
this pull request
Jun 17, 2026
) (#58) chore: default index templates to composable API; document DECIMAL mapping Flip use_legacy_template default from true to false so new users get the modern composable _index_template API (ES >= 7.8). #57 is still unreleased, so no shipped version is affected by the default change. The "skipped on ES < 7.8" warning now hints at setting 'use_legacy_template true'. Also document (#36) that DECIMAL columns are sent to Elasticsearch as strings -- BigDecimal cannot cross Fluentd's msgpack buffer, so it is stringified, which also preserves exact precision -- and should be mapped as double or scaled_float in a template, where Elasticsearch's coerce converts the numeric string to a number at index time. No value conversion is needed in the plugin. Update the README examples to the composable template format and the unit tests to the new default. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lets
mysql_replicator_elasticsearchinstall an index template on startup so newly created indices get the desired mappingbefore the first document locks in dynamic mapping.
This unblocks the mapping-dependent requests that value coercion can't reach:
not_analyzedfields ( How to set “index”: “not_analyzed” While push data from Mysql using mysql-replicator #20)geo_point(mysql and es datatype support #36) — dynamic mapping never infers itConfig (mirrors fluent-plugin-elasticsearch)
template_name+template_filemust be set together. Option names, defaults,and template-file formats match fluent-plugin-elasticsearch, so existing
templates/knowledge carry over.
Why this is the right mechanism for date-based indices
The template is a server-side rule: once installed, Elasticsearch applies it
to every new index matching
index_patterns, including the next day'sdate-rolled index (from #54) — no per-write or per-day action needed. This is
exactly why a template (not per-index creation) is used.
Behavior / safety
logged but never abort indexing.
use_legacy_template false) require ES ≥ 7.8; on olderES it is skipped with a warning. The default legacy path works on ES 6.x+, so
this is an opt-in, backward-compatible addition (no change to core
indexing or the supported ES range) → next release is a minor (v1.4.0).
Example template_file (legacy format, default)
{ "index_patterns": ["myindex-*"], "mappings": { "properties": { "message": { "type": "keyword" }, "location": { "type": "geo_point" } } } }Tests
5 new unit tests (WebMock): legacy install (default), composable install
(
use_legacy_template false+ ES 8.x), skip on ES < 7.8, skip when the templateexists and
template_overwrite false, and the must-be-set-together config error.Verified locally on Ruby 3.4 (
24 tests, 0 failures).Follow-ups
mysql and es datatype support #36, the
geo_pointhalf is covered here, thedecimal→numerichalf remains(value coercion).
🤖 Generated with Claude Code