Skip to content

feat: plugin-managed Elasticsearch index templates (#20, #36 geo_point) - #57

Merged
y-ken merged 1 commit into
masterfrom
feature/index-template
Jun 16, 2026
Merged

feat: plugin-managed Elasticsearch index templates (#20, #36 geo_point)#57
y-ken merged 1 commit into
masterfrom
feature/index-template

Conversation

@y-ken

@y-ken y-ken commented Jun 16, 2026

Copy link
Copy Markdown
Owner

Summary

Lets mysql_replicator_elasticsearch install an index template on startup so newly created indices get the desired mapping
before the first document locks in dynamic mapping.

This unblocks the mapping-dependent requests that value coercion can't reach:

Config (mirrors fluent-plugin-elasticsearch)

template_name        myindex_template
template_file        /etc/fluent/myindex_template.json
template_overwrite   false
use_legacy_template  true   # true: PUT /_template (ES 6.x+); false: PUT /_index_template (ES >= 7.8)

template_name + template_file must be set together. Option names, defaults,
and template-file formats match fluent-plugin-elasticsearch
, so existing
templates/knowledge carry over.

Why this is the right mechanism for date-based indices

The template is a server-side rule: once installed, Elasticsearch applies it
to every new index matching index_patterns, including the next day's
date-rolled index
(from #54) — no per-write or per-day action needed. This is
exactly why a template (not per-index creation) is used.

Behavior / safety

  • Installed once on the first write (after version detection); failures are
    logged but never abort indexing.
  • Composable templates (use_legacy_template false) require ES ≥ 7.8; on older
    ES it is skipped with a warning. The default legacy path works on ES 6.x+, so
    this is an opt-in, backward-compatible addition (no change to core
    indexing or the supported ES range) → next release is a minor (v1.4.0).

Example template_file (legacy format, default)

{
  "index_patterns": ["myindex-*"],
  "mappings": { "properties": {
    "message":  { "type": "keyword" },
    "location": { "type": "geo_point" }
  } }
}

Tests

5 new unit tests (WebMock): legacy install (default), composable install
(use_legacy_template false + ES 8.x), skip on ES < 7.8, skip when the template
exists and template_overwrite false, and the must-be-set-together config error.
Verified locally on Ruby 3.4 (24 tests, 0 failures).

Follow-ups

🤖 Generated with Claude Code

Add template_name / template_file / template_overwrite / use_legacy_template
options (names and defaults mirror fluent-plugin-elasticsearch) to install an
index template on the first write. The template is a server-side rule, so
Elasticsearch applies it to every new index matching its index_patterns,
including future date-rolled indices, with no per-write work.

This lets users control mappings that dynamic mapping cannot fix after the
fact: keyword / not_analyzed fields (#20) and geo_point (#36).

- use_legacy_template true (default) -> PUT /_template          (ES 6.x+)
- use_legacy_template false           -> PUT /_index_template    (ES >= 7.8)

Template-install failures are logged but never abort indexing. Version
detection is refactored to capture [major, minor] (needed for the composable
7.8 check) and shared with the existing _type-suppression logic.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@y-ken
y-ken merged commit 1db4c5d into master Jun 16, 2026
9 checks passed
@y-ken
y-ken deleted the feature/index-template branch June 16, 2026 12:24
y-ken added a commit that referenced this pull request Jun 17, 2026
) (#58)

chore: default index templates to composable API; document DECIMAL mapping

Flip use_legacy_template default from true to false so new users get the
modern composable _index_template API (ES >= 7.8). #57 is still unreleased,
so no shipped version is affected by the default change. The "skipped on
ES < 7.8" warning now hints at setting 'use_legacy_template true'.

Also document (#36) that DECIMAL columns are sent to Elasticsearch as strings
-- BigDecimal cannot cross Fluentd's msgpack buffer, so it is stringified,
which also preserves exact precision -- and should be mapped as double or
scaled_float in a template, where Elasticsearch's coerce converts the numeric
string to a number at index time. No value conversion is needed in the plugin.

Update the README examples to the composable template format and the unit
tests to the new default.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@y-ken y-ken mentioned this pull request Jun 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

How to set “index”: “not_analyzed” While push data from Mysql using mysql-replicator

1 participant