Add plugin SPI for dynamic field-type inference and dynamic-template types - #22607
Add plugin SPI for dynamic field-type inference and dynamic-template types#22607naykudev wants to merge 5 commits into
Conversation
Introduce a generic extension point so mapper plugins can participate in dynamic mapping for their own field types, without core hard-coding any plugin-specific type knowledge. A single hook, tryPluginInference(), fires in innerParseObject() before the token-type switch (covering arrays, objects, and scalars). It offers an unmapped field to two plugin mechanisms: - DynamicFieldTypeInferencer: inspects the buffered value and returns a mapping config to claim the field, or null to pass. - DynamicTemplateTypeHandler: backs a plugin-registered match_mapping_type (validated against the plugin registry when it is not a builtin XContentFieldType), completing the template config before the TypeParser builds the mapper. isConfigComplete() gates eager index-creation validation. Both receive a FieldValueParserSupplier, which hands out a fresh XContentParser over the buffered field bytes on demand — lazy, no boxing, preserves JSON fidelity. If no plugin claims the field, the buffered bytes are replayed through the existing path, so all current behavior is preserved and the hook fast-exits when no plugin is registered. Plugins register via new MapperPlugin SPI methods collected by IndicesModule into MapperRegistry. All new types are @experimentalapi. Signed-off-by: ved naykude <vnaykude@amazon.com>
PR Code Analyzer ❗AI-powered 'Code-Diff-Analyzer' found issues on commit 840945c. ⛔ Hard block: Issues at Medium severity or above will block this PR from merging.
The table above displays the top 10 most important findings. Pull Requests Author(s): Please update your Pull Request according to the report above. Repository Maintainer(s): You can Thanks. |
9eaca9f to
6f6a195
Compare
PR Reviewer Guide 🔍(Review updated until commit 000d534)Here are some key observations to aid the review process:
|
PR Code Suggestions ✨Latest suggestions up to 000d534 Explore these optional code suggestions:
Previous suggestionsSuggestions up to commit 2ae39c1
Suggestions up to commit 6dae7af
Suggestions up to commit 7863ebc
Suggestions up to commit 6f6a195
|
Per OpenSearch triage: instead of taking the first plugin claim by registration/load order, DocumentParser now consults ALL registered plugin template types and ALL inferencers for an unmapped field and throws if more than one claims it: - Two plugin dynamic-template types matching the same field -> throw. - Two field-type inferencers claiming the same field -> throw. Zero claims still fall through to existing behavior; exactly one is used as before. The exception names the conflicting claimants so the misconfiguration is clear. Adds PluginInferenceConflictTests covering both conflict paths and the single-match (no-throw) case. Signed-off-by: ved naykude <vnaykude@amazon.com>
6f6a195 to
7863ebc
Compare
|
Persistent review updated to latest commit 7863ebc |
|
Persistent review updated to latest commit 6dae7af |
|
❕ Gradle check result for 6dae7af: UNSTABLE Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #22607 +/- ##
============================================
- Coverage 71.39% 71.37% -0.03%
- Complexity 76760 76802 +42
============================================
Files 6148 6148
Lines 357951 358155 +204
Branches 52170 52208 +38
============================================
+ Hits 255556 255616 +60
- Misses 82024 82224 +200
+ Partials 20371 20315 -56 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Persistent review updated to latest commit 2ae39c1 |
|
❌ Gradle check result for 2ae39c1: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Cover the previously-untested fall-through and error paths in DocumentParser.tryPluginInference and RootObjectMapper eager validation: - Throwing inferencer is caught and the field falls through to normal inference (not a parse failure). - Inferencer returning a config with no "type", or an unknown type with no registered TypeParser, falls through instead of failing. - A template handler that reports its config complete but throws during adjustMappingConfig is treated as non-fatal at index creation (deferred). Adds PluginInferencerEdgeCaseTests and a throwing-handler case in PluginDynamicTemplateTests. Improves patch coverage on the new branches. Signed-off-by: ved naykude <vnaykude@amazon.com>
2ae39c1 to
000d534
Compare
|
Persistent review updated to latest commit 000d534 |
|
❌ Gradle check result for 000d534: null Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Description
This PR introduces a generic core interface and plugin SPI that lets any mapper plugin participate in dynamic mapping for its own field types. Core defines the extension points (
DynamicFieldTypeInferencer,DynamicTemplateTypeHandler,FieldValueParserSupplier) and the registration SPI onMapperPlugin; core itself only detects the JSON value and delegates the "is this mine, and what type is it" decision to whichever plugin registered. No plugin-specific type knowledge lives in core.The k-NN plugin is simply the first consumer of this SPI (auto-mapping vector fields and
match_mapping_type: "knn_vector"dynamic templates, in a companion PR), but the interface is deliberately generic — e.g. a geospatial plugin could register an inferencer that claims{"lat": .., "lon": ..}objects asgeo_pointthrough the exact same SPI, with zero further core changes.Motivation. Today dynamic mapping can only produce core's built-in field types. When an unmapped numeric array arrives, core parses it element-by-element and maps it as
float— it never looks at the array as a whole, and a plugin has no way to claim it. Likewise,match_mapping_typeonly accepts the built-inXContentFieldTypevalues, so a plugin type can't be targeted by a dynamic template. This change gives plugins a single, generic seam for both.What it adds (all
@ExperimentalApi):DynamicFieldTypeInferencer—inferFieldType(FieldValueParserSupplier) → Map<String,Object> | null. Called for an unmapped field with no matching template; the plugin inspects the value and returns a mapping config to claim it, ornullto pass.DynamicTemplateTypeHandler— backs a plugin-registeredmatch_mapping_type(validated against the plugin registry when it isn't a built-inXContentFieldType).adjustMappingConfig(...)completes the template config before theTypeParserbuilds the mapper;isConfigComplete(...)gates eager index-creation validation.FieldValueParserSupplier— hands the plugin a freshXContentParserover the buffered field bytes on eachget(). Lazy (no parser allocated unless the plugin reads it), preserves JSON fidelity, no boxing. Core stays free of any deserialized-representation contract.MapperPluginSPI methodsgetDynamicFieldTypeInferencers()andgetDynamicTemplateTypes(), collected intoMapperRegistrybyIndicesModuleat startup (same pattern asgetMappers()).How it hooks in. A single hook,
tryPluginInference(), fires inDocumentParser.innerParseObject()before the token-type switch — one placement covering arrays, objects, and scalars, so the SPI is genuinely generic. Resolution order: explicit mapping → plugin template → standard template → plugin inferencer → existing per-element fallback. If no plugin claims the field, the buffered bytes are replayed through the original path, so all existing behavior is preserved. When no plugin registers an inferencer or template type, the hook fast-exits before any buffering — zero overhead for clusters without such a plugin.Ambiguity is a hard error, not a silent pick. Rather than taking the first claim by registration/load order, core consults all registered plugin template types and all inferencers for an unmapped field and throws a clear
MapperParsingException(naming the conflicting claimants) if more than one claims it. Zero claims fall through as before; exactly one is used. Duplicatematch_mapping_typeregistration across plugins is rejected at node startup.Backwards compatibility. Strictly additive. The hook only fires for unmapped fields, only when a plugin is registered; existing dynamic templates, explicit mappings, and per-element inference are unchanged. All new types are
@ExperimentalApi.Scope note. This is the core SPI only. The k-NN implementation (inferencer +
knn_vectortemplate handler) is a companion PR in the k-NN repo and depends on this change; it cannot build against core until this merges and a snapshot publishes.Testing
PluginDynamicTemplateTests(28) — parsing, index-creation validation (registered/unregistered/typo/complete/incomplete/{name}placeholder), inference claims/declines for scalars/strings/booleans, precedence (explicit beats inference, builtin fallback), and fast-path exit.PluginInferenceConflictTests(3) — two inferencers claiming the same field throw; two plugin templates matching the same field throw; a single match resolves without throwing.IndicesModuleTests(+3) — plugins register inferencers/template types correctly; duplicate template-type registration throws at startup; empty by default.DynamicTemplateTests— plugin-type parse/serialization.DocumentParserTests(130) pass — regression guard for the changed hot path.All server mapper/indices suites green;
spotlessJavaCheckpasses.Related Issues
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.