ignite3: normalize OTLP metrics via a kit-local OTel collector (stable job=ignite3 + clean names)

[rustyrazorblade]

Background

The ignite3 kit (added in #684) collects metrics via Ignite 3's native OTLP exporter, which pushes to the shared control-node OTel collector → VictoriaMetrics. This works and the dashboard is functional, but the metric surface is raw:

• Unstable job label — Ignite's OTLP exporter stamps service.name = the cluster UUID, so metrics land under job="<uuid>" (and service_name="<uuid>"). The dashboard sidesteps this by keying on the stable cluster label instead of job, but it deviates from the job=<kit> convention every other kit follows.
• Generic, scope-labeled names — Ignite emits names like ActiveCount, Succeeded, Hits, Writes disambiguated only by otel_scope_name (e.g. thread.pools.*, sql.queries). Every dashboard query must carry an otel_scope_name filter to avoid collisions, which is fragile and verbose.

Proposal

Run Ignite's OTLP push through a kit-local intermediate OTel collector (a Deployment + config shipped in the ignite3 kit — NOT the shared observability stack, preserving kit separation). The collector:

1. Relabels job/service.name from the cluster UUID → ignite3 (restores the job=<kit> convention → stable, dashboard-friendly).
2. Normalizes metric names — fold otel_scope_name into the metric name to produce clean, collision-free series, e.g. ActiveCount{otel_scope_name="thread.pools.partitions-executor"} → ignite_thread_pool_partitions_executor_active_count. Then dashboard queries are clean ignite_* selectors with no scope-filter gymnastics.
3. Optionally drops high-cardinality per-partition noise.

Flow: Ignite OTLP → kit-local collector (transform/relabel) → VictoriaMetrics.

Notes / scope

• Establishes a reusable pattern for any future OTLP-push kit (other JVM databases) that can't set a stable service.name at the source.
• New pattern for this repo — no existing kit brings its own collector; kits currently use metrics.type: scrape or push to the shared collector.
• Once the names are normalized, the ignite3 dashboard panels should be simplified to the clean ignite_* names (drop the otel_scope_name filters).
• Deferred from feat: add Apache Ignite 3 kit (ignite3) #684 intentionally — the current raw-metric dashboard works; this is the polish pass. Owner to QA the renamed metrics + updated dashboard.