Add Remote Info to data collection to monitor remote cluster connections - #1148
Open
rtkkroland wants to merge 2 commits into
Open
Add Remote Info to data collection to monitor remote cluster connections#1148rtkkroland wants to merge 2 commits into
rtkkroland wants to merge 2 commits into
Conversation
Signed-off-by: rtkkroland <ken.roland@arcticwolf.com>
rtkkroland
force-pushed
the
update_remote_info
branch
from
March 26, 2026 15:36
3cc56ff to
dfb15d4
Compare
Author
|
@sysadmind Moved to this new PR with all new code instead of dealing with #1099 which you were right and was based off old code. |
|
Would like to see this make it into a release |
Contributor
|
It still looks like there are merge conflicts. Can you please resolve those so that I can do a review? |
Signed-off-by: Ken Roland <ken.roland@arcticwolf.com>
There was a problem hiding this comment.
Pull request overview
Adds a new collector to export Elasticsearch cross-cluster “remote info” connection stats (/_remote/info), enabling monitoring of remote cluster connectivity/connection counts, and resolves a metric name collision observed when running both indices and node-related collectors.
Changes:
- Introduces a new
collector.remote-infocollector that scrapes/_remote/infoand exports per-remote-cluster connection metrics. - Adds fixtures and unit tests for remote info across multiple ES versions (and empty response).
- Fixes a metric namespace/name collision by moving
indexing_is_throttledfromelasticsearch_indices_*toelasticsearch_index_stats_*in the indices collector (and updates tests accordingly).
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Documents the new collector.remote-info flag and required privileges; tidies related flag rows. |
| metrics.md | Adds the new remote info metrics to the metrics catalog. |
| fixtures/remote_info/empty.json | Fixture for empty /_remote/info response. |
| fixtures/remote_info/8.0.0.json | Fixture for ES 8.x-style remote info payload. |
| fixtures/remote_info/7.15.0.json | Fixture for ES 7.15 remote info payload with multiple remotes. |
| collector/remote_info.go | New collector implementation for /_remote/info and metric descriptors. |
| collector/remote_info_test.go | Tests collector output against fixtures and error scenarios. |
| collector/indices.go | Renames one metric to index_stats namespace to avoid collisions. |
| collector/indices_test.go | Updates expected output to match renamed metric. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+139
to
+149
| // Remote Info | ||
| for remoteCluster, remoteInfo := range rir { | ||
| for _, metric := range remoteInfoMetrics { | ||
| ch <- prometheus.MustNewConstMetric( | ||
| metric.Desc, | ||
| metric.Type, | ||
| metric.Value(remoteInfo), | ||
| metric.Labels(remoteCluster)..., | ||
| ) | ||
| } | ||
| } |
Comment on lines
+110
to
+156
| func TestRemoteInfoError(t *testing.T) { | ||
| // Test error handling when endpoint is unavailable | ||
| ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { | ||
| http.Error(w, "Internal Server Error", http.StatusInternalServerError) | ||
| })) | ||
| defer ts.Close() | ||
|
|
||
| u, err := url.Parse(ts.URL) | ||
| if err != nil { | ||
| t.Fatal(err) | ||
| } | ||
|
|
||
| c, err := NewRemoteInfo(promslog.NewNopLogger(), u, http.DefaultClient) | ||
| if err != nil { | ||
| t.Fatal(err) | ||
| } | ||
|
|
||
| expected := `` | ||
|
|
||
| if err := testutil.CollectAndCompare(wrapCollector{c}, strings.NewReader(expected)); err != nil { | ||
| t.Fatal(err) | ||
| } | ||
| } | ||
|
|
||
| func TestRemoteInfoJSONParseError(t *testing.T) { | ||
| // Test JSON parse error handling | ||
| ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { | ||
| w.Write([]byte("invalid json")) | ||
| })) | ||
| defer ts.Close() | ||
|
|
||
| u, err := url.Parse(ts.URL) | ||
| if err != nil { | ||
| t.Fatal(err) | ||
| } | ||
|
|
||
| c, err := NewRemoteInfo(promslog.NewNopLogger(), u, http.DefaultClient) | ||
| if err != nil { | ||
| t.Fatal(err) | ||
| } | ||
|
|
||
| expected := `` | ||
|
|
||
| if err := testutil.CollectAndCompare(wrapCollector{c}, strings.NewReader(expected)); err != nil { | ||
| t.Fatal(err) | ||
| } | ||
| } |
Author
|
@sysadmind Ready to go? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #1128
Replaces #1099
Collect remote info from a cluster set up for cross-cluster search. Allows verification of number of connections from remotes.
NOTE: When running with:
elasticsearch_exporter --es.uri=https://localhost:9200 --es.ssl-skip-verify --es.indices --collector.snapshots --es.shards --collector.data-stream --es.timeout=60s --es.all --log.format=json --web.listen-address=:9114 --web.telemetry-path=/metricsI was getting a Panic about the duplicated metric with varying help and labels. Changing theindicestoindex_statsresolved it, but not sure if it was preferred to match the labels and help instead.