CASMTRIAGE-9186 SMA 1.11.18 : No data is present in River Sensors dashboard

[sravani-sanigepalli]

Summary and Scope

The monitorSMA() health check in hmcollector used an ICMP ping to determine whether the SMA Kafka broker was reachable. This worked under Weave, which responded to ICMP for Kubernetes Services. Following the migration from Weave to Cilium, Cilium does not respond to ICMP for Service IPs, so the ping now fails unconditionally and the check permanently returns false.

However, it is worth noting that this behavior may not necessarily be related to the Weave-to-Cilium migration. It could also be related to the Kubernetes upgrade, specifically the fix for kubernetes/kubernetes#72236

Starting with Kubernetes v1.26, the following iptables rule was introduced into KUBE-IPVS-FILTER:

REJECT all -- anywhere anywhere ctstate NEW match-set KUBE-IPVS-IPS dst reject-with icmp-port-unreachable

This rule effectively means:

For any new connection to a Kubernetes Service IP that was not already handled by a specific IPVS rule above (for example, traffic targeting an unsupported or non-existent TCP/UDP port), return an ICMP Port Unreachable response.

Therefore, the observed ICMP ping failure could potentially be a consequence of the Kubernetes upgrade and the associated IPVS/iptables behavior, rather than being caused specifically by the migration from Weave to Cilium.

Since smaOK gates the polling loop in collectData(), all River telemetry polling was silently paused — no temperature, power, fan, or voltage data was published to the cray-telemetry-* Kafka topics, even though the SMA Kafka cluster was healthy and TCP-reachable on port 9092.

Fix - The check now checks the broker's TCP port (9092) instead of relying on ICMP, which accurately reflects whether hmcollector can actually produce messages to SMA.

Issues and Related PRs

• Resolves CASMTRIAGE-9186

Testing

Tested on:

• groot

Test description:

Testing:

Before the fix -

ncn-m001:~ # kubectl logs -n services -l app.kubernetes.io/name=cray-hms-hmcollector-poll --tail -1   | grep -E "SMA check|Monitoring SM
A|No brokers|Polling enabled"
{"level":"info","ts":1790171612.537989,"msg":"Monitoring SMA connectivity..."}
{"level":"info","ts":1790171612.5380785,"msg":"Polling enabled."}
{"level":"warn","ts":1790171612.550228,"msg":"SMA check: Connectivity is inactive.  Telemetry polling is paused."}

After the fix -

ncn-m001:~ # kubectl logs -n services -l app.kubernetes.io/name=cray-hms-hmcollector-poll --tail -1   | grep -E "SMA check|Monit
oring SMA|No brokers|Polling enabled"
{"level":"info","ts":1790173532.2865472,"msg":"Monitoring SMA connectivity..."}
{"level":"info","ts":1790173532.2866015,"msg":"Polling enabled."}
{"level":"warn","ts":1790173532.2982085,"msg":"SMA check: Connectivity OK.  Telemetry polling is running."}

Pull Request Checklist

• Version number(s) incremented, if applicable
• Copyrights updated
• License file intact
• Target branch correct
• Testing is appropriate and complete, if applicable
• Release Notes in docs-csm updated, if applicable

[harishkhpe]

LGTM

[mohansunderv-hpe]

LGTM