HMPPS Probation Search - OpenSearch cluster red

[marcus-bcl]

Service name

HMPPS Probation Search

Service environment

• Dev / Development
• Staging
• Prod / Production
• Other

Impact on the service

Users are unable to search through probation case records, impacting live service.

Problem description

OpenSearch cluster is in an invalid state due to IOPS maxing out, followed by JVM memory maxing out.

We have tested and found that scaling horizontally or vertically does not help with this issue. Cluster usage has not changed. Once it gets into this state, IOPS is maxed out no matter what the max IOPS is set to.

This has been resolved in the past by rebooting the affected OpenSearch data nodes.

More info: https://mojdt.slack.com/archives/C57UPMZLY/p1786345070197429

Previously raised here: #7168 and resolved by AWS Support Case 175947532200990.

Contact person

Marcus Aspin / #probation-integration-team

[jackstockley89]

I have created a support case with AWS

Case ID
178635630800853

[jackstockley89]

Hello,

Thank you for your patience while I investigated this further.

Analysis:

1. Cluster health is currently red, and has been in this state since 2026-08-07 14:55 UTC. Kibana/Dashboards health itself is green.

2. There was a node drop on the domain — node count went from 9 → 8 at 2026-08-07 15:05 UTC, and returned to 9 nodes at 2026-08-07 23:10 UTC. This node loss correlates directly with the onset of the red status.

[+] https://eu-west-2.console.aws.amazon.com/cloudwatch/home?region=eu-west-2#metricsV2?graph=(metrics(('AWS2fES~'ClusterStatus.red~'DomainName~'cloud-platform-1cc47140~'ClientId~'754256621582~(id~'m1))('.'Nodes'.'.'.'.(id~'m2yAxis'right)))view'timeSeriesstackedfalseregion'eu-west-2stat'Maximumperiod300start'-PT72Hend'P0D)&query=~'7bAWS2fES2cClientId2cDomainName7d20DomainName3d22cloud-platform-1cc4714022*20Nodes

3. JVM memory pressure is elevated across multiple data nodes.

[+] https://eu-west-2.console.aws.amazon.com/cloudwatch/home?region=eu-west-2#metricsV2?graph=(metrics(('AWS2fES~'JVMMemoryPressure~'DomainName~'cloud-platform-1cc47140~'NodeId~'pN8sy1rtQPW2f5exfX3xUg~'ClientId~'754256621582~(id~'m1))('...'G-ChW2llSq6pPPyQqHdaVw'.'.(id~'m2))('...'P_EtpCAKQWGq9TKWD04e1w'.'.(id~'m3))('...'vAOUgWrNTFCWR5Q6-kLftQ'.'.(id~'m4))('...'WWbafeyMQ5uBUB2BEH6kpg'.'.(id~'m5))('...'nsgUZSLLSOesG0s1INKjlQ'.'.(id~'m6)))view'timeSeriesstackedfalseregion'eu-west-2stat'Maximumperiod300)&query=~'7bAWS2fES2cClientId2cDomainName2cNodeId7d20JVMMemoryPressure20DomainName3d22cloud-platform-1cc4714022

4. Recent automated snapshots are completing as PARTIAL (e.g. snapshot at 2026-08-09 14:07 UTC — 108 of 116 shards; 2026-08-08 07:07 UTC — 107 of 117 shards).

Regarding Node Reboot:

I understand from your description, and from the previous Case 175947532200990, that this condition has been cleared in the past by rebooting the affected OpenSearch data nodes.

Please note the following requirement to restart the OpenSearch process on a node, or to reboot a node, the domain must not be in a red state, and all indexes must have replicas configured.
[+] https://docs.aws.amazon.com/opensearch-service/latest/developerguide/admin-options.html

If traffic is still high at the point the node comes back up, the CPU might spike immediately, resulting in the cluster going RED again.

In order to perform the node reboot, please reduce incoming traffic (indexing and search) to the domain first. Once the reboot is complete, you can resume traffic gradually while monitoring the cluster health. If you would like to proceed with the node reboot, please share the node ID(s) on which you would like us to perform the reboot.

Looking forward to your response.

If you have any other questions or concerns, please feel free to reach out. I am happy to assist you.

Thank you

We value your feedback. Please share your experience by rating this and other correspondences in the AWS Support Center. You can rate a correspondence by selecting the stars in the top right corner of the correspondence.

Best regards,
Neha R S.
Amazon Web Services

[marcus-bcl]

jackstockley89 I've ensured all indexes have at least one replica, and I have disabled indexing traffic. I'm reluctant to disable search traffic as well as searches are still working for a number of users.

Please could we ask them to restart the following nodes one-by-one?
G-ChW2llSq6pPPyQqHdaVw
P_EtpCAKQWGq9TKWD04e1w
nsgUZSLLSOesG0s1INKjlQ

Thanks!

[jackstockley89]

Hello Jack,

Thank you for writing back to us and providing the node IDs for the reboot.

I have escalated this case to our OpenSearch internal team to perform the node reboots one by one, as they have the advanced tools required. A member from our team will contact you with further updates. Until then, I will be keeping the case in Pending Amazon Action.

Please feel free to reach out if you have any questions or concerns. I will be happy to assist you.

Thank you!

We value your feedback. Please share your experience by rating this and other correspondences in the AWS Support Center. You can rate a correspondence by selecting the stars in the top right corner of the correspondence.

Best regards,
Neha R S.
Amazon Web Services

[jackstockley89]

From the case notes, I understand that the OpenSearch cluster (cloud-platform-1cc47140) was in a RED state due to IOPS saturation followed by JVM memory pressure maxing out on multiple data nodes, and you had requested a restart of the affected nodes to resolve the issue.

Observations:

---

1. The cluster was confirmed to be in RED status since 2026-08-07 14:55 UTC, with 61 unassigned shards and 4 RED indices (contact-semantic-search-b, .plugins-ml-jobs, .plugins-ml-task, .opendistro-job-scheduler-lock).
2. A node drop event occurred on 2026-08-07 15:05 UTC (node count went from 9 --> 8), which correlated directly with the onset of the RED status. The node returned at 2026-08-07 23:10 UTC, but the cluster remained RED due to IOPS saturation preventing proper shard recovery.
3. JVM memory pressure was critically elevated on the following data nodes:

• Node G-ChW2llSq6pPPyQqHdaVw: 93.1%
• Node P_EtpCAKQWGq9TKWD04e1w: 78%
• Node nsgUZSLLSOesG0s1INKjlQ: 77.4%

4. The IOPS saturation combined with high JVM pressure created a self-reinforcing cycle where garbage collection could not free memory efficiently, preventing shard allocation and recovery.

Actions Taken:

---

1. Per your approval, we performed a rolling restart of the three affected data nodes one by one:

• Node G-ChW2llSq6pPPyQqHdaVw (restarted first — highest JVM pressure at 93.1%)
• Node nsgUZSLLSOesG0s1INKjlQ (restarted second)
• Node P_EtpCAKQWGq9TKWD04e1w (restarted third)

2. After the first two restarts, the cluster moved from RED → YELLOW, confirming all primary shards were assigned and shard recovery was in progress.
3. After all three restarts completed, the cluster continued recovering. One additional data node experienced a brief instability (joining and leaving the cluster), but it self-stabilized once shard distribution evened out across the other recovered nodes.

Current Cluster Status (as of 2026-08-10 17:15 UTC):

---

• Cluster Health: GREEN
• Total Nodes: 9/9 (3 dedicated master + 6 data nodes)
• Active Shards: 307 (100%)
• Unassigned Shards: 0
• Initializing Shards: 0
• Pending Tasks: 0
• Relocating Shards: 4 (final rebalancing in progress, will complete shortly)

Summary:

---

• Trigger: IOPS saturation on EBS volumes
• Impact: JVM memory pressure escalated to critical levels (>90%), causing a node drop and unassigned primary shards
• Resolution: Rolling restart of affected data nodes cleared the JVM heap and IOPS backlog, allowing clean shard recovery

Recommendations:

---

1. As you have observed, this is a recurring pattern on this domain. I would recommend monitoring JVM memory pressure and IOPS metrics proactively. If JVM exceeds 85% consistently, it may indicate the need for a node restart before the cluster enters a RED state.
2. You may consider setting up CloudWatch alarms for:

• JVMMemoryPressure > 85% for sustained periods
• ClusterStatus.red = 1

[+] https://docs.aws.amazon.com/opensearch-service/latest/developerguide/cloudwatch-alarms.html

3. If this pattern continues to recur, it may be worth evaluating whether the current instance type (m7g.xlarge) provides sufficient memory headroom for your workload, particularly for the contact-semantic-search-b index (154GB primary). A
   larger instance type with more JVM heap would provide additional buffer against IOPS-induced pressure spikes.

[+] https://docs.aws.amazon.com/opensearch-service/latest/developerguide/sizing-domains.html

I hope the above provided information is helpful. The cluster is now healthy and fully operational. Please feel free to reach out if you have any questions or concerns and I would be glad to assist you further.

[jackstockley89]

Hi,

This is Ahmed from AWS Support. I will assist you with your case today.

I understand that your OpenSearch domain (cloud-platform-1cc47140) in eu-west-2 has been experiencing recurring cluster health degradation, where the cluster was restored to a GREEN state on August 10 at approximately 17:15 UTC after a rolling restart of affected data nodes, but went RED again shortly after at 18:26 UTC the same day. You are requesting a reboot of two specific nodes that are not visible in the AWS Console during the degraded state, and you are also seeking a root cause investigation into why random ReadIOPS saturation occurs without any corresponding workload change on your side.

We have completed the node reboot for the affected node (P_EtpCAKQWGq9TKWD04e1w). The node rejoined the cluster at approximately 12:50 UTC and is currently stable. All 9 nodes are present with 6 data nodes active, and active shards have increased from 239 to 259 with all 110 primary shards accounted for. The unassigned shard count has dropped from 22 to approximately 10, and the cluster pending task queue has drained. The node G-ChW2llSq6pPPyQqHdaVw was not rebooted as it had already restarted approximately one hour earlier and was healthy in the cluster; rebooting it again would have unassigned its shards and added recovery reads to volumes already under pressure.

Regarding the root cause investigation, we have identified that the underlying issue is EBS read-IOPS saturation. Your domain's EBS volumes are configured as gp3 with the default baseline of 3,000 IOPS. Our observations show that ReadIOPS is consistently running at or near this ceiling, while WriteIOPS remains near zero since indexing traffic has been stopped. The IOPS throttling has been continuous for the last 27 hours across multiple data nodes.

When IOPS is saturated at the EBS level, the cluster's internal coordination operations are delayed because they depend on timely disk writes. This causes data nodes to fall behind on cluster state updates, and the cluster manager removes them from membership to maintain consistency. The removed node then rejoins automatically, but the rejoin itself generates additional disk operations on the same saturated volume, which re-triggers the cycle. This self-reinforcing pattern explains why the previous rolling restart on August 10 only held for approximately 71 minutes before the cluster degraded again[1].

To permanently resolve this recurring issue, we recommend increasing the provisioned IOPS on the domain's gp3 EBS volumes from the current 3,000 to at least 6,000, which aligns with the m7g.xlarge sustained EBS baseline. This can be done through the AWS Console or CLI by modifying the domain's EBS options[2]. The cluster should be considered stable only once all 6 data nodes maintain continuous membership for at least 30 minutes with no unassigned shards. Longer term, the more effective approach would be moving to a memory-optimized instance type (such as the r-type family) which would provide more JVM headroom and reduce the I/O pressure that leads to this cycle[3].

Should you need any clarity or have any further queries with this issue, please feel free to reach out through email on the case correspondence and I will be sure to take a look

References:
[1] https://docs.aws.amazon.com/opensearch-service/latest/developerguide/handling-errors.html
[2] https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-configuration-changes.html
[3] https://docs.aws.amazon.com/opensearch-service/latest/developerguide/sizing-domains.html

We value your feedback. Please share your experience by rating this and other correspondences in the AWS Support Center. You can rate a correspondence by selecting the stars in the top right corner of the correspondence.

Best regards,
Ahmed A.
Amazon Web Services

[jackstockley89]

Hi Marcus,

Thank you for your patience while we investigated further.

I want to provide you with an update on the current status and the immediate next step we need from your side to move forward.

We can confirm that the configuration change you submitted to increase IOPS to 10,000 and throughput to 1,000 has failed. The cluster's current instability is preventing the blue/green deployment from proceeding, as it requires healthy node membership before it can provision new infrastructure[1].

To move forward, we need two things from your side. First, please temporarily stop all search and query traffic to the domain. This will remove the disk read pressure and allow the nodes to maintain stable membership. Second, once the cluster shows all 6 data nodes holding steady, please re-trigger the IOPS configuration change so it can pass validation and be applied.

Once the IOPS increase is in place and the cluster is stable, you can gradually resume traffic while monitoring the cluster's response.

In parallel, we have escalated your question regarding the sustained ReadIOPS to our internal OpenSearch service team. Specifically, we are investigating what is generating the disk reads that began around July 31st and whether this is related to the vector storage and semantic search functionality on this cluster. Your observation that other clusters without these features have not experienced the same issue has been included in the escalation. We will update you as soon as findings are available.

References:
[1] https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-configuration-changes.html

We value your feedback. Please share your experience by rating this and other correspondences in the AWS Support Center. You can rate a correspondence by selecting the stars in the top right corner of the correspondence.

Best regards,
Ahmed A.
Amazon Web Services