Allocation: Shared cache capacity monitor implemenation

[surya-estc]

• Add monitor implementation by tracking the previous and current commitments per node
• Calls reroute once there is a valid over-subscription condition
• Implements reroute throttling
• Closes https://github.com/elastic/elasticsearch-team/issues/4588
• Add unit tests
• Will update ITs once Allocation: Implement canRemain function for shared cache capacity decider #154863 is merged

[Copilot]

Pull request overview

This PR adds a stateless allocation monitor that watches per-node shared cache commitments (from ClusterInfo) and triggers allocation reroutes when search nodes newly cross the shared-cache capacity watermarks, along with a new dynamic throttle setting and accompanying unit tests.

Changes:

• Introduces SharedCacheCapacityMonitor to detect high/low watermark transitions in node cache commitments and call RerouteService#reroute.
• Wires the monitor into StatelessPlugin and exposes a new dynamic setting to rate-limit reroute requests.
• Adds SharedCacheCapacityMonitorTests covering transition classification, reroute triggering rules, throttling, and live setting updates.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
x-pack/plugin/stateless/src/main/java/org/elasticsearch/xpack/stateless/allocation/SharedCacheCapacityMonitor.java	New monitor implementation that evaluates cache commitment watermark transitions and triggers reroutes with throttling.
x-pack/plugin/stateless/src/main/java/org/elasticsearch/xpack/stateless/allocation/SharedCacheCapacityAllocationDecider.java	Adds cluster.routing.allocation.shared_cache_capacity.reroute_interval setting used by the monitor.
x-pack/plugin/stateless/src/main/java/org/elasticsearch/xpack/stateless/StatelessPlugin.java	Registers the new monitor as a ClusterInfoService listener and exposes the new setting via getSettings().
x-pack/plugin/stateless/src/test/java/org/elasticsearch/xpack/stateless/allocation/SharedCacheCapacityMonitorTests.java	New unit tests validating monitor decisions, reroute triggering behavior, throttling, and dynamic settings updates.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[surya-estc]

@nicktindall

1. Right now a new high-watermark or low-watermark transition inside the throttle interval is delayed because the monitor leaves lastNodeCommitments unchanged and lets a future ClusterInfo publication trigger the reroute after the throttle expires. That seems acceptable if we’re deliberately choosing a simple global minimum interval, since ClusterInfo should refresh soon anyway. Or, should we always bypass the throttle for any new transitions regardless of when the last reroute was triggered?

2. After a successful reroute, if the same nodes remain over the high watermark and ClusterInfo does not materially change, this monitor will not reroute again just because reroute_interval elapsed. That seems reasonable to me because if the first reroute did not change the commitments, another reroute with the same inputs likely will not achieve anything. I just want to confirm that this is the intended behavior.