Allow configuring Fleet agent index shards

[ycombinator]

Summary

Adds the node setting xpack.fleet.agents.index.number_of_shards and applies it when Elasticsearch creates the .fleet-agents system index. The setting defaults to one, preserving current behavior unless an environment opts in. Existing .fleet-agents indices are unchanged because their primary shard count is static.

Adds unit coverage for both the default and an explicitly configured four-primary layout.

Experiment and rollout

The Serverless QA and Staging overrides are in https://github.com/elastic/serverless-gitops/pull/155710. If the controlled Staging experiment is successful toward achieving 100k-agent scale, we will promote the setting override to Production. If it is unsuccessful, we will revert both this Elasticsearch change and the serverless-gitops overrides.

100k scale-test evidence

Tracking issue: https://github.com/elastic/ingest-dev/issues/8895

Test run: https://buildkite.com/elastic/observability-perf/builds/6690

During the large failure wave in this run, the sole .fleet-agents-7[0] primary became an Elasticsearch indexing hotspot. Its indexing thread-pool utilization reached 100.1% with 8,829 ms queue latency. Elasticsearch logged 18,358 slow bulk transport operations (p50 7.25 s, p95 13.36 s, max 13.81 s) and 527 .fleet-agents search-shard notification failures.

The same interval produced 22,148 Fleet Server context-deadline errors, 77,889 rejected bulk dispatches, and waves of 25,645 HTTP 429, 16,627 HTTP 500, and 2,224 HTTP 503 responses. Allowing multiple .fleet-agents primaries lets configured environments distribute this indexing load across more shard paths.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[elasticsearchmachine]

Hi @ycombinator, I've created a changelog YAML for you.

[github-actions]

🔍 Preview links for changed docs

⏳ Building and deploying preview... View progress

This comment will be updated with preview links when the build is complete.

[github-actions]

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

• Check out the cumulative docs guidelines
• Reach out in the #docs Slack channel