[elasticsearch] Add security_stats data stream#19938
Open
ebarlas wants to merge 4 commits into
Open
Conversation
Surface per-node security statistics from the Elasticsearch Security Stats API (GET /_security/stats, available in ES 9.2+) through Stack Monitoring. The first metric group is the Document Level Security (DLS) bitset cache: entries, memory, hits, misses, evictions, and hit/miss latency. The data stream mirrors the existing X-Pack per-node metrics streams (ccr, enrich): identical base-fields, ecs, and agent stream config, plus the same manifest shape. The field schema and sample event are transcribed from the merged Metricbeat security_stats metricset (elastic/beats#50674), which the agent runs to collect the data.
…tream # Conflicts: # packages/elasticsearch/changelog.yml # packages/elasticsearch/elasticsearch/transform/index_pivot/transform.yml # packages/elasticsearch/manifest.yml
Contributor
✅ Elastic Docs Style Checker (Vale)No issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
andrewkroh
added
documentation
pickypg
approved these changes
Jul 2, 2026
| # own deploy version instead of using the package-level 8.5.0 service. | ||
| image: "docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION:-9.2.0-SNAPSHOT}" | ||
| environment: | ||
| - "ES_JAVA_OPTS=-Xms1g -Xmx1g" |
Author
There was a problem hiding this comment.
Tested with 512m and applied change in commit aea224c
…docker/docker-compose.yml Co-authored-by: Chris Earle <pickypg@users.noreply.github.com>
Reduce the deploy service heap from 1g to 512m (verified sufficient for a single-node 9.2.0 cluster to boot and serve /_security/stats), per review suggestion. Skip the security_stats system test with a documented reason: the metricset requires Elastic Agent 9.3.5+/9.4.2+/9.5.0+, but package CI runs the minimum supported stack (8.10.1) where the metricset is absent, so the test collects zero documents and fails. Reword the docs "hit/miss behavior" to "cache efficiency" to satisfy the Elastic docs Vale style check.
|
✅ All changelog entries have the correct PR link. |
🚀 Benchmarks reportPackage
|
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
server |
5988.02 | 3773.58 | -2214.44 (-36.98%) | 💔 |
To see the full report comment with /test benchmark fullreport
💚 Build Succeeded
History
|
Contributor
|
Tick the box to add this pull request to the merge queue (same as
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed commit message
Surface per-node security statistics from the Elasticsearch Security
Stats API (GET /_security/stats, available in ES 9.2+) through Stack
Monitoring. The first metric group is the Document Level Security
(DLS) bitset cache: entries, memory, hits, misses, evictions, and
hit/miss latency.
The data stream mirrors the existing X-Pack per-node metrics streams
(ccr, enrich): identical base-fields, ecs, and agent stream config,
plus the same manifest shape. The field schema and sample event are
transcribed from the merged Metricbeat security_stats metricset
(elastic/beats#50674), which the agent runs to collect the data.
Checklist
changelog.ymlfile.Author's Checklist
querylogdata stream (1.21.0/1.21.1).security_statssystem test on a stack whose agent contains the metricset (9.3.5+/9.4.2+/9.5.0+). On older agents the system test will collect zero documents and fail, so a version guard may be needed.How to test this PR locally
This data stream requires an Elastic Agent build that contains the
security_statsmetricset (9.3.5+, 9.4.2+, or 9.5.0+). Older agents will collect nothing.A bare, security-enabled cluster reports the DLS bitset cache block with zero-valued counters, so the system test collects and validates documents without any DLS role/query fixture. This PR was validated with
elastic-package checkand a passingelastic-package test systemon a 9.5.0-SNAPSHOT stack.Related issues
Screenshots
N/A. This PR adds a data stream and its field schema only; no dashboards or configuration UI changes are included.