Skip to content

[Bug]: Elasticsearch native trace summaries undercount errors for HTTP 5xx spans #9323

Description

@HESleagacy

What happened?

Native Elasticsearch trace summaries report an incorrect error count when a span
contains only the http.response.status_code attribute with a value in the 5xx
range and has no explicit OpenTelemetry error status.

The full-trace fallback correctly classifies the span as an error, but the native
summary aggregation does not. This causes the native and fallback summary paths
to return different error counts for the same kind of trace.

Steps to reproduce

  1. Start Elasticsearch using the repository's Elasticsearch 9 compose file.

  2. Start Jaeger with native trace summaries enabled.

  3. Send one OTLP span with:

    • service.name: repro-http-5xx
    • http.response.status_code: 500
    • no explicit status.code
    • no error=true attribute
  4. Query /api/v3/trace-summaries for the repro-http-5xx service.

  5. Stop Jaeger and restart it with native summaries disabled:

    go run ./cmd/jaeger \
      --config cmd/jaeger/config-elasticsearch.yaml \
      --feature-gates=-jaeger.es.nativeTraceSummaries
  6. Run the same trace-summary query again.

The complete commands are documented above in this file.

Actual Behavior

With native summaries enabled, the response reports zero errors:

{
  "spanCount": 1,
  "errorSpanCount": 0,
  "services": [
    {
      "name": "repro-http-5xx",
      "spanCount": 1,
      "errorSpanCount": 0
    }
  ]
}

With native summaries disabled, the full-trace fallback reports one error:

{
  "spanCount": 1,
  "errorSpanCount": 1,
  "services": [
    {
      "name": "repro-http-5xx",
      "spanCount": 1,
      "errorSpanCount": 1
    }
  ]
}

The API omits zero-valued protobuf fields in the raw response; the displayed
values above were normalized with jq using // 0.

Expected behavior

Native Elasticsearch summaries should classify spans consistently with the
full-trace conversion path. A span with http.response.status_code=500 should
contribute one to both:

  • the trace-level errorSpanCount
  • the per-service errorSpanCount

Explicit status and error tags should retain their existing precedence and
behavior.

Relevant log output

These are the outputs collected during the local reproduction. The first query
was run against the fallback path and returned one error:


{
  "traceId": "6b73d1dc57a65e99076ff9039499e2c3",
  "spanCount": 1,
  "errorSpanCount": 1,
  "services": [
    {
      "name": "repro-http-5xx",
      "spanCount": 1,
      "errorSpanCount": 1
    }
  ]
}


The second query was run against the native summary path and returned zero
errors:


{
  "traceId": "6e3eb473c58607d9da14e6fdba200cb1",
  "spanCount": 1,
  "errorSpanCount": 0,
  "services": [
    {
      "name": "repro-http-5xx",
      "spanCount": 1,
      "errorSpanCount": 0
    }
  ]
}

Screenshot

Image

Additional context

  • Jaeger: local source build from main
  • Storage: Elasticsearch 9.x
  • Jaeger configuration: cmd/jaeger/config-elasticsearch.yaml
  • Endpoint: GET /api/v3/trace-summaries
  • Feature gate: jaeger.es.nativeTraceSummaries

Suspected Cause

The native Elasticsearch summary aggregation in
internal/storage/v2/elasticsearch/tracestore/core/summary.go currently counts
errors using only the explicit error=true tag.

The write path does not add that tag when a span has an HTTP 5xx attribute but no
explicit OpenTelemetry error status. In contrast,
internal/storage/v2/elasticsearch/tracestore/from_dbmodel.go infers
StatusCodeError from the HTTP status during normal span conversion.

The query service accepts native summary results directly and falls back only
when the backend returns errors.ErrUnsupported, so this mismatch is exposed to
API clients.

Relevant paths:

  • internal/storage/v2/elasticsearch/tracestore/core/summary.go
  • internal/storage/v2/elasticsearch/tracestore/from_dbmodel.go
  • cmd/jaeger/internal/extension/jaegerquery/querysvc/summary.go
  • cmd/jaeger/internal/extension/jaegerquery/querysvc/service.go

Jaeger backend version

No response

SDK

No response

Pipeline

No response

Stogage backend

No response

Operating system

Linux POP_OS

Deployment model

No response

Deployment configs

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions