CEL input: Add OTel tracing

[chrisberkhout]

Proposed commit message

CEL input: Add OTel tracing (#)

Instruments the CEL Input with OpenTelemetry tracing. Sampling is 100% -
all operation is covered. By default no exporter is set up and traces
will not be exported. Export can be configured to go to the console or
to an OTLP endpoint using the `grpc` (default) or `http/protobuf`
protocols.

Typically OTel tracing considers the whole process to be the "resource".
However, in this case the resource is the input instance. For that
reason a trace provider is created specifically for the input instance
and it is not explicitly set as the global tracer provider.

There is an extra environment variable to override any other
configuration and disable export for a specific input:
`BEATS_OTEL_TRACES_DISABLE=cel`.

Spans covering HTTP requests are enriched with attributes for request
and response headers, with values automatically (but configurably)
redacted to protect sensitive data.

Normal request logging and Filebeat logs will include span and trace IDs
that allow correlation with the OTel data. This is done in any location
to which we can pass a logger from the trace creation site. Other
Filebeat logging will lack the IDs. Because logger attributes are
append-only we pass around a logger with modified attributes rather than
modify attributes in a global logger.

Normal request logging had unused functionality for including a
`trace.id` field. That has been removed in favor of an OTel-specific
implementation that adds `trace.id` and `span.id` if there is a current,
valid span.

Requests initiated by CEL will have spans added by `otelhttp` and will
identify the correct parent span using trace data from the request
context. Since the relevant eval-time context is not propagated to those
requests by mito, cel-go[1] or oauth2[2], `ContextInjector` is used to
rewrite each request to include the current context as it is processed.

[1]: https://github.com/google/cel-go/issues/557
[2]: https://github.com/golang/oauth2/issues/262

There were a couple of things for which the initial approach changed:

• Use of https://pkg.go.dev/go.opentelemetry.io/contrib/exporters/autoexport to interpret OTel environment variables and set up the exporter was removed in favor of manual handling, which seems to be standard when using the Go SDK (unlike implementations in some other languages).
• The context with OTel tracing data needs to be propagated the HTTP client used by CEL so that HTTP spans are attached to the correct parent span. That was initially done with a change in Mito: Add HTTPWithContextFnOpts so requests can have eval-time context mito#118. That has been closed to avoid changing Mito. Now it is done in the CEL Input by having ContextInjector rewrite requests in the client used by CEL, which also solves the problem for OAuth2 requests.

There are some differences from the attribute and other names given in the planning document:

• cel.periodic.program_count
  → Changed to cel.periodic.execution_count to match cel.program.execution.
• cel.program.batch_count
  → Removed. It would only indicate whether an execution returned any events or not. Any other batching is internal to the CEL evaluation.
• cel.{periodic,program}.success
  → Removed, in favor of span status.
• cel.program.error_message
  → Not set. Uses SetStatus and RecordError instead.
• BEATS_OTEL_TRACING_DISABLE
  → Changed to BEATS_OTEL_TRACES_DISABLE to match OTEL_TRACES_EXPORTER and OTEL_EXPORTER_OTLP_TRACES_*.

Handling of span-specific context and loggers is somewhat cumbersome. Refactoring to extract separate functions from run for separate stages of processing will help to tidy this up and is planned as follow-up work: #48464.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

How to test this PR locally

You can use otel-desktop-viewer as simple receiver and viewer of OTel traces:

# Install it
go install github.com/CtrlSpice/otel-desktop-viewer@latest

# Run it. It will open its web UI
otel-desktop-viewer

# In another terminal, set it as the destination for OTel traces
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4317

In the terminal with those environment variables set, you can run the input with an example that includes OAuth2 and multiple requests per period, like this:

(cd x-pack/filebeat && go build) && ./x-pack/filebeat/filebeat run -c <(echo '
filebeat.inputs:
- type: cel
  enabled: true
  id: cel-1
  interval: 5s
  resource.url: https://api.ipify.org/?format=json&passwd=mysecretword
  program: |
    get(state.url).Body.as(body, state.with({
        "events": [body.decode_json()],
        "want_more": int(state.?runcount.orValue(1)) % 3 != 0,
        "runcount": int(state.?runcount.orValue(1)) + 1,
    }))
  resource.tracer.enable: true
  resource.tracer.filename: "x-pack/filebeat/logs/cel/http-request-trace-cel-*.ndjson"
  auth.oauth2.enabled: true
  auth.oauth2.client.id: someclientid
  auth.oauth2.client.secret: someclientsecret
  auth.oauth2.scopes: scope.me
  auth.oauth2.token_url: https://oauth-mock.mock.beeceptor.com/oauth/token/github
  auth.oauth2.endpoint_params:
    grant_type: client_credentials
  otel.trace.redacted:
    - User-Agent
  otel.trace.unredacted:
    - Authorization
output.elasticsearch:
  hosts: ["https://elasticsearch:9200"]
  username: "elastic"
  password: "changeme"
  protocol: "https"
  ssl.verification_mode: "none"
  preset: balanced
logging.level: debug
logging.to_stderr: true
')

You can also use Elastic Observability to receive and view OTel traces, but it involves a bit more setup.

Bring up the Elastic Stack:

elastic-package stack up -v

In Kibana, go to "Management > Integrations" and go to the "APM" integration page. Click "Manage APM integration in Fleet", then "Add Elastic APM". Under "Configure integration > Integration settings > General > Server configuration", change the Host and URL settings to use '0.0.0.0' instead of 'localhost'. Under "Where to add this integration?", choose "Existing hosts > Elastic Agent (elastic-package)". Then click "Save and continue".

Now, back in the terminal, find the IP address of the agent container.

docker ps # confirm the agent container name is elastic-package-stack-elastic-agent-1
AGENT="elastic-package-stack-elastic-agent-1"
AGENT_IP=$(docker inspect "$AGENT" \
  --format '{{ (index .NetworkSettings.Networks "elastic-package-stack_default").IPAddress }}')
echo "$AGENT_IP" # confirm the IP was found

Use that as the destination for OTel traces:

export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=grpc
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://$AGENT_IP:8200"

Then from the terminal with those settings you can run the input using example Filebeat configuration as above.

To view the exported traces in Kibana, go to "Observability > Applications > Traces".

Related

• Add OTEL metrics to cel input #47014
• Add HTTPWithContextFnOpts so requests can have eval-time context mito#118

Use cases

This tracing is to be used for troubleshooting, particularly for Agentless.

Screenshots

OTel traces for the CEL Input in Elastic Observability:

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b cel-otel-tracing upstream/cel-otel-tracing
git merge upstream/main
git push upstream cel-otel-tracing

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @chrisberkhout? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b cel-otel-tracing upstream/cel-otel-tracing
git merge upstream/main
git push upstream cel-otel-tracing

[github-actions]

🔍 Preview links for changed docs

• docs/reference/filebeat/filebeat-input-cel.md

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)