Is it Observable

Episode : OTTL in Action: Stop Shipping Junk Telemetry to Your Backend

This repository contains the files utilized during the tutorial presented in the dedicated IsItObservable episode related to the OpenTelemetry Transformation Language (OTTL).

This tutorial will deploy an OpenTelemetry Collector with OTTL pipelines that cover the most common transformation use cases: enrichment, PII redaction, log parsing, noise filtering, and cardinality control.

We will also utilize the following components:

the OpenTelemetry Operator
Dynatrace Operator to report the health of the cluster
The OpenTelemetry Demo
A custom workload generator that produces PII-laden telemetry for OTTL demos

All the observability data generated by the environment would be sent to Dynatrace.

Prerequisite

The following tools need to be install on your machine :

jq
kubectl
git
curl
Helm
k8S cluster having the admission controller enabled

Getting started

1. Dynatrace Tenant

1. Dynatrace Tenant - start a trial

If you don't have any Dynatrace tenant , then I suggest to create a trial using the following link : Dynatrace Trial Once you have your Tenant save the Dynatrace tenant url in the variable DT_TENANT_URL (for example : https://dedededfrf.live.dynatrace.com)

DT_TENANT_URL=<YOUR TENANT Host>

2. Create the Dynatrace API Tokens

The dynatrace operator will require to have several tokens:

Token to deploy and configure the various components
Token to ingest metrics and Traces

Operator Token

One for the operator having the following scope:

Create ActiveGate tokens
Read entities
Read Settings
Write Settings
Access problem and event feed, metrics and topology
Read configuration
Write configuration
Paas integration - installer downloader

Save the value of the token . We will use it later to store in a k8S secret

API_TOKEN=<YOUR TOKEN VALUE>

Ingest data token

Create a Dynatrace token with the following scope:

Ingest metrics (metrics.ingest)
Ingest logs (logs.ingest)
Ingest events (events.ingest)
Ingest OpenTelemetry
Read metrics

Save the value of the token . We will use it later to store in a k8S secret

DATA_INGEST_TOKEN=<YOUR TOKEN VALUE>

3. Spinup a k8S cluster with the Admission Controller enabled

Deploy most of the components

The application will deploy the entire environment:

NAME="observable-ottl"
chmod 777 deployment.sh
./deployment.sh  --clustername "${NAME}" --dturl "${DT_TENANT_URL}" --dtingesttoken "${DATA_INGEST_TOKEN}" --dtoperatortoken "${API_TOKEN}"

This deploys:

Cert Manager + OpenTelemetry Operator
Dynatrace Operator + DynaKube
Two OpenTelemetry Collectors via the OTel Operator CRD (base config — no OTTL yet):
- DaemonSet (oteld) — collects logs from nodes using the filelog receiver
- StatefulSet (otel) — receives traces, metrics, and logs via OTLP from applications
The OpenTelemetry Demo (generates realistic telemetry)

Deploy the Workload Generator

The OTel Demo does not produce PII (emails, credit cards, user IDs) in its telemetry. The workload generator creates logs and traces with realistic sensitive data — exactly what the OTTL demos need.

kubectl apply -f workload-generator/k8s-deployment.yaml

Architecture: Two Collectors, Two Roles

Understanding which Collector does what is key to following the OTTL examples:

┌─────────────────────────────────────────────────────────┐
│  Node                                                   │
│                                                         │
│  ┌─────────────┐      ┌─────────────────────────────┐  │
│  │ OTel Demo   │──────│ StatefulSet Collector (otel) │  │
│  │ Workload Gen│ OTLP │ receives OTLP data           │  │
│  └─────────────┘      │ (traces, metrics, logs)      │──── Dynatrace
│                       └─────────────────────────────┘  │
│  /var/log/pods/                                         │
│       │                                                 │
│  ┌────▼─────────────────────────────────┐              │
│  │ DaemonSet Collector (oteld)          │              │
│  │ collects node logs via filelog        │──────────────── Dynatrace
│  └──────────────────────────────────────┘              │
└─────────────────────────────────────────────────────────┘

We start with base Collectors (no OTTL). Then we progressively add OTTL use cases — each step upgrades the Collectors by applying a new set of CRDs.

Step-by-Step: OTTL Use Cases

Each step adds one OTTL use case on top of the previous. Apply the step folder to upgrade both Collectors:

kubectl apply -f kubernetes/step01-enrich/    # Step 1
kubectl apply -f kubernetes/step02-pii/       # Step 2
kubectl apply -f kubernetes/step03-parse/     # Step 3
kubectl apply -f kubernetes/step04-filter/    # Step 4
kubectl apply -f kubernetes/step05-complete/  # Step 5 (final)

Step 1: Enrich Your Data

Goal: Add missing context — derive service.name from K8s metadata, tag the environment, normalize HTTP methods to uppercase.

Where it runs: Both Collectors (transform/enrich processor)

Key OTTL statements (resource context):

# Derive service.name from K8s workload if missing
- set(resource.attributes["service.name"], resource.attributes["k8s.deployment.name"])
    where resource.attributes["service.name"] == nil and IsString(resource.attributes["k8s.deployment.name"])

# Tag environment based on namespace
- set(resource.attributes["deployment.environment"], "production")
    where IsMatch(resource.attributes["k8s.namespace.name"], "prod.*")

Key OTTL statements (span context):

# Normalize HTTP method to uppercase
- set(span.attributes["http.request.method"],
      ConvertCase(span.attributes["http.request.method"], "upper"))
    where span.attributes["http.request.method"] != nil

v0.120+ rule: Every path must be prefixed with its context name — resource.attributes[...] in resource context, span.attributes[...] in span context, log.body / log.attributes[...] in log context. Without the prefix, the collector auto-fixes it but warns you to rewrite.

Functions used: set(), ConvertCase(), IsMatch(), IsString()

Verify: In Dynatrace, check that traces from the OTel Demo have deployment.environment set and http.request.method is always uppercase (GET, not get).

Step 2: PII Redaction

Goal: Remove or mask sensitive data before telemetry leaves your infrastructure. Emails and credit cards are redacted from log bodies, user IDs are hashed (preserving correlation), and sensitive attributes are deleted.

Where it runs: Both Collectors (transform/pii processor)

Key OTTL statements — logs (log context, DaemonSet):

# Redact emails in log body
- replace_pattern(log.body, "\\b[\\w.-]+@[\\w.-]+\\.\\w+\\b", "[REDACTED_EMAIL]")
    where IsMatch(log.body, ".*@.*")

# Redact credit cards
- replace_pattern(log.body, "\\b\\d{4}[- ]?\\d{4}[- ]?\\d{4}[- ]?\\d{4}\\b", "[REDACTED_CC]")

# Hash user IDs (preserves correlation without identity)
- set(log.attributes["user.id"], SHA256(log.attributes["user.id"]))
    where log.attributes["user.id"] != nil

# Remove all sensitive attributes by pattern
- delete_matching_keys(log.attributes, "(?i).*authorization.*")
- delete_matching_keys(log.attributes, "(?i).*cookie.*")
- delete_matching_keys(log.attributes, "(?i).*token.*")

Key OTTL statements — traces (span context, StatefulSet):

# Redact SQL query values (keep structure, hide data)
- replace_pattern(span.attributes["db.statement"], "'[^']*'", "'***'")
    where span.attributes["db.system"] != nil

# Redact sensitive URL query parameters
- replace_pattern(span.attributes["url.full"],
    "(\\?|&)(password|token|secret|key|auth|api_key)=[^&]*",
    "$1$2=[REDACTED]")
    where span.attributes["url.full"] != nil

Functions used: replace_pattern(), SHA256(), delete_matching_keys(), IsMatch()

Verify: Check logs in Dynatrace — emails from the workload generator should appear as [REDACTED_EMAIL]. Check DB spans — SQL values should show '***'.

Test on ottl.run: Paste testing/sample-data/log-with-pii.json as input with:

transform:
  error_mode: ignore
  log_statements:
    - context: log
      statements:
        - replace_pattern(log.body, "\\b[\\w.-]+@[\\w.-]+\\.\\w+\\b", "[REDACTED_EMAIL]")
            where IsMatch(log.body, ".*@.*")

Step 3: Parse & Extract — The Cache Trick

Goal: Parse JSON and key=value log bodies into structured attributes. Uses the OTTL cache map to decode byte-array bodies, redact PII, then parse — all in a single pipeline.

Where it runs: StatefulSet (transform/parse processor) — because the workload generator sends JSON/key=value logs via OTLP, not via stdout

Key OTTL statements (log context):

# STEP 1: Decode body into cache (handles byte-array bodies)
- set(log.cache["decoded_body"], String(log.body))

# STEP 2: Redact PII BEFORE parsing into attributes
- replace_pattern(log.cache["decoded_body"],
    "\\b[\\w.-]+@[\\w.-]+\\.\\w+\\b", "[REDACTED]")
    where IsMatch(log.cache["decoded_body"], ".*@.*")

# STEP 3: Parse cleaned JSON from cache into attributes
- merge_maps(log.attributes, ParseJSON(log.cache["decoded_body"]), "upsert")
    where IsMatch(log.cache["decoded_body"], "^\\s*\\{")

# STEP 4: Extract severity from parsed level field
- set(log.severity_text, log.attributes["level"])
    where log.attributes["level"] != nil

# For key=value logs (no cache needed)
- merge_maps(log.attributes, ParseKeyValue(log.body, "=", " "), "upsert")
    where IsMatch(log.body, "^[a-zA-Z_]+=")

Functions used: set(), String(), replace_pattern(), ParseJSON(), merge_maps(), ParseKeyValue(), log.cache[]

Why the cache? The log body is often a byte array, not a string. Functions like replace_pattern() expect a path as target. The log.cache map is a temporary scratchpad — write to it, transform it, read from it. It disappears after the record is processed.

Verify: Logs from the workload generator that arrive as JSON strings (e.g., {"level":"ERROR","host":"db-primary",...}) should now have host, duration_ms, etc. as individual attributes in Dynatrace.

Test on ottl.run: Paste testing/sample-data/log-json-with-pii.json as input. Note: cache[] is a runtime-only concept and is not available in ottl.run. Test individual statements (ParseJSON, replace_pattern) separately.

Step 4: Kill the Noise

Goal: Drop telemetry nobody will look at — successful health checks, startup logs, internal Go runtime metrics.

Where it runs: Both Collectors (filter/noise processor)

Key OTTL statements — traces (StatefulSet):

# Drop successful health/readiness checks (keep failures!)
- IsMatch(name, "(?i)health|readiness|liveness")
    and status.code == STATUS_CODE_OK

# Drop Kubernetes probe spans
- IsMatch(attributes["http.user_agent"], "(?i)kube-probe.*")
    and status.code == STATUS_CODE_OK

OTTL gotcha: In the filter processor's span context, you are already inside the span — so it's name, not span.name, and status.code, not span.status.code. This is a common mistake that crashes the Collector.

Key OTTL statements — logs (DaemonSet):

# Drop debug/trace severity in production
- severity_number < SEVERITY_NUMBER_INFO
    and IsMatch(resource.attributes["deployment.environment"], "prod.*")

# Drop repetitive startup/shutdown noise
- IsMatch(body, "(?i)^(starting|started|stopping|stopped|initialized|ready|listening).*")

Key OTTL statements — metrics (StatefulSet):

# Drop internal Go runtime and collector self-metrics
- IsMatch(name, "process\\.runtime\\.go\\..*")
- IsMatch(name, "otelcol_.*")

Same principle for the metric context in the filter processor: use name, not metric.name.

Functions used: IsMatch(), severity constants (SEVERITY_NUMBER_INFO), status constants (STATUS_CODE_OK)

Verify: In Dynatrace, health check spans (GET /healthz) should no longer appear. Startup logs should be gone. The volume of metrics should decrease noticeably.

Step 5: Tame High Cardinality

Goal: Control exploding metric series by removing high-cardinality attributes (IPs, container IDs), normalizing URL paths, and bucketing status codes.

Where it runs: StatefulSet (transform/cardinality processor)

Key OTTL statements (datapoint context):

# Remove known high-cardinality attributes
- delete_key(datapoint.attributes, "net.peer.ip")
- delete_key(datapoint.attributes, "net.peer.port")
- delete_key(datapoint.attributes, "container.id")
- delete_key(datapoint.attributes, "url.full")

# Normalize URL paths: /api/users/12345 → /api/users/{id}
- replace_pattern(datapoint.attributes["url.path"], "/\\d+", "/{id}")

# Bucket status codes: 200 → "2xx"
- set(datapoint.attributes["http.response.status_class"],
      Concat([Substring(String(datapoint.attributes["http.response.status_code"]), 0, 1), "xx"], ""))
    where datapoint.attributes["http.response.status_code"] != nil
- delete_key(datapoint.attributes, "http.response.status_code")

# Safety net: cap total attributes (3rd arg = priority keys to keep)
- limit(datapoint.attributes, 15, [])

OTTL gotcha: limit() requires 3 arguments: limit(target, max, priority_keys[]). The third argument is an array of keys that should be kept even if the limit is exceeded. Pass [] if you have no priority keys.

Functions used: delete_key(), replace_pattern(), Concat(), Substring(), String(), limit()

Test on ottl.run: Paste testing/sample-data/high-cardinality-metric.json as input with:

transform:
  error_mode: ignore
  metric_statements:
    - context: datapoint
      statements:
        - delete_key(datapoint.attributes, "net.peer.ip")
        - delete_key(datapoint.attributes, "container.id")
        - replace_pattern(datapoint.attributes["url.path"], "/\\d+", "/{id}")

Verify: In Dynatrace, the http.server.request.duration metric should have far fewer unique time series. URLs like /api/users/12345 should appear as /api/users/{id}.

Testing Your OTTL Statements

1. OTTL Playground (ottl.run)

Before deploying any config, test your statements at ottl.run.

Sample data for each use case is in testing/sample-data/:

File	Use Case
`log-with-pii.json`	PII redaction (emails, credit cards, tokens)
`log-json-with-pii.json`	Parse & Extract (JSON body with PII)
`span-missing-context.json`	Enrichment + noise filtering (health checks)
`high-cardinality-metric.json`	Cardinality control

Important: ottl.run expects the processor config block, not the full Collector YAML. Use this format:

transform:
  error_mode: ignore
  log_statements:
    - context: log
      statements:
        - replace_pattern(body, "\\b[\\w.-]+@[\\w.-]+\\.\\w+\\b", "[REDACTED_EMAIL]")

2. Config Validation

Before applying a step, extract the Collector config from the CRD and validate it:

# Extract the config section from a step's CRD
yq '.spec.config' kubernetes/step05-complete/openTelemetry-manifest_statefulset.yaml > /tmp/otel-config.yaml
otelcol-contrib validate --config=/tmp/otel-config.yaml

This catches OTTL syntax errors, bad regex patterns, missing processors, and pipeline misconfigurations. Do this before every kubectl apply.

3. Debug Exporter

Both Collectors include a debug exporter. To see what the Collector is receiving and producing, check the pod logs:

# StatefulSet collector
kubectl logs -l app.kubernetes.io/component=oteld-statefulset -f
# DaemonSet collector
kubectl logs -l app.kubernetes.io/component=oteld-daemonset -f

Pipeline Order Matters

The order of processors in a pipeline is critical. Here is the recommended order and why:

memory_limiter → k8sattributes → resource → enrich → pii → parse → filter → cardinality → batch → export

memory_limiter first — protect the Collector from OOM
k8sattributes — enrich with K8s metadata (needed by downstream processors)
resource — set cluster name
transform/enrich — derive service.name, set environment (other processors may need these)
transform/pii — redact sensitive data BEFORE any parsing exposes it as attributes
transform/parse — parse log bodies into structured attributes
filter/noise — drop unwanted records (after enrichment, so filter conditions can use enriched fields)
transform/cardinality — reduce metric series (after filtering, so you don't waste cycles on dropped records)
batch — batch before export for efficiency

OTTL Functions Reference

Editors (modify data)

Function	What It Does	Example
`set(target, value)`	Set a value	`set(attributes["env"], "prod")`
`delete_key(map, key)`	Remove one key	`delete_key(attributes, "http.url")`
`delete_matching_keys(map, pattern)`	Remove keys by regex	`delete_matching_keys(attributes, "(?i).token.")`
`keep_keys(map, keys[])`	Keep only listed keys	`keep_keys(attributes, ["method", "status"])`
`replace_pattern(target, regex, replacement)`	Regex replace	`replace_pattern(body, "\\d{4}", "****")`
`merge_maps(target, source, strategy)`	Merge two maps	`merge_maps(attributes, ParseJSON(body), "upsert")`
`limit(map, max, priority_keys[])`	Cap number of attributes	`limit(attributes, 20, ["service.name"])`
`flatten(map, prefix)`	Flatten nested maps	`flatten(attributes, "app")`
`truncate_all(map, max_length)`	Truncate all strings	`truncate_all(attributes, 256)`
`append(target, value)`	Append to array	`append(attributes["tags"], "processed")`

Converters (transform values)

Function	What It Does	Example
`ConvertCase(value, case)`	Change case	`ConvertCase(attributes["method"], "upper")`
`SHA256(value)`	Hash value	`SHA256(attributes["user.id"])`
`ParseJSON(value)`	Parse JSON	`ParseJSON(body)`
`ParseKeyValue(value, d, pd)`	Parse key=value	`ParseKeyValue(body, "=", " ")`
`IsMatch(value, pattern)`	Regex test	`IsMatch(body, ".error.")`
`Concat(values[], delimiter)`	Join strings	`Concat([attributes["first"], attributes["last"]], " ")`
`Substring(value, start, length)`	Extract substring	`Substring(attributes["trace_id"], 0, 8)`
`String(value)`	Cast to string	`String(attributes["count"])`
`Int(value)`	Cast to integer	`Int(attributes["status_code"])`
`Now()`	Current timestamp	`Now()`
`Duration(value)`	Parse duration	`Duration("5s")`
`UUID()`	Generate UUID	`UUID()`
`Len(value)`	Length	`Len(attributes)`

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
dynatrace		dynatrace
image		image
kubernetes		kubernetes
opentelemetr-demo		opentelemetr-demo
testing/sample-data		testing/sample-data
workload-generator		workload-generator
README.md		README.md
deployment.sh		deployment.sh

Navigation Menu

Uh oh!

Repository files navigation

Is it Observable

Episode : OTTL in Action: Stop Shipping Junk Telemetry to Your Backend

Prerequisite

Getting started

1. Dynatrace Tenant

1. Dynatrace Tenant - start a trial

2. Create the Dynatrace API Tokens

Operator Token

Ingest data token

3. Spinup a k8S cluster with the Admission Controller enabled

Deploy most of the components

Deploy the Workload Generator

Architecture: Two Collectors, Two Roles

Step-by-Step: OTTL Use Cases

Step 1: Enrich Your Data

Step 2: PII Redaction

Step 3: Parse & Extract — The Cache Trick

Step 4: Kill the Noise

Step 5: Tame High Cardinality

Testing Your OTTL Statements

1. OTTL Playground (ottl.run)

2. Config Validation

3. Debug Exporter

Pipeline Order Matters

OTTL Functions Reference

Editors (modify data)

Converters (transform values)

Resources

About

Releases

Packages

Contributors

Languages

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

Is it Observable

Episode : OTTL in Action: Stop Shipping Junk Telemetry to Your Backend

Prerequisite

Getting started

1. Dynatrace Tenant

1. Dynatrace Tenant - start a trial

2. Create the Dynatrace API Tokens

Operator Token

Ingest data token

3. Spinup a k8S cluster with the Admission Controller enabled

Deploy most of the components

Deploy the Workload Generator

Architecture: Two Collectors, Two Roles

Step-by-Step: OTTL Use Cases

Step 1: Enrich Your Data

Step 2: PII Redaction

Step 3: Parse & Extract — The Cache Trick

Step 4: Kill the Noise

Step 5: Tame High Cardinality

Testing Your OTTL Statements

1. OTTL Playground (ottl.run)

2. Config Validation

3. Debug Exporter

Pipeline Order Matters

OTTL Functions Reference

Editors (modify data)

Converters (transform values)

Resources

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages