Multiline Framing Lab

One application exception can become sixteen unrelated log records.

This interactive lab models a line-oriented log collector that splits stack traces before enrichment. The first line keeps ERROR, trace_id, and exception.type; continuation lines become low-severity orphans with no correlation fields.

The default scenario emits 120 logical exceptions per minute, each containing 14 stack frames plus a message and cause line. A 60% rollout of the broken framer produces:

Signal	Broken line split	Correct framing
Logical exceptions	120	120
Exported records	1,200	120
Orphan records	1,080	0
Records dropped by a 1,000-entry limit	200	0
Exceptions with complete trace-linked stacks	40%	100%

The numbers are computed by the Ballerina model in main.bal, not hard-coded in the browser.

Why this happens

The OpenTelemetry log data model allows a log body to be a multiline string or a structured value, and defines correlation fields such as TraceId, SpanId, SeverityText, and attributes on each individual record.

A file reader still needs to decide where a record begins. The OpenTelemetry Collector Contrib file log receiver reads newline-delimited entries by default. Its multiline configuration changes the boundary using either a line_start_pattern or line_end_pattern.

This ordering matters:

application output
      ↓
record framing       ← the failure lives here
      ↓
severity parsing
      ↓
trace/exception enrichment
      ↓
collector queue

Once a stack frame has become its own record, downstream enrichment cannot infer that it belongs to the previous exception without another explicit reassembly rule.

Primary references:

• OpenTelemetry Logs Data Model
• OpenTelemetry Collector file log receiver multiline configuration

The repairs

The controls demonstrate two safe contracts:

1. Reassemble continuations — configure multiline framing before parsing severity, trace IDs, and exception attributes.
2. Emit structured JSON — serialize the exception and complete stack trace as one object so transport framing is unambiguous.

Both restore one record per logical exception, eliminate orphan frames, and keep the complete exception body available on the trace-correlated record.

What telemetry.sh reveals

The lab exposes the same failure through logs, metrics, and a correlated trace:

• lab.log.record_inflation compares exported records to logical exceptions.
• lab.log.orphan_records counts continuation-shaped bodies with no trace ID.
• lab.log.stack_context_coverage measures complete trace-linked exceptions.
• lab.log.dropped_records shows queue loss caused by artificial volume.
• sample log records preserve body, severity, trace ID, exception type, and drop state for inspection.

Try the investigation queries in the UI:

logs | where body contains " at " and trace_id == "" | count
logs | summarize records=count(), traces=dcount(trace_id) by service.name
logs | where severity_text == "ERROR" | project trace_id, exception.type, body

This is where correlated, full-fidelity telemetry is more useful than an isolated “error count” chart: the record count can rise while the useful context falls.

Run locally

Install Ballerina Swan Lake 2201.13.5 or a compatible Update 13 release, then:

make check
make run

Open http://localhost:8080.

The live endpoints are:

GET /healthz
GET /api/simulate
GET /api/telemetry

Example repairs:

curl 'http://localhost:8080/api/simulate?multilineParser=true'
curl 'http://localhost:8080/api/simulate?structuredJson=true'

Run with Docker

docker compose up --build

The multi-stage image compiles and tests the Ballerina package, then runs the generated JAR as an unprivileged user on a Java 21 runtime.

Repository layout

.
├── main.bal                 # model, OTLP-shaped payload, and HTTP service
├── tests/main_test.bal      # model invariants
├── tests/test_static.py     # UI, security, and asset checks
├── public/                  # dependency-free interactive workbench
├── Dockerfile
└── compose.yaml

License

MIT