Watermark Lag Lab

The window closed. The truth arrived 43 seconds later.

This interactive F# 10 / .NET 10 lab shows how event-time policy changes what a streaming dashboard can know. It runs one deterministic partitioned event stream through four notions of progress:

1. processing-time windows,
2. an eager watermark led by the fastest partition,
3. a conservative minimum watermark with no idle detection, and
4. bounded out-of-orderness plus an explicit idle timeout.

At the default workload, processing time puts 335,034 late events in the wrong window. The eager watermark drops 184,269 events. The conservative minimum preserves every event but leaves two-thirds of windows open and delays incident detection by 20 minutes. Bounded lateness with idleness detection preserves 100% event-time completeness with a 45-second alert delay.

Why this lab exists

Event time is when something happened. Processing time is when a particular operator saw it. A watermark is a progress signal: it says that the stream believes events at or before a timestamp have arrived and event-time windows may advance.

In a parallel stream, an operator commonly follows the lowest active input watermark. That creates two symmetric hazards:

• a fast partition can advance a naive global watermark and classify valid events from slower partitions as late, and
• an idle partition can hold back a minimum watermark forever, leaving accurate windows open and incidents unreported.

This lab makes the tradeoff visible through:

• event-time completeness,
• accepted, dropped, and mis-windowed late events,
• sealed versus stalled windows,
• maximum watermark lag,
• p95 alert delay,
• false-clear windows, and
• the percentage of incident windows that remain diagnosable.

The deterministic model lives in src/Simulation.fs. The service uses ASP.NET Core components included with .NET; there are no runtime package dependencies.

Run it

Requires the .NET 10 SDK, which includes F# 10.

make run

Open http://127.0.0.1:8080.

Choose another bind address:

HOST=0.0.0.0 PORT=9000 make run

Print the default model:

make json | jq

With Docker:

docker build -t watermark-lag-lab .
docker run --rm -p 8080:8080 watermark-lag-lab

The image is pinned to .NET 10.0.10 and runs as the non-root user supplied by Microsoft's official ASP.NET image.

Verify it

make check

The gate restores the pinned Fantomas development tool, verifies formatting, builds with warnings as errors, runs the F# semantic model tests, exercises HTTP routes and security headers, validates JavaScript syntax, and publishes a production artifact with its UI. CI separately builds and health-checks the non-root container.

API

GET /api/simulate accepts:

Query parameter	Default	Range
partitions	8	1–64
windows	30	10–120
window_seconds	60	5–300
events_per_second	200	1–10,000
out_of_order_percent	12	0–80
max_delay_seconds	90	0–600
idle_partitions	1	0–partitions − 1
idle_timeout_seconds	120	0–900
allowed_lateness_seconds	30	0–600
incident_drop_percent	35	0–90
alert_threshold_percent	20	5–80

Example:

curl 'http://127.0.0.1:8080/api/simulate?idle_partitions=3&idle_timeout_seconds=300' | jq

Instrumentation recipe

Make distributed time queryable:

event.time
event.arrival_time
event.lateness_ms
messaging.partition.id
stream.partition.idle
stream.partition.last_event_time
stream.watermark.time
stream.watermark.lag_ms
stream.allowed_lateness_ms
stream.window.state
stream.window.completeness
stream.late_event.action

With telemetry.sh, these fields let you distinguish “nothing happened” from “nothing has finalized,” find the partition holding back event time, and connect every dropped event or false clear to its watermark policy.

Model boundaries

This is an educational deterministic model, not a benchmark of one streaming framework. It isolates the semantics documented by systems such as Apache Flink:

• watermarks represent progress in event time,
• event-time windows tolerate out-of-order arrivals,
• multi-input operators follow the lowest active watermark,
• idle inputs must be excluded explicitly to avoid stalling progress, and
• bounded out-of-orderness turns completeness into a deliberate latency budget.

Read Apache Flink's documentation on generating watermarks and idle sources and debugging event time for the production semantics behind the experiment.

The application uses F# 10, which ships with .NET 10.

License

MIT