Churn Canary — Data Observability Demo

A lean, laptop-runnable demo for the workshop "Data Observability: Knowing What's Broken Before Users Do" (Parts 3 & 4).

It's a churn-prediction pipeline plus the checks and alerts that catch it when the data silently breaks — no errors, the model just quietly degrades, and a check flags it before any user notices. Runs entirely on your laptop: no cloud, no Docker, no paid services.

The idea in one line

One model, one pipeline, and the ways data can silently break it — each caught by one check, with a routed, severity-tiered alert, before any user notices.

Pillar	What breaks	What catches it
Freshness	data didn't refresh	dbt source freshness (watermark > SLA)
Volume	far fewer / more rows	dbt row-count test + z-score anomaly
Schema	a key column arrives empty/renamed	dbt not_null test
Distribution	a feature drifts	Evidently PSI report
Lineage	(not "broken" live)	dbt graph + impact analysis

---

Prerequisites

• Python 3.10+ · make · (macOS for the say-based rehearsal + slide PDFs use Node/npx)

Quick start

python3 -m venv .venv
source .venv/bin/activate       # (Windows: use WSL)
make setup                       # install pinned deps (once)
make run                         # generate data + build the feature table
make train                       # train the model + save the healthy drift reference
make status                      # confirm the pipeline is running & healthy
make alert                       # run all checks -> all green

make targets auto-use .venv, so they work even if the venv isn't activated.

---

Commands

Pipeline & health

Command	What it does
make setup	Install Python dependencies (once)
make run	Generate synthetic data, then build customer_features
make train	Train the churn model + save the healthy drift reference
make status	One-glance liveness: is the pipeline running + healthy?
make check	Freshness + data-quality tests (raw dbt view)
make check-drift	Distribution drift vs the healthy reference (Evidently PSI)
make alert	Run all checks and fire a Slack/console alert on any failure
make preflight	Pre-talk readiness check (deps, artifacts, healthy data)
make reset	Wipe everything and rebuild clean, healthy data
make clean	Delete the local database and build files

Observability extras

Command	What it does
make impact	Lineage impact analysis — what each source feeds downstream
make docs	Build + serve the dbt lineage graph (http://localhost:8080)
make slo	SLO / error-budget status per pillar (from run history)
make heartbeat	Dead-man's switch — alert if the monitor itself stops running

Break it (the demo)

Command	Trips
make inject-schema	column rename → NULL feature → not_null → 🔴 P1
make inject-volume	rows drop → row-count test → 🟠 P2
make inject-freshness	data > 24h stale → freshness error → 🔴 P1
make inject-freshness-warn	data aging 12–24h → freshness WARN → 🟡 P3
make inject-drift	support_sentiment shifts → Evidently → 🟠 P2
make inject-drift-tenure	tenure_days shifts → PSI > 0.25 → 🟠 P2

Rehearsal

Command	What it does
make slides	Render Part 3 + 4 decks to PDF (slides/dist/)
make narrate-part3 / narrate-part4	Read the talk aloud (macOS say)

---

Live demo — step by step

The rhythm never changes: break → detect → reset.

make inject-<x>   # 1. break the data (SILENT — no alert yet, by design)
make alert        # 2. detect it + fire the alert
make reset        # 3. back to healthy, ready for the next one

⚠️ make inject-* on its own does nothing visible — the alert only fires on make alert. That silence is the point.

Before the talk (off-screen)

source .venv/bin/activate
make preflight        # expect READY
make run              # regenerate fresh data (so freshness isn't already aging)
make status           # ✓ pipeline running & healthy
make alert            # four ✓, all green

• Bump the terminal font (18pt+); have the Slack channel visible.
• (Optional live Slack) put a webhook in .env — see Configuration below.

Step 1 — Show the pipeline is running + healthy

• Run: make status → ✓ last data load (fresh), ✓ row count, last monitor run.
• Run: make alert → four ✓, "all checks PASS".
• Say: "The pipeline's alive, data landed minutes ago, all green. Now let's break it."

Step 2 — Freshness 🔴 P1

• make inject-freshness → make alert
• See: 🔴 P1 · Freshness check failed — names the source, hours stale, SLA, and the downstream blast radius. Routed to on-call data engineer.
• make reset

Step 3 — Volume 🟠 P2

• make inject-volume → make alert
• See: 🟠 P2 · Volume check failed — actual count ("1,000 rows … ~1,000 short")
  • a statistical-anomaly line. Routed to data engineer.
• make reset

Step 4 — Schema 🔴 P1

• make inject-schema → make alert
• See: 🔴 P1 · Schema check failed — the exact column, 2,000 rows, sample IDs. (A cascade 🟠 P2 Distribution alert may also fire.) Routed to data engineer.
• Story: an upstream column rename (session_minutes → sessions_total) — the pipeline still builds, but the feature goes NULL.
• make reset

Step 5 — Distribution / the silent one 🟠 P2 (the punchline)

• Story: a marketing campaign brings new users → tenure_days shifts younger.
• make inject-drift-tenure
• make check → freshness passes, dbt tests PASS 9/9 — everything green.
• Say: "Every value valid, every test passes. If this were our whole safety net, we'd ship a rotting model."
• make alert → 🟠 P2 · Distribution check failed — "tenure_days: mean 372 → 149; PSI 4.29 (threshold 0.25)". Routed to ML engineer + model owner.
• make reset
• (make inject-drift — support_sentiment — is the alternate drift demo.)

Step 5.5 — Lineage / impact (optional)

• make impact → for each source, everything downstream it feeds (raw_customers → customer_features → churn_model, drift_monitor).
• make docs (run before the talk) → interactive DAG at localhost:8080.
• Say: "Lineage isn't a smoke detector — it's the map. If this source breaks, this is exactly what's affected."

Step 6 — Close

• make alert → all green again.
• Say: "Silent failures, caught before a user felt it. You can build this on Monday."

If something breaks

Problem	Fix
A command errors	make reset (rebuild clean healthy data, ~10s)
Data looks corrupt	make clean && make run && make train
"No reference" on drift	make train (on healthy data)
Slack didn't post	Fine — console alert is the fallback; keep going
Totally lost	make reset && make alert → show green → move on

Memorise one thing: make reset.

---

Observability model

Severity tiers

• 🔴 P1 — page now: Freshness (>24h), Schema — model serving impacted, routed to data engineer.
• 🟠 P2 — investigate: Volume, Distribution — degradation; drift routes to ML engineer + model owner.
• 🟡 P3 — worth a look: Freshness aging 12–24h — fires an alert but does not fail the run.

Production-grade alerting (all six principles met — see ALERTING-PRINCIPLES.md)

1. Threshold alerts for invariants (row-count band, not_null, freshness SLA)
2. Statistical/anomaly alerts, significance-gated (Evidently PSI + row-count z-score)
3. P1 / P2 / P3 severity tiers
4. Routing by owner (data engineer vs ML engineer + model owner)
5. SLO error-budget burn (make slo — alert only on sustained failure)
6. Dead-man's switch (freshness = pipeline; heartbeat.py = monitor)

---

Configuration — Slack (optional)

Alerts print to the console by default. To post to Slack, put a webhook in a gitignored .env (copy .env.example):

SLACK_WEBHOOK_URL=https://hooks.slack.com/services/XXX/YYY/ZZZ
# optional per-route channels:
SLACK_WEBHOOK_URL_DATA=...   # freshness / schema / volume
SLACK_WEBHOOK_URL_ML=...     # distribution / drift

---

Stack (lean — keep it this way)

DuckDB (one-file warehouse) · dbt Core (transforms + tests + lineage) · scikit-learn (churn model) · Evidently (drift, PSI) · Slack webhook (alerts) · Python + Makefile. Out of scope by design: Airflow, OpenLineage/Marquez, Prometheus/Grafana, Monte Carlo, Great Expectations, WhyLogs — the lean stack covers their detection roles for a laptop demo.

Project structure

churn-canary/
├── Makefile              # every command
├── requirements.txt      # pinned deps
├── generate_data.py      # synthetic data generator (fixed seed)
├── train_model.py        # churn model + drift reference snapshot
├── inject_failure.py     # the failure injections
├── check_drift.py        # Evidently PSI drift check
├── run_checks.py         # the monitor: runs all checks, routes + fires alerts
├── alert.py              # Slack/console alert sender (severity + routing)
├── metrics.py            # run history: SLO burn + z-score anomaly
├── heartbeat.py          # dead-man's switch for the monitor
├── impact.py             # lineage impact analysis from the dbt manifest
├── status.py             # pipeline liveness
├── preflight.py          # pre-talk readiness check
├── slides/               # Marp decks + spoken presenter scripts
└── dbt/
    ├── models/staging/   # sources.yml + stg_customers + stg_events
    ├── models/marts/     # customer_features + schema.yml (tests)
    ├── models/exposures.yml   # churn model + drift monitor (lineage consumers)
    └── tests/            # assert_row_count.sql (volume check)

Notes

• Fixed-seed generator → reproducible data, stable model + drift reference.
• Data ages: regenerate with make run shortly before presenting (freshness warns after 12h).