GA4 Anomaly Detector

Statistical anomaly detection over GA4 BigQuery exports, narrated by an LLM.

The pipeline does the math; the LLM only writes the prose. That ordering matters: in a controlled comparison published by Coupler.io, Google's own GA4 MCP server reported a 35% increase in traffic when the actual trend was a 60% decrease — because it handed the LLM raw query results and asked it to do the analysis. ga4-anomaly-detector flips that: STL, PELT, and JS-divergence detectors run over the data first, and the LLM only ever sees the structured findings.

📚 Related writeups

Long-form posts on this project at hugonissar.github.io:

• Statistical anomaly detection over GA4 BigQuery exports, narrated by an LLM.

What it produces

A markdown report you can pipe into Slack, email, a doc, or just cat:

# GA4 Anomaly Detector
*2021-01-01 → 2021-01-31 · `bigquery-public-data.ga4_obfuscated_sample_ecommerce`*

## Headline
Revenue stepped down by 38% starting 2021-01-19 and has held.

## Key findings
- **revenue** level shift on 2021-01-19: ~$8,200 → ~$5,100 (↓ -38% sustained over 14 days)
- **conversions** on 2021-01-22: 47 vs ~89 expected (↓ -47%, high severity)
- **sessions** on 2021-01-08: 4,820 vs ~3,100 expected (↑ +56%, medium severity)
- **page_views** level shift on 2021-01-12: ~22,400 → ~18,900 (↓ -16% sustained over 14 days)

## What changed in the mix
**revenue by source medium** (2021-01-18–2021-01-24 → 2021-01-25–2021-01-31)
- Gainer: `(direct) / (none)` 31% → 44% share
- Loser: `google / cpc` 28% → 17% share

## Watch list
- **active_users** mild deviation on 2021-01-26 (-12% vs expected)

The gap this fills

Approach	What you ask	What you get
GA4 UI / Looker Studio	"Was traffic down?"	A chart you have to read
GA4 SQL query libraries	"Show me anomalies"	A query you have to run and interpret
Raw-SQL-to-LLM (Google's MCP, ChatGPT + BigQuery, etc.)	"Anything weird this week?"	A narrative — sometimes wrong, because the LLM does the math
ga4-anomaly-detector	(same)	A narrative grounded in pre-computed statistics

The third row is the one that bites people in practice. The Coupler.io benchmark is one documented case; the same pattern shows up whenever an LLM is asked to spot trends in raw SQL output. The fix is structural: run statistics in code, hand the LLM only the findings.

Quick start

Tested on Python 3.12+. Requires a GCP project (for BigQuery) and access to a Gemini model (for narration — or pass --no-llm to skip).

git clone https://github.com/<you>/ga4-anomaly-detector.git
cd ga4-anomaly-detector
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# Auth to GCP (covers BigQuery; also covers Vertex AI if you go that route)
gcloud auth application-default login
export GOOGLE_CLOUD_PROJECT=your-gcp-project

You now have two options for the Gemini side, depending on whether you want to manage an API key or use the same gcloud auth you already set up.

Option A — AI Studio (API key, free tier). Fastest to get started. Get a key at aistudio.google.com/apikey.

export GEMINI_API_KEY=...
python cli.py sample

Option B — Vertex AI (gcloud auth, no API key). Uses your existing Application Default Credentials. No separate key to manage, single auth surface for both BigQuery and Gemini.

# Enable Vertex AI on your project (one time)
gcloud services enable aiplatform.googleapis.com \
    --project=$GOOGLE_CLOUD_PROJECT

# Either pass --vertex each time…
python cli.py sample --vertex

# …or set the env var once
export GOOGLE_GENAI_USE_VERTEXAI=true
python cli.py sample

Vertex AI defaults to the global endpoint (location="global"), which is required for Gemini 3 Preview models — they don't run on regional endpoints like us-central1 and you'll get a NOT_FOUND if you point at one. Global is also generally the right default because it routes to whichever region has capacity, reducing 429s. If you need to pin a region for data residency, set GOOGLE_CLOUD_LOCATION=europe-west4 (or your region of choice) and use a non-preview model that supports it.

Which one to pick. AI Studio is simpler for personal use and has a free tier. Vertex AI is better when you want unified auth, audit logs in Cloud Logging, or eventually to run this from a server. The two paths use the same SDK, the same model strings, and produce identical output — the only difference is auth.

The sample mode points at bigquery-public-data.ga4_obfuscated_sample_ecommerce (an obfuscated real ecommerce site Google publishes for testing) and defaults to a clean month of that data. A single run scans well under the BigQuery free tier.

To skip the LLM call and use the deterministic template renderer instead:

python cli.py sample --no-llm

This is the dev-loop command — same output shape, no tokens spent. Useful when tuning detector parameters.

Your own export

python cli.py run \
    --project-id your-project \
    --dataset analytics_123456789 \
    --start 2026-04-15 --end 2026-04-30 \
    --dimensions source_medium,device_category,country \
    --output weekly-report.md

Available dimensions are source_medium, device_category, country, and browser. Pass --dimensions "" to skip mix-shift detection entirely (it's the slowest part on large exports).

Use it from Claude Desktop

mcp_server.py exposes the analyze pipeline as an MCP tool that any compatible client can call — Claude Desktop, Cursor, Claude Code, Gemini CLI. The server returns the structured findings as JSON; the client's LLM narrates them based on what you asked. No second LLM call, no duplicated work.

Add this to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "ga4-anomaly-detector": {
      "command": "python",
      "args": ["/absolute/path/to/ga4-anomaly-detector/mcp_server.py"],
      "env": {
        "GOOGLE_CLOUD_PROJECT": "your-gcp-project"
      }
    }
  }
}

Restart Claude Desktop. The tool will show up in the tool picker (the slider icon next to the chat input). Then ask things like:

What anomalies were there in the GA4 sample dataset in January 2021?

Compare the last two weeks of my GA4 export (project my-project, dataset analytics_123456789). Anything to worry about?

Did the traffic mix shift on mobile devices last month?

Claude will call analyze_ga4, receive the structured findings, and narrate them. Because the LLM only sees pre-computed statistics, the failure mode where it reports "+35%" when the truth is "-60%" can't happen — the statistics are right or there's no finding at all.

How it works

Three detectors, each chosen because the alternative was wrong for this specific data shape.

Point anomalies — STL residual z-score. GA4 metrics are dominated by weekday/weekend cycles. A naive rolling-mean z-score flags every Saturday as anomalous. STL (Cleveland et al., 1990) decomposes the series into trend + weekly seasonal + residual; we z-score the residual. A day is flagged when its residual exceeds the --sigma-threshold (default 3.0; lower for noisy small-site data, higher for high-volume stable sites).

Change points — PELT with RBF cost. A one-day spike (campaign launch) is not a change point. A site migration that drops sessions and they stay dropped is. We use the PELT algorithm from the ruptures library with an RBF cost function, which is roughly scale-invariant — so the same --pelt-penalty (default 10.0) works for sessions (thousands) and conversion_rate (single digits) without rescaling. Higher penalty = fewer breaks detected.

Mix shifts — Jensen-Shannon divergence. Catches the class of problem GA4 dashboards hide: total sessions look flat, but direct doubled while organic collapsed. We compute the share-of-voice distribution of each dimension (e.g., source/medium) in two adjacent windows and measure the JS divergence between them. JS is bounded [0, ln 2] so the threshold (default 0.02) is interpretable and symmetric. Chi-square would give a p-value, but on high-traffic sites every shift is "significant" — useless for filtering.

The LLM (Gemini 3 by default via the google-genai SDK — AI Studio or Vertex AI, your pick; swap in AnthropicClient or anything else that implements the LLMClient Protocol) receives only the structured findings — never the daily metric values, never the raw event rows. The system prompt forbids speculation about causes that aren't corroborated by other findings.

Project layout

ga4-anomaly-detector/
├── anomalies.py     # STL + PELT + JS-divergence detectors, dataclass results
├── fetcher.py       # BigQuery → pandas, in the exact shapes anomalies expects
├── narrative.py     # AnomalyReport → markdown (LLM client + template fallback)
├── cli.py           # Wire-up; argparse subcommands `run` and `sample`
├── mcp_server.py    # MCP tools for Claude Desktop, Cursor, Gemini CLI
├── requirements.txt
├── LICENSE
└── README.md

The three core modules are deliberately independent — anomalies.py doesn't import BigQuery, narrative.py doesn't import either, and cli.py is the only thing that knows about all three. Each module has its own usage example in its docstring; you can use them directly from a notebook without going through the CLI.

Tuning knobs

The defaults are picked for medium-traffic ecommerce sites (the shape of the sample dataset). Tune from there:

• --sigma-threshold (default 3.0) — Lower means more sensitive. Drop to 2.5 for small-traffic sites where daily noise is high; raise to 4.0 for high-volume stable sites where you only want extreme deviations.
• --pelt-penalty (default 10.0) — Lower means more change points detected. If you're getting spurious breaks, raise. If real shifts are getting missed, lower.
• --mix-window-days (default 7) — Width of each comparison window for mix shifts. 7 = last week vs. the week before, anchored to the most recent date in the export.
• --known-events — Comma-separated dates the point-anomaly detector should ignore. Without this, a query containing Christmas / Thanksgiving / New Year will dutifully flag those days as -50% to -70% anomalies, which is technically true and substantively meaningless. The detector excludes these dates from both the noise-floor estimate and the flagged-anomaly list. Example: --known-events 2026-12-24,2026-12-25,2026-12-31,2027-01-01. The holidays Python package generates these lists by country if you don't want to hardcode them.
• --conversion-events (default purchase) — GA4 event names that count as conversions. The default matches ecommerce sites and the public sample. SaaS or lead-gen sites should override: --conversion-events sign_up,subscribe. Without this, non-ecommerce sites will report 0 conversions and the narrative will confidently say so.

A note on missing data: the detector linearly interpolates single-day gaps so STL has a clean grid to fit on, but synthesized dates are never flagged as anomalies or change points (we shouldn't claim data we made up is statistically significant). Multi-day gaps are reported in the log.

Roadmap

What's not yet built but should be:

• Tests against synthetic series with planted anomalies. The clean way to verify the detectors don't drift.

Contributions welcome on any of the above. Open an issue first for anything that touches the detector defaults or the system prompt — those are tuned, and changes need a sanity check against the public sample.

License

MIT. See LICENSE.