NetPulse

Open-source detector for Internet outages and BGP anomalies, evaluated against real RIPE RIS archive data with a public, reproducible benchmark: 7 / 7 labeled historical incidents detected, 0 false negatives.

🔴 Live monitor → netpulse-live.onrender.com — tapping the global BGP feed in real time right now.

Try it in 30 seconds — no API keys, no deploy, no native libs:

git clone https://github.com/pauti04/netpulse && cd netpulse
uv sync
uv run netpulse demo                  # 2008 YouTube hijack, bundled real data, ~1s
uv run netpulse demo --incident all   # roll up all 7 corpus incidents

The demo replays the canonical 2008 YouTube /24 hijack against a bundled slice of real RRC00 archive data and prints the detection — story, the actual hijacker AS-path pulled from the archive, and a color-coded verdict — in about a second. A hosted HTTP API (POST /detect/bgp) ships in the repo and deploys to any Docker host; see Deploy.

Live monitor

NetPulse also runs as a real-time service: netpulse live taps the RIPE RIS Live global BGP feed (~1,800 updates/sec), runs the detectors on a rolling window with auto-reconnect, and serves a self-refreshing status page + JSON feed of the anomalies it's flagging right now.

uv run netpulse live          # → http://localhost:8000  (status page + /live/recent)

In a 70-second sample against the live feed it ingested 149,246 BGP updates and surfaced 11 candidate routing anomalies (origin deaggregation bursts) with zero reconnects. It's a single always-on process — deploy it as one web service (see Deploy).

Real performance numbers (benchmark methodology in BENCHMARK.md): RPKI validate against 859k VRPs ≈ 43 µs / call (~23k / sec, after a 500× speedup from longest-prefix-match indexing). Route-leak detector over 1,000 real archived AS-paths ≈ 5.7 ms total. Feature extraction over a 51k-announce / 7.7k-prefix BGP window ≈ 39 ms.

End-to-end tour (demo · corpus · streaming-mode latency · live API)

Bundled-data demo (no setup, ~1 second)

Live BGP feed against the RIPE RIS Live WebSocket

Headline: 7 / 7 incidents detected · 0 µs streaming latency on sub-prefix hijacks

Five labeled BGP incidents drawn from primary sources (RIPE NCC, Cloudflare, ISC, BGPmon). The expected detector fires on 5 / 5. Indosat 2014 (the most recent addition) exercises BOTH branches of SubPrefixHijackDetector -- 3 exact-prefix alerts plus 16 sub-prefix alerts -- so the same detector logic covers MOAS-shape and more-specific-shape hijacks in one labeled case. The Google 2017 leak — previously reported as a GAP because the pair-direction valley-free check abstained on the AS15169↔AS4713 step — is now caught by the customer-cone-aware leak detector (customer_cone_leak): 4713 (NTT OCN) is not in cone(15169) (Google's 2017 cone has 10 ASes), so the path direction is not cone-monotone and the detector fires (123,749 on-target alerts). The full per-incident outcome table and the TP / FN / GAP methodology are in BENCHMARK.md and the working-note writeup at docs/paper.md.

On the two sub-prefix incidents, the per-record streaming-mode latency benchmark reports 0 µs from documented incident onset — the first qualifying update in the public RIS archive is the onset record itself. Rerun with uv run netpulse benchmark stream-latency --incidents data/incidents --baseline data/baselines/yt_rib_filtered.duckdb.

Headline result: multi-signal fusion on a real incident

On the 2018-11-12 MainOne → Google route leak, two independent observability signals both fire on the same window of real archive data and a small correlator binds them into one critical alert:

BGP signal (route_leak / CAIDA serial-2 20181101 snapshot):
  Paths inspected:                         7,411
  MainOne-shape leak alerts (37282→15169): 1,985

Atlas signal (msm 1999544 ping 8.8.8.8):
  Baseline median RTT (pre-21:06Z):           38.0 ms
  Window   median RTT (21:06–22:30Z):          49.9 ms  (1.31× baseline)

→ 1 fused critical alert  ·  reproducible via scripts/fusion_demo.py

The first AS37282 transit observation at RRC00 is 2018-11-12 21:12:16 UTC — to the second of BGPmon's published onset. BENCHMARK.md walks through the full methodology and the temporal-drift detail that matters: the route-leak detector requires the time-aligned CAIDA snapshot; with the current 2026-05 dataset the same query produces 0 alerts on this 2018 incident.

What it does

Pulls BGP updates from RIPE RIS or RouteViews, normalizes them into a DuckDB single-file store, and runs detectors over rolling windows. Eight detectors covering MOAS, sub-prefix hijack (RFC 6811-style supernet check), withdraw-spike, RPKI Origin Validation (RFC 6811), route-leak (RFC 7908 valley-free), customer-cone-aware route-leak, Atlas loss spike, and DNS reachability (active probes via dnspython). Plus a three-axis correlator that fuses BGP + Atlas RTT

• DNS failures on the same time window.

Try it now

git clone https://github.com/pauti04/netpulse && cd netpulse
uv sync                                  # core install (no native deps)
uv run netpulse demo                     # 2008 YouTube hijack (bundled)
uv run netpulse demo --list              # all 7 curated incidents
uv run netpulse demo --incident all      # play all 7 + roll-up table

Each demo prints a story panel describing the incident, pulls the actual hijacker AS path from the archive (with friendly AS names — AS17557 ←ORIGIN), runs the matching detectors (hijack vs leak via auto-dispatch), lands a color-coded verdict panel — HIJACK DETECTED, LEAK DETECTED, or Clean window — and prints the stream-latency from documented onset (0µs from onset is the headline number on the sub-prefix cases). Noise-filtered by default; add --all to see every alert.

Incident	Detectors fired	Verdict	Stream latency
youtube_2008	moas=2, subprefix_hijack=1	HIJACK DETECTED	0 µs
rostelecom_2017	subprefix_hijack=4 (financial network)	HIJACK DETECTED	1 s
google_ntt_leak_2017	customer_cone_leak=124,145	LEAK DETECTED	n/a²
indosat_2014	subprefix_hijack=19 (both branches)	HIJACK DETECTED	99 s¹
myetherwallet_2018	subprefix_hijack=5	HIJACK DETECTED	0 µs
mainone_google_leak_2018	route_leak=1,985, customer_cone_leak=4,100	LEAK DETECTED	n/a²
vodafone_idea_2024	route_leak=43, customer_cone_leak=1,015	LEAK DETECTED	n/a²

¹ Indosat's first AS4761 announcement is on a Bangladesh prefix not in the small hand-curated baseline; the detector fires on the first covered prefix 99 s later. Wider baseline coverage would drop this to 0 µs.

² Stream-latency only measured for sub-prefix hijacks; leaks use a different per-record harness.

Forensic deep-dive: how did it spread?

Where demo is a snapshot, netpulse explain reconstructs how the anomaly propagated across the global routing table — straight from the archive:

uv run netpulse explain                                  # 2008 YouTube hijack
uv run netpulse explain --incident mainone_google_leak_2018

It computes, over the same BGPStore the detectors read, how many independent RIS/RouteViews peers observed the event, how fast it reached them (a cumulative spread curve), and how many distinct AS paths carried it. The 2008 YouTube hijack reached 11/11 observable vantage points in 32 s via 21 distinct paths; the 2018 MainOne leak traversed 24/24 peers via 480 paths. Hijacks trace by origin (prefix originated by the attacker); leaks trace by transit (paths traversing the leaking AS). The math lives in forensics.py behind a pure, unit-tested assembler.

Unsupervised ML: detect hijacks without a baseline

The rule-based detectors need a RIB baseline of legitimate origins. Can the hijacks be surfaced from observable features alone, with no labels and no baseline? An Isolation Forest over 8 scale-invariant per-prefix features, evaluated as a ranking task on 37,269 real BGP observations from two incident windows:

Incident	Observations	Base rate	Isolation Forest AP	Lift	Rule baseline AP
Indosat 2014	33,034	11.0%	0.34	3.1×	0.18
Rostelecom 2017	4,235	3.1%	0.47	15.3×	0.04

The unsupervised scorer surfaces hijacks 3–15× better than random and beats the single-rule baseline — with no labels. Notably, a supervised "which AS is the attacker" classifier hits AUC ≈ 1.0 by learning origin volume (the culprit is just the biggest announcer) — label leakage, not skill; the unsupervised ranking framing avoids it and uses the right imbalanced-data metric (average precision + lift, not accuracy). Methodology + honest caveats →

uv sync --extra ml && uv run python scripts/ml_anomaly_eval.py

For a live tap of the global routing table:

uv run netpulse stream

connects to the RIPE RIS Live WebSocket, maintains a 1-minute rolling window of updates from every collector, and runs detectors every 10s. On a healthy Internet you'll see ~50k updates/30s and alerts on real anycast / multi-homed prefixes (Google's AS15169/19527 footprint, etc.) — exactly the noise floor BENCHMARK.md describes.

To run the detectors as a JSON HTTP API instead:

uv run netpulse serve --store data/youtube_2008.duckdb \
                     --baseline data/baselines/yt_rib_filtered.duckdb
curl -s -X POST http://127.0.0.1:8000/detect/bgp \
    -H 'Content-Type: application/json' \
    -d '{"start_iso":"2008-02-24T18:00:00Z","duration_s":3600}'

Returns the same alerts the CLI prints, as JSON.

How it works

flowchart LR
  RIS[RIPE RIS<br/>RouteViews<br/>RIPE Atlas] -- MRT / JSON --> Ingest[ingest]
  Ingest --> Store[(DuckDB)]
  Store --> Features[features<br/>per-window aggregation]
  Features --> M[MOAS]
  Features --> S[Sub-prefix<br/>hijack]
  Features --> W[Withdraw<br/>spike]
  Features --> RP[RPKI<br/>RFC 6811]
  Features --> RL[Route leak<br/>RFC 7908]
  Features --> A[Atlas<br/>loss spike]
  M --> Alerts[Alerts]
  S --> Alerts
  W --> Alerts
  RP --> Alerts
  RL --> Alerts
  A --> Alerts
  Alerts --> Out[stdout / FastAPI / RIS Live stream]

Loading

Each stage is a thin module that talks to the next through DuckDB rather than in-memory queues, so any stage can be replayed independently — which is what makes the historical benchmark reproducible.

ARCHITECTURE.md has the full system diagram and the design rationale: why DuckDB, why detectors are pure functions, why there are three leak detectors instead of one, and how the TP/FN/GAP corpus methodology works.

Headline numbers

Seven labeled historical incidents of distinct shape, scored on real RIPE RIS archive data:

Incident	Shape	Outcome	Catching detector
2008-02-24 YouTube /24 hijack	sub-prefix hijack	TP	subprefix_hijack
2014-04-02 Indosat / AS4761 MOAS	sub-prefix hijack	TP	subprefix_hijack
2017-04-26 Rostelecom financials	sub-prefix hijack	TP	subprefix_hijack
2018-04-24 MyEtherWallet	sub-prefix hijack	TP	subprefix_hijack
2018-11-12 MainOne → Google leak	RFC 7908 Type-1 leak	TP	route_leak
2017-08-25 Google → Verizon → NTT	RFC 7908 Type-1 leak	TP	customer_cone_leak
2024-09-30 Vodafone Idea AS55410	RFC 7908 Type-1 leak	TP	customer_cone_leak

Indosat 2014: 19 alerts — 3 exact-prefix + 16 sub-prefix — both branches of SubPrefixHijackDetector fire from a single labeled case. MyEtherWallet: all five hijacked /24s flagged correctly (205.251.192/193/195/197/199.0/24 from AS10297 vs Amazon AS16509's /23 supernets). Onset at RRC00: 2018-04-24 11:05:50 UTC. MainOne: 1,985 leak alerts on the actual AS37282→AS15169 path shape using time-aligned CAIDA serial-2 (20181101) data. Google/NTT 2017: 123,749 leak alerts via the customer-cone-aware detector — the pair-direction valley-free check abstained, but Google's 2017-08 customer cone (10 ASes) provably does not contain NTT OCN, so the step 15169→4713 is "uphill" and the path is not cone-monotone. See docs/paper.md §3.2 for the TP / FN / GAP methodology.

False-positive survey of the BGP hijack detector across 5 hours of real RRC00 data (1 hijack hour + 4 background hours, 13,961 distinct prefixes total) using a real RIB-derived baseline:

Detector	Hour with hijack	4 background hours
subprefix_hijack	1 alert (TP)	0 alerts
moas	10 alerts	~40 alerts/hour
withdraw_spike	0 alerts	0 alerts

Plus:

• Multi-signal fusion — on the MainOne 2018 leak, the BGP route-leak detector fires (1,985 alerts on the actual leak shape using time-aligned CAIDA serial-2 data) at the same time as RIPE Atlas median RTT to 8.8.8.8 jumps 1.31× above baseline (38.0 ms → 49.9 ms). MultiSignalCorrelator binds them into one critical alert, with an optional third axis for DNS-reachability alerts on the same window (netpulse ingest dns + netpulse detect dns). Reproducible: scripts/fusion_demo.py.
• RPKI Origin Validation (RFC 6811) — netpulse ingest rpki pulls Cloudflare's published rpki.json (859k VRPs in ~20 s via DuckDB-native bulk load) and the validator gives the standard Valid / Invalid / NotFound classification.
• netpulse stream — runs detectors against the RIPE RIS Live WebSocket in real time; alerts deduplicated by fingerprint within a 5-minute cooldown.
• netpulse serve — FastAPI HTTP, POST /detect/bgp returns alerts as JSON.
• netpulse benchmark replay — incidents declare their own bgp_store_path, so one command scores the whole corpus.
• netpulse benchmark stream-latency — per-record streaming-mode detection latency. 0 µs from onset on both labeled sub-prefix incidents.
• Cross-collector aggregation — netpulse detect bgp accepts --in repeatedly; multiple DuckDB stores are attached read-only and exposed as a single UNION ALL view (src/netpulse/storage/multi_store.py).

Reproduction commands and methodology: BENCHMARK.md.

Reading list

• docs/paper.md — paper-style working note: abstract, methodology, latency characterization, multi-signal correlator, honest limitations, future work.
• docs/grafana/ — ready-to-import Grafana 10+ dashboard for the /metrics surface.
• BENCHMARK.md — full methodology, per-hour FPR table, reproduction commands, and an honest note on what the latency number does and does not mean.
• docs/why-subprefix.md — why a same-prefix multi-origin (MOAS) check cannot catch the canonical 2008 YouTube hijack, and what does. Long-form draft for external publication: docs/blog/.
• docs/comparison.md — where NetPulse fits next to ARTEMIS, BGPmon, Cloudflare Radar, RIPEstat / bgp.tools.
• docs/architecture.md — module boundaries and data-flow conventions.
• docs/references.md — RFCs, primary incident sources, and detection literature this draws on.
• PROJECT.md — full project context, roadmap, and the hard rules (no fabricated incident data, no invented API shapes, no over-engineering).

Performance

Profiled, optimized, and load-tested — full numbers + reproduction in PERFORMANCE.md:

• Detection pipeline: 1.71M records/sec on a real 131K-record RIS window, after a 12× sub-prefix-detector speedup (memoized prefix parsing; 92% cache hit rate) lifted it 4× from 420K/sec.
• API: 28 ms p50 per request, scaling ~linearly with workers (3.75× on 4) — the stateless app + DuckDB read-only opens let worker processes share one store. Load testing caught a real concurrency bug (write-lock contention) before it ever hit production.
• RPKI validation: 43 µs/call over 859K records (500× indexed).

Deploy

A Dockerfile ships the FastAPI surface as a container. The image bakes in the bundled YouTube fixture + RIB baseline, so a fresh deployment answers POST /detect/bgp against the canonical incident with no setup.

Full local stack — API (4 workers) + Prometheus + Grafana with the metrics dashboard auto-loaded:

docker compose up --build      # API :8000 · Prometheus :9090 · Grafana :3000

The deployed /health endpoint reports the loaded baseline size; /ready gates on a DuckDB sanity check; POST /detect/bgp accepts {start_iso, duration_s} and returns the same alerts the CLI prints, as JSON.

Render (free tier, no card): a render.yaml Blueprint is committed — connect the repo at dashboard.render.com/blueprints and Render builds the Dockerfile onto a *.onrender.com URL in ~3 min.

Fly.io: a fly.toml is committed for an always-on deploy.

flyctl deploy --app=<your-name>     # builds Dockerfile, ships

To swap stores, mount different DuckDB files as a volume and override the container CMD.

Install for full BGP/Atlas pulls

brew install bgpstream                  # macOS; Linux: bgpstream.caida.org/docs/install
CFLAGS="-I$(brew --prefix)/include" \
LDFLAGS="-L$(brew --prefix)/lib" \
    uv sync --extra bgp                 # adds pybgpstream
uv sync --extra viz                     # adds matplotlib for chart regeneration

Status

Pre-v1; the BGP detection path is benchmarked end-to-end and the Atlas signal is wired up. Multi-signal fusion, additional incidents, and the streaming/dashboard surfaces are tracked in PROJECT.md and BENCHMARK.md.

Development

make install   # uv sync, including dev deps
make lint      # ruff check + ruff format --check + mypy strict
make test      # pytest (skips integration tests by default)
make ci        # everything CI runs

License

MIT — see LICENSE.