sourcepulse

Platform-agnostic silent log source detection. Learns each source's cadence and volume from your own logs, then alerts when a feed goes quiet or drops off trend. A dead log source is a blind spot the SOC doesn't know it has.

Zero runtime dependencies. Python 3.11+. Apache-2.0.

Why this exists

When a firewall stops forwarding, an agent silently dies, or a parser change eats a feed, nothing pages. Detections don't fire on logs that never arrive; dashboards render fine with yesterday's data. Every platform treats this as its own feature: Google SecOps has silent source monitoring, Splunk has the TrackMe app, Graylog publishes blog posts telling you to build it yourself. If your logs live anywhere else, or in three places at once, you have nothing.

sourcepulse is the missing standalone: point it at logs (files, directories, or stdin), let it learn what "normal" looks like per source, then run a check from cron and page on exit code.

Install

pip install .

Standard library only; nothing else comes with it.

Quick start

# 1. Learn per-source baselines from a week of history
sourcepulse baseline /var/log/remote/ --out sourcepulse-baseline.json

# 2. Check current logs against them (exit 1 when something is wrong)
sourcepulse check /var/log/remote/ --window 3600

Example check output:

[HIGH] silent: fw01.example.com  (no events for 4h12m, baseline p95 gap 1m, allowed 15m)
[MEDIUM] degraded: web02.example.com  (5 events in the last 1h00m, baseline ~60 (ratio floor 0.2))
[INFO] new: mystery.example.com  (source has no baseline (new feed, renamed host, or spoof))

14 sources baselined: 12 healthy, 2 alerting, 1 new

--json gives the same as machine-readable output.

How detection works

baseline builds a per-source profile from history using robust statistics (median, gap percentiles), so one outage inside the training window doesn't poison the profile: events per active hour, inter-arrival gaps (p50/p95/max), first and last seen.

check then applies three tests:

• silent (high): the time since the source's last event exceeds max(gap_multiplier * p95_gap, max_gap, min_gap_seconds). An hourly backup job is allowed hours of quiet; a chatty firewall is not.
• degraded (medium): the source is still talking, but its recent window rate fell below volume_ratio of its baseline hourly median. Only applies to sources chatty enough to judge (min_hourly_for_volume).
• new (info): events from a source with no baseline. Usually someone plugged something in; occasionally someone is spoofing a feed. Either way you want to know.

Exit codes: 0 all healthy (new sources alone don't fail), 1 at least one silent or degraded source, 2 usage error. That makes the cron recipe the whole deployment:

*/15 * * * * sourcepulse check /var/log/remote/ || notify-send-somehow

Input formats

--format auto (default) recognizes per line:

• syslog RFC 3164 (Aug 26 01:02:03 host app: msg, year borrowed from the clock with rollover handling) and RFC 5424
• JSON lines, trying common field names for time (@timestamp, time, ts, ...) and source (host, hostname, agent.name, ...); override with --time-field / --source-field (dotted paths work)
• CSV as timestamp,source[,...]

Timestamps may be ISO 8601 (any offset; normalized to UTC) or epoch seconds/milliseconds. Unparseable lines are counted and skipped, never fatal. --now overrides the clock for replays and tests.

Configuration

Optional .sourcepulse.toml next to where you run check (or --config path):

gap_multiplier = 3.0          # allowed gap = multiplier * p95 gap
min_gap_seconds = 900.0       # never alert on gaps shorter than this
volume_ratio = 0.2            # degraded below this fraction of baseline
min_hourly_for_volume = 5.0   # volume check only for chatty sources
window_seconds = 3600.0       # recent-activity window

Handling the baseline file

The baseline JSON contains your source names and cadence facts. Treat it as internal. If you ever need to share one (bug report, blog post), run it through opsec-scrub first; that is exactly what it is for, and this repository's own CI runs the same scan gate.

Limitations, stated plainly

• Baselines are static snapshots: rebuild them on a schedule (weekly cron) so drift doesn't stale them. There is no online learning.
• No seasonality model in v0.1: a source that is quiet every weekend needs gap_multiplier headroom or a weekday-built baseline. Robust gap percentiles absorb most of this in practice.
• Source identity is the parsed string: a renamed host is a new source plus a silent one, which is arguably the correct alarm.
• Reads files and stdin; it does not tail, and it makes no network connections. Feed it what your collector already writes to disk.

License

Apache-2.0. Copyright 2026 NorthQuinn Inc.