🔍 Log Anomaly Detection & Alerting System

A production-grade monitoring tool that ingests application/server logs, detects error-rate anomalies using statistical thresholding, fires real-time Telegram alerts, and serves a dark-themed Flask dashboard with SQLite-backed historical trend visualization.

---

✨ Features

Feature	Details
Multi-format Log Parsing	Apache/Nginx, Python logging, syslog — with auto-detection
Dual Anomaly Detection	Rolling Z-score (adaptive) + static rate threshold (fallback)
Severity Classification	LOW / MEDIUM / HIGH based on Z-score and error-rate thresholds
Real-time Telegram Alerts	Formatted messages with retry logic + cooldown deduplication
SQLite Storage	All anomaly events persisted with full metadata
Flask Dashboard	Dark-themed UI with Chart.js line/doughnut charts + filterable event table
Background Scheduler	APScheduler polls every 60s (configurable) — no cron needed
Sample Log Generator	Inject realistic error spikes for immediate demo

---

🏗️ Architecture

Log Files (disk)
      │
      ▼
 LogWatcher  ── polls new lines by byte offset (tail-like)
      │
      ▼
 LogParser   ── Regex → normalised pandas DataFrame
      │
      ▼
 Thresholder ── Rolling Z-score + static rate check → AnomalyEvents
      │
     YES
      ├──► SQLite  (store event)
      └──► TelegramAlert  (send notification) 📱
                │
 Flask Dashboard ◄──── SQLite (query history)

---

🚀 Quick Start

1. Clone & Install

git clone https://github.com/adhiharan-h/log-anomaly-dectection-system.git
cd log-anomaly-dectection-system

python -m venv venv
source venv/bin/activate        # Windows: venv\Scripts\activate
pip install -r requirements.txt

2. Configure

cp .env.example .env
# Edit .env with your Telegram credentials and preferences

Required .env values:

TELEGRAM_BOT_TOKEN=your_bot_token
TELEGRAM_CHAT_ID=your_chat_id

3. Generate Sample Logs (optional demo)

python tools/generate_sample_logs.py
# Creates logs/sample_apache.log, logs/sample_python.log, logs/sample_syslog.log
# Each file contains ~300 lines with 3 injected error spikes

4. Run

python run.py

• Dashboard: http://localhost:5000
• Telegram receives a startup ping + anomaly alerts as logs are processed

---

📁 Project Structure

log-anomaly-dectection-system/
├── app/
│   ├── ingestion/
│   │   ├── log_parser.py     # Multi-format log parsing
│   │   └── log_watcher.py    # File watcher with byte-offset tracking
│   ├── detection/
│   │   └── thresholder.py    # Z-score + rate anomaly detection
│   ├── alerting/
│   │   └── telegram_alert.py # Telegram Bot notifications
│   ├── storage/
│   │   ├── database.py       # SQLAlchemy ORM + AnomalyEvent model
│   │   └── repository.py     # CRUD operations
│   ├── dashboard/
│   │   ├── routes.py         # Flask API + HTML routes
│   │   └── templates/
│   │       └── index.html    # Dashboard UI (Chart.js)
│   └── scheduler.py          # APScheduler job definitions
├── tools/
│   └── generate_sample_logs.py
├── tests/
│   ├── test_parser.py
│   ├── test_thresholder.py
│   └── test_alerts.py
├── config.py
├── run.py
├── requirements.txt
└── .env.example

---

⚙️ Configuration Reference

Variable	Default	Description
TELEGRAM_BOT_TOKEN	—	Bot token from @BotFather
TELEGRAM_CHAT_ID	—	Your Telegram chat/user ID
LOG_DIR	./logs	Directory to watch for log files
LOG_FORMAT	auto	auto, apache, python, syslog
ZSCORE_THRESHOLD	3.0	Z-score above which a window is anomalous
ERROR_RATE_THRESHOLD	0.10	Fraction (0–1) above which a window is anomalous
ROLLING_WINDOW_MINUTES	30	Lookback window for rolling baseline
ALERT_COOLDOWN_MINUTES	5	Min time between alerts for the same source
POLL_INTERVAL_SECONDS	60	How often the scheduler polls for new lines
FLASK_PORT	5000	Dashboard port
DATABASE_URL	sqlite:///anomalies.db	SQLAlchemy DB URL

---

🔬 Detection Algorithm

Strategy A — Rolling Z-Score

1. Bucket log lines into 1-minute windows
2. Compute error count per bucket
3. Calculate rolling mean and std over the last N windows (configurable)
4. Z-score = (current_count - rolling_mean) / rolling_std
5. Flag windows where Z ≥ ZSCORE_THRESHOLD

Strategy B — Static Rate Threshold

• Flag any window where error_count / total_count ≥ ERROR_RATE_THRESHOLD

Severity

Condition	Severity
Z-score ≥ 4 or error rate ≥ 20%	🔴 HIGH
Z-score ≥ 3 or error rate ≥ 10%	🟠 MEDIUM
Otherwise	🟡 LOW

---

🤖 Telegram Setup

1. Open Telegram → message @BotFather → /newbot
2. Copy the bot token → paste in .env
3. Start a chat with your bot, then visit:

   https://api.telegram.org/bot<TOKEN>/getUpdates
   

4. Find "chat": {"id": ...} → paste that ID as TELEGRAM_CHAT_ID in .env

---

🧪 Running Tests

pytest tests/ -v

---

📜 CLI Options

python run.py                  # Full server (Flask + scheduler)
python run.py --once           # Run one detection cycle and exit
python run.py --no-telegram    # Skip startup Telegram ping

---

📊 Dashboard Preview

The dashboard features:

• Live stat cards — anomalies today, high-severity count, all-time total, most impacted source
• Error rate line chart — time-series with anomaly markers
• Severity distribution doughnut chart
• Filterable event table — filter by date range, source, and severity
• Auto-refresh every 60 seconds

---

🛠️ Tech Stack

• Python 3.11+, Pandas, SciPy/NumPy
• Flask 3 — dashboard API
• SQLAlchemy 2 + SQLite — persistent storage
• APScheduler 3 — background polling
• Telegram Bot API — real-time alerts
• Chart.js 4 — dashboard visualizations

---

📄 License

MIT License — see LICENSE