Skip to content

Amarusinggithub/southeastcon26_04_utechiee

Repository files navigation

Data Center Monitor - Architecture

IEEE SoutheastCon 2026 | UTech IEEE Student Branch

Real-time edge AI monitoring system for data center racks with anomaly detection and temperature forecasting, optimized for Raspberry Pi 5 deployment.


Table of Contents

  1. Project Overview
  2. Technology Stack
  3. System Architecture & Data Flow
  4. Machine Learning Models
  5. Synthetic Data Generation
  6. Developer Guide

1. Project Overview

The Data Center Monitor is an edge AI system designed to monitor data center server racks in real time. It ingests telemetry from physical sensors (temperature, power draw, airflow, utilization, humidity), detects anomalies using machine learning models, and forecasts future temperatures to provide early warnings. The system is built to run entirely on a Raspberry Pi 5, making it a lightweight, cost-effective, and self-contained monitoring solution.

Problem Statement

Data centers require constant monitoring to prevent hardware failures, overheating, and environmental hazards. Traditional solutions rely on centralized cloud processing, introducing latency and network dependency. This project brings AI inference to the edge, enabling:

  • Sub-second anomaly detection directly at the rack level.
  • Temperature forecasting up to 2 hours into the future.
  • Cold-start protection so new or restarted racks are monitored immediately.
  • Graceful degradation under resource constraints via system throttling.

Team Goals

  • Build a production-ready edge AI pipeline for the IEEE SoutheastCon 2026 hardware competition.
  • Demonstrate real-time ML inference on constrained hardware (Raspberry Pi 5, 2 GB RAM).
  • Support monitoring of up to 20 concurrent racks via an LRU caching strategy.
  • Provide a live dashboard with interactive charts and anomaly alerts.

2. Technology Stack

Backend

Technology Purpose Rationale
Python 3.9+ Primary language Broad ML library support, async capabilities
FastAPI Web framework & REST API Async-native, automatic OpenAPI docs, high performance
Uvicorn ASGI server Lightweight, production-ready server for FastAPI
SQLAlchemy (async) ORM & database abstraction Mature ORM with first-class async support
aiosqlite Async SQLite driver Non-blocking database access on single-board hardware
Redis In-memory data store Fast sliding window buffers, LRU rack tracking, anomaly log queues
Pydantic Data validation Type-safe schema enforcement for API payloads

Machine Learning & Inference

Technology Purpose Rationale
ai-edge-litert TFLite inference runtime Optimized for ARM CPUs (Raspberry Pi); replaces full TensorFlow
XGBoost Temperature forecasting Fast gradient-boosted trees; low memory footprint
scikit-learn Isolation Forest, preprocessing StandardScaler, Isolation Forest for cold-start detection
NumPy / Pandas Data manipulation Standard numerical and tabular data processing
joblib Model serialization Efficient persistence for sklearn and XGBoost models

Frontend & Visualization

Technology Purpose Rationale
Jinja2 HTML templating Server-side rendering for the dashboard
HTMX Interactive UI updates Lightweight alternative to full SPA frameworks
Plotly.js Interactive charts Real-time line charts for sensor data and forecasts
Server-Sent Events (SSE) Real-time streaming Unidirectional push from server to dashboard; simpler than WebSockets

DevOps & Testing

Technology Purpose Rationale
GitHub Actions CI/CD Automated testing on push/PR against Python 3.9 - 3.13
pytest / pytest-cov Testing & coverage Standard Python testing with coverage reporting
psutil System monitoring Tracks CPU, memory, and disk on the Raspberry Pi

3. System Architecture & Data Flow

3.1 File Structure

southeastcon26_04_utechiee/
|
+-- main.py                          # Entry point: starts FastAPI server
+-- run.sh                           # Bash helper: venv setup + launch
+-- requirements.txt                 # Python dependencies
|   +-- README.md              # This document
|
+-- src/                             # --- Core Application ---
|   +-- app.py                       # Singleton init: Redis, Streamer, Processor
|   +-- config.py                    # Global constants (features, paths, thresholds)
|   +-- ingest.py                    # FastAPI app + POST /telemetry/ endpoint
|   +-- dashboard.py                 # HTMX routes: /dashboard, /stream (SSE), /api/*
|   +-- database.py                  # Async SQLAlchemy engine + session factory
|   +-- models.py                    # ORM models (Telemetry, AnomalyLog, ForecastLog, etc.)
|   +-- schemas.py                   # Pydantic request schemas (TelemetrySchema)
|   +-- dal.py                       # Data Access Layer (all async DB operations)
|   |
|   +-- streamer.py                  # DataStreamer: state manager, buffer orchestration
|   +-- processor.py                 # DataProcessor: ML inference + anomaly diagnosis
|   +-- queue.py                     # Redis-backed SlidingWindowBuffer + AnomalyLogQueue
|   +-- sse.py                       # SSE publisher: pub/sub to dashboard clients
|   |
|   +-- anamoly_detector.py          # GRU TFLite inference wrapper
|   +-- forecaster.py                # XGBoost multi-horizon forecaster
|   +-- isolation_forest.py          # Isolation Forest cold-start detector
|   +-- system_montor.py             # Raspberry Pi resource monitor + throttling
|   |
|   +-- generate_data.py             # Synthetic data generator (training + live test)
|   +-- telemetry_sender.py          # Test client: sends CSV rows to API
|   +-- tflite_conversions.py        # TFLite conversion utilities (float16, int8, dynamic)
|   |
|   +-- train_gru_anomaly_detection_colab.ipynb    # GRU training notebook
|   +-- train_isolation_forest_colab.ipynb          # Isolation Forest training notebook
|   +-- train_xgboost_forecasting_colab.ipynb       # XGBoost training notebook
|
+-- models/                          # --- Pre-trained Models ---
|   +-- gru-model/
|   |   +-- gru_anomaly_detector.tflite             # Dynamic-range quantized GRU
|   |   +-- gru_anomaly_detector_int8.tflite         # Full INT8 quantized GRU
|   |   +-- model_config.pkl                         # {sequence_length, features, threshold}
|   |   +-- scaler.pkl                               # StandardScaler for normalization
|   +-- isolation-forest-model/
|   |   +-- isolation_forrest.pkl                    # Trained Isolation Forest
|   |   +-- isolation_forest_config.pkl              # {n_estimators, contamination, roc_auc}
|   +-- xgboost-model/
|       +-- xgboost_forecasters.joblib               # Dict of 4 XGBoost models (30m-120m)
|       +-- forecaster_config.pkl                    # {feature_cols, horizons, params}
|
+-- data/                            # --- Test Data ---
|   +-- training_data.csv            # 7-day multi-rack dataset with anomaly labels
|   +-- live_stream_test.csv         # 2-hour multi-rack live simulation dataset
|
+-- templates/
|   +-- index.html                   # HTMX dashboard template
|
+-- static/
|   +-- css/style.css                # Dashboard styling
|   +-- js/index.js                  # Plotly charts + SSE subscription logic
|
+-- tests/
|   +-- unit/test_config.py          # Config constant tests
|   +-- feature/test_dashboard.py    # Mock dashboard with synthetic data
|
+-- docs/
|   +-- ANOMALY_SCENARIOS.md         # Reference guide for anomaly patterns
|   +-- Data-Center-Monitor-Architecture.excalidraw
|   +-- Data-Center-Monitor-Architecture.png
|
+-- .github/workflows/System-ci.yml  # CI pipeline

Key Directories

Directory Responsibility
src/ All application logic: API, ML inference, database, streaming, data generation
models/ Serialized pre-trained model artifacts and their configurations
data/ CSV datasets for training and live-stream testing
templates/ + static/ Frontend: HTML template, JavaScript (Plotly/SSE), CSS
tests/ Unit and feature tests
docs/ Architecture docs, anomaly reference, diagrams

3.2 Data Pipeline

The system follows a linear pipeline from ingestion through inference to output:

Sensor/Test Client
       |
       | POST /telemetry/ (JSON)
       v
+----------------------------------------------+
|  1. INGESTION  (ingest.py)                   |
|  - Validate payload via TelemetrySchema      |
|  - Write raw reading to SQLite (async)       |
+----------------------------------------------+
       |
       v
+----------------------------------------------+
|  2. BUFFER MANAGEMENT  (streamer.py)         |
|  - Normalize reading via StandardScaler      |
|  - Add to Redis SlidingWindowBuffer          |
|  - On first access: backfill from DB         |
|  - LRU eviction if > 20 cached racks        |
+----------------------------------------------+
       |
       v
+----------------------------------------------+
|  3. MODEL INFERENCE  (processor.py)          |
|                                              |
|  IF buffer < 15 readings (cold-start):       |
|    -> Isolation Forest single-point detect   |
|                                              |
|  IF buffer >= 15 readings:                   |
|    -> GRU sequence anomaly detection         |
|                                              |
|  ALWAYS (unless throttled):                  |
|    -> XGBoost temperature forecast           |
|       (30m, 60m, 90m, 120m horizons)         |
|                                              |
|  IF system memory > 95%:                     |
|    -> Throttle: skip ML, keep telemetry      |
+----------------------------------------------+
       |
       v
+----------------------------------------------+
|  4. DIAGNOSIS & STORAGE                      |
|  - Classify anomaly cause (fan failure,      |
|    CPU overload, thermal runaway, etc.)       |
|  - Store anomaly to AnomalyLog table (async) |
|  - Push to Redis AnomalyLogQueue             |
+----------------------------------------------+
       |
       v
+----------------------------------------------+
|  5. OUTPUT  (sse.py -> dashboard.py)         |
|  - Publish SSE event "sensor-update"         |
|  - Dashboard receives real-time:             |
|    * Current sensor values                   |
|    * Anomaly status + probability            |
|    * Forecast series (4 horizons)            |
|    * Active model indicator (GRU/IF/None)    |
|    * Anomaly log history                     |
|  - Plotly.js renders interactive charts      |
+----------------------------------------------+

Step-by-Step Walkthrough

Step 1 - Ingestion: A sensor (or the test client telemetry_sender.py) sends a JSON payload to POST /telemetry/. The ingest.py endpoint validates it using the TelemetrySchema Pydantic model (fields: rack_id, temperature, utilization, humidity, airflow, power_draw). The reading is immediately written to the Telemetry SQLite table via the async DAL.

Step 2 - Buffer Management: The DataStreamer normalizes the reading through the GRU model's StandardScaler and appends it to a Redis-backed SlidingWindowBuffer keyed by rack_id. If this is the first reading for a rack, the buffer backfills up to 30 historical readings from the database. The buffer maintains a fixed window of the last MAX_LEN = 30 readings. An LRU tracker (Redis sorted set) evicts the oldest rack's buffer when the count exceeds MAX_CACHED_RACKS = 20.

Step 3 - Model Inference: The DataProcessor receives the current reading and the full buffer. It decides which model to run:

  • Cold-start (buffer has < 15 readings): The Isolation Forest runs single-point anomaly detection using raw (un-normalized) sensor values. At 10-second telemetry intervals, this cold-start period lasts up to 2.5 minutes — the reason the Isolation Forest exists as a bridge model.
  • Steady-state (buffer has >= 15 readings): The GRU model runs sequence-based anomaly detection on the normalized 15-step window. XGBoost forecasters predict temperatures at 30, 60, 90, and 120-minute horizons using the current raw reading.
  • Throttled (system memory > 95%): All ML inference is skipped; telemetry collection and DB writes continue.

Step 4 - Diagnosis & Storage: When an anomaly is detected, the processor diagnoses its likely cause based on threshold rules (e.g., airflow < 15 + temp > 60 = fan failure). Anomalies are persisted to the AnomalyLog table asynchronously and pushed onto the per-rack AnomalyLogQueue in Redis.

Step 5 - Output: The streamer publishes a sensor-update SSE event containing the current values, anomaly status, forecast series, active model indicator, anomaly log, and chart history. The JavaScript in index.js listens for these events and updates the Plotly charts and status banners in real time.


4. Machine Learning Models

4.1 Model Selection

The system uses three complementary models, each serving a distinct role:

Model Role Input When Active
GRU (Gated Recurrent Unit) Primary anomaly detector Sequence of 15 normalized readings After buffer fills (>= 15 readings)
Isolation Forest Cold-start anomaly detector Single raw data point During buffer warmup (< 15 readings, ~2.5 min)
XGBoost Temperature forecaster Single raw data point Always (unless throttled)

Why GRU? GRU is a recurrent neural network architecture that captures temporal dependencies across a sequence of readings. Unlike a simple threshold check, it can detect subtle drift patterns (e.g., gradual fan degradation) and compound failures by learning correlations between features over time. Compared to LSTM, GRU has fewer parameters (no separate cell state), making it faster and more memory-efficient on constrained hardware.

Why Isolation Forest? During the cold-start period (first 15 readings, ~2.5 minutes at 10-second intervals), the GRU buffer is not yet full. Isolation Forest provides immediate single-point anomaly detection without requiring a sequence. It works by isolating observations in a random forest; anomalies require fewer splits to isolate. It operates on raw feature values without normalization.

Why XGBoost? Gradient-boosted decision trees are lightweight, fast, and effective for tabular regression. With max_depth=4 and 100 boosting rounds, each model is small enough to run on the Pi without noticeable latency. Four separate models are trained for the four forecast horizons (30m, 60m, 90m, 120m).

4.2 GRU Anomaly Detector - Training Process

Notebook: src/train_gru_anomaly_detection_colab.ipynb

Data Preprocessing

  1. Feature columns: utilization, power_draw, airflow, temperature, humidity (5 features).
  2. Label conversion: Original labels {1, -1} are mapped to {0, 1} for binary classification (1 = anomaly, 0 = normal).
  3. Normalization: A StandardScaler is fitted on the full dataset (including anomaly readings to capture the full value range). The scaler is saved as scaler.pkl and used at inference time.
  4. Train/Test split: 80% train, 20% test (stratified, random_state=42). A further 15% of training data is held out for validation.

Sequence Creation

Sequences are created using a sliding window grouped by rack_id:

  • Window length: 15 timesteps (2.5 minutes at 10-second intervals).
  • For each window [i, i+15), the target label is the label at index i + 15.
  • Grouping by rack prevents cross-contamination between different racks' time series.

Model Architecture

Input: (batch, 15, 5)
  |
  +-- GRU Layer 1:    128 units, tanh, return_sequences=True, unroll=True
  +-- BatchNorm
  +-- Dropout(0.3)
  |
  +-- GRU Layer 2:    64 units, tanh, return_sequences=False, unroll=True
  +-- BatchNorm
  +-- Dropout(0.3)
  |
  +-- Dense:          32 units, ReLU
  +-- Dropout(0.2)
  |
  +-- Dense:          16 units, ReLU
  |
  +-- Output Dense:   1 unit, Sigmoid  -->  probability [0, 1]
  • unroll=True is set on both GRU layers to statically unroll the recurrence loop, which is required for TFLite conversion (TFLite does not support dynamic TensorList ops).

Hyperparameters

Parameter Value
Optimizer Adam
Learning rate 0.001
Batch size 128
Epochs 50 (with early stopping)
Loss function Binary Crossentropy
Class weights Balanced (auto-computed)
Early stopping patience 10 epochs (monitoring val_loss)
LR reduction patience 5 epochs, factor 0.5, min LR 1e-7

Threshold Selection

The optimal anomaly threshold is selected by maximizing the F1-score on the precision-recall curve:

f1 = 2 * (precision * recall) / (precision + recall)
threshold = argmax(f1)

The resulting threshold is saved in model_config.pkl and loaded at inference time.

Evaluation Metrics

  • Accuracy, Precision, Recall, AUC computed on the test set.
  • Confusion matrix (TN, FP, FN, TP).
  • Model checkpoint saved based on best val_auc.

4.3 Isolation Forest - Training Process

Notebook: src/train_isolation_forest_colab.ipynb

Data Preprocessing

  1. Feature columns: Same 5 features as GRU (utilization, power_draw, airflow, temperature, humidity).
  2. No scaling applied. Isolation Forest learns density in the original feature space. This is a deliberate design decision: the model benefits from the natural scale differences between features.
  3. Labels: Original {1, -1} convention kept (1 = normal, -1 = anomaly), matching sklearn's Isolation Forest output format.
  4. Train/Test split: 80/20, stratified, random_state=42.

Hyperparameters

Parameter Value
n_estimators 200 (tuned)
contamination Computed from training data anomaly ratio (tuned via sweep)
max_samples auto (min(256, n_samples))
max_features 1.0 (all features)
random_state 42
n_jobs -1 (all CPU cores)

Hyperparameter Tuning

Two grid sweeps are performed:

  • Contamination sweep: Values [0.01, 0.05, 0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40], optimizing F1-score.
  • n_estimators sweep: Values [50, 100, 150, 200, 300, 400, 500], using the best contamination.

Probability Mapping

The Isolation Forest's raw decision_function score is mapped to a [0, 1] probability:

anomaly_probability = clip(0.5 - raw_score, 0.0, 1.0)

Negative raw scores indicate anomalies (mapped closer to 1.0).

Evaluation Metrics

  • Classification report (per-class precision, recall, F1).
  • ROC AUC score.
  • Per-rack recall analysis across all 23 anomaly scenarios.
  • Permutation feature importance (10 repeats, F1-weighted scoring).

Model Export

  • isolation_forrest.pkl - joblib-serialized model.
  • isolation_forest_config.pkl - configuration dict (feature_cols, n_estimators, contamination, roc_auc).

4.4 XGBoost Forecaster - Training Process

Notebook: src/train_xgboost_forecasting_colab.ipynb

Data Preprocessing

  1. Feature columns: utilization, power_draw, airflow, temperature, humidity.
  2. Data sorted by rack_id and timestamp.
  3. Missing values checked (none expected in synthetic data).

Forecast Target Creation

For each horizon, the target Future_Temperature is created by shifting the temperature column backward:

Horizon Timesteps Shifted Time Offset
30m 180 180 x 10s = 30 minutes
60m 360 360 x 10s = 60 minutes
90m 540 540 x 10s = 90 minutes
120m 720 720 x 10s = 120 minutes

Rows where the shifted value is NaN (end of each rack's series) are dropped. One separate XGBoost model is trained per horizon.

Hyperparameters

Parameter Value
objective reg:squarederror
eval_metric RMSE
max_depth 4
eta (learning rate) 0.1
subsample 0.8
colsample_bytree 0.8
seed 42
num_rounds 100
early_stopping_rounds 10

Trees are kept shallow (max_depth=4) for computational efficiency on the Raspberry Pi.

Evaluation Metrics

  • RMSE (Root Mean Squared Error) in degrees Celsius.
  • MAE (Mean Absolute Error) in degrees Celsius.
  • R-squared score.
  • Error distribution histograms.
  • Feature importance by gain.

Model Export

  • xgboost_forecasters.joblib - dictionary of all 4 trained XGBoost models keyed by horizon name (30m, 60m, 90m, 120m).
  • forecaster_config.pkl - configuration dict (feature_cols, horizons, params, num_rounds, results).

4.5 Quantization

The GRU model is converted from Keras to TFLite with quantization for efficient edge deployment. The conversion logic is in src/tflite_conversions.py.

Available Quantization Strategies

Strategy File Technique Size Reduction Trade-offs
Dynamic Range gru_anomaly_detector.tflite Weights quantized to int8, activations remain float32 at runtime ~2x smaller Minimal accuracy loss; requires SELECT_TF_OPS for GRU ops
Float16 gru_anomaly_detector_float16.tflite Weights and activations in float16 ~2x smaller Very low accuracy loss; requires SELECT_TF_OPS
Full INT8 gru_anomaly_detector_int8.tflite Full integer quantization; weights and activations are int8 ~4x smaller Best speed on ARM CPUs; uses representative dataset for calibration; slight accuracy trade-off

Conversion Process

  1. Rebuild on CPU: The trained Keras model is rebuilt with unroll=True on CPU to eliminate CuDNN-specific ops that are incompatible with TFLite.
  2. Representative Dataset: For INT8 quantization, 100 samples from the test set are used as a calibration dataset to determine quantization ranges.
  3. Verification: Each converted model is loaded via the TFLite interpreter, tested with random input, and benchmarked for inference latency.

Production Deployment

The system currently uses the dynamic-range quantized model (gru_anomaly_detector.tflite) as the default. It is loaded at runtime via the ai-edge-litert package, which provides an optimized TFLite Interpreter tuned for ARM processors. The interpreter is configured with 4 threads (matching the Pi 5's quad-core CPU).


5. Synthetic Data Generation

Source: src/generate_data.py

The project uses a physics-informed synthetic data generator to produce realistic telemetry for training and testing. The generator creates multi-rack datasets with configurable anomaly scenarios injected at specific time windows.

Methodology

Base Signal Generation

For each rack, the generator creates a 7-day time series at 10-second intervals (60,480 steps per day):

  1. Utilization: A daily sinusoidal pattern (cos(2pi * (hour - 4) / 24)) with cumulative random walk drift, clipped to [10%, 90%].
  2. Power Draw: Linearly correlated with utilization: 100 + 200 * utilization + noise.
  3. Inlet Temperature: Sinusoidal daily cycle centered at 22 C with small noise.
  4. Airflow: Correlated with power draw: 20 + 0.15 * power_draw + noise (simulates cooling response).
  5. Temperature: Computed via a first-order thermal model:
    target = inlet_temp + k_thermal * (power_draw / airflow)
    temp[i] = 0.9 * temp[i-1] + 0.1 * target
    
    The thermal constant k_thermal = 8.0 and the exponential smoothing factor (0.9/0.1) create realistic thermal inertia.
  6. Humidity: Inversely correlated with temperature: 50 - (temp - 30) * 0.5 + noise.

Anomaly Injection

Each scenario modifies the base signals during a defined time window and sets the label to -1 (anomaly) for that period. 23 distinct scenarios are supported:

Category Scenarios
Cooling Failures Fan failure, gradual fan degradation, CRAC unit failure, blocked vents, hot aisle breach
Compute Overload CPU spike, crypto mining, overprovisioned VMs, memory leak
Power Anomalies Power supply failure, UPS battery drain, PDU overload, ground fault
Environmental Water leak, low humidity, external heat source, HVAC malfunction
Cascade/Compound Thermal runaway, failover overload, cooling + load storm
Hardware Degradation Thermal paste degradation, bearing failure, disk thrashing

Output Datasets

Two datasets are generated:

  1. training_data.csv - 7 days of data across 28 racks (5 normal + 23 with anomalies). Used for model training in the Colab notebooks.
  2. live_stream_test.csv - 2 hours of data across 26 racks (3 normal + 23 with anomalies). Anomalies are compressed into short time windows (10-60 minutes) appropriate for live testing. Generated on application startup if not present.

Labeling Rules

Data points are labeled as anomaly (-1) when any of the following conditions are true:

Condition Threshold
Temperature > 80 C
Airflow < 15 CFM
Utilization > 95% sustained
Power Draw > 320 W
Humidity < 20% or > 60%
Compound Temperature rising while airflow dropping
Efficiency Power draw does not match utilization pattern

Normal data (label 1) has all metrics within expected ranges with correlated physical relationships (higher utilization = higher power = higher temperature with proportional airflow response).


6. Developer Guide

Redis Setup

Installing Redis

# Linux (Ubuntu/Debian)
sudo apt update
sudo apt install redis-server

# macOS (Homebrew)
brew install redis

# Windows (WSL recommended)
# Install via WSL using the Linux instructions above

Starting Redis

# Linux (systemd)
sudo systemctl start redis-server

# macOS (Homebrew)
brew services start redis
# or run manually:
redis-server

# Verify Redis is running
redis-cli ping
# Should return: PONG

Stopping Redis

# Linux (systemd)
sudo systemctl stop redis-server

# macOS (Homebrew)
brew services stop redis
# or if running manually, use:
redis-cli shutdown

Resetting the Database and Redis

To delete the SQLite database and flush all Redis data (useful for a clean restart):

rm ./database.db && redis-cli FLUSHALL

Running the Application

# 1. Ensure Redis is running (see Redis Setup above)

# 2. Install dependencies
pip install -r requirements.txt

# 3. Start the server
python main.py

# Server starts at:
#   Dashboard: http://localhost:8000/dashboard
#   API Docs:  http://localhost:8000/docs
#   Telemetry: POST http://localhost:8000/telemetry/

Alternatively, use the helper script:

bash run.sh

Sending Test Telemetry

Use the test client to stream CSV data to the API:

python -m src.telemetry_sender

Or send a single reading via curl:

curl -X POST http://localhost:8000/telemetry/ \
  -H "Content-Type: application/json" \
  -d '{
    "rack_id": "RACK_01",
    "temperature": 55.0,
    "utilization": 65.0,
    "humidity": 42.0,
    "airflow": 40.0,
    "power_draw": 230.0
  }'

Common Development Tasks

Adding a New Sensor Feature

  1. Add the feature name to the FEATURES list in src/config.py.
  2. Add the field to TelemetrySchema in src/schemas.py and the Telemetry ORM model in src/models.py.
  3. Update the synthetic data generator in src/generate_data.py to produce values for the new feature.
  4. Retrain all three models using the updated training data (notebooks in src/).
  5. Update the dashboard template (templates/index.html) and JavaScript (static/js/index.js) to display the new metric.

Adding a New Anomaly Scenario

  1. Define the scenario's behavior in docs/ANOMALY_SCENARIOS.md.
  2. Add a new elif branch in generate_rack_data() in src/generate_data.py that modifies the appropriate base signals.
  3. Add the scenario to the scenarios list (for training data) and live_scenarios list (for live test data) in the same file.
  4. Regenerate training data: python -m src.generate_data.
  5. Retrain models using the updated CSV in the Colab notebooks.
  6. If the scenario has a distinct diagnostic signature, add a detection rule in DataProcessor._diagnose_anomaly() in src/processor.py.

Adding a New ML Model

  1. Create the training notebook in src/ following the existing naming convention (train_<model>_colab.ipynb).
  2. Save the trained model artifacts to a new directory under models/.
  3. Create an inference wrapper class in src/ (see src/anamoly_detector.py or src/forecaster.py as templates).
  4. Register the new model in src/app.py by instantiating it in create_streamer() and passing it to DataProcessor.
  5. Update DataProcessor.analyze_live_reading() in src/processor.py to call the new model at the appropriate point in the pipeline.
  6. Add model path constants to src/config.py.

Modifying the Dashboard

  • HTML template: templates/index.html (Jinja2 + HTMX).
  • Charts & SSE logic: static/js/index.js (Plotly.js). The EventSource connects to /stream and parses sensor-update events.
  • Styling: static/css/style.css.
  • Backend routes: src/dashboard.py (add new API endpoints or SSE event types).
  • SSE data shape: To add new fields to the SSE payload, update the sse.publish() call in DataStreamer.get_next_reading() in src/streamer.py.

Modifying the Database Schema

  1. Add or modify the ORM model in src/models.py.
  2. Add corresponding DAL functions in src/dal.py.
  3. Delete the existing database.db file (tables are auto-created on startup via Base.metadata.create_all).
  4. Restart the application.

Running Tests

# Run all tests
pytest

# Run with coverage
pytest --cov=src --cov-report=html

Key Configuration Constants

All tunable constants are centralized in src/config.py:

Constant Value Description
FEATURES ["utilization", "power_draw", "airflow", "temperature", "humidity"] Ordered list of sensor features
MAX_LEN 30 Redis buffer size (stores up to 30 readings for chart backfill; GRU uses the last 15)
MAX_CACHED_RACKS 20 Maximum concurrent rack buffers in Redis (LRU eviction)
ANOMALY_LOG_MAX_LEN 10 Maximum anomaly entries per rack in the log queue
REDIS_HOST localhost Redis server address
REDIS_PORT 6379 Redis server port
API_QUEUE_MAX_LEN 100 Rate-limiting queue capacity

Architecture Diagram

Data Center Monitor Architecture

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors