MONITORING.md

Monitoring the Model Manager System

Real-Time Monitoring

The system logs all activity to model_manager.log. You can monitor it in real-time using:

Option 1: Use the monitor script

./monitor.sh

Option 2: Use tail directly

tail -f model_manager.log

Option 3: Watch with timestamps

tail -f model_manager.log | while read line; do echo "$(date +%T) | $line"; done

Option 4: Filter specific components

# Watch only scheduler activity
tail -f model_manager.log | grep SCHEDULER

# Watch only job execution
tail -f model_manager.log | grep ENGINE

# Watch queue operations
tail -f model_manager.log | grep QUEUE

Log File Location

• File: model_manager.log (in the ModelManager directory)
• Format: [HH:MM:SS.mmm] [COMPONENT ] Message
• Components: SYSTEM, QUEUE, SCHEDULER, ENGINE, MONITOR, REGISTRY, API

What Gets Logged

• Job submissions and status changes
• Scheduler decisions (load/unload/execute)
• Execution progress
• Resource monitoring (VRAM, loaded models)
• Errors and warnings

Example Session

Terminal 1 - Run the system:

python main.py

Terminal 2 - Monitor in real-time:

./monitor.sh

Terminal 3 - Submit jobs:

python test_system.py

Stopping the Monitor

Press Ctrl+C in the monitoring terminal. This only stops the monitor, not the system.