Skip to content

task.md

Latest commit

 

History

History
152 lines (134 loc) · 8.44 KB

task.md

File metadata and controls

152 lines (134 loc) · 8.44 KB
Raw

SOC Platform Upgrade — Task Tracker

Phase 1: Auth Persistence Fix

  • Fix _load_users() to reload from disk (not cache at init)
  • Add atomic write in _save_users()
  • Add role field to user records (owner/admin/user)
  • Pre-seed users.json with owner account
  • Test: register → restart → login works

Phase 2: Secure Secret Manager

  • Create services/secret_manager.py
  • Implement st.secrets primary + Fernet-encrypted fallback
  • Add get_secret(), set_secret(), mask_secret(), list_secrets()
  • Migrate auth_service.py to use secret_manager
  • Test: secret read/write/mask round-trip

Phase 3: Owner-Only Admin Dashboard

  • Create pages/25_Admin.py with require_admin() guard
  • Implement User Management tab
  • Implement Secrets/Config management tab
  • Implement Audit Logs tab
  • Implement Incidents/Playbook controls tabs
  • Register admin page in dashboard.py navigation
  • Test: non-admin gets denied, admin sees all tabs

Phase 4: FCM + Combined Analysis Improvement

  • Increase training subsample to 15K
  • Add feature engineering (protocol/service one-hot encoding)
  • Add Hungarian algorithm for cluster-label alignment
  • Add Davies-Bouldin index to evaluate()
  • Build combined scoring: IF anomaly × FCM membership fusion
  • Add confusion matrix, ROC/PR curves to Analysis page
  • Report honest before/after metrics

Phase 5: MFA + Session Persistence

  • Add pyotp TOTP support in auth_service.py
  • Bug Fix: Resolved NameError (missing Tuple) in ml_engine/geo_predictor.py.
  • Grounding UBA: Learning from real SIEM data.
  • Grounding Executive Metrics: MTTR, MTTD, ROI, and SNR are now dynamic.
  • Ground Main Dashboard Metrics (Completed)
    • Refactor load_soc_data to use DatabaseService.
    • Add "Live System" / "Simulation Mode" indicator.
    • Fix empty state bug with robust fallback.
    • HOTFIX: Resolved "Blank Screen" regression caused by premature st.rerun().
  • Audit and Ground All Pages (Completed)
    • Audit and fix 05_Timeline.py (Added Live/Sim badges).
    • Audit and fix 08_Kill_Chain.py (Added Live/Sim badges).
    • Audit and fix 09_Dark_Web.py (Added Live/Historical badges).
    • Fix missing methods in services/threat_intel.py.
    • Verify transparency across all modules.
  • Final Verification
    • Run full system test (Manual verification recommended).
  • Bug Fix: Resolved NameError (missing time) in pages/08_Kill_Chain.py.
  • Bug Fix: Resolved NameError (missing imports) in pages/08_Kill_Chain.py.
  • Bug Fix: Resolved AttributeError in fuzzy_clustering.py.
  • Implement session token persistence (data/sessions.json)
  • Auto-login on valid session token
  • Add "Remember this device" option
  • Update requirements.txt with new dependencies

Phase 6: Final Verification

  • Run all tests
  • Verify no theme/style regressions
  • Git commit and push
  • Write walkthrough

Phase 7: Enhancements & Fixes (User Requested)

  • Fix Settings API Key mismatch (Gemini -> Groq)
  • Add AbuseIPDB configuration
  • Make Executive Dashboard dynamic (connect to DB)

Phase 8: ML Accuracy Boost (>95%)

  • Implement Supervised Fuzzy Centroid Classification in fuzzy_clustering.py
  • Tune Isolation Forest ensemble weights (95% RF / 5% IF)

Phase 9: API & Auth Fixes (Completed)

  • Implement reload_config() in ThreatIntelligence and AIAssistant
  • Update Settings page to reload services on save
  • Add Gmail SMTP & Twilio config fields to Settings
  • Implement persistent sessions.json token storage
  • Add "Remember this device" checkbox to Login page
  • Verify auto-login on restart logic

Phase 10: UI/UX & RBAC Polish (Verified)

  • Fix Admin panel syntax error (pages/25_Admin.py)
  • Update sidebar_manager.py (hide Admin page, bold headers)
  • Refactor Settings.py (Role-based view: System vs User)
  • Enhance Login.py with animations and better styling

Phase 12: Global API Integration (Current Focus)

  • Alerts Page: Enable AlienVault OTX & AbuseIPDB enrichment for all displayed alerts
  • Executive Dashboard: Replace static "0.0h" metrics with real calculations from incident_history.json
  • Threat Intel: Verify VirusTotal/AbuseIPDB real-time lookups
  • Logs: Ensure "Live Stream" actually reads from services.log_ingestor
  • Reports: Generate real PDF reports using fpdf and dynamic data
  • Settings: Validate API keys on save (check connectivity)

Phase 13: Advanced Analytics & ML Optimization (Current Focus)

  • Fuzzy C-Means: Improve clustering accuracy (currently ~52%) via feature scaling and hyperparameter tuning
  • Data Preprocessing: Implement RobustScaler/MinMaxScaler to handle outliers in NSL-KDD
  • Visualization: Enhance cluster visualization with PCA/t-SNE for better separability

Phase 14: Comprehensive API Integration & Live Data Updates

  • API Configuration Audit: Verified keys for VirusTotal, AbuseIPDB, OTX, Groq.
  • Service Layer Enhancements:
    • Updated services/threat_intel.py with real pulse matching and fallback logic.
    • Improved get_country_threat_counts with weighted geographic distribution.
  • Page Integration & Live Data:
    • pages/01_Dashboard.py: Wired up live stats and added manual Refresh.
    • pages/06_Threat_Intel.py: Integrated force_refresh and auto-feed.
    • services/background_monitor.py: Integrated with log_ingestor for live feed.
  • Verification: Verified end-to-end data flow and refresh functionality.
  • GitHub Sync: Pushing all changes to repository.

[x] Phase 14-B: Global API Integration & Continuous Data Refresh [x] ML Synchronization (Grounding Models in Reality) [x] Update geo_predictor.py to ingest real OTX pulse counts (Replace synthetic history). [x] Ensure fuzzy_clustering.py handles live SIEM data streams. [x] Continuous Updates (Apply to all 12 platform pages) [x] Executive Dashboard: Connect KPI cards (Uptime, MTTR) to real database statistics + Refresh button. [x] Geo Predictions: Implement Refresh and automated model re-prediction. [x] Kill Chain: Link "Active Threats" to live SIEM alerts + Refresh stage mapping. [x] Log Viewer: Implement non-blocking auto-refresh mechanism. [x] Final Verification: Confirm all pages update without manual script restarts.

Phase 15: Continuous Live Data Ingestion (User Requested)

  • Create Background Job: services/background_job.py to poll APIs independently.
  • Enhance siem_service: Add ingest_live_threats() method for higher volume/frequency.
  • Data Pipeline: Ensure new threats are auto-converted to SIEM events (Access Attempts/Blocks).
  • Verification: Run background job and verify 01_Dashboard.py updates without manual refresh.
  • Data Restoration: Backfilled 2,000 historical events to restore dashboard volume metrics.
  • Cloud Self-Healing: Implemented auto-seeding in siem_service.py to fix empty state on Streamlit Cloud.

Phase 16: Cloud-Ready Background Service

  • Create Thread Manager: Implement services/cloud_background.py using threading and st.cache_resource.
  • dashboard Integration: Start the background thread on 01_Dashboard.py load.
  • Verification: Ensure thread is singleton and doesn't duplicate on reload.

Phase 17: Inter-Page Consistency & Real-Time Sync (User Requested)

  • Unified Data Model: Ensure ingest_live_threats generates both Events AND Alerts (so Alerts page updates).
  • Page Auto-Refresh: Add "Live Mode" toggles or auto-refresh concepts to:
    • 03_Alerts.py
    • 05_Timeline.py
    • 06_Threat_Intel.py
    • 24_SIEM.py
  • Verification: Confirm that a single ingested threat appears on Dashboard, Alerts, SIEM, and Timeline.

Phase 18: Executive & Real-Time Expansion

  • Executive Dashboard: Add auto-refresh to 02_Executive.py and ensure SOCMonitor pulls live DB stats.
  • UBA Real-Time: Add auto-refresh to 12_UBA.py.
  • Threat Hunt: Connect "Hypothesis Hunt" in 10_Threat_Hunt.py to db.search_events() for real results.
  • Global Sync: Verify that a background threat block updates the Executive KPI cards immediately.