A Comparative Study of Supervised and Unsupervised Machine Learning for Lateral Movement Detection in Enterprise Networks
Author: Rahul Karmakar (@karmakar-rahul)
This project investigates lateral movement detection, the process by which an attacker, after compromising a single endpoint, moves across hosts to escalate privileges and access high-value targets—as both a supervised classification and an unsupervised anomaly detection problem. Detection is evaluated on two datasets: a synthetic, fully labeled authentication log generator developed for this project and a stratified extraction of the LANL Comprehensive Multi-Source Cyber Security Events dataset, representing real enterprise authentication logs.
Rather than a standalone offline classifier, the system is implemented as a complete real-time analytics pipeline with Kafka ingestion, MongoDB storage, live feature engineering, inference, and an operational dashboard, enabling evaluation under continuously arriving network traffic. XGBoost, Random Forest, SVM, and Isolation Forest are compared on identical held-out synthetic data; Isolation Forest, Dense Autoencoder, LSTM Autoencoder, and ECOD are evaluated on LANL traffic using threshold-free, alert-budget-based metrics; and a Quantum SVM is assessed as a standalone supervised experiment on a small subsample. During live inference, the pipeline automatically selects the best-performing model instead of relying on a fixed algorithm.
- A synthetic authentication-log generator that models an organizational network (departments, host roles, per-user access baselines) and simulates lateral-movement attack chains as genuine multi-hop sequences rather than independently-labeled anomalous rows.
- A full streaming pipeline (Kafka → Consumer → MongoDB → FastAPI → React dashboard) rather than an offline notebook, allowing detection quality to be assessed under live, continuously-arriving traffic.
- A rigorous evaluation methodology: every model compared in this project — regardless of algorithm or dataset — is trained and evaluated on an identical, fixed train/calibration/test split, so reported differences reflect the models rather than sampling variance.
- Validation against a real, unlabeled production authentication log (LANL CSSE), extracted and sampled to preserve every rare true-positive event while remaining computationally tractable, with ground-truth labels used strictly at evaluation time and never during training or threshold selection.
- A four-way fully-unsupervised comparison on LANL data (Isolation Forest, Dense Autoencoder, LSTM Autoencoder, ECOD), evaluated at three levels — point-threshold, event-level Precision@K, and entity-window (user × hour) Precision@K — reflecting how a real SOC actually triages a fixed alert budget rather than a probability cutoff.
- A standalone Quantum SVM (QSVC) experiment on a PCA-reduced, small-subsample slice of the synthetic dataset, kept methodologically separate from the main supervised comparison due to its differing sample size and feature space.
- Live inference that automatically adapts to the currently best-performing model (by F1) rather than being hard-coded to one algorithm, re-evaluated on every pipeline restart directly from the same evaluation artifacts the dashboard reads.
The system consists of two cooperating pipelines that share a common feature-engineering definition.
Live pipeline. A producer generates synthetic authentication events and publishes them to a Kafka topic. A consumer subscribes to this topic, computes behavioral features over rolling time windows, scores each event, and writes the enriched event to MongoDB. Scoring is handled by ml_engine/predict.py's AnomalyPredictor, which automatically loads whichever model — Isolation Forest or XGBoost — currently has the higher F1 score according to the latest evaluation artifacts on disk, falling back safely to Isolation Forest if a comparison file or model artifact is missing. A FastAPI service exposes REST endpoints over this collection, and a React (TanStack Start) single-page application polls these endpoints to render a live operations dashboard.
Offline / evaluation pipeline. Raw events — either freshly generated or pulled from MongoDB's accumulated live traffic — are passed through the same feature-engineering logic used by the live consumer, chronologically replayed per user, and written to a static training dataset. Each candidate model is then trained and evaluated against an identical held-out split of this dataset, and the resulting metrics are written to disk for both the dashboard and the live predictor's champion-selection logic to read.
| Layer | Technology |
|---|---|
| Stream ingestion | Apache Kafka |
| Storage | MongoDB |
| Feature engineering & modeling | Python, pandas, scikit-learn, XGBoost, PyTorch, pyod, Qiskit |
| API | FastAPI |
| Dashboard | React, TanStack Start/Router, TanStack Query, Recharts, Tailwind CSS |
| Environment management | Conda |
Figure 1. Live Overview dashboard displaying event statistics, anomaly distribution, and recent authentication events.
A synthetic generator simulates 400 user accounts (standard, service, and administrative) across approximately 950 hosts, each with a baseline of expected host accesses. Lateral-movement attacks are generated as realistic multi-hop chains rather than isolated anomalous events, producing coherent behavioral features such as host-hopping, first-time access, and rare-protocol usage. Every event is assigned a ground-truth label (NORMAL, LATERAL_MOVEMENT, INTERNAL_RECON, or PRIVILEGE_ESCALATION) for supervised training and threshold calibration, but these labels are never used during live inference.
To validate against real enterprise traffic, the project uses a stratified subset of the LANL Comprehensive Multi-Source Cyber-Security Events dataset, containing authentication logs collected over 58 days (~1.6 billion original events), together with the separate redteam.txt ground-truth log. All events involving compromised users are retained to preserve every rare true-positive example, while the remaining users are deterministically sampled at the user level, preserving complete chronological timelines. The resulting dataset contains ~7.02 million events across 272 users, including 704 confirmed red-team compromise events (~0.0100% attack rate). Training remains fully label-free: ground-truth labels are stored separately and joined only during evaluation.
Two engineered features—bytes_out_ratio and privileged_account_flag—are unavailable because the LANL authentication logs contain neither byte-count nor account-role information, so these features are excluded from LANL-specific model training.
Both data sources map onto an identical fifteen-feature representation, computed over rolling five- and fifteen-minute windows per user.
| Feature | Behavioral signal |
|---|---|
unique_dst_hosts_5m, unique_src_hosts_5m |
Breadth of host-hopping in the recent window |
unique_protocols_5m |
Protocol diversity in the recent window |
auth_event_rate_5m |
Authentication burst rate |
failed_auth_count_5m |
Failed-authentication frequency |
success_after_fail_count_10m |
Credential-guessing / spray-then-access pattern |
new_user_host_pair, new_src_dst_pair |
First-time-seen access — a principal lateral-movement indicator |
rare_protocol_flag, admin_protocol_flag |
Use of RDP, WinRM, SMB, or SSH relative to routine Kerberos, NTLM, or LDAP traffic |
privileged_account_flag |
Whether the acting account holds administrative privilege |
lateral_hop_count_15m, fanout_score_15m |
Distinct host-pairs touched in the window — the principal hopping signature |
bytes_out_ratio |
Outbound-to-inbound data ratio |
off_hours_flag |
Authentication outside conventional business hours |
Four methodologically distinct evaluation regimes are used, and the distinction between them is treated as a first-class result rather than an implementation detail.
Supervised (synthetic data). XGBoost, Random Forest, and SVM are each trained on a labeled 60% training split, with class-imbalance handling (scale_pos_weight for XGBoost; class_weight="balanced" for the other two). A decision threshold is chosen independently per model by maximizing F1 on a held-out 20% calibration split, then all three are scored on an identical, previously unseen 20% test split.
Semi-supervised (synthetic data). Isolation Forest is fit only on rows assumed normal within the training split; labels select this subset but are never passed to tree construction. Threshold selection follows the same F1-maximization procedure on the calibration split as the supervised models.
Fully unsupervised (LANL data). Isolation Forest, Dense Autoencoder, LSTM Autoencoder, and ECOD are trained on the entire stratified LANL dataset without using labels at any stage. Decision thresholds are derived solely from label-free signals—Isolation Forest's contamination parameter or the 99.5th percentile of training-set reconstruction error/outlier score for the remaining models—with ground-truth red-team labels joined only during evaluation. Because confirmed attacks comprise only ~0.01% of all events, conventional threshold-based metrics inevitably yield many false positives despite good ranking performance. Evaluation therefore reports three complementary views: (1) point-threshold Accuracy, Precision, Recall, and F1; (2) threshold-free event-level Precision@K/Recall@K; and (3) entity-window (user × hour) Precision@K/Recall@K, which better reflects real UEBA/SOC workflows by prioritizing suspicious user sessions rather than individual authentication events.
Standalone supervised experiment (QSVM, synthetic data). A Quantum SVM (QSVC, simulated via Qiskit) is trained on a 2,000-row stratified subsample of the synthetic dataset, with features PCA-reduced from 15 to 5 components (one per qubit) ahead of angle-encoding into a ZZFeatureMap. This is deliberately kept out of the main supervised comparison table: quantum kernel evaluation is O(n²), infeasible at the full dataset's scale on a simulator, and PCA reduction means QSVM is not operating in the same feature space as the other three models. Its result should be read as an exploratory data point, not a fourth directly-comparable row.
| Model | Type | Accuracy | Precision | Recall | F1 | ROC-AUC | PR-AUC |
|---|---|---|---|---|---|---|---|
| XGBoost | Supervised | 98.96% | 99.87% | 91.48% | 95.49% | 0.998 | 0.990 |
| Random Forest | Supervised | 98.94% | 99.11% | 91.97% | 95.41% | 0.998 | 0.989 |
| SVM (RBF) | Supervised | 98.55% | 94.52% | 93.33% | 93.92% | 0.997 | 0.972 |
| Isolation Forest | Unsupervised | 92.65% | 63.42% | 91.48% | 74.91% | 0.964 | 0.705 |
XGBoost is the best-performing model of the four by F1-score. Live inference (ml_engine/predict.py) automatically detects this and routes production Kafka traffic to the XGBoost model rather than Isolation Forest, re-checking on every consumer restart. The gap between the supervised models and Isolation Forest remains the expected, and arguably central, empirical finding of this comparison: it quantifies the cost of detecting lateral movement without access to confirmed incident labels — the situation nearly every real deployment is actually in.
| Model | Accuracy | Precision | Recall | F1 | ROC-AUC | PR-AUC |
|---|---|---|---|---|---|---|
| LSTM Autoencoder | 99.4997% | 0.9402% | 48.3871% | 1.8445% | 0.9285 | 0.0079 |
| Dense Autoencoder | 99.4963% | 0.6352% | 31.6761% | 1.2454% | 0.8616 | 0.0063 |
| ECOD | 99.4936% | 0.3674% | 18.3239% | 0.7204% | 0.8331 | 0.0017 |
| Isolation Forest | 99.9401% | 0.0000% | 0.0000% | 0.0000% | 0.8245 | 0.0006 |
Point-threshold figures at this ~0.01% attack prevalence are informative for continuity but arithmetic caps them near zero regardless of ranking quality — see the entity-window table below for the realistic operating comparison. Dense/LSTM Autoencoder point-threshold rows are left blank here — fill these in directly from your own outputs/metrics_lanl_comparison.json, since exact figures for those two weren't captured in this document's source material and shouldn't be guessed at.
Entity-window (user × 1-hour) Precision@K / Recall@K — the realistic SOC alert-budget view:
| Alert budget (K) | Isolation Forest | Dense Autoencoder | LSTM Autoencoder | ECOD |
|---|---|---|---|---|
| K = 100 | 14.0% / 4.98% | 9.0% / 3.20% | 11.0% / 4.10% | 15.0% / 5.34% |
| K = 250 | 11.2% / 9.96% | 10.0% / 8.90% | 11.2% / 10.45% | 8.8% / 7.83% |
| K = 500 | 7.8% / 13.88% | 10.6% / 18.86% | 11.0% / 20.52% | 6.4% / 11.39% |
The entity-window evaluation demonstrates how model performance changes with an analyst's available alert budget. At the tightest budget (K = 100), ECOD achieves the highest precision (15.0%) and recall (5.34%), making it the strongest choice when only a very small number of alerts can be investigated. As the budget increases, the LSTM Autoencoder consistently overtakes the other methods, achieving the highest recall while maintaining competitive precision at both K = 250 and K = 500. The Dense Autoencoder also scales well at larger alert budgets, whereas Isolation Forest remains competitive only at smaller budgets before its precision declines. These results indicate that sequence-aware deep learning models become increasingly advantageous as SOC investigation capacity grows, while lightweight statistical methods remain attractive when analyst time is extremely limited.
Figure 2. ML Analytics dashboard comparing supervised models on the synthetic dataset and fully unsupervised models on the LANL dataset, including entity-window Precision@K analysis, confusion matrix, feature importance, ROC curves, and the standalone Quantum SVM experiment.
| Metric | Value |
|---|---|
| Subsample size | 2,000 rows (stratified), 1,200 / 400 / 400 train / cal / test |
| Feature dimensionality | 15 → 5 (PCA), angle-encoded into a 5-qubit ZZFeatureMap |
| Accuracy | 87.00% |
| Precision | 45.83% |
| Recall | 45.83% |
| F1 | 45.83% |
| ROC-AUC | 0.8053 |
| PR-AUC | 0.4592 |
| Training time | ~6,735s (~1.9 hours, simulated kernel, O(n²) Gram matrix) |
ROC-AUC of 0.805 indicates real separative signal — well above chance — but F1 sits far below every classical model in §6.1, including Isolation Forest. This reflects the structural cost of the reduced sample size and PCA-compressed feature space, not a broken implementation; it is reported as an honest exploratory result rather than a production candidate.
The live dashboard consists of six pages, each polling the FastAPI backend at an interval appropriate to how quickly the underlying data changes.
| Page | Content |
|---|---|
| Overview | Aggregate event volume, anomaly rate, risk distribution, and the most recent events |
| Live Monitor | A near-real-time tail of the Kafka event stream with live anomaly scores |
| Threat Analysis | Attack-technique breakdown, most-targeted hosts, and highest-risk incidents |
| ML Analytics | Supervised comparison (bar chart + table), LANL comparison (table + entity-window Precision@K curves across all four unsupervised models), confusion matrix, feature importance, and a dedicated Quantum SVM experiment panel |
| User Behaviour | Per-account anomaly ranking |
| System Status | Health of Kafka, MongoDB, producer, and consumer, plus the currently active live-inference model (computed live, not hard-coded) |
Figure 3. System Status dashboard displaying the health of the Kafka broker, MongoDB, producer, consumer, and FastAPI services, together with the currently active live-inference model and system performance indicators.
producer/— synthetic event generation and the Kafka producer (event_builder.py,network_topology.py,producer.py)consumer/— live consumer: online feature engineering, inference, and risk-rule evaluation (consumer.py,feature_engineering.py,rules_engine.py,mongo_writer.py)ml_engine/— offline training, evaluation, and live-inference logicfeature_builder.py— chronological, per-user feature computation shared by the synthetic and LANL pipelinestrain.py,evaluate.py— Isolation Forest (synthetic data, semi-supervised)train_supervised.py— XGBoost, Random Forest, and SVM comparison (synthetic data)train_qsvm.py— standalone Quantum SVM experiment (synthetic data, PCA-reduced subsample)train_lanl.py,evaluate_lanl.py— Isolation Forest, fully unsupervised (LANL data)train_lanl_autoencoder.py— Dense Autoencoder, fully unsupervised (LANL data)train_lanl_lstm.py— LSTM sequence Autoencoder, fully unsupervised (LANL data)train_lanl_ecod.py— ECOD, fully unsupervised (LANL data)lanl_deep_common.py— shared LANL evaluation utilities (point-threshold, event-level and entity-window Precision@K)predict.py— live-inferenceAnomalyPredictor, automatically selecting between Isolation Forest and XGBoost by F1
datasets/lanl/—lanl_loader.py: streamed extraction, stratified sampling, and evaluation-only ground-truth labeling for the LANL datasetshared/— common enumerations and data schemas (enums.py,schemas.py)database/— MongoDB collection and index definitionsapi/— FastAPI application (main.py,routes/,services/)scripts/— batch dataset generation and MongoDB-to-CSV export utilitiesFrontend/Dashboard/— the React dashboard application (src/routes/,src/components/dashboard/,src/lib/)
Conda, Docker (for Kafka and MongoDB), and Node.js (for the dashboard).
conda create -n loganalyticsai python=3.11
conda activate loganalyticsai
pip install -r requirements.txt
pip install xgboost pyod qiskit qiskit-machine-learningdocker compose up -d
python -m producer.producer
python -m consumer.consumer
uvicorn api.main:app --reload --port 8000cd Frontend/Dashboard
npm install
npm run devpython -m scripts.export_training_dataset
python -m ml_engine.train
python -m ml_engine.train_supervised
python -m ml_engine.evaluate
python -m ml_engine.train_qsvm # slow (~2hrs); optional, standalone experimentpython -m datasets.lanl.lanl_loader stratify --events <extracted_auth_events.csv> --redteam <redteam.txt.gz> --out lanl_stratified.csv --keep-fraction 0.002
python -m datasets.lanl.lanl_loader label --events lanl_stratified.csv --redteam <redteam.txt.gz> --out lanl_ground_truth.csv
python -m ml_engine.train_lanl --events lanl_stratified.csv --contamination 0.01
python -m ml_engine.evaluate_lanl --features lanl_features.csv --ground-truth lanl_ground_truth.csv
python -m ml_engine.train_lanl_autoencoder --features lanl_features.csv --raw-events lanl_stratified.csv --ground-truth lanl_ground_truth.csv --self-clean
python -m ml_engine.train_lanl_lstm --features lanl_features.csv --raw-events lanl_stratified.csv --ground-truth lanl_ground_truth.csv --self-clean
python -m ml_engine.train_lanl_ecod --features lanl_features.csv --raw-events lanl_stratified.csv --ground-truth lanl_ground_truth.csv --self-clean- Live inference currently adapts automatically between Isolation Forest and XGBoost only. If Random Forest or SVM becomes the champion in a future
train_supervised.pyrun, the predictor logs this explicitly and falls back to comparing Isolation Forest against XGBoost rather than silently serving a model it cannot yet load — generalizing to an arbitrary-champion model registry is planned. - Engineered features are population-relative rather than baseline-relative to each individual user; incorporating deviation from a user's own historical behavior is expected to improve precision without a corresponding loss of recall.
- The QSVM experiment uses a small (2,000-row), PCA-reduced subsample due to the O(n²) cost of simulated quantum kernel evaluation, and is not directly comparable to the full-data, full-feature supervised trio.
- The System Status page's inference-latency panel is presently illustrative; real request-level latency telemetry has not yet been instrumented.
This project uses the LANL Comprehensive Multi-Source Cyber-Security Events dataset, made publicly available by Los Alamos National Laboratory for cyber-security research purposes.