Network Intrusion Detection with CIC-IDS-2017

Overview

Network intrusions can go undetected for weeks or months, giving attackers time to exfiltrate data or establish persistent access. Automated classification of network flows enables security teams to triage events at scale — faster and more consistently than manual review alone.

This project builds machine learning models to automate network intrusion detection. The data consist of 2.8 million network flows generated by the Canadian Institute of Cybersecurity for the CIC-IDS-2017 benchmark dataset (see data/README.md). The models classify flows as benign or an attack type.

Model Performance

We aimed to train a multi-class model with the ability to detect any one of 11 classes (benign or one of 10 attack types). A random forest model (see models/final_random_forest_weights.skops) achieved recall >0.96 on all classes, including recall >0.99 on eight classes.

For bots and web attacks, the model had excellent recall (0.9872 and 0.9679, respectively) but poor or moderate precision (0.0325 and 0.6563, respectively). These attacks are difficult to distinguish using flow telemetry alone. Flow records summarize transport-layer and behavioral statistics about connections but do not contain application payloads. Attacks such as SQL injection are often only visible in application-layer data (Layer 7), such as HTTP request bodies.

We prioritized recall (probability of classifying a flow that is an attack as an attack) over precision (probability of a flow classified as an attack being an attack) so that more attacks would be detected. The practical tradeoff when emphasizing recall is that security analysts will spend additional time on false positives. This could be tuned to reduce alert fatigue.

Below, the confusion matrix is column-normalized (by true label), so the diagonal values correspond to recall for each attack type. We see that, when bots and web attacks were misclassified, they typically were misclassified as benign. We also see that web attacks were misclassified as denial of service attacks, which is unsurprising since all of these attack types target HTTP servers.

Attack Types

As published, CIC-IDS-2017 includes 15 classes. We dropped or combined classes with low support because results for these classes would be unreliable. Specifically, we dropped Heartbleed attacks (n=11) and Infiltration attacks (n=36). We also combined SQL injection attacks (n=21) and cross-site scripting attacks (n=652) with brute-force web attacks (n=1,507) as one web-attack class. This left 11 classes (benign and 10 attack types).

Class Imbalance

A major challenge in security data science is class imbalance. The vast majority (~80%) of these samples were labeled benign, and the remaining samples were distributed among 10 classes. We considered both SMOTE and class weights for this problem, finding similar performance between the two and preferring class weights for two reasons. First, class imbalance is real and models should reflect this reality. Second, weights allow security team to focus detection based on a threat model and risk assessment.

Feature Engineering

We dropped duplicative and zero-variance features. We also replaced destination_port with three indicator variables -- is_ftp, is_http, and is_ssh -- using the function indicate_service() in src/data/build_features.py. The original destination_port feature included 53,805 unique values. Many of these were ephemeral ports that were not associated with a particular service. Encoding every port was not desirable. Furthermore, just six well-known ports (20, 21, 22, 53, 80, 443) accounted for ~75% of the flows. Five of these six (excluding 53/DNS) were ports for services (FTP, SSH, HTTP) targeted by attack types in the dataset. These ports were encoded.

Getting Started

Install the Project

You can download the final model (models/final_random_forest_weights.skops) or, to run the data and model pipelines yourself, install the project as follows.

git clone https://github.com/tylerjssmith/cic-ids-2017.git
cd cic-ids-2017

Install requirements.

pip install -r requirements.txt

Download Data

The raw data need to be downloaded to data/raw/. The training pipeline will process the data and save them to data/processed. See data/README.md for how to obtain the raw data.

Process Data

A function (src/pipeline.py) is used to orchestrate each stage of the machine learning lifecycle based on configuration files. The following uses data.yml to load and process the raw data, saving data splits to data/processed/.

python3 src/pipeline.py --config config/data.yml

Train Models

Next, the data splits are loaded and piped into train_models() following train_smote.yml or train_weights.yml. Results, including fitted models, are saved as .pkl files in models/.

python3 src/pipeline.py --config config/train_smote.yml
python3 src/pipeline.py --config config/train_weights.yml

Test Models

Next, the data splits and results files are loaded and piped into evaluate_models() following evaluate_smote.yml or evaluate_weights.yml. Results are saved as .pkl files in models/.

python3 src/pipeline.py --config config/evaluate_smote.yml
python3 src/pipeline.py --config config/evaluate_weights.yml

Finalize Model

A chosen model from the training .pkl file is finalized by retraining it on the full data using finalize_model(). The final model is saved as a compressed .skops file for deployment using save_model(). save_model() includes a smoke test to confirm the model will serve predictions and saves a SHA-256 hash file so model integrity can be verified before deployment.

python3 src/pipeline.py --config config/finalize_model.yml

Next Steps

A future version will add probability calibration and threshold tuning, with a function to select per-class thresholds by specifying a minimum precision and maximizing recall subject to that constraint. This would allow users to explicitly control the precision-recall tradeoff for each attack type based on their threat model and analyst capacity.