This project looks at a specific problem in network intrusion detection: when you have far more "normal" traffic than "attack" traffic in your data, and you use a technique like SMOTE to balance things out, does it actually help with the attacks that are hard to catch, or is it mostly just padding out the easy ones?
The code here answers that question directly, using three real intrusion detection datasets, five different oversampling techniques, and a method that splits every attack sample into a difficulty category before measuring anything.
Picture a network intrusion dataset. Most of the traffic is normal. A small fraction is an actual attack. If you train a model on this data as-is, it will often just learn to guess "normal" most of the time, because that's right most of the time, and it'll still score reasonably well on paper while missing a lot of real attacks.
To fix this, people use oversampling: you generate synthetic examples of the attack class so the model sees roughly equal numbers of both classes during training. SMOTE is the most common technique for this. There are several variants too, like Borderline-SMOTE, ADASYN, SMOTE-Tomek, and SMOTE-ENN, each with a slightly different strategy for where and how it creates new synthetic samples.
Here's the issue this project addresses: when people evaluate whether these techniques worked, they almost always report a single number for the whole attack class, like F1 score or recall. That number hides something important. Not every attack sample is equally hard to detect. Some attacks look nothing like normal traffic and are trivial to catch. Some sit right at the edge, mixed in with normal-looking traffic, and are genuinely ambiguous. And some are such extreme outliers that they're surrounded almost entirely by normal traffic in the data. Averaging across all three of these situations into one F1 score tells you very little about which one an oversampling technique is actually fixing.
Instead of one blanket score, every attack sample gets sorted into one of three groups, based on its immediate neighborhood:
- Safe - this sample's nearest neighbors are also attack samples. It's easy to tell apart from normal traffic.
- Danger - this sample's neighbors are a mix of attack and normal samples. It's sitting right on the boundary between the two classes. This is the genuinely hard case.
- Noise - every single neighbor of this sample is normal traffic. It's isolated, and might be an unusual attack variant or might just be a messy data point.
The grouping is done using a k-nearest-neighbors approach: for each attack sample, look at its 5 closest neighbors (using all the data, not just other attacks), and count how many of those neighbors are normal traffic. Zero means Safe, some but not all means Danger, and all of them means Noise.
Once every sample has a group, error rates get measured separately within each group. That's what makes it possible to say something like "this technique cut the Danger-region error rate by 90%, but barely touched the Noise-region error rate" instead of just reporting one number that blends everything together.
Three widely-used benchmark datasets for network intrusion detection, each trimmed down to 200,000 rows so they're directly comparable in scale:
| Dataset | Total rows | Attack samples | Normal samples | Ratio (normal:attack) | Features |
|---|---|---|---|---|---|
| WUSTL-IIOT-2021 | 200,000 | 14,579 | 185,421 | 12.72 : 1 | 43 |
| UNSW-NB15 | 200,000 | 25,297 | 174,703 | 6.91 : 1 | 39 |
| CIC-IDS2017 | 200,000 | 41,599 | 158,401 | 3.81 : 1 | 68 |
WUSTL-IIOT-2021 focuses on industrial IoT network traffic. UNSW-NB15 is general-purpose network traffic covering a wide range of attack types. CIC-IDS2017 focuses on a handful of specific attacks (this project uses the Monday, Tuesday, Wednesday, and Friday-DDoS files specifically, not all eight files in the original collection).
The raw dataset files aren't included in this repository. They're large (CIC-IDS2017 alone is close to two million rows before trimming) and GitHub isn't built for hosting files that size. Download them here instead:
- WUSTL-IIOT-2021: https://www.cse.wustl.edu/~jain/iiot2/index.html
- UNSW-NB15: https://research.unsw.edu.au/projects/unsw-nb15-dataset
- CIC-IDS2017: https://www.unb.ca/cic/datasets/ids-2017.html
The code used to be one long script. It's now split into separate files, each handling one part of the pipeline, so it's easier to read, easier to test individual pieces, and easier for someone unfamiliar with the project to figure out where a particular piece of logic lives.
region_analysis/
├── config.py all settings in one place: dataset paths, random seed,
│ model hyperparameters, colors used in the figures
├── data_loaders.py three functions, one per dataset, that read the raw CSVs
│ and turn them into a clean, scaled feature table
├── regions.py the core logic that sorts each attack sample into
│ Safe, Danger, or Noise
├── dataset_stats.py computes basic facts about each dataset: how imbalanced
│ it is, how many features it has, how much the two
│ classes overlap statistically
├── sensitivity.py re-runs the region sorting at a couple of different
│ neighborhood sizes, to check the result isn't a fluke
│ of one particular setting
├── models.py defines the six conditions tested (no resampling,
│ plus five oversampling techniques) and the three
│ classifiers (Random Forest, XGBoost, Logistic
│ Regression), along with shared scoring functions
├── ablation.py tests what happens if you train a model using only
│ Safe samples, only Danger samples, or a mix, to see
│ whether Danger samples are actually useful on their own
├── synthetic_placement.py checks where each oversampling technique's newly
│ generated samples actually end up landing (Safe,
│ Danger, or Noise)
├── stats_tests.py runs statistical tests to check whether the
│ differences between techniques are real or just
│ random variation
├── experiment.py the main experiment loop: splits data into
│ train/test, applies each oversampling technique,
│ trains each classifier, records results
├── plots.py generates the seven summary charts
├── save_tables.py writes every result out as a CSV file
└── main.py runs the whole pipeline start to finish;
this is the file you actually execute
output/ results from a completed run: every CSV table and
every chart the pipeline produces
requirements.txt the exact list of Python packages needed
.gitignore tells git to skip the raw data folder and the
virtual environment folder
README.md this file
The data/ folder isn't part of the repository. You create it yourself locally after downloading the datasets (see the "Running it yourself" section below).
Running main.py walks through these stages, once per dataset:
1. Load and clean the data. Each dataset has a different structure, so each has its own loader function in data_loaders.py. This step handles things like: finding the correct label column, converting text labels into a simple 0/1 (normal/attack), dropping columns that aren't useful features (like timestamps or IP addresses), handling missing or infinite values, and scaling all the features so they're on a comparable numeric range.
2. Sort attack samples into Safe, Danger, and Noise. This is the regions.py logic described above, run at the default neighborhood size of 5.
3. Double-check the sorting is stable. The same sorting is repeated with neighborhood sizes of 10 and 15 (sensitivity.py), to confirm the Safe/Danger/Noise split doesn't change wildly just because a different number was picked for "how many neighbors to check."
4. Split into training and test sets. An 80/20 split, done in a way that keeps the same proportion of attacks in both halves.
5. Try every combination of oversampling technique and classifier. Six resampling conditions (no resampling, SMOTE, Borderline-SMOTE, ADASYN, SMOTE-Tomek, SMOTE-ENN) times three classifiers (Random Forest, XGBoost, Logistic Regression) gives eighteen combinations tested per dataset. This happens in experiment.py.
6. Measure error rate separately for each region. For every combination above, the test set predictions get broken down by whether the true sample was Safe, Danger, or Noise, and the error rate is calculated for each group separately.
7. Test whether Danger samples are actually useful on their own. ablation.py retrains models using only certain subsets of the training data (only Safe samples, only Danger samples, or a combination) to see how much each group actually contributes to model performance.
8. Check where synthetic samples land. synthetic_placement.py looks at the new samples each oversampling technique generates and checks whether they end up in the Safe, Danger, or Noise region. This turns out to explain a lot about why some techniques help more than others.
9. Run statistical tests. stats_tests.py checks whether the differences between techniques are statistically meaningful, using a Friedman test across all techniques and Wilcoxon signed-rank tests comparing each technique against the no-resampling baseline.
10. Save everything. save_tables.py writes out CSV files for every result, and plots.py generates seven charts summarizing the findings visually.
Across every dataset and every technique, error in the Safe region stays under 1% almost without exception. Whatever is going wrong with these models, it isn't happening on the easy cases.
Oversampling reliably helps the Danger region. The most dramatic example: on UNSW-NB15, SMOTE-ENN brought Danger-region error down from 17.09% with no resampling to 0.49% after resampling. Similar improvements show up across the other datasets and techniques, particularly with ADASYN and Borderline-SMOTE.
The Noise region behaves differently, and less predictably. Sometimes a technique fixes it dramatically (SMOTE-ENN cuts UNSW-NB15's Noise error from 46.3% down to under 2%). Sometimes it barely moves at all (several techniques leave CIC-IDS2017's Noise error above 19%, even after resampling). This isn't something you can predict just by knowing how imbalanced a dataset is.
Looking at where synthetic samples land explains the difference. Borderline-SMOTE and ADASYN specifically aim their new synthetic samples at the Danger boundary, which is exactly why they're good at reducing Danger-region error. But across every technique tested, almost none of the synthetic samples land in the Noise region. None of these techniques were built to target isolated outlier samples specifically, so it makes sense that Noise-region error doesn't respond the same way.
Danger samples need Safe samples around them to be useful. The ablation results show that training a model using only Danger-region samples causes a sharp drop in performance compared to training on Safe-plus-Danger together. Danger samples carry real information about where the decision boundary sits, but they don't work well in isolation.
Statistical significance is genuinely limited with only three classifiers. The tests in this project compare Random Forest, XGBoost, and Logistic Regression, which is a small enough number that broad statistical claims should be treated cautiously. Some comparisons come back significant, others don't, and that's worth keeping in mind when reading the numbers rather than treating every result as settled fact.
Full numbers for every technique, classifier, and dataset combination live in output/region_results_full.csv. A more digestible summary, averaged by technique and region, is in output/region_error_summary.csv.
Step 1 — Get the code.
git clone https://github.com/janapati2005/ids-region-oversampling-analysis.git
cd ids-region-oversampling-analysisStep 2 — Set up a Python environment.
python -m venv venv
venv\Scripts\activate # on Windows
source venv/bin/activate # on Mac or Linux
pip install -r requirements.txtStep 3 — Download the datasets and organize them.
Download the three datasets from the links above, and arrange them like this inside a new data folder:
data/
├── wustl_iiot_2021.csv
├── unsw-nb15/
│ ├── UNSW-NB15_1.csv
│ ├── UNSW-NB15_2.csv
│ ├── UNSW-NB15_3.csv
│ └── UNSW-NB15_4.csv
└── cicids2017/
├── Monday-WorkingHours.pcap_ISCX.csv
├── Tuesday-WorkingHours.pcap_ISCX.csv
├── Wednesday-workingHours.pcap_ISCX.csv
└── Friday-WorkingHours-Afternoon-DDos.pcap_ISCX.csv
Step 4 — Point the code at your data.
Open region_analysis/config.py and update these four lines to match where you put the data folder:
WUSTL_PATH = r"data\wustl_iiot_2021.csv"
UNSWNB15_DIR = r"data\unsw-nb15"
CICIDS2017_DIR = r"data\cicids2017"
OUTPUT_DIR = r"output"Step 5 — Run it.
cd region_analysis
python main.pyIf a dataset's files aren't found at the path you set, the pipeline prints a warning and moves on to the next dataset instead of stopping, so you can test with whichever datasets you already have on hand.
A note on runtime. With the default setting of 200,000 samples per dataset, a full run across all three datasets takes somewhere in the range of 3 to 6 hours on an ordinary CPU, mostly because of how long Random Forest and Logistic Regression take to train on the larger resampled training sets. If you just want to confirm everything runs correctly before committing to a full run, open region_analysis/config.py and temporarily lower MAX_SAMPLES to something like 5000. That finishes in a few minutes and is a good way to catch path or environment problems early.
- Every random process (the train/test split, all three classifiers, all five oversampling techniques) uses the same fixed seed, 42, so re-running the pipeline should produce very similar results each time.
- The train/test split is stratified, meaning the proportion of attack samples is kept consistent between the training set and the test set.
- Model settings are fixed and printed at the start of every run: Random Forest uses 100 trees with a maximum depth of 15, XGBoost uses 100 trees with a maximum depth of 6 and a learning rate of 0.1.
- Small numeric differences between separate runs (typically within a percentage point) are normal and come from floating-point and library-version differences in the underlying resampling and machine learning libraries. The overall pattern described above holds consistently across runs, even when the exact decimal values shift slightly.
This project only looks at binary classification: attack versus normal. It doesn't break attacks down into different types, so it's an open question whether the same Safe/Danger/Noise pattern would hold if you separated out specific attack categories instead of lumping them all into one class.
The way regions are defined depends on how the data is scaled beforehand. This project scales every dataset consistently before doing the region sorting, and the sensitivity check at different neighborhood sizes suggests the overall pattern holds up, but a different scaling approach could in principle shift some samples across region boundaries.
Comparing only three classifiers limits how strongly any statistical claims can be made. A useful next step would be running the same analysis with a wider range of classifiers to see if the pattern holds more broadly.
The Noise region is the part of this project that's least well understood. It's clear that current oversampling techniques generally don't help much here, and it's clear that this varies a lot by dataset, but there isn't yet a good explanation for why some datasets respond better than others to techniques like SMOTE-ENN. That open question is probably the most interesting direction for extending this work: designing an oversampling approach specifically aimed at isolated, Noise-region samples, rather than the boundary-focused approach that most current techniques take.