Network Security Threat Classifier (Phishing Detection)

End-to-end phishing URL classification project that demonstrates a production-style ML workflow: data ingestion, validation + drift checks, feature preprocessing, model training + experiment tracking, and interactive inference (batch + single) with explainability.

The goal is to classify a URL instance as Phishing or Legitimate using 30 engineered signals commonly found in phishing datasets (e.g., URL length, HTTPS token, SSL state, anchor URL patterns).

Contents

• Highlights
• What This Project Does
• How It Works
• Streamlit Demo
• Screenshots
• Dataset
• Project Structure
• Quickstart (Local)
• Configuration
• API (FastAPI)
• Training (Pipeline)
• Inference schema
• Labels

Highlights

• Data: MongoDB ingestion + schema validation/drift checks
• ML: preprocessing with KNNImputer, multi-model training, MLflow tracking
• Serving: Streamlit UI (streamlit_app.py) + FastAPI (app.py)
• Demo modes: batch prediction (CSV) and single prediction (interactive + SHAP)

What This Project Does

• Train a supervised binary classifier on phishing signals and persist deployable artifacts (preprocessor + model).
• Serve predictions in two ways:
  • a Streamlit "recruiter demo" UI with batch scoring, single-case scoring, and explainability
  • a FastAPI service for CSV-based batch predictions
• Validate incoming data against a schema and generate a drift report to help catch distribution shifts.

How It Works

At a high level, the training pipeline is implemented in networksecurity/pipeline/training_pipeline.py and orchestrates:

1. Data ingestion (networksecurity/components/data_ingestion.py)
   • Pulls records from a MongoDB collection (MONGO_DB_URL)
   • Writes a feature store CSV and creates train/test splits
2. Data validation (networksecurity/components/data_validation.py)
   • Validates the schema and runs a KS-test drift check
   • Writes a drift report YAML to the artifacts directory
3. Data transformation (networksecurity/components/data_transformation.py)
   • Fits a KNNImputer preprocessing pipeline
   • Saves the fitted preprocessor to final_model/preprocessor.pkl
4. Model training + tracking (networksecurity/components/model_trainer.py)
   • Trains multiple classifiers (including XGBClassifier) with hyperparameter search
   • Logs metrics to MLflow (and initializes DagsHub MLflow tracking if configured)
   • Saves the trained model to final_model/model.pkl

Streamlit Demo

The Streamlit UI (streamlit_app.py) is designed for a clean walkthrough and supports:

• Batch Prediction: upload a CSV with 30 feature columns and download predictions
• Single Prediction + SHAP: score one instance, show confidence, and plot top feature impact (when SHAP is available)
• Performance Dashboard: view core metrics, confusion matrix, and a classification report on reference data
• Architecture + Tech Stack: a narrative tab for explaining the pipeline and tooling

Screenshots

Main starting page

Batch CSV prediction

Single prediction + SHAP explainability

Performance dashboard

Architecture + tech stack

Pipeline diagram

Dataset

• Sample dataset: NetworkData/phisingData.csv (includes the target column Result).
• Training expects the same 30 feature columns shown in Inference schema.
• Label handling:
  • The raw dataset uses Result values like -1 and 1.
  • During transformation, the pipeline maps -1 to 0 so the final labels are 0 (Phishing) and 1 (Legitimate).

Project Structure

• networksecurity/: pipeline code (ingestion, validation, transformation, training, utilities)
• streamlit_app.py: recruiter-ready UI (batch + single + explainability + dashboard)
• app.py: FastAPI service (/train, /predict) and Swagger docs
• main.py: run the training pipeline from the CLI
• data_schema/schema.yaml: expected columns + types
• final_model/: saved artifacts used for inference (model.pkl, preprocessor.pkl)
• assests/: README screenshots and diagrams (folder name matches the repo)

Quickstart (Local)

pip install -r requirements.txt
streamlit run streamlit_app.py

If you don’t have artifacts in final_model/ yet, run training once:

python main.py

Configuration

Set MONGO_DB_URL (used for ingestion/training), for example by creating a local .env file:

MONGO_DB_URL="your-mongodb-connection-string"

API (FastAPI)

• Start the API:

python app.py

• Open docs at http://localhost:8000/docs
• Training endpoint: GET /train
• Batch prediction endpoint: POST /predict (CSV upload)

Training (Pipeline)

Run the training pipeline directly:

python main.py

Artifacts are saved under final_model/ (e.g., final_model/model.pkl, final_model/preprocessor.pkl).

Inference schema

Expected 30 input feature columns

having_IP_Address, URL_Length, Shortining_Service, having_At_Symbol, double_slash_redirecting, Prefix_Suffix, having_Sub_Domain, SSLfinal_State, Domain_registeration_length, Favicon, port, HTTPS_token, Request_URL, URL_of_Anchor, Links_in_tags, SFH, Submitting_to_email, Abnormal_URL, Redirect, on_mouseover, RightClick, popUpWidnow, Iframe, age_of_domain, DNSRecord, web_traffic, Page_Rank, Google_Index, Links_pointing_to_page, Statistical_report

Labels

• 0 -> Phishing
• 1 -> Legitimate