By combining advanced feature extraction with a robust Random Forest classifier, it provides real-time risk assessment and deep network intelligence for any given URL.
Preview of the Mailharpoon URL analysis workflow
- 🧠 Intelligent Detection: Uses a Random Forest model trained on the UCI Phishing Dataset to classify URLs based on 30+ distinct technical features.
- 🌐 Network Intelligence: Performs real-time DNS records, SSL/TLS certificate validation, and WHOIS domain age lookups.
- 📸 Visual Preview: Safely captures a full-page screenshot of the target site using Playwright to identify visual deception.
- 🛡️ Reputation Checks: Integrates with reputation APIs to check domains against known blacklists.
- ⚡ High Performance: Fast inference engine built with FastAPI and an interactive, modern dashboard powered by Streamlit.
- 🐳 Deployment Ready: Fully containerized with Docker and Docker Compose for seamless deployment in any environment.
The system follows a modern, containerized microservices architecture:
- Frontend (Streamlit): High-level UI for user interaction and report generation.
- Backend (FastAPI): Heavy-lifting service for feature extraction, external lookups, and model inference.
- Intelligence Layer: Asynchronous tasks for screenshot capturing and multi-source data gathering.
The project classifies URLs using a Random Forest classifier, which was selected after evaluating multiple different algorithms. Random Forest achieved the best performance, demonstrating excellent robustness against overfitting, strong generalization, and the ability to effectively handle a highly diverse set of features (ranging from URL structures to network signals). These results directly power the phishing prediction system used in the application.
This section highlights the training and validation results for the selected Random Forest model.
Comparison of training versus testing performance
The Train vs. Test comparison displays highly stable performance metrics across both datasets. The minimal drop-off in performance on the unseen test data indicates extremely low overfitting and confirms the reliability of the model.
Cross-validation results for the Random Forest model
The cross-validation results reinforce the model's reliability. With consistently high and tight metric clusters across different data splits, the classifier proves excellent generalization capabilities.
- Frontend: Streamlit
- Backend: FastAPI, Uvicorn, Pydantic, Playwright, Requests
- Machine Learning: Scikit-Learn, Joblib, NumPy, Pandas
- DevOps: Docker, Docker Compose
mailharpoon/
├── backend/ # FastAPI Microservice
│ ├── src/ # Source code (API, Logic, Config)
│ ├── models/ # Trained ML models & feature sets
│ └── Dockerfile # Backend containerization
├── frontend_streamlit/ # Streamlit Web App
│ ├── pages/ # Multi-page application structure
│ ├── images/ # UI assets
│ └── Dockerfile # Frontend containerization
├── docker-compose.yml # Multi-container orchestration
├── .env.example # Deployment environment template
└── requirements.txt # Full project dependencies
- Python 3.11+
- Docker & Docker Compose (Recommended)
The fastest way to get the entire system up and running:
# Clone the repository
git clone https://github.com/Andrej-Art/Mailharpoon.git
cd Mailharpoon
# Launch the system
docker-compose up --buildThe frontend will be available at http://localhost:8501, and the backend at http://localhost:8000.
# Install dependencies
pip install -r requirements.txt
# Start Backend
cd backend/src
python main.py
# Start Frontend (in a new terminal)
cd frontend_streamlit
streamlit run app.pyPOST /predict-url
Request:
{
"url": "https://suspicious-site.com/login",
"model": "rf_full",
"extended": true
}Response:
{
"prediction": "Malicious",
"phishing_probability": 0.86,
"legit_probability": 0.14,
"model_used": "rf_full"
}Andrej Artuschenko
- GitHub: @Andrej-Art
- Email: andrejart95@gmail.com