FindCars NZ (this repo: CarAuction-Analytics-NZ) is a data platform tracking damaged-vehicle auction listings from Manheim New Zealand. Three years of daily scraping (1,131 days and 660,000+ raw records) served as an interactive dashboard, open datasets, and a free static JSON API.
🔗 Live site: findcars.prasanthsasikumar.com 📖 The story: a three-part blog series: the data · the analytics · the prediction model 🔮 Price predictor: nz-car-auction.streamlit.app
- 📊 Interactive Analytics Dashboard - Visualize price trends, manufacturer statistics, and damage patterns
- 📈 Real-Time Listings - View latest vehicle auction data with interactive filtering
- 🔌 RESTful API - Programmatic access to all data with multiple endpoints
- 🧹 Data Cleaning Pipeline - Automated processing of raw scraper data
- 📥 Multiple Export Formats - CSV, Parquet, and SQLite database exports
- 🤖 Daily Automated Scraping - Fresh data collected every day
| Metric | Value |
|---|---|
| Time Period | May 27, 2023 - present (updated daily) |
| Total Days | 1,131+ days |
| Missing Days | 8 (99%+ coverage) |
| Total Records | 660,000+ raw / 460,000+ cleaned |
| Manufacturers | 500+ |
| Data Fields | 20+ attributes per vehicle |
- 2023-05-30
- 2023-07-19
- 2024-02-28 through 2024-03-04 (6 consecutive days)
- Python 3.13 or higher
- pip package manager
- Virtual environment (recommended)
- Clone the repository
git clone https://github.com/prasanthsasikumar/CarAuction-Analytics-NZ.git
cd CarAuction-Analytics-NZ- Create and activate virtual environment
# Windows
python -m venv .venv
.venv\Scripts\activate
# Linux/Mac
python3 -m venv .venv
source .venv/bin/activate- Install dependencies
pip install -r requirements.txt- Run the application
# Development server
python app_main.py
# Production server with Gunicorn (Linux/Mac)
gunicorn -c gunicorn_config.py app_main:app- Access the application
http://localhost:8000
CarAuction-Analytics-NZ/
├── app_main.py # Main Flask application
├── clean_data.py # Data cleaning pipeline
├── main.py # Web scraper (daily job)
├── requirements.txt # Python dependencies
├── gunicorn_config.py # Production server config
├── data/
│ ├── raw/ # Daily CSV files (1,131+)
│ ├── processed/ # Cleaned datasets
│ │ ├── car_auction_public.csv
│ │ ├── car_auction_public.parquet
│ │ └── car_auction_data.db
│ └── *.csv # Additional data files
├── templates/ # HTML templates
│ ├── index_main.html # Landing page
│ ├── today.html # Listings view
│ ├── analytics.html # Analytics dashboard
│ ├── api_docs.html # API documentation
│ └── about.html # About page
├── src/
│ ├── scrapers/ # Scraper modules
│ └── utils/ # Utility functions
└── docs/ # Documentation
https://findcars.prasanthsasikumar.com/api/v1
GET /api/v1/stats/overviewReturns aggregated statistics from the most recent 30 days.
Response:
{
"total_listings": 45000,
"unique_vehicles": 12000,
"manufacturers": 500,
"avg_price": 2500.50,
"median_price": 1800.00,
"top_manufacturers": {
"Toyota": 8500,
"Mazda": 6200
}
}GET /api/v1/stats/price-trendsReturns weekly price trends (all days in each week) with average daily listing counts.
GET /api/v1/search?manufacturer=Toyota&max_price=5000Parameters:
manufacturer- Filter by manufacturermodel- Filter by modelmax_price- Maximum pricemin_price- Minimum price
GET /api/v1/download/latest # Latest raw CSV
GET /api/v1/download/processed # Cleaned datasetFor complete API documentation, visit: /api-docs
The project includes a comprehensive data cleaning pipeline that:
- Loads all 1,131+ daily CSV files
- Removes duplicates based on vehicle attributes
- Cleans price and mileage data
- Standardizes manufacturer and model names
- Adds derived features (vehicle age, price changes)
- Exports to multiple formats
Run the cleaning pipeline:
python clean_data.pyOutput files:
data/processed/car_auction_public.csv- Cleaned CSV (460K records)data/processed/car_auction_public.parquet- Parquet formatdata/processed/car_auction_data.db- SQLite databasedata/processed/DATA_SUMMARY.txt- Summary statistics
The scraper runs daily to collect fresh auction data.
Manual scraping:
python main.pySchedule with cron (Linux/Mac):
# Add to crontab -e
0 2 * * * cd /path/to/CarAuction-Analytics-NZ && /path/to/.venv/bin/python main.py- Market Research - Analyze pricing trends and market dynamics
- Price Prediction - Train ML models for vehicle valuation
- Academic Research - Study depreciation patterns and damage types
- Business Intelligence - Inform inventory decisions for dealerships
- Personal Use - Track specific vehicles and get price alerts
Backend:
- Python 3.13
- Flask (Web framework)
- Pandas (Data processing)
- BeautifulSoup4 (Web scraping)
- Gunicorn (Production server)
Frontend:
- Bootstrap 5
- Chart.js (Visualizations)
- DataTables (Interactive tables)
- Font Awesome (Icons)
Data Storage:
- SQLite
- Parquet (PyArrow)
- CSV
- Configure Gunicorn
# gunicorn_config.py
bind = '0.0.0.0:8000'
workers = 4
module = 'app_main:app'- Start Gunicorn
gunicorn -c gunicorn_config.py app_main:app- Setup Nginx (optional)
server {
listen 80;
server_name your-domain.com;
location / {
proxy_pass http://127.0.0.1:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}This project is licensed under the MIT License - see the LICENSE file for details.
Contributions are welcome! Please feel free to submit a Pull Request.
- Fork the project
- Create your feature branch (
git checkout -b feature/AmazingFeature) - Commit your changes (
git commit -m 'Add some AmazingFeature') - Push to the branch (
git push origin feature/AmazingFeature) - Open a Pull Request
This data is for informational and research purposes only. Prices and availability should be verified directly with Manheim NZ. This project follows ethical web scraping practices and respects robots.txt.
- Implement XGBoost price prediction model, live at nz-car-auction.streamlit.app
- Add email price alerts for tracked vehicles
- Expand to other auction platforms
- Build recommendation engine
- Mobile app for iOS/Android
- Export to Excel/Google Sheets
Prasanth Sasikumar - prasanthsasikumar.com
Project Link: https://github.com/prasanthsasikumar/CarAuction-Analytics-NZ
⭐ If you find this project useful, please consider giving it a star!