🏎️ Formula 1 Data Pipeline (2019–2024)

Automated end-to-end data pipeline that scrapes and merges Formula 1 race data (2019–2024) from Wikipedia and the Ergast API, generating a clean dataset to analyze the relationship between pit-stop efficiency and race performance across seasons.

---

📂 Project Structure

.
├── main.py                     # Orchestrates the entire pipeline
├── Apartado_1.py               # Scrapy spider for race classifications (Wikipedia)
├── Apartado_2.py               # API data collection (Ergast pit stops & drivers)
├── Apartado_3.py               # Data merging and consolidation
├── Apartado_4.ipynb            # Analytical notebook (visualizations & correlations)
├── Apartado_4_analisis.pdf     # Final analytical report
├── requirements.txt            # Install dependencies
├── .gitignore                  # Recommended: ignore generated data and caches
├── proyecto_final/             # Auto-generated folders with raw CSVs (per year)
└── DatosF1_2019-2024.csv       # Auto-generated final dataset

---

⚙️ How It Works

1️⃣ Data Collection — Wikipedia (Scrapy)

• Crawls each season’s Wikipedia page (2019–2024).
• Finds the “Grands Prix” table and extracts race links.
• Scrapes every “Race classification” table into structured CSV files.

2️⃣ Data Collection — Ergast API

• Retrieves pit-stop data (number and duration) and driver info from the Ergast API.
• Handles missing values via:
  • Imputation (mean/median substitution)
  • Race-level medians for anomalous entries
• Adjusts special cases (Verstappen → #1 after 2021, Bearman, Lawson, etc.).

3️⃣ Data Fusion

• Merges Scrapy and API CSVs by DriverNumber.
• Adds metadata (Year, RaceNumber) and outputs the unified dataset DatosF1_2019-2024.csv.

4️⃣ Analysis

• Visualizes top teams’ performance evolution (Red Bull, Mercedes, Ferrari, McLaren).
• Examines how pit-stop times correlate with total points (≈ −0.29).
• Highlights trends influenced by FIA regulation changes.

---

🧠 Key Insights

• Teams with faster pit stops generally achieve higher points totals.
• Driver participation varies each season due to substitutions, injuries, or strategy.
• FIA regulation changes can temporarily increase total race times.

---

🧰 Technologies Used

Category	Tools / Libraries
Scraping	Scrapy, pandas
API	requests, Ergast API
Data Processing	NumPy, pandas
Automation	subprocess, os, time
Analysis	Jupyter Notebook, Matplotlib, Seaborn
Language	Python 3.10+

---

🏁 How to Run the Pipeline

1. Clone the repository

   git clone https://github.com/SantiCoAr/f1-data-pipeline.git
   cd f1-data-pipeline-2019-2024

2. Create and activate a virtual environment

   python -m venv venv
   venv\Scripts\activate    # Windows
   # or
   source venv/bin/activate   # macOS/Linux

3. Install dependencies

   pip install -r requirements.txt

   (If you don’t have this file, just install scrapy, pandas, numpy, and requests.)

4. Run the pipeline

   python main.py

   This will:

   • Create the Scrapy project
   • Collect race data from Wikipedia
   • Fetch pit-stop data from the API
   • Merge all CSVs into a single file DatosF1_2019-2024.csv

5. (Optional) Explore the results:

   jupyter notebook Apartado_4.ipynb

---

📈 Output

• Final dataset: DatosF1_2019-2024.csv
  → Contains 6 seasons × 1000+ race entries with driver performance and pit-stop stats.
• Analysis report: Apartado_4_analisis.pdf
  → Summary of findings and correlations.

---

🧩 Example Columns

Column	Description
DriverId	Unique driver identifier
DriverNumber	Permanent driver number
NPitstops	Number of pit stops in the race
MedianPitStopDuration	Median pit-stop time (seconds)
Constructor	Team / car constructor
Points	Championship points earned
Year	Season year
RaceNumber	Round number of the season

---

⚠️ Known Limitations

• The Ergast API may temporarily block requests (HTTP 429 – Too Many Requests) if queried too fast.
  → The script includes a short delay and can be extended with retry logic.
• Some races (e.g., 2021 Belgian GP) may lack pit-stop data and are skipped automatically.

👤 Author

Santiago Córdoba
📧 santicordobaartieda@gmail.com
💼 LinkedIn: https://www.linkedin.com/in/santiago-córdoba-artieda-019420359/