Cricket Scraper

A Python-based cricket data scraping system that collects match information, squads, live match updates, and scorecards from CREX and stores the data in MongoDB Atlas.

---

Overview

Cricket Scraper automates the process of collecting cricket match data from CREX. The project is designed with a modular structure, making it easy to maintain and extend.

The scraper currently supports:

• Match URL collection
• Match information scraping
• Squad and Playing XI extraction
• Live match updates
• Scorecard collection
• MongoDB Atlas integration

---

Features

Match Discovery

• Fetches match URLs from the CREX schedule page
• Supports multiple matches in a single execution

Match Information

• Collects match details
• Venue information
• Match metadata
• Playing XI information

Squad Collection

• Extracts squad details
• Stores team-wise lineup information

Live Match Tracking

• Captures live score updates
• Tracks ongoing match information

Scorecard Collection

• Retrieves batting scorecards
• Retrieves bowling scorecards
• Stores score summaries

Database Integration

• MongoDB Atlas support
• Automatic document updates using upsert operations
• Separate collections for different data types

---

Technology Stack

Technology	Purpose
Python	Core Development
Selenium	Web Scraping
MongoDB Atlas	Database
PyMongo	MongoDB Integration
WebDriver Manager	Browser Driver Management
Python Dotenv	Environment Configuration

---

Project Structure

CricketScraper/
│
├── database/
│   └── mongodb.py
│
├── schemas/
│   ├── match_schema.py
│   ├── squad_schema.py
│   ├── live_schema.py
│   └── scorecard_schema.py
│
├── scrappers/
│   ├── match_list_scraper.py
│   ├── match_info_scraper.py
│   ├── squads_scraper.py
│   ├── live_scraper.py
│   └── scorecard_scraper.py
│
├── utils/
│   ├── driver.py
│   ├── logger.py
│   ├── config.py
│   ├── helpers.py
│   └── constants.py
│
├── logs/
│
├── main.py
├── requirements.txt
├── .env.example
└── README.md

---

Installation

1. Clone the Repository

git clone <repository-url>
cd CricketScraper

2. Create Virtual Environment

python -m venv venv

3. Activate Virtual Environment

Windows

venv\Scripts\activate

Linux / macOS

source venv/bin/activate

4. Install Dependencies

pip install -r requirements.txt

---

Configuration

Create a .env file in the root directory.

MONGO_URI=your_mongodb_connection_string

Example:

MONGO_URI=mongodb+srv://username:password@cluster.mongodb.net/

---

Running the Scrapers

Run All Scrapers

python run_all.py

Run Individual Scrapers

python scrappers/match_list_scraper.py

python scrappers/match_info_scraper.py

python scrappers/squads_scraper.py

python scrappers/live_scraper.py

python scrappers/scorecard_scraper.py

---

Database Collections

matches

Stores:

• Match ID
• Match URL
• Match Information

squads

Stores:

• Squad Information
• Playing XI Details

live_scores

Stores:

• Live Match Updates
• Current Match State

scorecards

Stores:

• Batting Scorecard
• Bowling Scorecard
• Match Summary

---

Data Flow

CREX Schedule
      │
      ▼
Match URLs
      │
      ▼
Match Information
      │
      ▼
Squads
      │
      ▼
Live Scores
      │
      ▼
Scorecards
      │
      ▼
MongoDB Atlas

---

Sample Output

You can add screenshots here after running the project.

![Matches Collection](images/matches.png)

![Live Scores Collection](images/live_scores.png)

![Scorecards Collection](images/scorecards.png)

---

Future Improvements

• Redis Caching
• Celery Task Scheduling
• Docker Support
• REST API Integration
• Structured Data Extraction
• Automated Scheduling
• Unit Testing
• CI/CD Pipeline

---

Notes

• MongoDB Atlas is used as the primary database.
• Configuration values are managed using environment variables.
• The current implementation stores scraped page content and can be extended to extract structured statistics.
• The project is designed to be easily extended with caching, scheduling, and API layers in future versions.