---

🌍 Country Data Web Scraper

A simple and efficient Python web scraper that extracts country information (name, capital, population, and area) from a public website.

---

📌 Project Overview

This project demonstrates practical web scraping using Python, including:

• Sending HTTP requests with requests.Session
• Parsing HTML using BeautifulSoup
• Extracting structured data safely
• Formatting large numbers using numerize
• Storing results in both Python objects and CSV format

---

🚀 Features

• ✅ Uses requests.Session() for persistent connections
• ✅ Custom headers to mimic real browser behavior
• ✅ Robust error handling for network failures
• ✅ Safe data extraction (avoids crashes on missing elements)
• ✅ Formats large numbers (e.g., 1000000 → 1M)
• ✅ Structured output (list of dictionaries)
• ✅ Exports data to CSV using pandas
• ✅ Clean console output for readability

---

🛠️ Technologies Used

• Python 3
• requests
• BeautifulSoup (bs4)
• lxml
• numerize
• pandas

---

📂 Project Structure

.
├── scraper.py              # Main scraping script
├── scraped_countries.csv  # Output file (generated after run)
└── README.md               # Documentation

---

⚙️ Installation

1. Clone the repository

git clone https://github.com/your-username/country-scraper.git
cd country-scraper

2. Install dependencies

pip install requests beautifulsoup4 lxml numerize pandas

---

▶️ Usage

Run the script:

python scraper.py

---

🧠 How It Works

1. Creates a requests.Session with custom headers

2. Sends a GET request to the target website

3. Parses HTML content using BeautifulSoup

4. Locates all country containers (div.country)

5. Extracts:

   • Country name
   • Capital
   • Population
   • Area

6. Safely handles missing or invalid data

7. Converts large numbers into readable format using numerize

8. Stores results in a list of dictionaries

9. Converts data into a pandas DataFrame

10. Saves the data as a CSV file

---

📊 Example Console Output

Response Status Code: 200
Number of countries found: 250

Country: Afghanistan, Capital: Kabul, Population: 38.9M, Area(km²): 652.9K
Country: Albania, Capital: Tirana, Population: 2.8M, Area(km²): 28.7K

---

📦 Returned Data Format

[
    {
        "Name": "Afghanistan",
        "Capital": "Kabul",
        "Population": "38.9M",
        "Area_km²": "652.9K"
    },
    ...
]

---

📁 Output File

After execution, a CSV file is generated:

scraped_countries.csv

• Encoding: utf-8-sig (Excel-friendly)

• Columns:

  • Name
  • Capital
  • Population
  • Area_km²

---

⚠️ Error Handling

• Handles request failures using try-except
• Uses response.raise_for_status() for HTTP errors
• Prevents crashes from missing HTML elements
• Returns an empty list if scraping fails

---

🔮 Future Improvements

• Add logging instead of print statements
• Implement retry & backoff mechanism
• Add support for multiple pages (pagination)
• Export data to JSON/Excel formats
• Turn into a reusable Python package
• Add CLI arguments (e.g., output format, file name)

---

📜 License

This project is open-source and available under the MIT License.

---

🙌 Acknowledgment

Data sourced from: https://www.scrapethissite.com/pages/simple/

---

👨‍💻 Author

Mustak Absar Khan GitHub: https://github.com/MustakAbsarKhan

---