Magicbricks Property Analysis

A complete data science pipeline that scrapes residential property listings from Magicbricks, cleans the raw data, and performs exploratory and statistical analysis to extract pricing insights across six major Indian cities.

Demo Recording: Watch on Google Drive

---

Dataset

Property	Detail
Source	Magicbricks (scraped via Python)
Records	5,650 residential property listings
Cities	Hyderabad, Pune, Chennai, Mumbai, Gurgaon, Noida
File	magicbricks_properties.xls (stored as CSV)

Columns collected: City, Title, Society, BHK, Area (sqft), Bathrooms, Furnishing, Status, Age, Price (INR), Listing URL

---

Repository Structure

magicbricks-analysis/
│
├── Magicbricks_Simple_Project11.ipynb   # Web scraper
├── Magicbricks_EDA__1_.ipynb            # Exploratory data analysis
├── magicbricks_adv_stats.ipynb          # Advanced statistics
├── magicbricks_properties.xls           # Collected dataset (CSV format)
└── README.md

---

Notebooks

1. Web Scraper

Magicbricks_Simple_Project11.ipynb

Builds the dataset from scratch by scraping Magicbricks property listings across multiple cities. Written for beginners — every line is commented.

• Sends HTTP requests with a browser User-Agent header using requests
• Parses HTML with BeautifulSoup to locate property cards
• Extracts BHK, area, price, furnishing, and status using re (regex)
• Deduplicates listings via a URL-based set before saving to CSV
• Includes a polite time.sleep(2) delay between requests

2. Exploratory Data Analysis

Magicbricks_EDA__1_.ipynb

Cleans the raw data and explores it visually with matplotlib and seaborn.

Cleaning steps: lowercase normalization, removal of unusable columns, filtering to sensible area (200–10,000 sqft) and price (5 lakh–50 crore) ranges, and derivation of price-per-sqft.

Univariate analysis: price and area distributions, BHK counts, furnishing and status breakdowns.

Bivariate analysis: area vs. price scatter, city-wise price boxplots, BHK vs. price trends, median price-per-sqft bar charts, and a correlation heatmap.

3. Advanced Statistics

magicbricks_adv_stats.ipynb

Goes beyond basic EDA into formal statistical analysis using scipy.

• Descriptive statistics: mean, median, mode, variance, standard deviation, coefficient of variation
• Five-number summary and IQR-based outlier detection
• Skewness and kurtosis for distribution shape assessment
• Probability distribution fitting (normal distribution)
• Hypothesis testing: independent t-test, one-way ANOVA, and chi-square test
• Missing value heatmap and city-level summary tables

---

Setup

# 1. Clone the repository
git clone https://github.com/your-username/magicbricks-analysis.git
cd magicbricks-analysis

# 2. Install dependencies
pip install requests beautifulsoup4 pandas numpy matplotlib seaborn scipy

# 3. Launch Jupyter
jupyter notebook

Run the notebooks in order: scraper first, then EDA, then advanced stats. If you already have the dataset, skip directly to the EDA notebook.

---

Key Findings

• Property prices are right-skewed. The mean is noticeably higher than the median, driven by a small number of high-value listings. The median is the more reliable measure of a typical price.
• Mumbai and Gurgaon consistently show the highest median price-per-sqft among the six cities.
• Area and BHK count are positively correlated with price, though price-per-sqft varies significantly by city independent of size.
• Unfurnished listings form the majority of the dataset, while furnished homes command a premium.
• Roughly half of all listings are ready-to-move, with the remainder under construction.

---

Technologies Used

Library	Purpose
requests	HTTP requests for scraping
BeautifulSoup	HTML parsing
re	Regex-based field extraction
pandas	Data manipulation and cleaning
numpy	Numerical operations
matplotlib / seaborn	Visualizations
scipy	Statistical tests and distribution fitting

---

Notes

• The .xls file is a plain CSV despite its extension. Open it with pd.read_csv(), not an Excel reader.
• Scraping is rate-limited to two seconds per page to avoid being blocked. Respect Magicbricks' terms of service before running the scraper.
• The Bathrooms column was dropped during cleaning due to a high rate of missing and implausible values in the raw scrape.