Medical Insurance Cost Prediction - Exploratory Data Analysis

Overview

This project performs comprehensive Exploratory Data Analysis (EDA) on a medical insurance dataset to understand the factors influencing insurance costs. The analysis includes data cleaning, statistical summaries, visualizations, correlation analysis, and feature selection using chi-square tests for categorical variables.

Dataset

The dataset used is insurance.csv, which contains information about medical insurance costs and related factors. It includes the following columns:

• age: Age of the primary beneficiary
• sex: Gender of the insurance contractor (male/female)
• bmi: Body mass index, providing an understanding of body weight relative to height
• children: Number of children covered by health insurance
• smoker: Smoking status (yes/no)
• region: Residential area in the US (northeast, northwest, southeast, southwest)
• expenses: Individual medical costs billed by health insurance

The dataset contains 1,338 records with no missing values.

Project Structure

.
├── insurance.csv          # Raw dataset
├── l.ipynb               # Jupyter notebook with complete EDA
└── README.md             # Project documentation

Technologies Used

• Python 3.x
• pandas: Data manipulation and analysis
• numpy: Numerical computations
• matplotlib: Basic plotting
• seaborn: Statistical data visualization
• scipy: Statistical functions (chi-square test)

Installation and Setup

1. Ensure Python 3.x is installed on your system
2. Install required packages:

   pip install pandas numpy matplotlib seaborn scipy

3. Clone or download this repository
4. Open the Jupyter notebook l.ipynb in Jupyter Lab or Jupyter Notebook

Analysis Steps

1. Data Loading and Initial Exploration

• Load the dataset using pandas
• Display first few rows and dataset shape
• Check data types and missing values

2. Statistical Analysis

• Generate descriptive statistics for numerical columns
• Check for null values (none found in this dataset)

3. Data Visualization

• Histograms and box plots for numerical variables
• Count plots for categorical variables
• Correlation heatmaps
• Scatter plots showing relationships between variables

4. Data Preprocessing

• Label encoding for binary categorical variables (sex, smoker)
• One-hot encoding for multi-category variables (region)
• BMI categorization into weight classes

5. Feature Selection

• Correlation analysis for numerical features
• Chi-square test for categorical features to determine statistical significance
• Binning of target variable (expenses) for categorical analysis

Key Findings

Statistical Summary

• Age: Range from 18 to 64 years, mean ~39 years
• BMI: Range from 15.96 to 53.13, mean ~30.66
• Children: 0 to 5 children, mean ~1.09
• Expenses: Range from $1,122 to $63,770, mean ~$13,270

Correlation Analysis

Strong positive correlations observed between:

• Expenses and age
• Expenses and BMI
• Expenses and smoking status

Chi-square Test Results

The analysis evaluates the relationship between categorical features and binned expense categories:

Feature	Chi-square Statistic	P-value	Decision
smoker	High value	< 0.05	Keep (significant)
sex	Varies	Check p-value	May keep/drop
region_*	Varies	Check p-value	May keep/drop
bmi_category_*	Varies	Check p-value	May keep/drop

Usage

1. Open l.ipynb in Jupyter Notebook or Jupyter Lab
2. Run cells sequentially to reproduce the analysis
3. View visualizations and statistical outputs
4. Modify parameters as needed for further exploration

Example Code Snippets

Loading and Viewing Data

import pandas as pd

# Load dataset
df = pd.read_csv('insurance.csv')

# View first 5 rows
print(df.head())

# Check dataset shape
print(f"Dataset shape: {df.shape}")

Basic Statistical Analysis

# Descriptive statistics
print(df.describe())

# Check for missing values
print(df.isnull().sum())

Data Visualization Example

import matplotlib.pyplot as plt
import seaborn as sns

# Distribution of expenses
plt.figure(figsize=(10, 6))
sns.histplot(df['expenses'], bins=30, kde=True)
plt.title('Distribution of Insurance Expenses')
plt.xlabel('Expenses ($)')
plt.ylabel('Frequency')
plt.show()

Correlation Analysis

# Correlation matrix
correlation_matrix = df.corr()
plt.figure(figsize=(10, 8))
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', center=0)
plt.title('Correlation Matrix')
plt.show()

Visualizations Included

1. Histograms: Distribution of age, BMI, children, and expenses
2. Box Plots: Outlier detection for numerical variables
3. Count Plots: Distribution of categorical variables (sex, smoker, region)
4. Scatter Plots: Relationships between expenses and other variables
5. Correlation Heatmap: Overall correlation between numerical features
6. Pair Plots: Multivariate relationships

Feature Engineering

• BMI Categories: Classified into Normal weight, Overweight, and Obese
• Expense Binning: Divided into 4 equal frequency bins for categorical analysis
• One-hot Encoding: Region variable converted to dummy variables
• Label Encoding: Sex and smoker variables converted to binary

Future Work

This EDA serves as a foundation for predictive modeling. Potential next steps include:

1. Model Development: Build regression models to predict insurance costs
2. Feature Engineering: Create additional derived features
3. Model Evaluation: Compare different algorithms (Linear Regression, Random Forest, etc.)
4. Hyperparameter Tuning: Optimize model performance
5. Deployment: Create a web application for cost prediction

Contributing

Feel free to fork this repository and contribute improvements to the analysis or add predictive modeling components.

License

This project is for educational purposes. Please check the dataset source for any licensing restrictions.