Pan-Cancer AI: Dataset & Predictive Pipeline

This project leverages the CDC Behavioral Risk Factor Surveillance System (BRFSS) survey data to drive a multi-tiered cancer risk assessment platform.

Dataset: processed_dataset/refined_cancer_dataset.csv

The dataset was custom-engineered to provide a clean, standardized foundation for machine learning.

1. Data Sources

• BRFSS 2021 (LLCP2021.csv): ~100k records sampled.
• BRFSS 2024 (brfss_survey_data_2024.csv): ~100k records sampled.
• Total Merged Records: 195,940

2. Engineering Process

The data was processed using a multi-step extraction pipeline:

• Filtering: Removal of records with invalid age groups or critical missing demographic data.
• Standardization: Variable names were mapped from technical CDC codes (e.g., _AGEG5YR, _BMI5) to human-readable clinical labels (e.g., Age_Group, BMI).
• Cleaning: BMI values were normalized, and categorical codes were simplified for ML compatibility.
• Merging: Harmonized variables across different survey years (handling column name changes between 2021 and 2024).

3. Feature Indicators (25+ Variables)

The dataset includes exhaustive health indicators across several categories:

• Demographics: Age, Sex, BMI.
• Tobacco Use: Cigarette smoking intensity (100+), current usage, and E-cigarette history.
• Physical Activity: Regular exercise habits and leisure time activity.
• Screenings: History of Mammograms, Cervical, Colorectal, and Lung cancer screenings.
• Immunizations: HPV Vaccination, Flu shots, and Pneumonia shots.
• Chronic Conditions: History of Diabetes, COPD, Depression, Kidney Disease, and Stroke.

4. Granular Cancer Labeling

Records were labeled based on specific cancer diagnoses:

• Major Types: Lung, Breast, Colon, Melanoma, Skin (Non-Melanoma), etc.
• Mapping: Over 30 CDC cancer codes were mapped to a unified Cancer_Label column.

AI Model Usage (Tier 1)

The backend uses a Random Forest Classifier ensemble trained on this merged dataset.

• Model Goal: Predict the specific risk level for top cancer concerns (Lung, Breast, Skin, Colorectal) based on the user's complete health profile.
• Integration: The models are exported as .pkl files and loaded via joblib in the FastAPI backend.
• Hybrid Scoring: The AI predictions are combined with population-wide empirical baselines (CDC base risk) to provide a nuanced, data-driven risk percentage.

How to Use

1. Tier 1: User fills out the comprehensive health form.
2. Analysis: The AI model calculates probabilities across multiple cancer types.
3. Tier 2: The platform routes the user to a specialized clinical biomarker form based on the highest AI-flagged risk.