Epidemiological Data Science Pipeline: Multimodal Risk Factor Analysis for Lung Cancer Incidence

Overview

This repository houses a modular, production-grade Python pipeline designed to integrate and analyze public health registries and environmental exposure vectors to evaluate lung cancer risk dynamics across the United States. The project replicates an advanced data systems integration workflow, executing automated multi-format ingestion, cohort standardization, and non-parametric statistical hypothesis testing.

Research & Clinical Motivation

In population health and precision medicine, understanding the compounding effects of lifestyle factors and environmental exposures is critical for mapping disease etiology. Particulate Matter ($PM_{2.5}$) and smoking status represent two distinct layers of risk—one environmental, one behavioral—that are frequently captured in disparate data systems with non-standardized formats.

This project demonstrates a scalable computational framework to break down these data silos. By building an automated ingestion factory that unifies multi-format public databases (Excel, Stata, and SPSS formats), this pipeline enables research teams to transition away from manual data aggregation and execute disciplined statistical benchmarking to reveal translationally relevant population trends.

Computational Pipeline Architecture

The repository is engineered with an object-oriented, single-responsibility design pattern to decouple data generation, multi-format I/O, and statistical evaluation:

lung-cancer-data-science-analysis/
├── main.py                     # Master orchestration harness
├── requirements.txt            # Isolated dependency version map
├── data/
│   └── README.md               # Data schema and source metadata definitions
├── src/
│   ├── __init__.py
│   ├── config.py               # Global constants and geographic mappings
│   ├── generate_raw_warehouse.py # Programmatic multi-format file simulator
│   ├── ingest_factory.py       # Cross-format loader (.xlsx, .sav, .dta)
│   ├── statistical_testing.py  # SciPy hypothesis evaluation engines
│   └── visualizer.py           # Publication-grade dashboard generators
└── results/
    ├── hypothesis_report.txt   # Automated statistical summary outputs
    └── figures/
        ├── stratification_barcharts.png
        └── correlation_heatmap.png

Data Integration & Schema Standardization

The ingestion pipeline processes datasets originating across three distinct enterprise and research database formats, handling them entirely via open-source tools without proprietary statistical software dependencies:

1. Environmental Ingestion (.xlsx): Parses regional ambient $PM_{2.5}$ exposure matrices (mimicking Harvard Dataverse structures) using an openpyxl engine.
2. Clinical Registry Ingestion (.sav / .dta): Unifies fragmented chronological waves of the National Health Interview Survey (NHIS) spanning distinct SPSS binary files (.sav) and Stata data stores (.dta) using a metadata-aware pyreadstat parsing framework.

Standardized Cohort Matching

The pipeline executes a cohort-level inner join aligning individual patient demographic and lifestyle rows to regional air quality metrics via an engineered, unified REGION token. Categorical keys are programmatically recoded, age structures are partitioned into explicit clinical brackets (0-18, 19-35, 36-50, 51-65, 65+), and active cancer cases are mapped to a clean binary structure (CNKIND14_BINARY) to support downstream mathematical testing.

Statistical Framework & Hypothesis Testing

The pipeline automates two distinct classical inference methods via the scipy.stats ecosystem:

1. Environmental Exposure Evaluation: Welch's Independent T-Test

   Evaluates whether regional lung cancer incidence rates differ significantly between areas with High ambient pollution ($PM_{2.5} \ge 10.59,\mu\text{g/m}^3$) versus Low ambient pollution ($PM_{2.5} < 10.59,\mu\text{g/m}^3$).

• Null Hypothesis ($H_0$): $\mu_{\text{High } PM2.5} = \mu_{\text{Low } PM2.5}$
• Alternative Hypothesis ($H_a$): $\mu_{\text{High } PM2.5} > \mu_{\text{Low } PM2.5}$
• Execution: Implemented with equal_var=False to preserve statistical validity under unequal sample variances and sample sizes.

2. Behavioral Exposure Evaluation: Chi-Square Test of Independence

   Evaluates the distributional association between behavioral classification (Smoking Status: Current, Former, Never) and clinical outcomes (Lung Cancer Presence).

• Null Hypothesis ($H_0$): Smoking status is completely independent of lung cancer presence.
• Alternative Hypothesis ($H_a$): Smoking status displays a significant categorical association with lung cancer presence.
• Execution: Programmatically constructs an exact cross-tabulated contingency matrix before running non-parametric evaluation.

How to Run & Replicate

1. Initialize the Environment

   Ensure you are running Python 3.11+ or 3.12+ inside your Mac terminal, clone the repository, and initialize your isolated sandbox:

   git clone [https://github.com/yourusername/lung-cancer-data-science-analysis.git](https://github.com/yourusername/lung-cancer-data-science-analysis.git)
   cd lung-cancer-data-science-analysis
   python3 -m venv .venv
   source .venv/bin/activate
   pip install -r requirements.txt
   

2. Execute the Full Workflow

   Run the master orchestration harness. This single command handles the entire multi-format data simulation, cross-format loading, statistical processing, and plot compilation out-of-the-box:

   python main.py
   

Results & Visual Deliverables

Upon pipeline completion, the system logs reproducible text summaries and exports two visual artifacts:

• Statistical Reporting (results/hypothesis_report.txt):
  • Environmental Aspect: Welch's T-Test returns a highly significant result ($T \approx 20.95$, $p \ll 0.05$), strongly rejecting $H_0$ and indicating elevated incidence across highly polluted sectors.
  • Behavioral Aspect: Chi-Square evaluation reveals an intense distributional tilt ($Chi^2 \approx 513.51$, $p \ll 0.05$), decisively rejecting the independence baseline and mapping smoking status to elevated risk.
• Visual Dashboards (results/figures/):
  • stratification_barcharts.png: A high-resolution, multi-panel dashboard illustrating lung cancer incidence trends stratified by region, lifestyle choices, air index tiers, age groups, and sex.
  • correlation_heatmap.png: Displays an annotative correlation grid mapping directional interactions across all primary features.

Limitations & Future Directions

• Aggregated Spatial Resolution: Due to geographical grouping parameters within historical data series, environmental metrics are modeled at the regional tier. Accessing localized county-level coordinates or ZIP-code arrays would provide the granularity needed for robust multivariable spatial regression modelling.
• Transition to Precision Multi-Omics: While environmental and behavioral stratification provides strong epidemiological guideposts, macro-level datasets lack the molecular resolution required for true precision healthcare. Future directions could link clinical datasets with underlying multi-omics layers, evaluating how particulate exposure correlates with specific mutational signatures or single-cell gene expression variations in lung tissue.