Summary of cdsp

.gitignore

@@ -5,3 +5,5 @@ venv/
 *.db
 *.idea
 *.ruff_cache
+.DS_Store
+.DS_Store

.vscode/settings.json

@@ -122,5 +122,6 @@
       "source.fixAll.ruff": "explicit",
       "source.organizeImports.ruff": "explicit"
     }
-  }
+  },
+  "cSpell.words": ["Kaggle"]
 }

0_domain_study/README.md

@@ -1 +1,83 @@
 # Domain Research
+
+## Background  
+
+In this milestone, our team explored several promising project ideas across
+various domains, including education, employment, migration, financial literacy,
+and the supply chain. Reaching a consensus was challenging, as each team member
+was passionate about different topics. However, after thorough discussion and a
+group vote, we agreed to focus on the U.S. retail supply chain. This topic
+aligns with our collective interests and represents a real-world issue that we
+are eager to analyze collaboratively, with the goal of identifying potential
+solutions.
+
+## Problem Statement  
+
+In recent years, the U.S. retail supply chain has experienced a dramatic
+transformation driven by the rise of e-commerce, heightened customer expectations
+for fast delivery, and increasing urbanization. While innovations such as
+same-day shipping and real-time package tracking have become common, delivery
+delays remain a persistent and costly challenge for retailers.
+
+From industry giants like Amazon and Walmart to small online businesses, delayed
+shipments erode customer trust, inflate operational costs, and increase the
+volume of returns. According to *Statista* (2023), over 80% of American
+consumers expect delivery within two days, and nearly 70% report dissatisfaction
+when packages are delayed even once. Seasonal surges (e.g., holidays) and
+location-specific issues (e.g., traffic congestion in Los Angeles or winter
+storms in Chicago) further hinder timely fulfillment.
+
+This issue is multifactorial, involving transportation bottlenecks, logistics
+coordination failures, carrier limitations, weather anomalies, and infrastructure
+weaknesses. The U.S. transportation system ranks only 17th globally in
+efficiency, and the country faces a shortage of more than 80,000 truck drivers
+(*ATA*, 2023). Delivery delays are not just inconvenient—they are systemic and
+require data-driven strategies for mitigation.
+
+## Research Questions
+
+### Main Research Question
+
+- What are the key factors that contribute to delivery delays in the retail supply
+  chain, and how can they be mitigated?
+
+### Supporting Research Questions
+
+- What transportation and environmental factors most significantly contribute to
+  delivery delays in the U.S. retail supply chain, and how accurately can machine
+  learning models predict these delays?
+- How do environmental and event-based anomalies (e.g., weather, COVID-19) affect
+  last-mile delivery delays in urban retail supply chains, and how accurately can
+  machine learning models predict these delays?
+- What are the major transportation and logistics-related causes of delivery delays
+  in the U.S. retail supply chain, and how can predictive models help identify
+  high-risk deliveries?
+- How can machine learning be used to predict and reduce delivery delays in the U.S.
+  retail supply chain based on traffic, weather, and logistics data?
+- How can machine learning models improve the accuracy of demand forecasting in
+  retail supply chains during seasonal fluctuations?
+
+## Literature Review
+
+- Ahmad, M. A., & Al-Bazi, A. (2023, May). *Disruptions in supply chain
+  transportation: A literature review.* 4th International Conference on
+  Administrative & Financial Sciences (ICAFS 2023).
+  [DOI link](https://doi.org/10.24086/ICAFS2023/paper.894)
+
+- Müller, N., Burggräf, P., Steinberg, F., Sauer, C. R., & Schütz, M. (2025).
+  *An analytical review of predictive methods for delivery delays in supply
+  chains.* Supply Chain Analytics, 11, 100130.
+  [DOI link](https://doi.org/10.1016/j.sca.2025.100130)
+
+## Data Sources
+
+### Kaggle
+
+- [https://www.kaggle.com](https://www.kaggle.com)
+
+### Additional Reference Links
+
+- [NRF: Call to Address Port Congestion](https://nrf.com/media-center/press-releases/nrf-calls-white-house-address-port-congestion-challenges)  
+- [Port Congestion & White House Response – Splash247](https://splash247.com/biden-pressured-to-fix-us-port-congestion-issues/)  
+- [U.S. Truck Driver Shortage – Truck News](https://www.trucknews.com/human-resources/u-s-is-short-78000-drivers-ata-says/1003170001/)  
+- [PYMNTS: Lack of Real-Time Data in Grocery Retail](https://www.pymnts.com/news/retail/2025/65percent-of-grocery-retailers-lack-real-time-supply-chain-data/)

0_domain_study/brainstorming.md

@@ -0,0 +1,192 @@
+
+# 📝 Introduction
+
+ Group 22 engaged in a thoughtful brainstorming and evaluation process to
+ identify a high-impact, data-driven project. Our goal was to choose a research
+ question that is relevant, actionable, and supported by available datasets.
+ Throughout this process, we explored a wide range of domains including
+ **education, humanitarian issues, financial technology, employment, supply
+ chain management, and the impact of this very program on career outcomes**
+.
+
+The following is a structured summary of the ideas we discussed, along with
+their associated research questions and sub-questions. This documentation reflects
+the depth of our collaboration and the progression toward selecting our final
+project focus.
+
+---
+
+## 💡 Brainstormed Project Ideas & Research Questions
+
+### 1. **Impact of the MIT Emerging Talent Program**
+
+- **Main Research Question:**  
+  *How does participation in the MIT Emerging Talent Certificate Program affect
+  students’ career trajectories, skills development, and income levels?*
+
+- **Sub-questions:**
+  - What types of technical and soft skills do participants report gaining?
+  - Is there measurable income growth or increased job stability post-program?
+  - How does the impact vary by background (e.g., prior education, immigration
+    status, refugee experience)?
+
+### 2. **Humanitarian + Education: Refugee Access to Education**
+
+- **Main Research Question:**  
+  *To what extent do digital education initiatives (e.g., mobile learning,
+  online classrooms) reach refugee learners, and how effective are they?*
+- **Sub-questions:**
+  - What are the key technological and infrastructural barriers?
+  - How does access vary by region or age group?
+  - What role do NGOs or international bodies (e.g., UNHCR) play in implementation?
+
+---
+
+### 3. **Career Transition & Education Outcomes**
+
+- **Main Research Question:**  
+  *To what extent do structured career transition programs affect job attainment
+  and income levels among participants?*
+- **Sub-questions:**
+  - What demographic factors influence program success?
+  - What role does mentorship or job placement support play?
+  - Are income improvements sustained over time?
+
+---
+
+### 3. **Mortgage Applications & Interest Rate Impact (FinTech)**
+
+- **Main Research Question:**  
+  *How do changes in central bank interest rates affect the volume and type of
+  mortgage applications over time?*
+- **Sub-questions:**
+  - Which loan types are most sensitive to rate changes?
+  - How do responses differ across income or credit groups?
+- **Data Source:** [FRED Economic Data](https://fred.stlouisfed.org/)
+
+---
+
+### 4. **Employee Turnover in SMEs**
+
+- **Main Research Question:**  
+  *What factors contribute most to early employee turnover, and how can we identify
+  employees at risk of leaving within their first year?*
+- **Sub-questions:**
+  - Does department or role type affect risk?
+  - How does onboarding quality impact retention?
+- **Data Source:** IBM HR Analytics (Kaggle)
+
+---
+
+### 5. **Bank Deposit Trends During Uncertainty**
+
+- **Main Research Question:**  
+  *How do economic shocks (e.g., inflation, layoffs, geopolitical events) impact
+  personal and business deposit trends?*
+- **Sub-questions:**
+  - Are small businesses more reactive than individuals?
+  - What lag exists between news events and deposit behavior?
+
+---
+
+### 6. **Student Debt & Career Choice**
+
+- **Main Research Question:**  
+  *How does student loan burden influence graduates’ career paths and earnings?*
+- **Sub-questions:**
+  - Are graduates avoiding certain professions due to debt?
+  - How does debt load correlate with postgraduate income?
+- **Data Source:** [College Scorecard](https://collegescorecard.ed.gov/data/)
+
+---
+
+### 7. **Remote vs. On-Site Work & Productivity**
+
+- **Main Research Question:**  
+  *How does work location (remote vs. on-site) impact employee productivity?*
+- **Sub-questions:**
+  - Are certain job types more suitable for remote work?
+  - How do project completion times compare?
+
+---
+
+### 8. **Gender Gap in Tech Employment**
+
+- **Main Research Question:**  
+  *What does data reveal about gender disparities in tech employment, and how have
+  they evolved over time?*
+- **Sub-questions:**
+  - Are certain roles or levels more unequal?
+  - Have diversity initiatives made measurable impact?
+
+---
+
+### 9. **Financial Literacy & Spending Behavior**
+
+- **Main Research Question:**  
+  *How does financial literacy affect individual saving and spending behavior?*
+- **Sub-questions:**
+  - Does financial education lead to better budgeting habits?
+  - How does literacy vary across age or region?
+- **Data Source:** OECD Financial Literacy Surveys
+
+---
+
+### 10. **AI & Job Market Evolution**
+
+- **Main Research Question:**  
+  *How have job posting requirements for software developers changed since the
+  emergence of AI tools (e.g., ChatGPT)?*
+- **Sub-questions:**
+  - Has demand shifted toward specific skills or languages?
+  - Are companies reducing headcount due to automation?
+  - How is AI adoption affecting employment trends in the U.S. tech sector?
+  - Are U.S. companies investing enough in upskilling/reskilling?
+
+---
+
+### 11. **Technical Certifications & ROI**
+
+- **Main Research Question:**  
+  *Which technical certifications (AWS, Google Cloud, Microsoft, etc.) provide the
+  highest return on investment in terms of salary and job opportunities?*
+- **Sub-questions:**
+  - Which certifications are most in-demand by region?
+  - What are typical salary boosts post-certification?
+
+---
+
+### 12. **Migration Trends & Global Talent Flow**
+
+- **Option 1:** Global Migration Trends and Drivers (2000–2025)  
+- **Option 2:** Global Competition for Talent: Skilled Labor Migration  
+  in the 21st Century  
+- **Potential Questions:**
+  - What economic and geopolitical factors drive skilled migration?
+  - Which countries are gaining or losing talent, and why?
+
+---
+
+## ✅ Preliminary Research Question: Supply Chain Project
+
+After two rounds of team voting and discussion, we selected the following as our
+final project focus:
+
+### 🧪  Preliminary Project Title
+
+### Understanding and Reducing Delivery Delays in the U.S. Retail Supply Chain
+
+### 🔬  Preliminary Research Question
+
+*What are the key factors that contribute to delivery delays in the U.S. retail supply
+chain, and how can they be mitigated?*
+
+---
+
+## 📍 Conclusion
+
+This document outlines the full range of ideas explored by Group 22 as part of our
+collaborative data science project. We carefully considered research value, data
+availability, and relevance before selecting our final focus. After thoughtful
+discussion and democratic voting, we agreed to pursue a project investigating
+**supply chain delivery delays in the U.S. retail sector**.

1_datasets/README.md

@@ -1 +1,107 @@
 # Datasets
+
+## Logistics and Supply Chain Dataset
+
+### Source
+
+- **Platform**: [Kaggle](https://www.kaggle.com/datasets/datasetengineer/logistics-and-supply-chain-dataset)
+- **Uploader**: `datasetengineer`  
+- **Listed Author**: `Austin Lasseter`  
+- **Access**: Publicly available for educational and research use  
+
+---
+
+### Structure and Content
+
+- Covers logistics operations in Southern California **(Jan 2021–Jan 2024)**
+- Provides **hourly** data on transport, warehousing, and routing  
+- Focuses on high-traffic urban corridors  
+- Useful for analyzing efficiency, risks, and delivery performance
+
+#### Features
+
+- **Timestamp**: Date and time of the logistics event  
+- **Vehicle GPS Latitude / Longitude**: Vehicle’s geolocation coordinates  
+- **Fuel Consumption Rate**: Liters of fuel consumed per hour  
+- **ETA Variation (hours)**: Gap between estimated and actual arrival  
+- **Traffic Congestion Level**: Congestion level on a 0–10 scale  
+- **Warehouse Inventory Level**: Units currently in warehouse  
+- **Loading/Unloading Time**: Time in hours to load/unload cargo  
+- **Handling Equipment Availability**: Equipment status (0 = no, 1 = yes)  
+- **Order Fulfillment Status**: Fulfillment on time (0 = no, 1 = yes)  
+- **Weather Condition Severity**: Weather impact (0–1 scale)  
+- **Port Congestion Level**: Port congestion level (0–10 scale)  
+- **Shipping Costs**: Cost of shipping in USD  
+- **Supplier Reliability Score**: Reliability score (0–1 scale)  
+- **Lead Time (days)**: Average supplier delivery time  
+- **Historical Demand**: Past logistics service demand (units)  
+- **IoT Temperature**: Temperature from sensors (°C)  
+- **Cargo Condition Status**: Cargo condition (0 = poor, 1 = good)  
+- **Route Risk Level**: Risk rating for the route (0–10 scale)  
+- **Customs Clearance Time**: Hours required for customs processing  
+- **Driver Behavior Score**: Driver pattern score (0–1 scale)  
+- **Fatigue Monitoring Score**: Driver fatigue score (0–1 scale)  
+
+#### Target Variables
+
+- **Disruption Likelihood Score**: Probability of disruption (0–1)  
+- **Delay Probability**: Likelihood of delivery delay (0–1)  
+- **Risk Classification**: Risk category — Low, Moderate, High  
+- **Delivery Time Deviation**: Hours deviated from expected delivery  
+
+---
+
+### Data Collection
+
+Collected from:
+
+- GPS tracking systems  
+- IoT sensors on vehicles and warehouses  
+- Warehouse management systems  
+- External sources (e.g., weather and port data)  
+
+The dataset is anonymized to preserve privacy while retaining analytical value.
+
+---
+
+### Possible Limitations
+
+- The dataset is not specific to the **retail sector**, which may limit direct
+alignment with our focus on retail supply chains.  
+- All data is sourced from **Southern California**, potentially limiting
+generalizability to other U.S. regions with different infrastructure and
+logistics environments.
+
+---
+
+### Relevance to Our Project
+
+This dataset supports our investigation of **delivery
+delays in the U.S. retail supply chain** through:
+
+- **Delay-focused target variables** such as delay probability and delivery time
+deviation
+- **Contextual features** that influence delays, including:
+  - Traffic congestion  
+  - Weather conditions  
+  - Route risk  
+  - Inventory levels and equipment availability  
+- **General logistics operations** applicable to retail supply chains, despite
+not being retail-specific
+
+---
+
+### Recreating the Dataset
+
+To recreate the cleaned dataset:
+
+1. **Download the raw CSV**  
+   From [Kaggle](https://www.kaggle.com/datasets/datasetengineer/logistics-and-supply-chain-dataset)
+   File: `dynamic_supply_chain_logistics_dataset.csv`
+
+2. **Run the cleaning script**  
+   - Drops 3 irrelevant columns: `vehicle_gps_latitude`, `vehicle_gps_longitude`,`fuel_consumption_rate`
+   - Confirms no missing or duplicate values
+   - Saves output as [`cleaned_dataset.csv`](../1_datasets/cleaned_and_processed_data/cleaned_dataset.csv).
+
+The script is available in [`2_data_preparation/clean_dataset.py`](../2_data_preparation/clean_dataset.py).