Author: Muthazir A
Institution: SRM Institute of Science and Technology, Vadapalani
Dataset: Data Science Job Postings & Skills (2024)
This project analyses the structure of the data science job market across four countries — the United States, United Kingdom, Canada, and Australia — using a Kaggle dataset of 12,217 real job postings. The study identifies in-demand skills, uncovers relationships between skills through network analysis, and segments the skill landscape using unsupervised machine learning. Findings are presented through an interactive Power BI dashboard alongside a reproducible analytical pipeline of 14 Jupyter notebooks.
The data science job market is characterised by a rapidly evolving and fragmented skill landscape. Job seekers and academic institutions often lack clarity on which skills are truly in demand, how skills relate to one another, and how demand is structured across experience levels and geographies. This project addresses these gaps by providing a data-driven analysis of skill demand, skill co-occurrence networks, and demand-based skill clustering.
| Property | Detail |
|---|---|
| Source | Kaggle — Data Science Job Postings & Skills (2024) |
| Total job postings | 12,217 |
| Total skill mentions | 315,000+ |
| Countries covered | United States, United Kingdom, Canada, Australia |
| Roles covered | Data science-related roles across all experience levels |
The dataset aggregates job postings from multiple platforms and includes structured fields for job title, location, job level, job type, and associated skills per posting.
- Python is the most in-demand skill, appearing in 4,815 job postings — present in approximately 1 in 12 postings across all roles.
- SQL is the second most demanded skill (4,612 postings), confirming it as a non-negotiable baseline requirement alongside Python.
- Machine Learning, Data Visualization, and Tableau round out the top five, reflecting strong demand for both modelling and communication of insights.
- The top 5 skills collectively account for 47% of total skill demand, while the remaining skills make up 53% — indicating a concentrated but not monopolised market.
Skill demand follows a three-tier structure:
- High demand — 12,700 mentions across dominant skills
- Medium demand — 13,400 mentions across mid-tier specialisations
- Low demand — 11,300 mentions across the long tail of niche skills
- Python + SQL is the most common skill combination, appearing together in 2,498 job postings.
- Machine Learning + Python (1,750) and Python + R (1,417) follow closely, reflecting the centrality of Python as a connective skill across all data science sub-disciplines.
- Network analysis of 434 unique skills and 3,984 co-occurrence relationships revealed that Python holds the highest betweenness centrality (0.276), confirming its role as the primary bridge skill across all communities.
K-Means clustering (K=5, silhouette score = 0.987) identified five distinct skill demand segments:
- Dominant skills — Python and SQL, disproportionately high demand
- Core data science skills — Data Analysis, Machine Learning, Data Visualization
- High demand tools — AWS, Tableau, R, Spark, Java
- Mid demand specialisations — Snowflake, Power BI, Hadoop, Scala, Azure
- Standard skills — 2,817 niche and complementary skills forming the long tail
The United States accounts for the largest share of job postings, followed by the United Kingdom, Canada, and Australia.
The project follows an 14-stage analytical pipeline:
| Notebook | Description |
|---|---|
| 01_data_inspection | Initial exploration of raw data structure and quality |
| 02_data_cleaning | Standardisation and deduplication of job postings |
| 03_skills_cleaning | Normalisation of skill labels and removal of noise |
| 04_data_integration | Merging job postings and skills into a unified dataset |
| 05_skill_explosion | Exploding multi-skill rows into one row per skill |
| 06_skill_eda | Exploratory analysis of skill frequency and distribution |
| 07_skill_trends | Analysis of skill demand patterns |
| 08_emerging_skills | Identification of skills with rising demand |
| 09_validation | Data quality checks and statistical validation |
| 10_dashboard_exports | Exporting aggregated data for Power BI |
| 11_skill_cooccurrence | Computing skill co-occurrence pairs |
| 12_skill_clustering | K-Means clustering of skills by demand profile |
| 14_network_analysis | Graph construction, centrality metrics, community detection |
Job Market Intelligence/
│
├── data/
│ ├── raw/ # Original Kaggle CSV files
│ ├── cleaned/ # Post-cleaning datasets
│ └── processed/ # Analysis-ready exports
│
├── notebooks/ # Jupyter notebooks (01–14)
├── dashboard/ # Power BI .pbix file and PDF export
├── docs/ # Charts, network graph, data dictionary
└── requirements.txt # Python dependencies
The dashboard contains three pages:
Page 1 — Job Market Intelligence Dashboard
Overview of total job postings (12K), total skill mentions (315K), top skill (Python), and top country (United States). Includes a ranked bar chart of the top 15 most in-demand skills and a breakdown of demand by skill category (Programming, BI Tools, Big Data, Cloud).
Page 2 — Skill Relationship Analysis
Displays the most common skill combinations found in real job postings. Python + SQL leads with 2,498 co-occurrences, followed by Machine Learning + Python and Python + R. Reveals the technology stacks most commonly expected by employers.
Page 3 — Market Concentration of Skills
Analyses how skill demand is distributed across tiers. Shows that the top 5 skills account for 47% of all demand, while the remaining skills collectively contribute 53% — evidence of a concentrated but accessible market structure.
Requirements
- Python 3.12+
- Jupyter Notebook or JupyterLab
Setup
# 1. Clone or extract the project folder
cd "Job Market Intelligence"
# 2. Install dependencies
pip install -r requirements.txt
# 3. Launch Jupyter
jupyter notebookRun order
Execute notebooks in numerical order from 01_data_inspection through to 14_network_analysis. Each notebook reads from the previous stage's output in data/processed/.
Dashboard
Open dashboard/Job Market Intelligence Dashboard.pbix in Power BI Desktop. If data connections need refreshing, point them to the CSV files in data/processed/.
pandas
numpy
matplotlib
seaborn
scikit-learn
networkx
pyvis
scipy
jupyter
- The dataset is sourced from Kaggle and aggregated from multiple job platforms, which may introduce collection bias toward certain geographies or role types.
- Skill labels were normalised during cleaning, but some variation in terminology across postings may remain (e.g. "ML" vs "Machine Learning").
- The analysis is cross-sectional — it captures a snapshot of demand for 2024 and does not account for real-time market changes.
- Salary data was not available in the dataset, limiting analysis of the economic dimension of skill demand.
- Integration of a live job posting scraper to enable real-time skill tracking
- Expansion to additional countries and non-data-science roles
- Inclusion of salary data to quantify the economic value of individual skills
- Development of a personalised skill gap recommender based on a user's current skill set
Submitted in partial fulfilment of the requirements for the final year project — SRM Institute of Science and Technology, Vadapalani.