Job Market Analysis

A data analytics project exploring U.S. job posting trends for data-focused roles between November 2022 and November 2025. The analysis identifies which skills are actually in demand, how that demand shifts over time, and where remote and onsite postings diverge.

Dataset

Data Analyst Job Postings (Pay, Skills, Benefits) by Luke Barousse — Google Search results for data analyst job postings, updated daily on Kaggle.

Key findings

• SQL is the single most requested skill, appearing in 49.8% of data analyst postings — more common than any other tool or language in the dataset.
• SQL and Python are the strongest skill pairing. They co-occur in 14,112 postings (24% of all postings), the highest joint-frequency pair in the co-occurrence matrix — evidence that employers expect analysts to query and script, not just one or the other.
• The SQL/Python ranking flips depending on role. For Data Analyst postings (n=46,780), SQL leads (~49%) and Python sits well behind (~27%, 4th place). For Data Scientist postings (n=3,562), that order inverts — Python leads at over 70% and SQL is 2nd (~53%). The two roles prioritize the same two core skills in opposite order.
• The remote-vs-onsite skill comparison is included as an exploratory breakdown (see below) — a real but modest pattern, not yet distilled into a single statistically confirmed conclusion.

What this project does

• Cleans and prepares raw job posting data from data/raw/gsearch_jobs.csv
• Extracts structured skill data from tokenized job descriptions
• Computes demand trends for top skills over time
• Compares remote and onsite skill requirements
• Builds a co-occurrence matrix showing which skills most often appear together
• Classifies job titles into role categories and compares skill profiles across categories

Repository layout

Job_Market_Analysis/
├── data/
│   ├── raw/
│   │   └── gsearch_jobs.csv      # gitignored
│   └── processed/
│       └── clean_jobs_data.csv
├── notebooks/
│   └── analysis.ipynb
├── outputs/
│   ├── top_skills.png
│   ├── skill_trends.png
│   ├── cooccurrence.png
│   ├── remote_vs_onsite.png
│   └── skills_by_role.png
├── requirements.txt
├── .gitignore
├── LICENSE
└── README.md

Note on data/: the data/ folder is excluded via .gitignore (raw data is too large and updates daily on the source). The path is shown above so you know where to place the downloaded CSV before running the notebook.

Analysis outputs

Most frequently requested skills across all postings. SQL leads at 49.8%.

---

Monthly demand trends for the top skills over the Nov 2022–Nov 2025 window. There's a sharp, single-month spike around Aug–Sep 2024 (Power BI and SQL both jump abruptly, then revert the following month) that hasn't been explained — flagged here as an anomaly worth investigating (e.g. duplicate postings or a scrape artifact for that period) rather than a confirmed market signal.

---

Skill co-occurrence matrix. SQL + Python is the highest-frequency pairing (14,112 postings).

---

Skill frequency compared between remote and onsite listings. Exploratory — shows the breakdown by category; no single isolated difference has been confirmed as statistically significant yet.

---

Top skills broken down by role category. SQL leads for Data Analyst, Business Analyst, Data Engineer, and Marketing Analyst postings — but Python leads for Data Scientist postings (>70%), with SQL second (~53%). The skill hierarchy is role-dependent, not universal.

Data processing workflow

1. Load raw CSV data into a pandas DataFrame
2. Select relevant columns and keep a clean subset
3. Parse description_tokens from string form into Python lists
4. Convert date_time to datetime and extract year/month
5. Flag salary availability and compute skill counts
6. Classify job titles into role categories
7. Explode skills into one row per skill for detailed analysis

How to run

cd Job_Market_Analysis
# For Windows
venv\Scripts\activate
# For Mac/Linux
source venv/bin/activate
pip install -r requirements.txt

Download the dataset from the Kaggle link above and place it at data/raw/gsearch_jobs.csv. Then open notebooks/analysis.ipynb in Jupyter and run the cells in order. The notebook generates the charts in outputs/ and writes cleaned data to data/processed/.

Analysis Notes & Limitations

• The core, confirmed finding: SQL is required in roughly half of all data analyst postings (49.8%), and when a posting requires a second language, it's almost always Python (SQL+Python is the top co-occurring pair at 14,112 postings).
• The SQL/Python priority flips by role: Data Analyst postings rank SQL above Python (~49% vs ~27%); Data Scientist postings rank Python above SQL by a wide margin (>70% vs ~53%). Data Engineer postings also rank SQL above Python, but the gap there is narrow (~64% vs ~57%) — a weaker version of the same pattern, not independent confirmation of it. This came from cross-checking the overall skill ranking against role-segmented data rather than stopping at the aggregate result.
• Salary data in this dataset has substantial missingness, so salary analysis was treated as a secondary angle rather than a primary finding — a deliberate scoping decision.
• The remote-vs-onsite breakdown is included as an exploratory visual. It shows a real but modest pattern (onsite postings request most named tools at slightly higher rates than remote postings) that hasn't been confirmed as statistically significant.
• There's an unexplained single-month spike in the trends chart (Aug–Sep 2024) that has not been root-caused. It's flagged here rather than attributed to a specific cause, since no underlying check (e.g. duplicate postings, scrape artifacts for that period) has been run yet.

Notes

• clean_jobs_data.csv in data/processed/ is available for downstream reporting or BI tools.
• The notebook is the primary analysis driver and can be extended with additional role filters, time windows, or skill-group comparisons.

License

This project is available under the MIT License.