A/B Testing News Headlines: Which Headline Works Better?

An analysis pipeline that fetches live news article data from the NewsAPI, cleans it, assigns articles into A/B test groups based on headline length, simulates click engagement, and runs a statistical significance test to determine whether headline length affects reader clicks.

---

Project Context

You are a data analyst for a media platform experimenting with article headline strategy and post timing. The question being tested:

Do longer headlines drive more clicks than shorter ones?

• Group A (Short): Headlines at or below the median character length
• Group B (Long): Headlines above the median character length

---

Dataset

Source

Live data fetched from the NewsAPI — US top headlines, general category, up to 100 results per request.

Shape

34 articles × 8 columns (after API fetch; duplicates removed before analysis)

Columns

Column	Type	Description
source	object	Publisher name (extracted from nested dict)
author	object	Article author (10 missing values)
title	object	Article headline — primary variable
description	object	Short article summary (2 missing)
url	object	Article URL
urlToImage	object	Thumbnail image URL (2 missing)
publishedAt	object	Publication timestamp
content	object	Article body excerpt (4 missing)

Engineered Columns

Column	Description
headline_length	Character count of title
group	A/B group assignment: 'A (Short)' or 'B (Long)' based on median split
clicks	Simulated engagement metric (see Simulation section)

---

Methodology

1. API Integration and Data Retrieval

• Fetches up to 100 US top headlines from NewsAPI (/v2/top-headlines)
• Stores raw response as a DataFrame and exports to raw_news_data.csv

2. Data Cleaning

• Inspected data types, shape, and missing values
• Extracted source name from nested dict column
• Added headline_length feature
• Dropped duplicate articles by title
• Imputed missing author values with the column mode

3. Group Assignment — A/B Logic

• Computed median headline length across all articles
• Split articles into Group A (≤ median) and Group B (> median)
• Each group received exactly 17 articles

4. Engagement Simulation

• Simulated click counts using seeded normal distributions:
  • Group A (Short): Normal(μ=100, σ=25)
  • Group B (Long): Normal(μ=150, σ=30)
• Clipped negative values to 0 and rounded to whole numbers

5. Exploratory Analysis

• Headline length statistics by group
• Headline length distribution histogram (with KDE)
• Headline length boxplot by group
• Simulated clicks boxplot by group
• Average clicks bar chart by group
• Top 10 news sources bar chart

6. A/B Test Execution

• Two-sample independent t-test (scipy.stats.ttest_ind) on simulated clicks
• Significance threshold: α = 0.05

---

Dependencies

pip install requests pandas numpy matplotlib seaborn scipy

Library	Purpose
requests	NewsAPI HTTP calls
pandas	Data manipulation
numpy	Simulation, group assignment
matplotlib	Chart rendering
seaborn	Statistical visualisations
scipy.stats	Two-sample t-test

---

Running the Notebook

1. Ensure a valid NewsAPI key is set in Cell 3 (the current key may expire).
2. Install all dependencies listed above.
3. Run all cells top-to-bottom — API fetch happens must succeed before downstream cells execute.
4. Output files (raw_news_data.csv, cleaned_articles.csv) and chart images are saved to the working directory.

Note on reproducibility: Simulated click data uses np.random.seed(42), so results are deterministic given the same article set. However, the article set itself changes with each live API call.

---

Key Results Summary

Metric	Group A (Short)	Group B (Long)
Articles	17	17
Mean headline length	66.9 chars	102.5 chars
Simulated mean clicks	~100	~150
T-statistic	-5.382
P-value	0.0000065
Statistically significant?	Yes (p < 0.05)

See REPORT for the full analysis narrative, interpretation, and limitations.