PEH Data Collection and Analysis Project

This repository contains a comprehensive data collection and analysis pipeline for studying homelessness and housing issues across multiple cities. The project collects data from Reddit, X (Twitter), news articles, and meeting minutes, then processes and analyzes this data for research purposes.

Project Overview

The project focuses on collecting and analyzing data related to homelessness and housing issues from multiple sources across various cities. Data is collected from:

• Reddit: Comments and submissions from city-specific subreddits
• X (Twitter): Posts from the X API
• News Articles: Articles from LexisNexis API
• Meeting Minutes: City council meeting transcripts

Data Structure

data/
├── baltimore/
│   ├── reddit/
│   │   ├── all_comments.csv
│   │   ├── filtered_comments.csv
│   │   ├── filtered_comments_deidentified.csv
│   │   └── statistics.csv
│   ├── x/
│   │   ├── posts_english_2015-2025.csv
│   │   ├── posts_english_2015-2025_rt.csv
│   │   ├── posts_english_2015-2025_rt_deidentified.csv
│   │   └── statistics.csv
│   ├── newspaper/
│   │   ├── lexisnexis.csv
│   │   ├── baltimore_filtered.csv
│   │   ├── baltimore_filtered_deidentified.csv
│   │   └── statistics.csv
│   └── meeting_minutes/
│       ├── BuffaloTranscripts1of3/
│       ├── BuffaloTranscripts2of3/
│       ├── BuffaloTranscripts3of3/
│       ├── meeting_minutes_lexicon_matches.csv
│       ├── meeting_minutes_lexicon_matches_deidentified.csv
│       └── statistics.csv
├── buffalo/
├── elpaso/
├── fayetteville/
├── kzoo/
├── portland/
├── rockford/
├── sanfrancisco/
├── scranton/
├── southbend/
└── data_summary/
    └── data_summary_by_city.csv

Cities Covered

• Baltimore, MD
• Buffalo, NY
• El Paso, TX
• Fayetteville, NC
• Kalamazoo, MI
• Portland, OR
• Rockford, IL
• San Francisco, CA
• Scranton, PA
• South Bend, IN

Setup

1. Environment Setup

# Create virtual environment
python3 -m venv venv

# Activate virtual environment
# On macOS/Linux:
source venv/bin/activate
# On Windows:
# venv\Scripts\activate

2. Install Dependencies

pip install -r requirements.txt

3. Download spaCy Model

python -m spacy download en_core_web_sm

4. Environment Variables

Create a .env file in the root directory with the following variables:

# Reddit API credentials
REDDIT_CLIENT_ID=your_client_id_here
REDDIT_CLIENT_SECRET=your_client_secret_here
REDDIT_USER_AGENT=your_user_agent_here

# News API credentials
NEWS_API_KEY=your_news_api_key_here

# LexisNexis API credentials
LEXISNEXIS_API_ID=your_lexisnexis_api_id_here
LEXISNEXIS_API_KEY=your_lexisnexis_api_key_here

Scripts Overview

Data Collection Scripts

Reddit Data Collection

# Edit the script to change the subreddit name
# In scripts/get_reddit_data.py, modify line 9:
# subreddit_name = "southbend"

python scripts/get_reddit_data.py

• Collects comments and submissions from city-specific subreddits
• Filters for relevant keywords related to homelessness and housing
• Outputs: all_comments.csv, filtered_comments.csv
• Note: Subreddit name must be modified in the script before running

X (Twitter) Data Collection

# Collect data for specific city
python scripts/get_twitter_data.py --city "san francisco"

# Collect data for all cities
python scripts/get_twitter_data.py

# Count tweets only (no data collection)
python scripts/get_twitter_data.py --count-only

• Collects posts containing relevant keywords
• Filters for English content and retweets
• Outputs: posts_english_2015-2025.csv, posts_english_2015-2025_rt.csv

News Data Collection

# LexisNexis - specific city
python scripts/get_lexisnexis_data.py southbend

# LexisNexis - all cities
python scripts/get_lexisnexis_data.py

# News API (San Francisco only)
python scripts/get_news_api_data.py

• Collects news articles from LexisNexis and News API
• Filters for relevant keywords
• Outputs: lexisnexis.csv, filtered news files

News Processing and Filtering

# Filter news articles by paragraph and keywords
python scripts/filter_lexisnexis_by_paragraph.py

# Filter specific cities
python scripts/filter_lexisnexis_by_paragraph.py --cities southbend portland

# Process long articles (shorten around keywords)
python scripts/process_long_articles.py --max-paragraphs 1

# Process specific cities with custom parameters
python scripts/process_long_articles.py --cities baltimore,portland --max-paragraphs 2 --max-sentences 10

# Sample news articles with processed data
python scripts/sample_news_with_processed_articles.py --samples-per-city 50

# Sample specific cities with custom output
python scripts/sample_news_with_processed_articles.py --cities baltimore,portland --output-dir custom_samples

• Filtering: Filters news articles by paragraph and keyword matches
• Processing: Shortens long articles around relevant keywords using paragraph context detection
• Sampling: Creates manageable samples from processed articles while preserving few-shot examples
• Outputs: {city}_filtered.csv, {city}_processed_articles.csv, sampled_lexisnexis_news.csv

Meeting Minutes Processing

# Process meeting minutes for specific city
python scripts/get_meeting_minute_paragraphs.py --city southbend

# Process San Francisco meeting minutes
python scripts/get_meeting_minutes_san_francisco.py

• Processes meeting minutes transcripts
• Extracts paragraphs containing relevant keywords
• Outputs: meeting_minutes_lexicon_matches.csv

Data Processing Scripts

Deidentification

# Deidentify all data types
python scripts/deidentify_text.py

# Deidentify specific data type
python scripts/deidentify_text.py --type reddit

# Deidentify specific city
python scripts/deidentify_text.py --cities southbend,portland

• Removes personally identifiable information from text data
• Uses spaCy for named entity recognition
• Outputs: *_deidentified.csv files

Statistics Generation

# Generate statistics for all cities
python scripts/generate_statistics.py

# Generate statistics for specific city
python scripts/generate_statistics.py --city southbend

• Creates statistics.csv files in each subfolder
• Counts keyword matches from the lexicon
• Provides file size and row count information

Analysis Scripts

Data Summary

python scripts/data_summary_by_city.py

• Generates summary statistics across all cities
• Outputs: data_summary_by_city.csv

News Processing and Analysis

# Filter news articles by paragraph and keywords
python scripts/filter_lexisnexis_by_paragraph.py

# Filter specific cities
python scripts/filter_lexisnexis_by_paragraph.py --cities southbend portland

# Process long articles (shorten around keywords)
python scripts/process_long_articles.py --max-paragraphs 1

# Process specific cities with custom parameters
python scripts/process_long_articles.py --cities baltimore,portland --max-paragraphs 2 --max-sentences 10

# Sample news articles with processed data
python scripts/sample_news_with_processed_articles.py --samples-per-city 50

# Sample specific cities with custom output
python scripts/sample_news_with_processed_articles.py --cities baltimore,portland --output-dir custom_samples

• Filtering: Filters news articles by paragraph and keyword matches
• Processing: Shortens long articles around relevant keywords using paragraph context detection and sentence segmentation
• Sampling: Creates manageable samples from processed articles while preserving few-shot examples
• Outputs: {city}_filtered.csv, {city}_processed_articles.csv, sampled_lexisnexis_news.csv

Keyword Analysis

python scripts/add_keywords_to_reddit.py
python scripts/add_keywords_to_x_deidentified.py

• Adds keyword analysis to deidentified datasets
• Identifies which keywords appear in each entry

Data Sampling Scripts

Comprehensive Data Sampling

# Sample 50 posts per city (default)
python scripts/sample_all_data.py

# Sample 100 posts per city
python scripts/sample_all_data.py --samples-per-city 100

# Copy ALL data (not samples) to separate files
python scripts/sample_all_data.py --mode all --output-dir complete_dataset

# Sample 25 posts per city to custom directory
python scripts/sample_all_data.py --samples-per-city 25 --output-dir small_sample

• Samples or copies all data types (Twitter, meeting minutes, Reddit, newspaper)
• Default sample size: 50 per city
• Creates separate CSV files for each data type (due to different column structures)
• Outputs individual files + combined sample file for analysis

Sampling Mode Output:

• gold_standard/sampled_twitter_posts.csv
• gold_standard/sampled_meeting_minutes.csv
• gold_standard/sampled_reddit_comments.csv
• gold_standard/sampled_newspaper_articles.csv
• gold_standard/combined_sample.csv (with data_type column)

All Data Mode Output:

• output_dir/all_twitter_posts.csv
• output_dir/all_meeting_minutes.csv
• output_dir/all_reddit_comments.csv
• output_dir/all_newspaper_articles.csv

Complete Dataset Summary Statistics

Data Type	Cities with Data	Total Records
Twitter Posts	10	4,282
Meeting Minutes	8	9,181
Reddit Comments	10	32,413
Newspaper Articles	10	2,577
Total	10	48,453

Note: Actual numbers vary by city and data availability. Meeting minutes are only available for cities with public transcripts.

Data Sources and Keywords

Lexicon Keywords

The project uses a predefined lexicon of keywords related to homelessness and housing:

KEYWORDS = [
    'homeless', 'homelessness', 'housing crisis',
    'affordable housing', 'unhoused', 'houseless',
    'housing insecurity', 'beggar', 'squatter', 
    'panhandler', 'soup kitchen'
]

Data Sources

1. Reddit: City-specific subreddits and regional subreddits
2. X (Twitter): Public posts containing relevant keywords
3. News Articles: LexisNexis and News API articles
4. Meeting Minutes: City council meeting transcripts

News Processing Features

Article Filtering

• Paragraph-based filtering: Extracts paragraphs containing relevant keywords
• Duplicate detection: Removes duplicate paragraphs across articles
• Content validation: Ensures articles contain meaningful content

Long Article Processing

• Paragraph context detection: Identifies natural paragraph breaks in articles
• Sentence segmentation fallback: Uses spaCy for sentence-level processing when paragraphs aren't detected
• Keyword-focused shortening: Creates focused versions around relevant keywords
• Configurable limits: Set maximum paragraphs or sentences per processed article

Processing Methods

1. Paragraph Method: Detects paragraph breaks using multiple regex patterns
2. Sentence Method: Falls back to spaCy sentence segmentation when paragraphs aren't found
3. Context Preservation: Maintains surrounding context around keyword matches
4. Segment Creation: Creates separate segments for articles with multiple keyword locations

Sampling with Few-Shot Preservation

• Few-shot example detection: Automatically finds and preserves specific examples from the original dataset
• Balanced sampling: Ensures representation across cities and sources
• Quality preservation: Maintains article metadata and processing information

File Naming Conventions

• all_*.csv: Raw collected data
• filtered_*.csv: Data filtered for relevant keywords
• *_processed_articles.csv: News articles processed and shortened around keywords
• *_deidentified.csv: Data with PII removed
• *_lexicon_matches.csv: Meeting minutes with keyword matches
• statistics.csv: Summary statistics for each subfolder
• sampled_*.csv: Sampled data for analysis
• all_*.csv: Complete datasets (when using --mode all)

Data Processing Pipeline

1. Collection: Raw data collected from various sources
2. Filtering: Data filtered for relevant keywords
3. News Processing: Articles shortened around keywords using paragraph context detection
4. Deidentification: PII removed for privacy protection
5. Analysis: Statistics and keyword analysis generated
6. Summary: Cross-city analysis and reporting
7. Sampling: Create manageable datasets for analysis (50 samples per city by default)

Privacy and Ethics

• All data is deidentified to protect individual privacy
• Meeting minutes are public records
• Social media data is publicly available
• News articles are from public sources
• No personally identifiable information is retained in processed datasets

Usage Examples

Complete Pipeline

# 1. Collect Reddit data
python scripts/get_reddit_data.py

# 2. Collect X data
python scripts/get_twitter_data.py

# 3. Collect news data
python scripts/get_lexisnexis_data.py

# 4. Filter and process news articles
python scripts/filter_lexisnexis_by_paragraph.py
python scripts/process_long_articles.py --max-paragraphs 1

# 5. Process meeting minutes (if available)
python scripts/get_meeting_minute_paragraphs.py

# 6. Deidentify all data
python scripts/deidentify_text.py

# 7. Generate statistics
python scripts/generate_statistics.py

# 8. Create samples for analysis
python scripts/sample_all_data.py --samples-per-city 100

Analysis Workflow

# Generate comprehensive statistics
python scripts/generate_statistics.py

# Create data summary
python scripts/data_summary_by_city.py

# Create samples for analysis
python scripts/sample_all_data.py

# View statistics for specific city
cat data/southbend/reddit/statistics.csv

Troubleshooting

Common Issues

1. spaCy model not found: Run python -m spacy download en_core_web_sm
2. API rate limits: Check API credentials and wait between requests
3. Memory issues: Process cities individually for large datasets
4. File not found errors: Ensure data directories exist before processing

News Processing Issues

1. Long article processing: Use --max-paragraphs and --max-sentences parameters to control article length
2. Paragraph detection failures: The script automatically falls back to sentence-based processing
3. Memory usage: Process cities individually for large news datasets
4. Few-shot examples not found: Check that processed articles contain the expected examples

Performance Tips

• Use --n_process parameter for parallel processing
• Process cities individually for large datasets
• Monitor disk space for large CSV files
• Use filtered datasets for analysis to reduce processing time
• Use sampling for analysis to work with manageable file sizes

Contributing

1. Follow the existing code structure
2. Add appropriate error handling
3. Update documentation for new features
4. Test with sample data before processing full datasets

License

This project is for research purposes. Please ensure compliance with data source terms of service and privacy requirements.