#NLPrimaries

An analysis of tweets mentioning the 2020 Democratic presidential candidates through the use of natural language processing

Project Partner: Julia Chong

Goal

Analyze tweets mentioning (not written by) the 2020 Democratic presidential candidates, in efforts to see how each candidate is viewed and key topics that are being discussed

File Directory

• twint_scraper.ipynb - code to use Twint to scrape tweets from Twitter
• sentiment_analysis.py - sentiment analysis code
• text_analysis.py - tokenization, lemmetization, TF–IDF, and subjectivity analysis code
• tweet_LDA.ipynb - code for LDA of tweets

Data Cleaning/Preprocessing

Thousands of tweets for each candidate

Twint - An advanced Twitter scraping tool

• Make sure nest-asyncio is installed, imported, and applied

1. Each tweet contains only one candidate's name and/or Twitter handle
2. Tweets are from verified accounts only to contain number of tweets
3. Removed links and images using regular expressions, stopwords, and non-English from tweets using langdetect

Sentiment Analysis

VADER (Valence Aware Dictionary and sEntiment Reasoner) Sentiment Analysis - a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media

• Follow documentation to install, import, and implement
• Get positive, negative, neutral, and compound polarity scores for tweets about each candidate

  

• Obtain average polarity score in each category for each candidate
• Mark each tweet as either positive or negative depending on which score is higher
• Create dictionary with average number of positive and negative tweets for each candidate to use for visualization

  

Subjectivity Analysis

TextBlob - provides access to common text-processing operations

Subjectivity is a float within the range [0.0, 1.0] where 0.0 is very objective and 1.0 is very subjective

• Follow documentation to install, import, and implement
• Obtain sentiment for each tweet
• Use regular expressions to capture float values and then convert these strings to floats

  

TF–IDF — Term Frequency Inverse Document Frequency

Use Scikit-Learn's Feature Extraction method

• Tokenize and Lemmatize all of the tweets first using NLTK functions, and make all of the tokens lowercase
• Remove NLTK's set of stopwords and add our own stopwords to remove
• Convert a collection of text documents to a matrix of token counts
• Use TfidfVectorizer(), TfidfTransformer(), CountVectorizer(), fit_transform(), and fit() functions to get TF–IDF vector for each document (candidate's tweets)

  

Word Clouds

Leverage NLTK FreqDist() function

• Displays the most commonly used words in tweets about each candidate, where the bigger the word appears in the cloud, the more commonly it is used:

  

  

  

Topic Modeling with LDA (Latent Dirichlet Allocation)

Used pyLDAvis - Python library for interactive topic model visualization

• Create one list of all of the tweets across candidates and eliminate given and implemented stopwords
• Run CountVectorizer() and TfidfVectorizer() functions on the tweets, followed by the LatentDirichletAllocation() function
• Create topic models using either TF or TF–IDF

LDA separates the tokens in the tweets based on underlying, unobserved similar topics to help users interpret them. Check out the demo below:

Demo

---

Check out my blog on Medium about this project!