NLP.Rmd

---
title: "NLP"
author: "Nicole Schlosberg"
---

## Libraries
```{r}
#Make sure you install and load the following libraries

library(tm)
library(SnowballC)
library(wordcloud)
library(ggplot2)
library(tidyverse) #You will need the full tidyverse package not tidyr and dyplr separately
library(topicmodels)

#IF USING A MAC PLEASE RUN THIS CODE
Sys.setlocale("LC_ALL", "C")
```

## In the class-notes folder you will find real csv files exported from real student's note taking in this class. Import all document files and the list of weeks file
```{r}
#Create a list of all the files, then loop over file list importing them and binding them together
D1 <- list.files(path = "/Users/nicoleschlosberg/Documents/workspace/Git/natural-language-processing/class-notes/",
               pattern = "*.csv", 
               full.names = T) %>% 
    map_df(~read_csv(., col_types = cols(.default = "c"))) 
```

## Step 1 - Clean
```{r}
#Separate out the variables of interest
D1 <- select(D1, Title, Notes)

#Remove the htlm tags from your text
D1$Notes <- gsub("<.*?>", "", D1$Notes)
D1$Notes <- gsub("nbsp", "" , D1$Notes)
D1$Notes <- gsub("nbspnbspnbsp", "" , D1$Notes)
D1$Notes <- gsub("<U+00A0><U+00A0><U+00A0>", "" , D1$Notes)

#Merge the weeks data with your notes data so that each line has a week attributed to it 
W1<- read.csv("week-list.csv",header=TRUE)
D1<- left_join(D1,W1,by=c("Title"="Title"))

#Also remove readings not belonging to the class (IE - that are NA for week)
D1 <- na.omit(D1)
```

## Step 2 - Process text using the tm package
```{r}
#Convert the data frame to the corpus format that the tm package uses
corpus <- VCorpus(VectorSource(D1$Notes))
#Remove spaces
corpus <- tm_map(corpus, stripWhitespace)
#Convert to lower case
corpus <- tm_map(corpus, tolower)
#Remove pre-defined stop words ('the', 'a', etc)
corpus <- tm_map(corpus, removeWords, stopwords('english'))
#Convert words to stems ("education" = "edu") for analysis, for more info see  http://tartarus.org/~martin/PorterStemmer/
corpus <- tm_map(corpus, stemDocument)
#Remove numbers
corpus <- tm_map(corpus, removeNumbers)
#remove punctuation
corpus <- tm_map(corpus, removePunctuation)
#Convert to plain text for mapping by wordcloud package
corpus <- tm_map(corpus, PlainTextDocument, lazy = TRUE)

#Convert corpus to a term document matrix - so each word can be analyzed individually
tdm.corpus <- TermDocumentMatrix(corpus)

#Note: we won't remove plural words here, plural words in English tend to be highly irregular and difficult to extract reliably
```

What processing steps have you conducted here? Why is this important? Are there any other steps you should take to process your text before analyzing?

ANSWER: Here I have done the preprocessing and mining of the data, thus creating the proxy of a proxy. I tokenized the text thus cut it into useful chunks, cleaned the chunks so to remove stuff not needed (e.g., punctuations), and removed stop words (e.g., "the","and"). To analyze, I will need to classify the data into positive, negative, and neutral parts before training the model.

## Step 3 - Find common words
```{r}
#The tm package can do some simple analysis, like find the most common words
findFreqTerms(tdm.corpus, lowfreq=500, highfreq=Inf)
#We can also create a vector of the word frequencies that can be useful to see common and uncommon words
word.count <- sort(rowSums(as.matrix(tdm.corpus)), decreasing=TRUE)
word.count <- data.frame(word.count)
#Look at the word.count dataframe
```

## Generate a Word Cloud

### ColorBrewer
ColorBrewer is a useful tool to help you choose colors for visualizations that was originally built for cartographers. On the ColorBrewer website (http://colorbrewer2.org/#) you can test different color schemes or see what their preset color schemes look like. This is very useful, especially if you are making images for colorblind individuals. 
```{r}
#Define the colors the cloud will use
col=brewer.pal(6,"Dark2")
#Generate cloud, make sure your window is large enough to see it
wordcloud(corpus, min.freq=500, scale=c(5,2),rot.per = 0.25,
          random.color=T, max.word=45, random.order=F,colors=col)
```

# Sentiment Analysis

### Match words in corpus to lexicons of positive & negative words
```{r}
#Upload positive and negative word lexicons
positive <- readLines("positive-words.txt")
negative <- readLines("negative-words.txt")

#Search for matches between each word and the two lexicons
D1$positive <- tm_term_score(tdm.corpus, positive)
D1$negative <- tm_term_score(tdm.corpus, negative)

#Generate an overall pos-neg score for each line
D1$score <- D1$positive - D1$negative
```

## Using ggplot Generate a visualization of the mean sentiment score over weeks, remove rows that have readings from other classes (NA for weeks). You will need to summarize your data to achieve this.
```{r}
D2 <- D1 %>% group_by(week) %>% summarise(score = mean(score))
ggplot(D2, aes(x=week, y=score)) + geom_point() + xlab("Weeks") + ylab("Score") + ggtitle("Mean Sentiment Score Over Weeks")
```

# LDA Topic Modelling

Using the same csv file you have generated the LDA analysis will treat each row of the data frame as a document. Does this make sense for generating topics?

ANSWER: In the case of this document, yes because each row represents a week and each week is a different subject or simply put a different topic.

```{r}
#Term Frequency Inverse Document Frequency
dtm.tfi <- DocumentTermMatrix(corpus, control = list(weighting = weightTf))

#Remove very uncommon terms (term freq inverse document freq < 0.1)
dtm.tfi <- dtm.tfi[,dtm.tfi$v >= 0.1]

#Remove non-zero entries
rowTotals <- apply(dtm.tfi , 1, sum) #Find the sum of words in each Document
dtm.tfi2   <- dtm.tfi[rowTotals> 0, ] #Divide by sum across rows

#Identify rows with zero entries
which(rowTotals %in% c(0))

#Remove these rows from original dataset
D1 <- D1[-c(980,988),]

#Generate LDA model, k is the number of topics and the seed is a random number to start the process
lda.model = LDA(dtm.tfi[-c(980,988),], k = 5, seed = 150)

#Which terms are most common in each topic
terms(lda.model, k = 10) 

#Identify which documents belong to which topics based on the notes taken by the student
D1$topic <- topics(lda.model)
```

What does an LDA topic represent? 

ANSWER: LDA topics are made up of words, thus it represents the words or common words within documents. It tries to figure out the distribution of words between the documents by looking at what similarities are there. Since we set the number of topics to 5 that is why we have 5 topics and each represents a percentage of words similar in the documents. (Topic 1 having a higher percentage than Topic 5). Each topic numbering is just a unique idea with a meaningful ID to be useful in comparing them. The stronger a topic is represented in a certain document, the higher that percent distribution is. 

# Final Task 

Find a set of documents, perhaps essays you have written or articles you have available and complete an LDA analysis of those documents.

```{r}
#Import data (each document contains the notes and key phases I collected from articles read for a cognitive development course)
a1<-read.csv("CoreKnowledge.csv",header=TRUE)
a2<-read.csv("SpontaneousGesturesInfluenceStrategyChoicesProblemSolving.csv",header=TRUE)
a3<-read.csv("UsingSymbols.csv",header=TRUE)

#Clean data and combine documents
a1<-add_column(a1,"Doc1")
a1<-a1[-c(10:17),-c(2,3,4,5)]
names(a1)<-c("Notes","Title")
a2<-add_column(a2,"Doc2")
names(a2)<-c("Notes","Title")
a3<-add_column(a3,"Doc3")
names(a3)<-c("Notes","Title")
A1 <- bind_rows(a1,a2,a3)

#Also remove any na
A1 <- na.omit(A1)

#Convert the data frame to the corpus format that the tm package uses
corpus <- VCorpus(VectorSource(A1$Notes))
#Remove spaces
corpus <- tm_map(corpus, stripWhitespace)
#Convert to lower case
corpus <- tm_map(corpus, tolower)
#Remove pre-defined stop words ('the', 'a', etc)
corpus <- tm_map(corpus, removeWords, stopwords('english'))
#Convert words to stems ("education" = "edu") for analysis, for more info see  http://tartarus.org/~martin/PorterStemmer/
corpus <- tm_map(corpus, stemDocument)
#Remove numbers
corpus <- tm_map(corpus, removeNumbers)
#remove punctuation
corpus <- tm_map(corpus, removePunctuation)
#Convert to plain text for mapping by wordcloud package
corpus <- tm_map(corpus, PlainTextDocument, lazy = TRUE)

#Convert corpus to a term document matrix - so each word can be analyzed individually
tdm.corpus <- TermDocumentMatrix(corpus)

#The tm package can do some simple analysis, like find the most common words
findFreqTerms(tdm.corpus, lowfreq=500, highfreq=Inf)
#We can also create a vector of the word frequencies that can be useful to see common and uncommon words
word.count <- sort(rowSums(as.matrix(tdm.corpus)), decreasing=TRUE)
word.count <- data.frame(word.count)

#Sentiment Analysis
#Upload positive and negative word lexicons
positive <- readLines("positive-words.txt")
negative <- readLines("negative-words.txt")

#Search for matches between each word and the two lexicons
A1$positive <- tm_term_score(tdm.corpus, positive)
A1$negative <- tm_term_score(tdm.corpus, negative)

#Generate an overall pos-neg score for each line
A1$score <- A1$positive - A1$negative

#Use ggplot to generate a visualization of the mean sentiment score over the document
A2 <- A1 %>% group_by(Title) %>% summarise(score = mean(score))
ggplot(A2, aes(x=Title, y=score)) + geom_point() + xlab("Title") + ylab("Score")

#LDA Topic Modeling
#Term Frequency Inverse Document Frequency
dtm.tfi <- DocumentTermMatrix(corpus, control = list(weighting = weightTf))

#Remove very uncommon terms (term freq inverse document freq < 0.1)
dtm.tfi <- dtm.tfi[,dtm.tfi$v >= 0.1]

#Remove non-zero entries
rowTotals <- apply(dtm.tfi, 1, sum) #Find the sum of words in each Document
dtm.tfi2   <- dtm.tfi[rowTotals> 0, ] #Divide by sum across rows

#Identify rows with zero entries
which(rowTotals %in% c(0)) #none in this case

#Generate LDA model, k is the number of topics and the seed is a random number to start the process
lda.model = LDA(dtm.tfi, k = 3, seed = 150)

#Which terms are most common in each topic
terms(lda.model, k = 10) 

#Identify which documents belong to which topics based on the notes taken by the student
A1$topic <- topics(lda.model)
```

Does the method group documents as you would expect?

ANSWER: Yes it does a reasonable grouping of the documents. I know that document 1 and document 3 were more similar while reading than document 2 was, and the LDA topics showed something similar. Topics 1 and 3 were mostly of document 1 and document 3, and Topic 2 was mostly document 1 and document 2. When I describe it as mostly, I am refering to lda.modal and which topic was "assigned" to each document.