Whats-For-Dinner-Data-Science-Project/datapreprocess.html at main · LegoMasterBuilder/Whats-For-Dinner-Data-Science-Project · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
<!DOCTYPE HTML>
<!--
	Miniport by HTML5 UP
	html5up.net | @ajlkn
	Free for personal and commercial use under the CCA 3.0 license (html5up.net/license)
-->
<html>
	<head>
		<title>PH Twitter Fake News Analysis</title>
		<meta charset="utf-8" />
		<meta name="viewport" content="width=device-width, initial-scale=1, user-scalable=no" />
		<link rel="stylesheet" href="assets/css/main.css" />
		<style>
			pre code {
				background-color: #eee;
				border: 1px solid #000;
				display: block;
				padding: 1px;
				text-align: left;
			}
		</style>
	</head>
	<body class="is-preload">
		<!-- Nav -->
			<nav id="nav">
				<ul class="container">
					<li><a href="index.html">Home</a></li>
					<li><a href="about.html">About</a></li>
					<li><a href="datapreprocess.html">Data</a></li>
					<li><a href="methods.html">Methods</a></li>
					<li><a href="visualization.html">Visualization</a></li>
					<li><a href="results.html">Results</a></li>
					<li><a href="conclusions.html">Conclusions</a></li>
				</ul>
			</nav>

		<!-- Home -->

		<!-- Results -->
		<article id="results" class="wrapper style3">
			<div class="container">
				<header>
					<h2>Data Preprocessing</h2>
					<p>Find out how the raw data was prepared and transformed<br/>before it can be used for analysis or machine learning tasks.</p>
				</header>


				<h3>Handling Missing Values/Ensuring No Missing Values</h3>
				<p>To ensure there are no missing values in the dataset, empty cells in the dataframe was replaced with a "NULL" string as such</p>
				<pre>
					<code>
	import pandas as pd

	data = pd.read_csv('Marcos Food Situation.csv')
	data = data.replace(r'^\s*$', "NULL", regex=True)
					</code>
				</pre>

				<h3>Handling Outliers and Ensuring Formatting Consistency (date, labels, etc.)</h3>
				<p>To only keep tweets that were posted from 2016 to 2022, the following code was used</p>
				<p>Outliers, specifically those that were tweeted beyond the years of 2016 and 2022.</p>
				<pre>
					<code>
	from datetime import datetime

	# Convert dates into Datetime Object
	newDates = []
	for index, row in data.iterrows():
		newDate = datetime.strptime(data['Date posted'][index], '%d/%m/%Y %H:%M')
		newDates.append(newDate)

	# Update 'Date posted' column with Dates in Datetime Object Format
	data['Date posted'] = newDates
	# Take only tweets that were posted between 2016 and 2022 inclusive
	mask = (data['Date posted'] >= '2016-01-01 00:00') & (data['Date posted'] <= '2022-12-31 00:00')
	data = data.loc[mask]
					</code>
				</pre>

				<p>By making it a datetime object, the outliers were easily identified and removed. The dates on the data set was properly arranged in a such a way that it would look like the following: YYYY-DD-MM HH:MM:SS</p>
				<pre>
					<code>
	import numpy as np
	import pandas as pd

	mask = (data['Date posted'] > '01/06/2018 00:00') & (data['Date posted'] <= '25/06/2020 00:00')
	datanew = data.loc[mask]
	datanew.shape
					</code>
				</pre>

				<h3>Normalization/standardization/scaling</h3>
				<p>Creating and adding standardized columns of the 'Following' and 'Followers' columns in the dataset</p>
				<pre>
					<code>
	# Create column called 'Following (Standardized)' which is the standardized version of the 'Following' column
	data['Following (Standardized)'] = (data['Following'] - data['Following'].mean())
	data['Following'].std()

	# Create column called 'Followers (Standardized)' which is the standardized version of the 'Followers' column
	data['Followers (Standardized)'] = (data['Followers'] - data['Followers'].mean())
	data['Followers'].std()

	# Create column called 'Likes (Standardized)' which is the standardized version of the 'Likes' column
	data['Likes (Standardized)'] = (data['Likes'] - data['Likes'].mean())
	data['Likes'].std()
					</code>
				</pre>

				<h3>Categorical data encoding</h3>
				<p>In order to process correctly categorical data, it must undergo categorical data encoding. Namely, the account types will be assigned with 1 for identified accounts (or accounts that have real names, bio and/or verified), 2 for anonymous accounts (or accounts that have pseudonyms, aliases, or untraceable names/bio) and 3 for media accounts (or those used by news outlets/personalities)</p>
				<p>Moreover, tweet types are categorical encoded as follows: Text is assigned 1, Images are assigned 2, Tweets with videos are assigned 3, tweets with URLs are assigned 4 and replies are assigned 5</p>
				<p>Lastly, the content types are assigned as follows: Rational content is 1, Emotional content is 2 and Transactional content is 3. </p>
				<pre>
					<code>
	# Categorizing Account Type
	def categorizeAccount(word):
		if word == 'Identified':
			return 1
		elif word == 'Anonymous':
			return 2
		return 3

	# Categorizing Tweet Type
	def categorizeTweet(word):
		if word == 'Text':
			return 1
		elif word == 'Image':
			return 2
		elif word == 'Video':
			return 3
		elif word == 'URL':
			return 4
		return 5

	# Categorizing Content Type
	def categorizeContent(word):
		if word == 'Rational':
			return 1
		elif word == 'Emotional':
			return 2
		return 3
					</code>
				</pre>
			</div>
		</article>

		<!-- Data -->
		<article id="data" class="wrapper style3">
			<div class="container">
				<header>
					<h2>Natural Language Processing (NLP)</h2>
					<p>To further make sense of the data, the preprocessed data</br>
						uses NLP technique to understand the human language better.</p>
				</header>
				<h3>Tokenization and lower casing</h3>
				<p>In order to transform our unstructured text data into a form fit for data analysis, there are several steps of preprocessing that need to be done. We begin with importing the data.</p>
				<pre>
					<code>
	import pandas as pd
	data = pd.read_csv('g23dataset.csv')
	data = data.iloc[:150]
	data.head()
					</code>
				</pre>
				<pre>
					<code>
	data.shape
					</code>
				</pre>

				<h3>Stop words removal/Stemming and lemmatization</h3>
				<p>It is important to tokenize the text to make sure NLP libraries can process the raw text. The Natural Language Toolkit (NLTK) python module will be used to tokenize the text. The text is transformed to lowercase to make sure the same word will not be treated as separate tokens during parsing. Punctuation and emoji will also be removed since they are not relevant in determining the main keywords used in the tweets. The stopwords will be removed using NLTK as a basis before tokenizing the text. To make sure that words with similar meanings are not treated as distinct tokens, it is also important to lemmatize the tokens. Lemmatization simplifies the word to its base meaning, known as its lemma, in order to have better results in data analysis. This allows similar words such as "better" and "good" to be processed as similar tokens. All of these preprocessing methods will be applied to the tweets in the dataframe.</p>
				<pre>
					<code>
	# Import NLTK and make sure the relevant libraries are downloaded
	import nltk
	nltk.download('punkt')
	nltk.download('stopwords')
	nltk.download('wordnet')
					</code>
				</pre>

				<pre>
					<code>
	# Extract the tweets
	tweets = data['Tweet Translated'].dropna()

	# Convert all tweets to lowercase
	tweets = tweets.str.lower()

	# Remove punctuation from tweets, remove stopwords then tokenize
	from nltk.corpus import stopwords
	from nltk.stem import WordNetLemmatizer
	lemmatizer = WordNetLemmatizer()

	processed_text = []

	for text in tweets:
		# Punctuation Removal
		tok = ''.join([c for c in text if c.isalnum() or c == ' '])

		# Tokenization
		tok_list = nltk.word_tokenize(tok)

		# Stopword Removal
		tok_list = [word for word in tok_list if word not in stopwords.words('english')]

		# Lemmatization
		tok_list = [lemmatizer.lemmatize(word) for word in tok_list]

		processed_text.append(tok_list)

	tweets = pd.DataFrame([processed_text])

	tweets = tweets.transpose()
	tweets.head()
					</code>
				</pre>
			</div>
		</article>

		<!-- Team -->
		<article id="team" class="wrapper style4">
			<footer>
				<ul id="copyright">
					<li>&copy; Nutribuns. All rights reserved.</li><li>Design: <a href="http://html5up.net">HTML5 UP</a></li>
				</ul>
			</footer>
		</article>

		<!-- Scripts -->
			<script src="assets/js/jquery.min.js"></script>
			<script src="assets/js/jquery.scrolly.min.js"></script>
			<script src="assets/js/browser.min.js"></script>
			<script src="assets/js/breakpoints.min.js"></script>
			<script src="assets/js/util.js"></script>
			<script src="assets/js/main.js"></script>

	</body>
</html>