a challenge to classify factuality of text without external reference- just from the words. Tested on a Kaggle dataset.
The challenge of classifying statements into factual/non-factual/unimportant is that different statements of the same category do not have to strongly sound the same in tone. Consider the example: “If he's serious about what he's saying, then the only place he can go to balance that budget is to raid the Social Security Trust Fund, and he tried that in 1985, and I think he's going to try it again." In the train dataset, this is labelled as factual (1). Yet, it contains opinion-words like ‘think’ that could just as easily be seen as non-factual classifications in other cases.
To keep this challenge simple and interesting: the scope was limited to 3 types of classifiers:
A Naive Bayes classifier (NB), a multi-layer perceptron network (MLP) and a logistic regression (LR) classifier.
(Precision score, out of 1)
baseline NB model: A NB model with just tokenised input and no further processing or engineering gave the score 0.65228
baseline with stemming: 0.62025
baseline with lemmatisation: 0.67012
baseline with Laplace smoothing: 0.64949
baseline MLP: google-news-300 word2vec embeddings. Model was 2-layered, with 128 and 64 neurons and default parameters; score 0.77492
baseline MLP with custom w2v: custom w2v trained on train set. Identical hyperparams. Score: 0.75377
baseline MLP with POS tagging: 0.75720
baseline MLP but grid search to set hyperparams: 0.78279
3 layer MLP: 3 hidden layers of 256,128, and 64 neurone; score: 0.79243
baseline LR with tf-idf features: 0.77603
The best model was the 3-layer MLP, with score 0.79243.
In the NB models, stemming was seen to worsen the score, probably because it is too aggressive.
Distinct words might collapse into the same stem (e.g university, universe -> univers), reducing discriminative ability.
Conversely, lemmatisation is less aggressive, and it improved results. As improved results, it likely means that words with the same root contributed generally to the same classification.
In the MLP, added information from POS tagging surprisingly didn't improve results. Adding POS increased the dimensionality of the vector, which might have contributed to overfitting.
Ultimately, it appears that the MLP, with many neurons and interconnections, is the most adept at capturing complex relationships in the data, enabling it to perform the best.