Data & Evaluation

##About the Data

Participants will be presented with a training and a test set. The training set will be composed of 2,237 instances, and the test set of 88,221. The data was collected through a survey, in which 400 annotators were presented with several sentences and asked to select which ones they did not understand the meaning of. The training set is composed by the judgments of 20 distinct annotators over a set of 200 sentences, while the test set is composed by the judgments made over 9,000 sentences by only one annotator. In the training set, a word is considered to be complex if at least one of the 20 annotators judged them so. [src]

Take a look at the SemEval's readme or click here for more information on the data and its formatting convention.

In this project both datasets were used in cross validation experiments. The smaller training data were used to determine the quality of different features and classifiers quickly, before checking these findings on the annotated test data.

---

##Classification Performance
To evaluate the tools's performance, the measures Accuracy, Precision, Recall and F-Measure were calculated on different configurations. In the following, evaluation results will be summarised. The actual data can be accesssed here.

Generally speaking, classification results generated by the experiments are rather poor.

Performance was better on the smaller dataset, which might be in part due to the fact that the ratio of complex words (26,37%) was larger compared to the bigger test dataset (4,68% complex words). In feature extraction, it is advisable to focus on features exclusively (or at least predominantly) present in complex words.

####Experiment Setup
Both experiments used identical parameter spaces in order to keep them comparable, only the classifiers were varied. The following experiment parameters yielded the best results:

ExperimentType: Cross Validation
Labeling Mode: Single Label
Feature Mode: Unit

Mandatory Preprocessing: Tokenisation, Lemmatisation, POSTagging, (lookup based) Frequency Tagging
Features Extracted: Word Length (in characters), POS, Character NGrams (min:2, max:4)

More details on the algorithms can be found in the Code] Section.
Regarding the number of folds, 10 seemed to yield best results while still maintaining a managable runtime.

#####ZeroR (Majority Class Classifier)

As mentionied before, the ZeroR Algorithm has proven to be unuseful in complex word identification. This should come at no surprise, as complex words are less frequently used than non-complex ones. Moreover, no insights can be gained as to which factors influence word difficulty.
The results serve as a good example for the potentially misleading nature of classifiers' Accuracy Rating. Despite its acceptable accuracy, ZeroR is entirely unsuited for complex word identification.

Accuracy: 73,60%
Precision: 0%
Recall: 0%
F-Measure: 0,0

#####Naive Bayes

Metrics are slightly better for the Naive Bayes classifier. Of all the classifiers, its recall value was the highest (65,01%). It was the only classifier, to actually overestimate the number of complex words, making it the best choice, when trying to find all complex words in a difficult text (when the ratio of complex words is relatively high). Amongst these, there will be, however an overwhelming majority of non-complex words, due to its low precision (13% on the large corpus).

Small Corpus(n = 2044)
Accuracy: 54,72%
Precision: 37,47%
Recall: 65,01%
F-Measure: 47,54%

Large Corpus (n = 84090)
Accuracy: 86,22%
Precision: 13,00%
Recall: 34,13%
F-Measure: 0,188

(Runtime on large corpus: ~15 Minutes)

#####J48 (C4.5 Decision Tree)

The J48 Algorithm produced the best overall results (F-Measure: 0,501 on the small Corpus). Its runtime is, however, substantially higher than that of Naive Bayes and the Random Forest . Especially on the Large corpus, this was a problem, as a single could take multiple hours. Its precision on the large corpus was surprisingly high (43,56%), compared to the other metrics.

Small Corpus(n = 2044)
Accuracy: 54,72%
Precision: 37,47%
Recall: 65,01%
F-Measure: 0,500

Large Corpus (n = 84090)
Accuracy: 94,95%
Precision: 43,56%
Recall: 10,16%
F-Measure: 0,165

(Runtime on large corpus: ~180 Minutes)

#####Random Forest

While having the weakest overall performance on the small dataset (F-Measure: 0,424), the Random Forest Decision Tree Algorithm actually has the best F-Measure on the large Dataset ( F-Measure: 0,213 ). Even though the performance on large corpora is still rather poor, the Random Forest algorithm is the most promising candidate in terms of performance/runtime ratio.

Small Corpus(n = 2044)
Accuracy: 68,57%
Precision: 50,29%
Recall: 36,69%
F-Measure: 0,424

Large Corpus (n = 84090)
Accuracy: 94,95%
Precision: 43,56%
Recall: 10,16%
F-Measure: 0,213

(Runtime on large corpus: ~70 Minutes)