The repository contains the settings, inter-annotator agreement evaluation and analysis of two manual sentiment annotation campaigns, intended to enrich the Slovenian ParlaMint (ParlaMint-SI) corpus, part of the ParlaMint corpus family.
In addition, the repository includes experiments to aggregate sentence-level sentiment to utterance via lanugage-agnostic methods, to extend annotation to other ParlaMint corpora.
The sentence annotation campaign was conducted in the scope of ParlaSent 1.0 dataset production. The resulting sentence-level annotations for Slovenian are available in the the multilingual sentiment dataset of parliamentary debates ParlaSent 1.0.
The utterance-level annotations for Slovenian parliamentary debates are available in the Speech-level sentiment annotation dataset of Slovenian parliamentary debates ParlaSent-SI 1.0.
Both sentence-level and utterance-level annotation followed similar campaign settings. In both campaigns, the annotations were produced by the same two annotators according to the 6-class schema originally proposed by Batanović et al. (2020):
- Positive for sentences/utterances that are predominantly positive
- Negative for sentences/utterances that are predominantly negative
- M_Positive for sentences/utterances that convey an ambiguous sentiment or a mixture of sentiments, but lean more towards the positive sentiment
- M_Negative for sentences/utterances that convey an ambiguous sentiment or a mixture of sentiments, but lean more towards the negative sentiment
- P_Neutral for sentences/utterances that only contain non-sentiment-related statements, but still lean more towards the positive sentiment
- N_Neutral for sentences/utterances that only contain non-sentiment-related statements, but still lean more towards the negative sentiment.
The final annotation for each utterance was determined in a separate reconciliation session, where the annotators reviewed their disagreements and agreed on the final tag. The 3-class labels (Positive, Negative, Neutral) were also provided.
- Sentence-level campaign: 2,600 annotated sentences.
- Utterance-level campaign: 1,000 annotated utterances (i.e., full speeches).
Inter-annotator agreement was evaluated using Krippendorff’s α. For utterance-level sentiment, the dataset includes both procedural (i.e. spoken by the session chair) and non-procedural parliamentary utterances; therefore, agreement is reported separately for the full dataset and for the non-procedural subset.
Sentence-level annotations and their corresponding inter-annotator agreement evaluations for individual phases and the overall dataset are available in the Sentences folder. Utterance-level annotations and agreement evaluations are available in the Utterances folder.
To avoid needing separate training and testing data for each ParlaMint corpus (other than Slovenian), we experimented with heuristic approaches for aggregating sentence-level sentiment predictions to the utterance level. We used the sentence-level sentiment scores provided for the annotation of the ParlaMint-SI corpus, generated by the multilingual parliamentary sentiment regression model XLM-R-ParlaSent (Motchak et al., 2024).
We evaluated the following five rule-based approaches and two machine-learning baselines:
- Mean – baseline aggregation using the arithmetic mean of sentence-level scores.
- Median – baseline aggregation using the median of sentence-level scores.
- Word-based average – weighted mean of sentence-level scores, where weights correspond to sentence length in words.
- Character-based average – weighted mean of sentence-level scores, where weights correspond to sentence length in characters.
- Position-based average – weighted aggregation giving weight to sentences based on their position within the utterance.
- Sentiment intensity-based average – weighted aggregation in which weights reflect the absolute sentiment intensity of individual sentences.
- Linear SVM – supervised regression model trained to predict utterance-level sentiment from sentence-level features.
- Random Forest – ensemble-based supervised regression model for utterance-level sentiment prediction.
Experiments, models and sentiment predictions used for subsequent annotation of the entire ParlaMint-SI corpus are available in the Aggregation folder.