N2: Comparative Drift in Brazilian Political Discourse

This repository contains the experiment pipeline and active STIL manuscript for an exploratory comparison of lexical change signals in Brazilian Portuguese political discourse. The study compares a lexical-statistical TF-IDF baseline, aligned slice-specific Word2Vec models, and contextual BERT on one shared panel.

The paper does not claim external semantic ground truth. Its main contribution is the comparison of method agreement, disagreement, interpretation, and cost.

Start Here

• Documentation guide: docs/README.md
• Current paper status: docs/paper/paper_status.md
• Project overview: docs/project/project_overview.md
• Frozen numerical snapshot: docs/results/frozen_results_snapshot_2026_03_24.md
• Active manuscript: 2026S1_STIL_conceptDrift-article/main.tex

Repository Layout

• 2026S1_STIL_conceptDrift-article/ contains the LaTeX manuscript, bibliography, and paper figures.
• src/stil_semantic_change/ contains the reusable pipeline implementation.
• run/ contains Hydra configuration, pipeline entry points, and local outputs.
• tests/ contains unit and integration tests.
• docs/ contains active project, paper, experiment, pipeline, research, and result documentation.
• RawDatasets/ contains local datasets and is intentionally ignored by Git.
• post-hoc/ contains external-probe and Google Trends analyses.

Frozen Experiment

The paper-facing source of truth is:

run/outputs/experiments/brpolicorpus_floor_yearly/ba65fe5b9cce

This run contains the prepared corpus, Word2Vec and TF-IDF scores, the 55-lemma comparison panel, contextual BERT scores, and cross-method agreement outputs. Do not use run 8e15dc2372c5 as an immutable prepared-artifact source because an aborted forced rerun touched its prepared directory.

The frozen shared panel contains 15 Word2Vec-led candidates, 15 TF-IDF-led candidates, 20 stable controls, and 5 theory seeds. The exact results are recorded in docs/results/frozen_results_snapshot_2026_03_24.md.

Setup and Verification

Install the project and development dependencies:

uv sync --group dev

Run the test suite:

uv run pytest

Run the toy end-to-end pipeline:

uv run python run/pipeline/main.py dataset=toy_brpolicorpus_yearly task=run_yearly_core \
  model.vector_size=40 model.window=3 model.negative=3 model.min_count=1 \
  model.epochs=20 model.replicates=1 alignment.min_anchor_words=2 \
  selection.min_occurrences_per_slice=2 selection.min_documents_per_slice=1 \
  selection.min_slice_presence_ratio=0.66

Build the paper:

cd 2026S1_STIL_conceptDrift-article
latexmk -pdf -interaction=nonstopmode main.tex

The generated main.pdf and auxiliary LaTeX files are local build artifacts.

Documentation Policy

Use docs/README.md as the documentation index. Active documents live in the topic folders under docs/; completed plans and review records live under docs/archive/. One-off chat prompts and superseded status files should not be kept as active documentation.