ViFeedback NLP Service

Vietnamese student-feedback classification: sentiment (3 classes) and topic (4 classes) on UIT-VSFC, and a CPU service for the sentiment model.

The benchmark. It compares TF-IDF with fine-tuned PhoBERT under a measurement protocol strict enough to have overturned several of this project's own claims (ADR-015, ADR-018).
The research after it. Pre-registered cycles asked what limits the minority class and what survives real input: typing without diacritics, informal text, and other universities' posts.
The published model. Datk4/vifeedback-sentiment-phobert on the Hugging Face Hub, with a verified release manifest.

At a glance


Served model	PhoBERT-base (12 layers) trained with diacritic/teencode augmentation and consistency anchored on a frozen teacher (ADR-042, ADR-043); served at 198.9 MB: FP16 storage and a vocabulary cut to the 17,500 entries its text uses, ≥ 99% label agreement with the full graph on every held-out set (ADR-045). A 6-layer student (185 MB, 1.8x faster) is released beside it (ADR-039)
Sentiment, test, 5 seeds	Served H10b 0.824 (the recipe without the consistency training 0.830 ± 0.007: −0.006 [−0.019, +0.007], not distinguishable); the student 0.817; research best 0.837 with VnCoreNLP
Robustness	Diacritics restored before the model: stripped validation 0.686 → 0.857 (ADR-031). Out-of-scope input flagged by a topic detector: AUROC 0.922 within another university's forum (ADR-034)
Latency	Raw text in, served pipeline p95 24.9 ms on a laptop CPU (the full FP32 graph 26.9 ms in the same session; the student 12.3 ms; target 30 ms)
Reproducible	A clean clone reproduces the published model's validation macro-F1 (0.8617, H10b) in about 2–3 min, each downloaded serving file checked against `SHA256SUMS`; CI runs it on every change to the workflow
Research	Cycles 0–4 closed, Cycle 5 run; 47 ADRs, 163 weight-updating runs in the ledger; Cycle 5 declared (cycle5.yaml)

Result

Test split, 5 seeds, PhoBERT-base + word segmentation. The test set was untouched until Gate G4, and every evaluation since is logged (28 in all, test_evaluations.log).

Two pipelines are reported separately, because the best research score and the deployed configuration are not the same artifact.

Sentiment, test, 5 seeds	Macro-F1	Weighted F1	Neutral F1
PhoBERT-base + VnCoreNLP (best measured)	0.8373 ± 0.0031	0.9391 ± 0.0020	0.5955 ± 0.0070
PhoBERT-base + pyvi	0.8288 ± 0.0108	0.9369	0.5714
PhoBERT-base + pyvi + diacritic/teencode augmentation (the 12-layer release until 2026-09-30, ADR-027)	0.8296 ± 0.0071	0.9365	0.5757
The same + anchored consistency on real typing (what the service runs, ADR-043)	0.8237 (5 seeds)	0.9345 (seed 42)	0.5504 (seed 42)
6-layer student distilled from the 12-layer model (ADR-040)	0.8168 (5 seeds)	0.9320 (seed 42)	0.5455 (seed 42)
TF-IDF B3 + dev-fitted priors	0.7450	0.8817	0.4207

Topic, test, 5 seeds	Macro-F1	Weighted F1	Others F1
PhoBERT-base + pyvi	0.8038 ± 0.0045	0.8907	0.5533
TF-IDF B4 LinearSVC	0.7423	0.8596	0.4790

No state-of-the-art claim is made. BamiBERT (2026) reports UIT-VSFC sentiment F1 83.41 and topic F1 79.90. Its averaging convention, seed protocol and model selection are not stated in enough detail to assume they match these, so it is context, not a ranking.

What the locked test split did establish is worth more:

On dev the same model scored 0.8670, which reads like a comfortable win.
The dev-to-test drop is −0.0298, and the TF-IDF baseline dropped −0.026 in the same direction. The gap belongs to the split, not the model.

Why macro-F1

UIT-VSFC sentiment is 4.32% neutral. On the real test split, a classifier that never predicts neutral but is otherwise perfect scores accuracy 0.947, weighted F1 0.922, macro-F1 0.649. Only macro-F1 notices that an entire class is missing, and most published UIT-VSFC results report the other two. (Pinned by tests/unit/test_metrics.py.)

What the service does

raw text ─► lowercase, NFC ─► diacritic restorer* ─► pyvi segmentation ─► PhoBERT (ONNX, FP16 storage) ─► label, probabilities
                                                                    └──► TF-IDF topic detector ─► in_scope, scope_score
* only when the input is essentially unaccented (ADR-031)

curl -s localhost:8000/v1/classify -H 'content-type: application/json' \
  -d '{"texts":["giang vien nhiet tinh voi sinh vien", "tuyển dụng thực tập sinh marketing"]}'

Each prediction carries label, confidence, probabilities, in_scope and scope_score.

confidence and probabilities are calibrated: the logits are divided by a temperature fitted on validation (ADR-041), which changes no label.
The service never refuses an input: in_scope: false is information for the caller. The scope flag was validated on NEU-ESC topics; it caught only 2 of 12 AI-authored off-topic probes, so callers should not treat in_scope: true as proof that a comment concerns a course.
The restorer and the scope detector are refused unless their SHA-256 matches the release manifest. /version reports the model file's hash, and make docker-e2e checks it inside the image.
/healthz, /readyz, /version and /metrics cover operations.
Input is bounded at the edge: at most 64 texts of 2,000 characters (HTTP 422), and a request body over 1 MiB is refused before it is parsed (HTTP 413, MAX_BODY_BYTES). A caller's x-request-id is echoed only if it is a short plain id (ADR-047).

The limits are stated in the model card:

The labels follow UIT-VSFC's policy, where a suggestion counts as negative.
The detector judges topic from words.
Other universities' posts score far lower (0.44 macro-F1 on NEU-ESC validation).
Real informal typing changes about 12% of labels between a comment and its normalization (18% before ADR-042).

Research cycles

An external review (REVIEW_AND_RESEARCH_PLAN) set the workflow: observe an error → propose competing explanations → design a controlled intervention → evaluate independently → record the decision and its limits. Each cycle's hypotheses, controls and decision rules are committed before its first run (configs/experiments/).

Cycle	Question	What it settled
0	What does the deployed model get wrong?	Neutral errors are confident (66–72% at p ≥ 0.9), so this is not a threshold problem. Probabilities are overconfident (T ≈ 1.5). No diacritics collapses macro-F1 from 0.863 to 0.268
1	Four declared hypotheses	Neutral is not fixable in the classifier head. Augmentation doubles no-diacritic robustness at no clean cost, confirmed on test. XLM-R's deficit is partly preprocessing. A shared encoder brings no gain
2	Confirmation on a frozen, SHA-256-pinned challenge set	Topic stacking fails. The service switched to the augmented model. An instruction LLM beats the encoder on neutral, but only on constructed text (H7)
3	The served model on real input (NEU-ESC forum posts, ViLexNorm comments)	The service lowercases and restores diacritics (NEU-ESC unaccented posts 0.270 → 0.374). 17% of labels flip on real typing. A lexicon and careful INT8 fail their rules
4	Student text from other institutions	In-domain data helps only with one head per label policy (+0.111 on NEU-ESC), but it failed two declared conditions, so it was not released. `in_scope` now comes from a topic detector. INT8 passed non-inferiority but not fidelity (91.4% label agreement, batch-dependent), so it was not released
5	Real typing, then a smaller model (the owner's order)	H10 not passed: its consistency training removed flips by calling informal text negative (ADR-038). H10b passed: anchored on a frozen teacher, flips fell from about 18% to 12% with no collapse and test unchanged at five seeds (ADR-042). H11 passed: a 6-layer student, 185 MB and 1.8x faster, 0.013 below on test (ADR-039, ADR-040), released. Calibrated confidence served (ADR-041). The owner chose H10b for the service (ADR-043), served at 198.9 MB (ADR-045). H12′ passed its five-seed, held-out NEU-ESC rule (+0.0474 macro-F1, 95% CI [+0.0385, +0.0564]); its candidate release gate is pending. H12 still needs authorised, independently labelled data from a third institution

Per-cycle tables: STATUS and EXPERIMENT_MATRIX § 5. Technical report: RESEARCH_REPORT. Current completion checklist (Vietnamese): TINH_TRANG_HOAN_THANH.

What was got wrong, and corrected

The decision log records every occasion where measurement or review overturned a plan, including retractions of results already written up as successes.

ADR	What was claimed	What measurement showed
008	The TF-IDF baseline would reach ~0.70 macro-F1	It reached 0.78. Three pre-registered ranges were falsified, all low, and the success criteria were revised against the measured baseline
012	Removing segmentation would be the biggest latency win	Segmentation costs 0.6% of p95 and helps accuracy (+0.0234 macro-F1, 5/5 seeds)
015	Threshold tuning is "the single largest lever", +0.035	Cross-fitted, the gain is zero. Retracted
018	We refuted a published paper's conclusion on segmentation	We did not. Its conclusion is conditional, and our result agrees with it. Caught by external review
020	The export verified its artifact	It checked parity on raw text while the service feeds segmented text. It was replaced by a staged release gate with a manifest
032	The out-of-scope score (AUROC 0.977) detects off-topic input	It detected another institution: within one forum its AUROC was 0.573. Replaced by a topic detector (ADR-034)
036	INT8 passed non-inferiority, so it can ship	It changed 8.6% of labels and depended on the batch. The release gate's fidelity rule held it back
038	Consistency training cut real-typing label flips from 16.6% to 0.3%	By calling 99.4% of informal comments negative. A guard on real student posts caught it
040	The distilled student matches the 12-layer model (validation)	On test it is 0.013 lower in every seed; 73 neutral validation sentences could not resolve it. Stated, not hidden

Quick start

git clone https://github.com/ThanhDatVN/ViFeedback-NLP-Service.git
cd ViFeedback-NLP-Service
python scripts/setup_venv.py   # .venv with the pinned versions; CUDA torch if an NVIDIA GPU is found
.venv\Scripts\Activate.ps1      # Windows  (Linux/macOS: source .venv/bin/activate)
cp .env.example .env           # optional keys (HF_TOKEN, OPENAI_API_KEY); .env is git-ignored
make data             # fetch UIT-VSFC at its pinned revision + run the integrity suite
make test             # about 400 fast tests
make baseline         # TF-IDF ladder
make train            # fine-tune PhoBERT (needs a GPU; about 5 min per seed on an RTX 3050)

make help lists every target, and targets use .venv automatically. Keys live in .env only:

HF_TOKEN is used only for publishing;
OPENAI_API_KEY is used only for the LLM reference measurements (H7), and UIT-VSFC text is never sent to an API.

Reproduce the published model (CPU only, no checkpoint)

pip install -e ".[serve]"
vifeedback data fetch
vifeedback serve reproduce     # Hub download, SHA-256 checks, validation macro-F1 0.8617

reproduce.yml runs this from a clean clone in CI (2.3 min, install included).

Serve

make export           # checkpoint -> ONNX in staging -> verified on full validation -> released,
                      # then the diacritic restorer and the scope detector attached (each hash-checked)
make docker-e2e       # build the image, mount the release, check /readyz, SHA-256s and golden labels
make docker && make docker-run

The runtime image is 519 MB. It carries no scikit-learn, SciPy or transformers: a small PhoBERT BPE tokenizer and pyvi's CRF were verified identical on all 49,141 corpus texts.

Research studies

Every study is one CLI call, writing to results/studies/:

vifeedback study tables              # result tables + paired comparisons from the registry
vifeedback study robustness          # perturbation suites, slices, negation probe
vifeedback study challenge           # Cycle 2 H6 on the frozen challenge set
vifeedback study neu-esc-confirm     # Cycle 3 confirmation on real student posts
vifeedback study b4prime             # Cycle 4 topic-aware scope detector
vifeedback study audit-report        # analyse the filled neutral-audit sheet

The Kaggle notebooks are for models that exceed a 4.29 GB GPU (KAGGLE_GUIDE).

Repository

├── src/vifeedback/
│   ├── cli/           one module per command group; `study` split by research cycle (study_cycle1…5)
│   ├── data/          loading · integrity · leakage · profiling
│   ├── preprocess/    normalizers · segmentation backends · diacritic restorer · variants
│   ├── models/        TF-IDF ladder (B0–B5) · cross-fitted prior tuning
│   ├── training/      fine-tuning loop · losses · augmentation · multi-task · domain (H8)
│   ├── evaluation/    metrics · bootstrap · decisions · calibration · robustness · audit · scope
│   │                  external data · LLM reference · INT8 power
│   ├── inference/     ONNX export · release gate · PhoBERT BPE tokenizer · publish · reproduce
│   └── serving/       FastAPI app · pipeline · scope detector (numpy)
├── configs/           model configs · experiments/ = declared cycles (cycle1–5.yaml) + run ledger
├── data/              challenge set and probes (the project's own); corpora fetched, never committed
├── docs/              see docs/README.md
├── results/           registry, per-run metrics, studies (see results/README.md)
├── notebooks/         EDA and results (executed) · three Kaggle notebooks
├── scripts/           venv setup · Docker end-to-end · tokenizer and segmenter equivalence checks
├── tests/             unit · data · contract · integration · packaging guards
└── models/            local weights, gitignored (see models/README.md)

Documentation map: docs/README.md. The ones to start with:

Document	Contents
STATUS	Progress by gate and cycle, open problems, what runs next
NEXT_PLAN	Cycle 5 and the open targets, with research sources
RESEARCH_REPORT	Technical report: questions, method, findings, negative results, limitations
DECISIONS	47 ADRs: every plan correction forced by measurement or review
EXPERIMENT_MATRIX	Run-ID scheme and every result table
EVALUATION_PROTOCOL · EVALUATION_DATA	What every number is held to; what counts as evidence
DATA_CARD	UIT-VSFC measured from the files, and what is never committed
HUONG_DAN_THU_CONG	(Tiếng Việt) The owner's manual tasks, step by step

Measurement protocol

What every number in this repository is held to (EVALUATION_PROTOCOL):

Five fixed seeds, mean ± std. A single-seed number is never a headline.
Paired comparisons. Seed-paired runs and paired bootstrap over the same examples. A single model's dev macro-F1 carries about ±0.027, because 73 neutral examples carry a third of the macro average. A paired difference resolves far more tightly (corrected after review, R7).
Declared before run. Hypotheses, controls, metrics and decision rules are committed before the first run of each cycle, and the rules are code (evaluation/decisions.py).
Cross-fitting. Anything fitted on the evaluation set is cross-fitted before its benefit is reported (ADR-015; optimism bias measured at +0.012 to +0.036).
Test discipline. The test set is used at gates only, and every use is logged. Later cycles confirm on new data (a frozen challenge set, then NEU-ESC test, each use logged).
Traceability. Every number is traceable to a run_id in results/registry.csv.
Negative results stay. Failed hypotheses are committed with the same detail as passed ones.

Reference machine

All latency figures come from one documented machine; a number from a cloud VM does not enter the registry.


CPU	AMD Ryzen 5 6600H · 6C/12T · AVX2 · no AVX512-VNNI
GPU	RTX 3050 Laptop, 4.29 GB · about 65 s per epoch for PhoBERT-base
Compute to date	196 registry rows; 169 weight-updating attempts (167 completed), at least 22.3 logged GPU-hours (ledger)

Single-sentence latency, model only, two steady passes (vifeedback study latency):

Configuration	p50	p95	vs L0, p50 / p95	Quality
L0 PyTorch FP32, pad to 96	117.4 ms	120.5 ms	1×	reference
L1 PyTorch FP32, dynamic padding	46.7 ms	64.1 ms	2.5× / 1.9×	identical
L3 ONNX FP32 (served)	15.5 ms	33.6 ms	7.6× / 3.6×	identical (parity 8.2e-5)
L4 ONNX INT8 dynamic	9.0 ms	20.4 ms	13.1× / 5.9×	neutral F1 −0.088: not released

Absolute times shift with the laptop's power state, and the speed-up ratios reproduce, so the ratios are the claim. The 33.6 ms session was a slow machine state: later sessions measured the served model at 19–20 ms p95, and the full served pipeline at 26.9 ms (ADR-032). Three sessions on an idle machine will re-measure the pipeline with the scope detector.

Status

Every external-review item (R1–R12) is closed (STATUS § 8).

CV snippet

Built a reproducible Vietnamese feedback classification benchmark on UIT-VSFC; PhoBERT + VnCoreNLP reached sentiment test macro-F1 0.837 ± 0.003 across five seeds against 0.745 for a tuned TF-IDF baseline, with minority-class analysis, preprocessing ablations and per-segmenter latency measured on documented hardware.

The served H10b recipe scores 0.8237 macro-F1 on test across five seeds. Its seed-42 checkpoint is the deployed artifact. Rules for quoting: ROADMAP § 9.

License and citation

Code: MIT, see LICENSE.
Published weights: CC BY-NC-SA 4.0, a non-commercial research artifact, in line with UIT-VSFC's research-purpose release.
Data: no corpus text is included here. UIT-VSFC, NEU-ESC and ViLexNorm are fetched under their own terms (DATA_CARD § 11).

Cite Nguyen et al. (KSE 2018) for the corpus and Nguyen & Nguyen (Findings of EMNLP 2020) for PhoBERT.

Navigation Menu

Repository navigation

About

Repository files navigation

ViFeedback NLP Service

At a glance

Result

Why macro-F1

What the service does

Research cycles

What was got wrong, and corrected

Quick start

Reproduce the published model (CPU only, no checkpoint)

Serve

Research studies

Repository

Measurement protocol

Reference machine

Status

CV snippet

License and citation

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
configs		configs
data		data
docs		docs
models		models
notebooks		notebooks
results		results
scripts		scripts
src/vifeedback		src/vifeedback
tests		tests
.dockerignore		.dockerignore
.env.example		.env.example
.gitattributes		.gitattributes
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
docker-compose.yml		docker-compose.yml
pyproject.toml		pyproject.toml
requirements-lock.txt		requirements-lock.txt

Latest commit

History

Folders and files

Repository files navigation

ViFeedback NLP Service

At a glance

Result

Why macro-F1

What the service does

Research cycles

What was got wrong, and corrected

Quick start

Reproduce the published model (CPU only, no checkpoint)

Serve

Research studies

Repository

Measurement protocol

Reference machine

Status

CV snippet

License and citation

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages