TechChat admission-chatbot reproducibility package

This repository accompanies the preprint “Reproducible Evaluation of Lightweight Intent Classifiers for a University Admission Enquiry Chatbot in Ghana.”

TechChat was developed in 2024 to answer prospective students' questions about admissions at Kwame Nkrumah University of Science and Technology (KNUST), Ghana. This release focuses on the non-generative research pipeline evaluated in the paper:

question -> tokenization and lemmatization -> binary bag of words
         -> intent classifier -> predefined response retrieval

Gemini or other generative response rewriting is not part of this benchmark.

Main result

All models were evaluated on the same untouched test partition of 3,444 questions across 358 eligible intent classes.

Model	Accuracy	Macro F1	Training time
Multinomial Naive Bayes	80.20%	0.753	0.10 s
Logistic regression	95.27%	0.927	11.47 s
Linear SVM	95.33%	0.930	11.66 s
Original neural architecture	94.86%	0.916	1,212.76 s

Timing values are specific to the Windows CPU environment used for the reported run.

Repository structure

.
├── data/
│   ├── raw/intents/                 # Original anonymized JSON intent files
│   ├── cleaned_intents.json         # Conservative cleaned research dataset
│   ├── quarantined_conflicts.json   # Exact questions assigned to multiple labels
│   ├── dataset_audit.json           # Machine-readable audit summary
│   └── dataset_partitions.csv       # Frozen 70/15/15 benchmark partitions
├── docs/                             # Audit, methods, protocol, and results notes
├── paper/                            # Current manuscript and arXiv metadata
├── results/                          # Reports, predictions, matrices, and training history
├── src/
│   ├── audit_dataset.py             # Rebuild the cleaned data from raw intent files
│   └── run_benchmark.py             # Reproduce the model comparison
├── CITATION.cff
├── requirements.txt
└── README.md

No .env file, API key, database credential, application session, private interaction log, or identifiable participant record is included.

Dataset summary

• Original intent records: 406
• Original unique intent labels: 364
• Original pattern instances: 23,486
• Same-label duplicate instances removed: 215
• Cross-label pattern instances quarantined: 293
• Cleaned patterns: 22,978
• Primary benchmark: 22,957 patterns across 358 eligible intents
• Frozen split: 16,069 train / 3,444 validation / 3,444 test

Six intents with fewer than seven cleaned patterns were excluded from the primary benchmark because they could not reliably populate all three stratified partitions.

Setup

Python 3.12 is recommended for matching the reported environment.

python -m venv .venv

Activate the environment, then install the pinned dependencies:

python -m pip install -r requirements.txt

The first run may download the NLTK punkt, wordnet, and omw-1.4 resources.

Rebuild the audited dataset

The raw files are anonymized and included so that the cleaning procedure can be inspected and rerun.

python src/audit_dataset.py --source data/raw --out data/rebuilt

The script merges records with identical intent tags, removes exact canonical duplicates within a tag, and quarantines a question whenever its canonical text has multiple labels. It does not silently choose a label for ambiguous questions.

Run the classical baselines

python src/run_benchmark.py --download-nltk

Subsequent runs do not need the download flag:

python src/run_benchmark.py

To keep the published result files unchanged while testing a local run, choose another output directory:

python src/run_benchmark.py --results-dir local-results

Run the neural reproduction

python src/run_benchmark.py --neural

The neural run uses the original 100 epochs and batch size of 5. On CPU, the reported run took approximately 20 minutes. The command also reruns the three classical baselines so that all outputs come from one consistent execution.

Reproducibility controls

• Fixed seed: 42
• Exact duplicate control before splitting
• Exact cross-label conflicts quarantined
• Stratified 70/15/15 train/validation/test split
• Vocabulary fitted using training data only
• Same frozen partitions for every classifier
• Accuracy, macro and weighted metrics, top-3 accuracy, timing, per-class reports, and prediction-level outputs retained

See the evaluation protocol and results report for details.

Ethical and data-origin note

During the original project, current KNUST students voluntarily contributed examples of common admission questions and gave consent for their responses to inform the system. The retained question patterns contain no participant identifiers and cannot be traced to individuals. The project team also consulted the KNUST Admissions Office for institutional information and clarification. No applicant records or identifiable interaction logs are included in this release.

Admission information can change. The responses in the dataset reflect the project-development context and must not be treated as current official KNUST policy. Users should consult official KNUST sources for operational admission decisions.

Authors

• Prince Frimpong — pfrimpong11@st.knust.edu.gh
• Akinie Samuel Aterh — saakinie@st.knust.edu.gh
• Andrew Selasi Agbemenu — asagbemenu@knust.edu.gh

Citation

Use the citation metadata in CITATION.cff. The record identifies the authors, artifact title, version, and preferred manuscript citation; publication identifiers can be added in a later repository release.

Licence

This repository uses a dual-licence arrangement:

• Source code under src/: MIT License
• Dataset, documentation, figures, tables, manuscript, and other non-code materials: Creative Commons Attribution 4.0 International

See the linked licence files for the applicable terms and attribution requirements.