Team research project (PSC, École Polytechnique, 2022–2023) on the security and interpretability of transformer models: detecting backdoored ("trojaned") models — models that behave normally until a hidden trigger flips their behaviour.
Full write-up (French, 38 pages): Final_report.pdf
A backdoored model is indistinguishable from a clean one on normal inputs: we generated 3,500+ toy GPT-style language models (clean and infected with 4 different trigger types) and measured 99.7% average clean accuracy alongside a 99.2% attack success rate — behaviour alone does not reveal the backdoor. The question: can a detector tell infected models apart just by querying them?
- Model zoo generation (whatIs/) — pipeline to mass-produce clean and trojaned character-level language models (4 trigger types), distributed across machines.
- Meta-network detection — an MNTD-style detector: a meta-network learns to classify a model as clean/infected from its answers to learned queries, adapted here from image classifiers to NLP models.
- Image baselines (MNIST/, ABS/) — backdoor injection and detection experiments on MNIST classifiers, including the ABS approach, plus a review of NeuralCleanse, SPECTRE and MNTD.
- Interpretability experiments (gpt/) — probing what trojaned transformers learn internally.
Detector performance on 350 held-out models:
| Metric | Score |
|---|---|
| Accuracy | 0.747 |
| Precision | 0.770 |
| Recall | 0.688 |
| F1 | 0.727 |
| ROC AUC | 0.746 |
Per-trigger detection precision ranged from 100% ("x first last") to 24% ("a first x last"), and dropped to 20% on a trigger type unseen during training — the method validates the approach but does not generalise to novel attacks, and costs ~3h of compute per meta-network. Both limitations are discussed in the report.
Guilherme Vieira Manhaes, Mohamed Hachem Ouertani, Benjamin Gras, Mateo Fornieles, Lemine Cherif — PSC INF03, École Polytechnique, April 2023.