MO KD History

KD History

Pattern

- 2020-05 **[Name]** paper_full_name [[Paper](paperlink)] [[Code](codelink)] [[Note](notelink)] #tags

2006-08 [Model Compression] Model Compression [Paper] [Code] [Note] #response-based
2014-12 [FitNets] FitNets: Hints for Thin Deep Nets [Paper] [Code] [Note] #feature-based
2015-03 [KD] Distilling the Knowledge in a Neural Network [Paper] [Code] [Note] #response-based
2016-06 [Seq-KD] Sequence-Level Knowledge Distillation [Paper] [Code] [Note] #response-based #seq2seq
2016-12 [AT] Paying More Attention to Attention (Attention Transfer) [Paper] [Code] [Note] #feature-based
2017-06 [DML] Deep Mutual Learning [Paper] [Code] [Note] #online
2018-05 [BAN] Born-Again Neural Networks [Paper] [Code] [Note] #self-distillation
2019-04 [RKD] Relational Knowledge Distillation [Paper] [Code] [Note] #relation-based
2019-09 [TinyBERT] TinyBERT: Distilling BERT for Natural Language Understanding [Paper] [Code] [Note] #response-based #nlp
2019-10 [DistilBERT] DistilBERT, a distilled version of BERT [Paper] [Code] [Note] #response-based #nlp
2020-06 [Survey] Knowledge Distillation: A Survey [Paper] [Code] [Note] #survey
2023-06 [GKD] On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes [Paper] [Code] [Note] #online #llm
2025-01 [DeepSeek-R1] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL [Paper] [Code] [Note] #response-based #llm
2025-02 [Scaling Laws] Distillation Scaling Laws [Paper] [Code] [Note] #theory #llm

2021-2023 这段时间以对比学习/图结构等增量式变体为主，没有找到一篇公认的里程碑式工作，这里不强行补一条——等后续有更明确的共识节点再补充。

按"学什么"（对应知识蒸馏 - Trend 的分类）给上面的条目归个类，方便按类型查找：

分类	含义	代表条目
#response-based	匹配 Teacher 最终输出（软化 logits）	Model Compression、KD、Seq-KD、TinyBERT、DistilBERT、DeepSeek-R1
#feature-based	匹配中间层激活/特征图	FitNets、AT
#relation-based	匹配样本/层之间的相对关系	RKD
#online	Teacher/Student 同时训练、互相指导	DML、GKD
#self-distillation	模型自己兼任 Teacher 和 Student	BAN
#theory / #survey	理论分析或综述，不提出新方法	Survey、Scaling Laws