Goal
Support knowledge transfer from a teacher model.
Design decisions to be determined experimentally
1. Logit sampling
Sample fixed number of logits from the teacher according to estimated probability distribution. The number should be implemented as a hyperparameter.
2. Scaling the distillation loss
We need a hyperparameter to scale distillation loss in relation to main language modeling loss.
3. Practical considerations
- On-the-fly calculation requires allocating GPU memory for the presumably very large teacher
- Pre-computing logits -> incredibly large file sizes (e.g., 1 teratoken -> couple of terabytes)
- Some middle ground? Asynchronous teacher pre-computing logits on multiple batches, plus clean-up after they have been used?
4. Off-policy vs. on-policy distillation
Off-policy is the classic distillation, where a teacher signal guides student learning on data.
In on-policy distillation, teacher judges the student's own generations. See generalized knowledge distillation
OdenseNLP will run experiments to determine best practices, and if both are needed and in what combination
5. Vocabulary mismatch
Integrate methods that support the case when the teacher has a different vocabulary then the student.
There is some prior work on this: 1 2
OdenseNLP will run experiments to see if their findings can be replicated, and if it is still useful.
6. Future work
- On-policy KD could serve as a data-free post-training method (tbd)
Benefits
- Allows us to use a powerful teacher model
- Training integration should be straightforward, at least for the off policy case
Definitions of Done
- Level 1: Teacher model with matching vocabulary can be used for off-policy distillation
- Level 2: Generalized knowledge distillation (incl on-policy) with arbitrary teacher that has a matching vocabulary.
- [Maybe] Level 3: For both off and on-policy distillation, we can specify an arbitrary teacher, even with mismatching vocabulary.
Why merge this
- Immediate value Stabilize training with a powerful teacher and improve final student model performance, compared to no-teacher-signal
- Low risk Teacher signal will be implemented as optional
- Maintains philosophy: Simple, explicit, debuggable - no hidden complexity
Goal
Support knowledge transfer from a teacher model.
Design decisions to be determined experimentally
1. Logit sampling
Sample fixed number of logits from the teacher according to estimated probability distribution. The number should be implemented as a hyperparameter.
2. Scaling the distillation loss
We need a hyperparameter to scale distillation loss in relation to main language modeling loss.
3. Practical considerations
4. Off-policy vs. on-policy distillation
Off-policy is the classic distillation, where a teacher signal guides student learning on data.
In on-policy distillation, teacher judges the student's own generations. See generalized knowledge distillation
OdenseNLP will run experiments to determine best practices, and if both are needed and in what combination
5. Vocabulary mismatch
Integrate methods that support the case when the teacher has a different vocabulary then the student.
There is some prior work on this: 1 2
OdenseNLP will run experiments to see if their findings can be replicated, and if it is still useful.
6. Future work
Benefits
Definitions of Done
Why merge this