Experiment code for fusing multiple specialist teachers (math / code / general / creative) into one small student via on-policy distillation (TRL GKD: generalized JSD, on-policy student rollouts), and for diagnosing when the teachers' capabilities conflict.
All scripts assume they live flat in one project directory and are launched
on a SGE/qsub cluster (BU SCC). Each Python entry point has a matching
*.qsub (and often a *_1b.qsub for the 1B student). The qsubs cd into the
project dir, activate a venv, set HF_HOME/HF_DATASETS_CACHE, then call the
.py. Model weights, HF caches, datasets, logs and result outputs are not
in this repo.
Ordered mixing → separation. Each is a standalone GKD trainer variant.
| script | method | how teachers are combined |
|---|---|---|
mopd_route.py |
route (JSD) | hard per-prompt routing to one teacher |
mopd_endorse.py |
endorse | per-token endorsement routing |
mopd_cluster.py |
clmc / clmcg | route to a teacher cluster, mixture within cluster |
mopd_confmix.py |
conf_prob | probability-space mixing by teacher confidence |
mopd_confgap.py |
confgap | probability-space mixing by confidence gap |
mopd_gapsel.py |
gap-select | gap-based teacher selection |
mopd_projconf.py |
projconf | all-teacher gradient PCGrad projection |
mopd_clustproj.py |
cproj / dsum | cluster + within-cluster projection (--incluster proj) or summation (--incluster sum) |
mopd_multiteacher.py |
naive | equal/weighted multi-teacher baseline |
mopd_projected.py |
projected | earlier gradient-projection variant |
opd_single.py |
single | single-teacher specialist (reference) |
trl_opd_starter.py |
starter | minimal TRL GKD starter / sanity |
train_teacher.py(+train_teacher*.qsub) — SFT the per-domain specialist teachers (1B and 7B, SFT / RL / L40S variants).
| script | measures |
|---|---|
eval_students.py (eval_1b.qsub, eval_all.sh) |
student benchmarks |
lmeval.qsub, compile_lmeval.py, compile_0v5.py |
lm-eval-harness (GSM8K, HumanEval, MMLU, …) + result compile |
eval_domainKL.py (domainKL.qsub) |
per-domain teacher-KL fidelity (student vs teacher) |
eval_ppl.py (ppl.qsub) |
perplexity |
gen_creative.py (gencr.qsub) |
held-out WritingPrompts story generation for LLM-judge (local models, no API key) |
student_vs_teachers.py |
student-vs-teacher comparison |
eval_pairs.txt |
eval pair list |
activation_paths.py(+activation_paths*.qsub) — per-layer activation measurements: DirCos (displacement-direction cosine), CKA, and neuron-Jaccard, in raw and across-teacher-centered variants. This is the source of the two-bloc conflict structure ({math,code} vs {general,creative}).diag_pcgrad.py,diag_winwin.py,diag_gapfrac.py,diag_lossmetric.py,diag_compat.py,diag_domclf.py(+ qsubs) — gradient-conflict / win-win / domain-classifier probes.compare_inits.py,probe_*_1b.qsub,sweep_1b.sh,race_check.sh— initialization comparison, weight-sweep probes, sweep driver, race check.
build_flashattn.qsub,run_verl_opd.qsub,opd_sanity*.qsub,compare_cpu.qsub— environment build and sanity jobs.- (The third-party
verl/framework used by some jobs is not vendored here.)
Measured on the MLP intermediate activation (input to each layer's
mlp.down_proj, width d_mlp) on base-relative displacements
Δ_k = a_teacher_k − a_base, pooled over all non-padding probe tokens
(≤ 8,000 per layer, a global aggregate — not per-prompt sets unioned). Each
neuron is scored by mean absolute displacement s_j = mean_t |Δ(t,j)|;
the active set is the top-256 (hard count, not a threshold); Jaccard =
|Nx ∩ Ny| / |Nx ∪ Ny|. The centered variant first subtracts the
across-teacher mean displacement Δ̃_k = Δ_k − (1/K)Σ_c Δ_c, then applies the
identical rule.
report/ renders the comprehensive results report from collected metrics — see
report/README.md.