Why Self-Distillation Is Taking Over LLM Post-Training (w/ the Researchers Behind It)
Deep Learning with Yacine · 91:19
On-policy self-distillation gets around the sparse-reward problem in RL methods like GRPO. The same model plays both student and teacher: the teacher also sees feedback (an error message, a demonstration, a user's fol...