Firehose

This paper introduces a new method called Weakening-Induced On-Policy Self-Distillation that helps improve reasoning models without needing a strong teacher model. Practitioners might care because it can be used to improve model performance even when a good teacher model is not available.

This paper improves a common technique in language model training called Multi-Token Prediction (MTP) by removing a noisy component that interferes with learning, resulting in faster and more accurate predictions. Practitioners may care because this improvement can lead to better performance in applications such as language translation and text generation.