Papers

Filtered to distillation · clear filter

Browse by term

continual learning 91reinforcement learning 43vision-language models 18benchmarking 12benchmarks 11robotics 11large language models 10language models 9world models 9video generation 6generative models 5multimodal models 5natural language processing 5recursive self-improvement 5video world models 5agents 4dialogue systems 4diffusion models 4foundation models 4multimodal large language models 4on-policy distillation 4reasoning 4self-distillation 4self-supervised learning 4conversational AI 3evaluation benchmarks 3human-computer interaction 3LLMs 3multi-agent systems 3policy optimization 3

Matching papers

On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics

168 upvotes · 28 SEP 2026 · Julianna Piskorz, Antonin Berthon, Mihaela van der Schaar

This paper compares the effectiveness of on-policy and off-policy learning in a distillation setting, finding that rollout policy is not as crucial as previously thought, and that learning rate and token-level KL direction play more significant roles in task performance and forgetting.

reinforcement learningdistillationcontinual learningKL divergencestrong-to-weak transfer

Self-Supervised Visual On-Policy Distillation

158 upvotes · 14 AUG 2026 · Yijiang Li, Yijun Liang, Yunjie Tian et al.

This paper proposes a new method called Self-Supervised Visual On-Policy Distillation (S^2VOPD) that generates learning signals from asymmetric augmented views of images, allowing for effective on-policy learning without privileged information. Practitioners might care about this paper because it presents a simple yet effective way to improve performance on various perception benchmarks.

visual augmentationon-policy learningself-supervised learningdistillationreinforcement learning