Papers

Filtered to mixture-of-experts · clear filter

Browse by term

continual learning 83reinforcement learning 50large language models 12benchmarking 11benchmarks 11language models 11vision-language models 11robotics 7natural language processing 6world models 6generative models 5recursive self-improvement 5attention mechanisms 4diffusion Transformers 4multi-agent systems 4multimodal learning 4multimodal models 4on-policy distillation 4self-distillation 4self-supervised learning 4transformers 4video generation 4vision-language-action models 4agent-based systems 3agentic models 3agentic search 3autonomous systems 3coding agents 3diffusion models 3image generation 3

Matching papers

SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

86 upvotes · 1 SEP 2026 · Shaowen Wang, Ge Zhang, Kairong Luo et al.

This paper investigates how to make Looped Transformers more efficient by scaling their depth while keeping the same number of parameters and compute usage, and it finds that this approach can lead to significant performance gains. Practitioners might care about this because it could lead to faster and more accurate models in resource-constrained environments.

Looped TransformersMixture-of-ExpertsTransformersscaling lawsefficient neural networksneural architecture search