Papers

Filtered to attention mechanisms · clear filter

Browse by term

continual learning 83reinforcement learning 50large language models 12benchmarking 11benchmarks 11language models 11vision-language models 11robotics 7natural language processing 6world models 6generative models 5recursive self-improvement 5attention mechanisms 4diffusion Transformers 4multi-agent systems 4multimodal learning 4multimodal models 4on-policy distillation 4self-distillation 4self-supervised learning 4transformers 4video generation 4vision-language-action models 4agent-based systems 3agentic models 3agentic search 3autonomous systems 3coding agents 3diffusion models 3image generation 3

Matching papers

Metis: Memory Foundation Model

256 upvotes · 29 JUL 2026 · Zeyu Zhang, Ziliang Guo, Yihang Sun et al.

This paper introduces a new type of AI model called memory foundation models, which allows the model to learn and retain information internally, rather than relying on external memory modules. This could be useful for practitioners who want to build more efficient and flexible AI agents.

memory foundation modelsmultimodal modelsnative memorymodel compressionattention mechanismscontinual learning

Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning

166 upvotes · 3 SEP 2026 · Heng Wang, Jielin Qiu, Wenting Zhao et al.

This paper proposes a new method for managing the memory of large language models, called Random Attention, which improves their performance on tasks that require extended reasoning by reducing memory usage. Practitioners caring about efficient model deployment might find this approach useful.

language modelsKV cacheattention mechanismsmemory efficiencyreasoning tasks

Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformers

68 upvotes · 21 JUL 2026 · Maohua Li, Qirui Li, Yanke Zhou et al.

This paper helps us understand how text-to-image diffusion transformers work by analyzing the role of "template tokens" in generating images from text prompts. Practitioners might care because it shows how to improve the efficiency of these models without sacrificing their performance.

diffusion transformerstext-to-image modelscausal interpretabilityattention mechanismssemantic routing

Language Models Can Control Their Own Attention

49 upvotes · 2 SEP 2026 · Namgyu Ho, Huzama Ahmad, Woosung Koh et al.

This paper introduces a new attention mechanism, Declarative Attention (DA), that allows language models to more efficiently control their attention by specifying where they need to focus during generation, rather than scanning the entire context. Practitioners might care because this could lead to significant reductions in computation and memory usage, making large language models more practical for real-world applications.

language modelsattention mechanismssparse attentioncontextual generationmodel efficiency