Papers

Filtered to attention sparsification · clear filter

Browse by term

continual learning 83reinforcement learning 50large language models 12benchmarking 11benchmarks 11language models 11vision-language models 11robotics 7natural language processing 6world models 6generative models 5recursive self-improvement 5attention mechanisms 4diffusion Transformers 4multi-agent systems 4multimodal learning 4multimodal models 4on-policy distillation 4self-distillation 4self-supervised learning 4transformers 4video generation 4vision-language-action models 4agent-based systems 3agentic models 3agentic search 3autonomous systems 3coding agents 3diffusion models 3image generation 3

Matching papers

SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

50 upvotes · 11 SEP 2026 · Zhiwei Li, Lei Zhu, Hao Gu et al.

This paper proposes a new method to sparsify attention in Transformers, called Simple Attention Sparsification (SAS), which optimizes context ranking end-to-end with the language modeling loss. Practitioners might care because SAS can improve performance on downstream tasks by using attention budgets more effectively.

attention sparsificationTransformersend-to-end optimizationcontext rankingcontinual learning