Firehose

13 SEP 2026 · Paper

Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training

This paper addresses the issue of memory peak allocation in Mixture-of-Experts (MoE) models during long-context training, proposing four different techniques to reduce memory usage without compromising performance. Practitioners might care about these techniques to train larger MoE models with longer context lengths.

mixture-of-experts memory-efficient training long-context models GPU optimization model parallelism activation management

Browse by tag

Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training