Firehose

29 OCT 2026 · Paper

Your LR Schedule May Not Be Principled: Effective Learning Rate Scheduling in Weight-Normalized Training

This paper shows that the learning rate schedule used in weight-normalized training is more important than the normalization itself, and proposes a new schedule called AIR that targets the effective learning rate trajectory. Practitioners might care about this because it could help them improve the performance of their models by optimizing their learning rate schedules.

weight-normalized training effective learning rate learning rate scheduling affine scheduling

Browse by tag

Your LR Schedule May Not Be Principled: Effective Learning Rate Scheduling in Weight-Normalized Training