Continual Learning Mechanisms Compose for Long-Horizon Memorization
This paper explores a way for language models to remember information from previous updates, even after many changes, and finds that combining different techniques can lead to much better retention than using any one technique alone.