Filtered to policy optimization · clear filter
This paper proposes a method to improve language model training by allocating credit to individual tokens within a response, allowing for more nuanced evaluation of model performance. Practitioners may care about this method as it can lead to better language model performance, especially in tasks that require specific formatting or semantic choices.
This paper develops a new method for improving reasoning language models, called β-OPSD, which combines policy optimization and self-distillation to improve stability and performance. Practitioners might care about this method because it provides a more efficient and effective way to improve language model reasoning abilities.
This paper investigates the use of the Muon optimizer in reinforcement learning (RL) post-training and finds that it can significantly improve the success rate of RL agents, especially when combined with other techniques like policy optimization and advantage estimation. Practitioners in RL may care about this research to explore new ways to improve the performance of their agents.
This paper addresses a problem in reinforcement learning (RL) where algorithms with PPO-Clip suffer from "exploration collapse", where they fail to explore new areas of the environment. The authors propose a new method called Riemannian Isometric Policy Optimization (RIPO) that helps to balance exploration and exploitation by using a more accurate geometric metric.
This paper proposes a new method for advantage shaping in reinforcement learning called Contrastive Policy Optimization (CPO), which uses contrastive disagreement between reference-guided and vanilla generation distributions to indicate correctness. Practitioners might care because it can improve the effectiveness of reinforcement learning methods in generating correct responses.