Filtered to benchmarking · clear filter
This paper introduces a new AI model called BDH-CQ that can learn from context and reason iteratively without verbalizing its steps. Practitioners might care because this model achieves a high level of performance while being very efficient, which could be useful for applications where resources are limited.
This paper addresses a problem in training spatial vision-language models (VLMs) where the model becomes specialized in answering certain types of questions, but loses its ability to perform well on others. A new method called ChronoReplay is proposed to improve the model's ability to generalize across different question types.
This paper introduces a benchmark, StreamEval, to evaluate interactive streaming video generation models, which is essential for enabling real-time interaction with generated video in applications like streaming services. Practitioners caring about the development of interactive video generation systems will find this paper useful.
This paper creates a benchmark (HarnessDev) to test whether large language models (LLMs) can design and improve their own execution infrastructure, called the agent harness, which affects their performance. Practitioners might care because it explores how models can adapt to changing environments and potentially improve efficiency.
This paper creates a benchmark to test how well coding agents can build agents that meet their users' needs, and it finds that most agents fail to deliver what they promise. Practitioners might care because it highlights the need for better evaluation methods to ensure agents are built to meet user requirements.
This paper introduces AREX, a self-improving agent that recursively refines its answers by verifying intermediate results and using the partially verified state to guide subsequent refinement, aiming to improve the efficiency of deep research. Practitioners might care about AREX's approach to reducing the cost of discovery and verification in complex research tasks.
This paper introduces EarlyEval, a method to reduce the cost of evaluating large language model (LLM) agents by predicting their outcomes early, allowing for earlier termination of agent runs. Practitioners may care because it can significantly reduce the computational cost of agent development.
This paper introduces DataFlex-RL, an evaluation platform for comparing different data policies in reinforcement learning with verifiable rewards (RLVR), and finds that uniform data policy leads to better performance and is more reproducible than other methods. Practitioners might care about this because it provides a way to compare different data policies and ensure that their RLVR models are performing well.
This paper creates a simulated environment where a human and AI assistant interact, allowing researchers to test how well AI assistants can guide humans in the physical world. Practitioners might care because it could lead to more reliable AI assistants that can truly help people with tasks like remembering what to do.
This paper explores a new approach to transferring capabilities from strong AI models to weaker ones at test time, rather than just during training. Practitioners might care about this because it could lead to more efficient and effective use of powerful models in real-world applications.
This paper develops a method to efficiently scale agent research loops, allowing for more effective self-improvement and reusable improvements across diverse environments. Practitioners might care about this research because it could lead to significant cost savings and improved performance in automated code completion and generation tasks.
This paper evaluates how well multimodal agents can generate videos that include synchronized code, sounds, and on-screen events, and why a practitioner might care because it could lead to more realistic video generation in applications like educational content, music videos, and video games.