Filtered to benchmarking · clear filter
This paper introduces a new AI model called BDH-CQ that can learn from context and reason iteratively without verbalizing its steps. Practitioners might care because this model achieves a high level of performance while being very efficient, which could be useful for applications where resources are limited.
This paper creates a benchmark (HarnessDev) to test whether large language models (LLMs) can design and improve their own execution infrastructure, called the agent harness, which affects their performance. Practitioners might care because it explores how models can adapt to changing environments and potentially improve efficiency.
This paper introduces AREX, a self-improving agent that recursively refines its answers by verifying intermediate results and using the partially verified state to guide subsequent refinement, aiming to improve the efficiency of deep research. Practitioners might care about AREX's approach to reducing the cost of discovery and verification in complex research tasks.
This paper introduces EarlyEval, a method to reduce the cost of evaluating large language model (LLM) agents by predicting their outcomes early, allowing for earlier termination of agent runs. Practitioners may care because it can significantly reduce the computational cost of agent development.
This paper introduces DataFlex-RL, an evaluation platform for comparing different data policies in reinforcement learning with verifiable rewards (RLVR), and finds that uniform data policy leads to better performance and is more reproducible than other methods. Practitioners might care about this because it provides a way to compare different data policies and ensure that their RLVR models are performing well.
This paper explores a new approach to transferring capabilities from strong AI models to weaker ones at test time, rather than just during training. Practitioners might care about this because it could lead to more efficient and effective use of powerful models in real-world applications.
This paper creates a database and search engine to help researchers and developers find and understand AI benchmarks, making it easier to evaluate and compare AI systems. Practitioners can use Benchmark Radar to discover and evaluate benchmarks, reducing the time and effort spent searching for relevant evaluations.
This paper introduces Ouroboros, a self-improving AI agent that develops its own tools and code through a process of reviewed commits, allowing it to learn and adapt over time. A practitioner might care about Ouroboros because it demonstrates a potential approach to creating autonomous AI systems that can improve themselves.
This paper introduces SWE-bench Science, a benchmark for evaluating coding agents' performance in repairing scientific software, and identifies common failure mechanisms that hinder their success. Practitioners may care about this research as it can help improve the reliability and reproducibility of scientific findings by developing more effective coding agents.
This paper introduces a new framework for training educational language models, specifically designed to evaluate their ability to understand curriculum knowledge and its visual presentation. Practitioners might care about this work because it aims to improve language models' performance in educational settings.
This paper introduces a benchmark to measure how well large language models (LLMs) can prepare training data, and how well they evaluate the quality of that data. Practitioners might care because improving data preparation can lead to better model performance.