Filtered to benchmarks · clear filter
This paper proposes a method to scale automatic research agents by replacing environment execution with a world model, which can reduce training costs and improve performance. Practitioners might care about this approach because it can accelerate training times and lead to better results for complex AI tasks.
This paper introduces VGI-bench, a new benchmark for evaluating video generation models' visual reasoning capabilities, and finds that current models can solve some visually grounded tasks but still struggle with reliability. Practitioners may care about developing more reliable video generation models that can perform better on this benchmark.
This paper introduces a new benchmark for video generation tasks that require both achieving a desired outcome and maintaining semantic consistency with a reference image. Practitioners might care about this research because it can help evaluate and improve the performance of video generation models in real-world applications.
This paper proposes a framework, LongHorizon-Harness, to help large language model agents tackle long-horizon tasks by explicitly tracking task states and verifying facts from the environment. Practitioners might care because it can improve the performance of these agents on real-world tasks.
This paper introduces LoopArena, a benchmark for evaluating how well a model can guide a coding agent through a long-running task, and finds room for improvement in long-horizon loop control. Practitioners working on AI development might care because it could lead to more efficient and reliable development processes.
This paper proposes a method to optimize prompts for multi-agent systems by identifying which agent's modification resolves a failure, and then using that agent's output as supervision to extract a fine-grained gradient. Practitioners might care because it could improve the performance of large language model-based multi-agent systems.
This paper develops a standardized evaluation method for computer-using agents (CUAs) to ensure they fulfill task instructions, using vision-language models (VLMs) as judges. Practitioners can benefit from this work by using reliable and cost-effective reward signals for CUA training.
This paper proposes a new approach to Large Language Models (LLMs) that uses diffusion to speed up generation without sacrificing quality, allowing for faster and more efficient language processing. Practitioners may care about this research if they need to process large amounts of text quickly and efficiently.
This paper introduces ASI-Bench, a new benchmark for evaluating AI systems' ability to explore new knowledge, create new ideas, and conduct scientific research autonomously. Practitioners might care about ASI-Bench because it aims to push the limits of current AI systems and accelerate the development of artificial superintelligence.
This paper presents two search agents, Iris-mini and Iris-pro, trained to solve complex search tasks using reinforcement learning and self-supervised learning. Practitioners might care because these models achieve state-of-the-art results on various benchmarks, demonstrating the potential of AI-powered search agents in real-world applications.
This paper proposes a framework for training agents to create 3D open worlds based on user queries, and evaluates its performance using a large benchmark dataset. Practitioners might care about this research because it can help develop more capable multimodal agents that can understand user intent and generate realistic 3D environments.