Filtered to vision-language models · clear filter
This paper evaluates whether vision-language models can act through a physical body and how they can make decisions about what actions to take, without being hindered by issues like balance and motor control. Practitioners in AI and robotics might care because understanding how models interact with their physical bodies can help improve their ability to navigate and interact with the world.
This paper develops a new type of AI model that can understand and interact with both images and text in real-time, without needing a huge amount of training data. Practitioners may care about this model because it can be used in applications where fast and efficient visual perception is required.
This paper investigates how to scale large language models to also understand and interact with the physical world by training them on multiple types of data from scratch, allowing them to reason about both text and images. Practitioners might care because this could lead to more robust and versatile AI systems that can handle a wider range of tasks.
This paper develops a framework, SpatialCLI, to help vision-language models (VLMs) better understand and use visual tools to make better decisions. By training VLMs to reason with spatial tools and then internalize those capabilities, SpatialCLI can improve the performance of VLMs in tasks that require visual reasoning.
This paper proposes a new method for fine-tuning vision-language models on robot demonstrations to improve their performance on real-world tasks, by preventing the overwrite of pre-trained representations and aligning language and action predictions. Practitioners may care about this work because it aims to improve the generalizability and robustness of vision-language-action policies in real-world applications.
This paper introduces Modus, a new type of any-to-any model that can predict any modality from any combination of others using only a decoder, without the need for specialized heads or losses. This approach can support a wide range of applications, such as chained generation and cross-modal self-verification, and has shown strong out-of-the-box performance on various benchmarks.
This paper introduces HPD-Parsing, a new approach to document parsing that uses hierarchical parallel decoding to improve efficiency and throughput. Practitioners in natural language processing and computer vision might care because it could lead to faster and more accurate document parsing models.
This paper introduces Robostral Navigate, a vision-language model that enables robots to navigate using only a single monocular RGB camera, making it more scalable and cost-effective for deployment across various robotic platforms. Practitioners might care about this because it can simplify navigation tasks for robots in real-world environments.
This paper introduces SceneActBench, a benchmark for evaluating the ability of vision-language model agents to perform actions in 3D scenes, and analyzes the performance of 11 different VLM configurations on this benchmark. Practitioners may care about this research because it helps to identify the limitations of current VLM agents in performing complex tasks in 3D environments.
This paper proposes a new approach to improve vision-language models for visual retrieval, which can handle long visual contexts and large numbers of distractors. Practitioners might care because it can lead to better performance on image and video benchmarks.