Computer vision and multimodal models
70 notes, newest first. Includes the vault tags computer vision, vision-language models, multimodal, multimodal learning, diffusion models, image generation.
GeoReform: Reflective Formalization Evolution for Multimodal Geometry Problem Solving
The preprint proposes GeoReform, which treats geometric formalization as an optimizable policy rather than a fixed parser output. It collects failed rollouts, diagnoses representation defects, and mutates how entities…
HRIL: Learning Multimodal Synergy via Higher-Order Tensor Modeling
The preprint frames multimodal synergy as higher-order dependence that can be lost by pairwise fusion. HRIL uses an empirical cross-moment tensor with Tucker decomposition and a synergy-aware regularizer. The authors…
VISTA: A Visual Harness for Reasoning in an Interactive World
The technical report introduces a visual harness that gives a multimodal model long-horizon visual context through direct perception, lossless visual memory, active retrieval, and input reorganization. The abstract…
WorldAuditBench: Interactive 3D World Auditing with Multimodal Agents
The preprint presents 213 anomaly tasks across 13 Unreal Engine 5 and three.js environments, covering five anomaly families. It evaluates five frontier models under a fixed exploration budget using VLA-led exploration…
C3M: Cross-Session Multimodal Memory Maintenance for Long-Horizon Tasks
The preprint proposes bounded query-blind memory for long-horizon tasks. It describes relation-aware updates for complementary and incompatible records, budgeted routing that selects index pages and expands source…
Breaking the weakest link to evade vision language models
The authors investigate the vulnerability of Vision Language Models (VLMs) to adversarial attacks, which can disrupt or force the models' interpretation of visual inputs. They propose a gradient-based attack method…
Neurosymbolic Embodied Agents
The authors propose a neurosymbolic agent that combines vision-language models with symbolic planning for executable long-horizon household tasks. The agent consists of two phases: the first involves a vision-language…
UniTAC: Universal Task-Aware Compression via Weighted Distortion Measures
Researchers propose UniTAC, a universal image compression system for physical AI systems like autonomous vehicles and robots. The system can adapt to changing tasks without retraining, using task-specific importance…
Beyond Trial-and-Error: Agentic Optimization for Image-to-Video Adherence
Researchers present the 'Agentic Self-Improvement' framework to address challenges in professional workflows with modern Image-to-Video (I2V) models. The framework uses a two-stage approach: an iterative prompt…
DreamFly: Causal Memory and Receding-Horizon Diffusion Planning for Aerial Vision-Language Navigation
Researchers propose DreamFly, a framework for aerial vision-language navigation that addresses challenges like limited historical context, short planning horizons, and unreliable implicit termination. The framework…
Modern Backbones Improve Multi-task DETR for Mammography Classification and Lesion Localization
The study evaluates a multi-task DETR framework that simultaneously predicts the malignancy of mammogram images and locates lesions, using modern backbones such as ConvNeXtV2 and DINOv3. The results show that these…
Multimodal Model Diffing for Feature Discovery and Control
Researchers developed a framework called MMDiff to help understand, control, and audit the behavior of Multimodal Large Language Models (MLLMs). The framework identifies features that are altered by multimodal…
NeuroRefiner: Morphology-Aware Multi-Agent Refinement for 3D Fluorescence Microscopy Neuron Segmentation
The study introduces a new method called NeuroRefiner for accurately segmenting neurons in 3D fluorescence microscopy images. The method overcomes challenges posed by the sparse and elongated morphology of neurons…
Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Models
Researchers present the Evaluation Agent framework, a system that efficiently evaluates visual generative models using human-like strategies. It decomposes evaluation requests into sub-aspects, generates prompts…
iARCS: Iterative Agentic RL for Controllable 3D Scene Generation
The paper presents iARCS, an iterative agentic reinforcement learning framework for enhancing the performance of synthetic 3D scene generators by adapting them to meet task-specific functional constraints, such as…
NSF-HRPT: Neural Semantic Field meets Hierarchical Risk Perception Tree for Safety-Critical Scenario Assessment
Researchers propose NSF-HRPT, a framework that uses a Neural Semantic Field (NSF) to model scene semantics, trajectory predictions, and probabilistic Time-to-Collision (TTC) distributions from simulation data. During…
When and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding
Researchers propose EcoFrame, a framework for adaptively scheduling visual evidence selection in long videos for efficient video understanding by vision-language models (VLMs). The method uses the VLM's inference…
EgoGenesis: Egocentric World-Action Modeling with Online Anchored Projective Memory and Action-3D RoPE
The authors present a method called EgoGenesis for creating synthetic manipulation videos for embodied AI using egocentric video. The method uses a pretrained video generation prior and introduces two geometry-aware…
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
The research introduces OSReward, a new benchmark for evaluating the reliability of Vision-Language Models (VLMs) that judge Computer-Using Agents' (CUAs) trajectories. The study finds that even state-of-the-art VLMs…
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
The paper introduces OSReward, a benchmark for evaluating the reliability of vision-language models (VLMs) used as judges for computer-using agents (CUAs). The benchmark consists of CUA trajectories from various agent…
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
The research introduces OSReward, a new benchmark for evaluating the reliability of vision-language models (VLMs) used to judge computer-using agents' trajectories. The study finds that state-of-the-art VLMs have a…
OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
The research introduces OSReward, a new benchmark for evaluating the reliability of vision-language models (VLMs) used to judge computer-using agents (CUAs). The study finds that even state-of-the-art VLMs have a…
Multimodal fusion of visual and morphometric features for avian bone classification
Researchers developed a framework that combines convolutional neural network-based image analysis with standardized morphometric data for classifying bird bones. The model achieved 86% accuracy in identifying skeletal…
GraphVid: Interactive Graph-Controllable Video Generation
The researchers introduce GraphVid, a model that generates interactive videos using structured interaction graphs, allowing for flexible and precise control. The model shows strong performance compared to Motion-I2V in…
MIRROR: Learning from the Other View for Multi-Modal Reasoning
The paper presents MIRROR, a method to improve multi-modal reasoning in AI models by learning from different views (text, image, combined text and image) of the same problems. The study reveals that vision-language…
MIRROR: Learning from the Other View for Multi-Modal Reasoning
Researchers have created a new approach called Modality-Informed Reciprocal Reasoning Optimization (MIRROR) to improve the reasoning abilities of both large language models and vision-language models. The approach uses…
HAS: Highlight-guided Attention Steering for Multimodal LLM Video Summarization
The paper proposes a new method called Highlight-guided Attention Steering (HAS) for video summarization using Multimodal Large Language Models (M-LLMs). HAS aims to enhance the understanding of videos by focusing…
O-VAD: Industrial Video Anomaly Detection through Object-Centric Tracking and Reasoning
Researchers have developed a new method for detecting abnormal objects and events in industrial settings using an object-centric tracking and reasoning approach. This method is designed to overcome the limitations of…
SGA: Plug&Play Geometric Verification for Educational Video Synthesis
Researchers have proposed a new tool called Symbolic Geometric Agent (SGA) to improve the spatial correctness and visual legibility of educational videos generated by Large Language Models (LLMs). The SGA intercepts…
Mask-Aware Policy Gradients for Diffusion Language Models
The paper presents a method for improving the performance of large language models in tasks like mathematical reasoning and coding by using reinforcement learning. This method addresses the challenge of extending…
UESF-Bench: Benchmarking and Probing for Unified Embodied Seeking and Following
The authors introduce the Unified Embodied Seeking and Following Benchmark (UESF-Bench), a new benchmark for embodied agents that allows them to search for and follow language-described targets in dynamic environments…
UR-VC: Unsupervised Robotic Value Correction for Time-Derived Progress Proxies
Researchers introduce an offline method called Unsupervised Robotic Value Correction (UR-VC) to correct time-derived progress labels used by modern robot learning systems. The method is based on a regularity in…
A Multimodal Dataset for Large Language Model Applications in the Energy Domain
The paper introduces a new open-access dataset called mAIEnergy which is a multimodal corpus designed to support Large Language Model (LLM) applications in the energy sector. It contains various types of data including…
Evidence-Backed Video Question Answering
The authors propose Evidence-Backed Video Question Answering (E-VQA), a new task that requires models to not only provide answers but also the specific spatial and temporal evidence from videos. They introduce…
MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents
The paper introduces MM-ToolSandBox, a new evaluation framework for visual tool-calling agents. It supports multi-image, multi-turn tasks and offers a stateful execution environment with over 500 tools across 16…
MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents
Researchers introduce MM-ToolSandBox, a new evaluation framework for visual tool-calling agents. This framework supports over 500 tools across 16 application domains and can handle multi-image, multi-turn tasks with…
Omni-Decision: A Progressive Evidence-State Agent System for Omni-Modal QA
The paper introduces Omni-Decision, a system aimed at answering questions where evidence is spread across multiple modalities like videos, audio, images, web pages, and computation results. The system maintains a…
Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos
The text discusses a new proactive assistance system named Vinci2, which provides assistance in continuous egocentric videos by considering the user's history, current activity, and whether help would be welcome. The…
Cognitive-structured Multimodal Agent for Multimodal Understanding, Generation, and Editing
The paper proposes a new multimodal agent called the Cognitive-structured Multimodal Agent. This agent is designed to improve long-horizon multimodal dialogue by externalizing visual information into an Episodic Visual…
OmniFood-Bench: Evaluating VLMs for Nutrient Reasoning and Personalized Health Advice
The paper introduces OmniFood-Bench, a new benchmark for evaluating Large Vision-Language Models (VLMs) in the context of personalized healthcare and dietary management. The benchmark is designed to assess the ability…
OpenCoF: Learning to Reason Through Video Generation
The paper introduces OpenCoF, a framework that includes a new dataset and a fine-tuned video model for improving Chain-of-Frame (CoF) reasoning in video generation. The authors argue that existing models lack specific…
SLORR: Simple and Efficient In-Training Low-Rank Regularization
The paper introduces SLORR, a simple and efficient in-training low-rank regularization framework for neural networks. It aims to improve compressibility without significant accuracy loss, while avoiding the limitations…
Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF
The paper proposes two methods for improving the efficiency of Reinforcement Learning from Human Feedback (RLHF) when applied to diffusion models, which are often inefficient due to high feedback requirements. The…
When Prompts Ignore Structure: Graph-Based Attribute Reasoning for Calibrated VLMs
The paper proposes ARGTCA, a method to improve the calibration of Vision-Language Models (VLMs) by using Graph Attention Networks to capture inter-attribute dependencies and reduce overconfidence during test-time…
Bridging Physical Reasoning and Task Generalization via Visual Action Outcome Reasoning Alignment
Researchers have developed a new method called VAORA (Visual Action Outcome Reasoning Alignment) to improve the performance of vision-language models in interactive physical reasoning, especially in unseen tasks and…
VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection
The paper introduces VendorBench-100, a benchmark for evaluating deepfake image detection models across three paradigms: commercial APIs, zero-shot vision-language models (LLMs), and open-source detectors. The…
VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection
A study introduces VendorBench-100, a cross-paradigm benchmark for deepfake image detection that evaluates different models using a common protocol. The benchmark includes a diverse set of edge cases and uses the…
MECoBench: A Systematic Study of Multimodal Agent Collaboration in Embodied Environments
The research paper introduces MECoBench, a benchmark for evaluating the collaboration abilities of multimodal agents in visually grounded environments. The study explores three key findings: collaboration generally…
TreeAgent: A Generalizable Multi-Agent Framework for Automated Bias Labeling in Forestry via Compiled Expert Rules and Vision-Language Models
Researchers propose a multi-agent system (MAS) called TreeAgent to automate bias labeling in forestry via expert decision trees and Vision-Language Models (VLMs). The MAS aims to reduce the need for human annotation…
TreeAgent: A Generalizable Multi-Agent Framework for Automated Bias Labeling in Forestry via Compiled Expert Rules and Vision-Language Models
Researchers have proposed a multi-agent system that uses expert decision trees and Vision-Language Models (VLMs) to automate the bias classification in tree height in forestry remote sensing. The system mitigates VLM…
TreeAgent: A Generalizable Multi-Agent Framework for Automated Bias Labeling in Forestry via Compiled Expert Rules and Vision-Language Models
Researchers propose a system called TreeAgent that uses artificial intelligence (AI) to classify tree heights in forestry remote sensing. The system combines decision trees created by experts with Vision-Language…
GROW$^2$: Grounding Which and Where for Robot Tool Use
Researchers introduce GROW$^2$, a system that helps robots use tools creatively by grounding which object to use as a tool and where to apply it, even when the object is not intended for that purpose. This system uses…
ManimAgent: Self-Evolving Multimodal Agents for Visual Education
The research presents ManimAgent, a self-evolving multimodal agent that learns from its failures and successes in generating Python code using the Manim library to render mathematical animations. The agent stores these…
Automating Potential-based Reward Shaping with Vision Language Model Guidance
Researchers introduce a new method called VLM-PBRS for automating potential-based reward shaping using vision language model (VLM) feedback. The aim is to improve reinforcement learning agents' ability to learn from…
Automating Potential-based Reward Shaping with Vision Language Model Guidance
Researchers have developed a new method called VLM-PBRS for automating potential-based reward shaping in reinforcement learning agents. This method uses vision language models (VLMs) to learn the potential function…
Automating Potential-based Reward Shaping with Vision Language Model Guidance
Researchers have developed a new method for reinforcement learning agents using vision language models (VLMs). The approach called VLM-guided PBRS learns a potential function from VLM feedback to address the challenge…
Confidence-Aware Tool Orchestration for Robust Video Understanding
The researchers propose a framework called Robust-TO for video understanding that addresses the 'Blind Trust Problem' in video reasoning language models. The proposed method incorporates per-frame trustworthiness into…
Confidence-Aware Tool Orchestration for Robust Video Understanding
The paper presents a new framework called Robust-TO for video understanding that addresses the issue of 'Blind Trust Problem' in current language models. These models assume every input frame is equally reliable, but…
Look-Before-Move: Narrative-Grounded World Visual Attention in Dynamic 3D Story Worlds
Researchers have developed a new camera planning framework called Look-Before-Move for embodied AI operating in dynamic 3D environments. The goal is to enable the camera to actively decide what to observe, not just…
Safe Autoregressive Image Generation with Iterative Self-Improving Codebooks
The authors propose a method for safe image generation using autoregressive unified multimodal models. They introduce Iterative Self-Improving Codebooks to eliminate harmful mappings in the codebook that could generate…
EG-VQA: Benchmarking Verifiable Video Question Answering with Grounded Temporal Evidence
Researchers have developed a new benchmark called Evidence-Grounded Video Question Answering Benchmark (EG-VQA) to evaluate the performance of video question answering models. The benchmark is unique because it…
NoContactNoWorries: Estimating Contact through Vision and Proprioception for In-Hand Dexterous Manipulation
Researchers have developed a transformer-based multimodal framework named NoContactNoWorries that allows robots to infer binary contact states from RGB-D vision and the robot's proprioception. This approach offers a…
OrbitForge: Text-to-3D Scene Generation via Reconstruction-Anchored Video Synthesis
The researchers introduce OrbitForge, a method that converts single text-generated videos into 3D scenes using frozen video priors and per-prompt Gaussian Splatting reconstruction optimization. The method uses 3D…
Data Selection Through Iterative Self-Filtering for Vision-Language Settings
Researchers propose a new method to select data for training neural networks in large scale, noisy datasets. The Self-Filtering method iteratively trains a CLIP model on an evolving dataset, balancing clean samples and…
Distribution-Aware Diffusion-LLM for Robust Ultra-Long-Term Time Series Forecasting
The authors propose a new method called Diffusion-LLM for time series forecasting using Large Language Models (LLMs). The proposed framework integrates a conditional diffusion model into an LLM-based forecasting…
HyperQuant: A Rate-Distortion-Optimal Quantization Pipeline for Large Language and Diffusion Models
Researchers present HyperQuant, a quantization pipeline for large language and diffusion models, which outperforms other methods in terms of compression rate and quality across various experiments. The method combines…
Kamera: Unified Position-Invariant Multimodal KV Cache for Training-Free Reuse
Researchers propose a new method called Kamera for efficient use of KV cache by multimodal agents during reasoning iterations. The method avoids the need to re-encode from scratch and reduces the need for multiple…
RS-Gen: A Multi-Stage Agentic Framework for Reasoning and Search-Augmented Image Generation
The paper introduces RS-Gen, a multi-stage image agent framework that aims to improve the performance of existing image generation and editing models by introducing a 'Questioning-and-Solving' closed-loop mechanism for…
Semantic Browsing: Controllable Diversity for Image Generation
Researchers have developed a new method for generating diverse images using text-to-image models. Instead of relying on stochastic variations within the model, they induce diversity directly at the text level. The…
VideoAgent: All-in-One Framework for Video Understanding and Editing
Researchers have developed a new framework called VideoAgent to automate video editing tasks. The system can handle diverse editing operations and understand long videos, addressing limitations in existing automated…
