Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Confidence-Aware Tool Orchestration for Robust Video Understanding

Source

arXiv

arxiv.org/abs/2606.26904v1

Read the source

Source

Plain-Language Brief

The paper presents a new framework called Robust-TO for video understanding that addresses the issue of 'Blind Trust Problem' in current language models. These models assume every input frame is equally reliable, but under certain perturbations like motion blur or occlusion, they can suffer significant accuracy drops. Robust-TO integrates per-frame trustworthiness into its reasoning process and organizes various visual perception tools to return evidence in a shared format with a calibrated reliability score. The framework outperforms existing methods on two video reasoning benchmarks, both on clean inputs and under corruption types.

What the Document Says

  • The paper introduces Robust-TO, a framework for video understanding that addresses the 'Blind Trust Problem' in current language models.
  • Robust-TO organizes heterogeneous visual perception tools under a unified evidence interface.
  • Each tool receives a sub-query and a set of trustworthy frames based on a reliability-relevance score.
  • The framework returns evidence with a concrete prediction, temporal grounding, and a calibrated reliability score.
  • During reasoning, the scores guide evidence weighting in a three-tier synthesis process and define a confidence-cost GRPO reward.
  • On two video reasoning benchmarks, Robust-TO achieves 56.4% average accuracy on clean inputs, outperforming the strongest open-source baseline by 10.6%p and Gemini-2.5-Pro (46.2%).
  • Under corruption types, Robust-TO maintains 54.3% average accuracy, 5.8%p above the strongest open-source baseline.

Key Concepts Mentioned

  • video reasoning language models
  • Blind Trust Problem
  • Robust-TO framework
  • per-frame trustworthiness
  • heterogeneous visual perception tools
  • reliability-relevance score
  • calibrated reliability score
  • three-tier synthesis process
  • confidence-cost GRPO reward
  • video reasoning benchmarks
  • corruption types

Claims Made by the Document

  • Current language models assume every input frame is equally reliable.
  • Robust-TO addresses the 'Blind Trust Problem' in current language models.
  • Robust-TO outperforms existing methods on two video reasoning benchmarks.

Evidence Used

  • Not stated in source

Methods / Approach

  • Three-tier synthesis process
  • confidence-cost GRPO reward

Limitations Stated by the Authors

Not stated

Limitations Noted by the Agent

Agent interpretation — not from the source.

The paper does not discuss potential limitations of the proposed method.

Open Questions

  • What are the specific perturbations considered as 'realistic corruption types'?
  • How is the 'reliability-relevance score' calculated?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The paper does not provide detailed information on the specific perturbations or how the reliability-relevance score is calculated.

Raw Abstract / Excerpt

Video reasoning language models implicitly assume that every input frame is equally reliable. This leads to what we term the Blind Trust Problem: under realistic perturbations such as motion blur, glare, or occlusion, frontier video reasoning models can suffer 15-30%p accuracy drops on real-world embodied benchmarks, while remaining unaware that their visual evidence has been degraded. To address this challenge, we propose Robust-TO, an agentic video understanding framework that explicitly integrates per-frame trustworthiness into every stage of reasoning. Robust-TO organizes heterogeneous visual perception tools under a unified evidence interface. Each tool receives a sub-query derived from the original question and a set of trustworthy frames selected by the reliability-relevance score. It returns evidence in a shared format: a concrete prediction (e.g., a bounding box, motion trajectory, recognized text, or action label), temporal grounding, and a calibrated reliability score. During reasoning, these calibrated scores guide evidence weighting in a three-tier synthesis process (high/medium/low) and define a confidence-cost GRPO reward that jointly optimizes correctness, evidence reliability, and efficiency. On two video reasoning benchmarks spanning eight tasks, Robust-TO achieves 56.4% average accuracy on clean inputs, surpassing the strongest open-source baseline by 10.6%p and outperforming Gemini-2.5-Pro (46.2%). Under five realistic corruption types, Robust-TO maintains 54.3% average accuracy, 5.8%p above the strongest open-source baseline, while exhibiting the smallest clean-to-corrupted accuracy drop among all compared methods.