Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Evidence-Backed Video Question Answering

Source

arXiv

arxiv.org/abs/2607.11862v1

Read the source

Source

Plain-Language Brief

The authors propose Evidence-Backed Video Question Answering (E-VQA), a new task that requires models to not only provide answers but also the specific spatial and temporal evidence from videos. They introduce ST-Evidence, a human-verified benchmark for this task, and create a large dataset called ST-Evidence-Instruct. Fine-tuning models on this dataset improves explainable video understanding compared to traditional methods.

What the Document Says

  • The authors discuss Evidence-Backed Video Question Answering (E-VQA), a task where models must provide both an answer and the specific spatio-temporal evidence from videos.
  • ST-Evidence is introduced as the first human-verified benchmark for this task.
  • The authors create a large dataset called ST-Evidence-Instruct to support E-VQA.
  • Evaluations of state-of-the-art models on ST-Evidence reveal a gap between QA accuracy and true visual perception that can't be bridged by scaling alone.
  • Fine-tuning grounded Video LLMs on the ST-Evidence-Instruct data improves explainable video understanding compared to traditional methods, with significant gains over corresponding size-matched UniPixel baselines.

Key Concepts Mentioned

  • Video Large Language Models (Video LLMs)
  • Question Answering (QA)
  • Explainability
  • Spatio-temporal evidence
  • ST-Evidence
  • E-VQA
  • ST-Evidence-Instruct
  • fine-grained grounding
  • high-level reasoning

Claims Made by the Document

  • Current Video LLMs operate as black boxes.
  • Existing explainability efforts are insufficient for complex video dynamics.
  • E-VQA requires models to provide spatio-temporal evidence along with answers.
  • ST-Evidence is the first human-verified benchmark for pixel-level grounding in video QA.
  • Evaluations reveal a gap between QA accuracy and true visual perception that can't be bridged by scaling alone.
  • Fine-tuning grounded Video LLMs on ST-Evidence-Instruct improves explainable video understanding.

Evidence Used

  • The authors discuss the need for models to provide both answers and spatial/temporal evidence.
  • The introduction of ST-Evidence as a human-verified benchmark.

Methods / Approach

  • Creation of E-VQA, ST-Evidence, and ST-Evidence-Instruct.
  • Fine-tuning grounded Video LLMs on the ST-Evidence-Instruct data

Limitations Stated by the Authors

The authors acknowledge that scaling alone fails to bridge the gap between QA accuracy and true visual perception.

Limitations Noted by the Agent

Agent interpretation — not from the source.

It is not clear if other methods besides fine-tuning grounded Video LLMs on ST-Evidence-Instruct could also improve explainable video understanding.

Open Questions

  • What other methods could potentially improve explainable video understanding?
  • How does the performance of E-VQA compare to traditional QA methods in various scenarios?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The authors focus on improving explainability and fine-grained grounding in video QA. They introduce a new task, E-VQA, and a human-verified benchmark, ST-Evidence, as well as a large dataset, ST-Evidence-Instruct.

Raw Abstract / Excerpt

Current Video Large Language Models (Video LLMs) excel in question answering (QA) but largely operate as black boxes, providing textual answers without verifiable visual grounding. Existing explainability efforts rely on textual rationales or sparse bounding boxes, which struggle to capture complex video dynamics such as occlusions and non-rigid deformations. We propose Evidence-Backed Video Question Answering (E-VQA), a novel task requiring models to jointly output a semantic answer and precise spatio-temporal evidence: temporal segments and dense, tracked object segmentation masklets. To support this, we introduce ST-Evidence, the first human-verified benchmark for both discriminative and generative pixel-level grounding. Evaluations of state-of-the-art models reveal a critical decoupling between QA accuracy and true visual perception that scaling alone fails to bridge. To address this, we develop scalable, automated generation pipelines to create ST-Evidence-Instruct, a 160k-scale dataset bridging high-level reasoning with fine-grained grounding. Fine-tuning grounded Video LLMs on this data yields substantial gains over the corresponding size-matched UniPixel baselines (e.g., +27.2 t-mean and +13.8 J&F on a 7B model), establishing a robust baseline for explainable, evidence-backed video understanding. Code and data are available at https://github.com/SalesforceAIResearch/EVQA.