Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

When and Where to Look: Adaptive Visual Evidence Scheduling for Efficient Long Video Understanding

Source

arXiv

arxiv.org/abs/2608.03918v1

Read the source

Source

Plain-Language Brief

Researchers propose EcoFrame, a framework for adaptively scheduling visual evidence selection in long videos for efficient video understanding by vision-language models (VLMs). The method uses the VLM's inference feedback to determine when to expand the frame budget and where to search for additional candidate evidence. It achieves better accuracy-efficiency trade-offs compared to existing methods on various datasets, with improved speed and comparable accuracy.

What the Document Says

  • EcoFrame is a training-free framework for adaptive visual evidence scheduling.
  • It uses the VLM's inference feedback for budget scheduling and candidate proposal.
  • Entropy-gated budget scheduling stops early when current evidence is sufficient or expands the frame budget otherwise.
  • Attention-guided candidate proposal converts frame-level attention into a temporal prior for dense local search.
  • The method demonstrates better accuracy-efficiency trade-offs on multiple VLM backbones across several datasets.
  • On Qwen2.5-VL, EcoFrame achieves an average accuracy of 64.4 and provides a $1.85 imes$ speedup over AKS and BOLT.
  • Compared to the agent-based A.I.R., EcoFrame maintains comparable accuracy with up to a $13.5 imes$ inference speedup.

Key Concepts Mentioned

  • vision-language models (VLMs)
  • EcoFrame
  • adaptive visual evidence scheduling
  • entropy-gated budget scheduling
  • attention-guided candidate proposal

Claims Made by the Document

  • EcoFrame achieves a better accuracy--efficiency trade-off.
  • On Qwen2.5-VL, EcoFrame surpasses BOLT at 63.5 with a $1.85 imes$ speedup.
  • Compared to A.I.R., EcoFrame maintains comparable accuracy with up to a $13.5 imes$ inference speedup.

Evidence Used

  • Experiments on Video-MME, LongVideoBench, and MLVU

Methods / Approach

  • entropy-gated budget scheduling
  • attention-guided candidate proposal

Limitations Stated by the Authors

UNKNOWN (Not stated in source)

Limitations Noted by the Agent

Agent interpretation — not from the source.

The study does not discuss potential limitations or challenges of the proposed method.

Open Questions

  • What are the potential limitations or challenges of EcoFrame?
  • How does EcoFrame perform on other benchmark datasets and tasks?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The text mentions experiments on specific datasets but does not provide detailed information about the evaluation methodology.

Raw Abstract / Excerpt

Efficient long-video understanding requires vision--language models (VLMs) to reason over a small number of frames selected as sparse visual evidence. Existing relevance-based methods rely on static one-shot selection with fixed frame budgets and candidate pools, while agent-based schedulers achieve adaptivity through costly multi-round reasoning and interactive search. We propose EcoFrame, a training-free framework for low-overhead query-adaptive visual evidence scheduling. EcoFrame leverages the VLM's inference feedback to determine when to increase the frame budget and where to search for additional candidate evidence. Specifically, entropy-gated budget scheduling uses output uncertainty to stop early when the current evidence is sufficient or progressively expand the frame budget otherwise. Meanwhile, attention-guided candidate proposal converts frame-level attention into a temporal prior, enabling dense local search in informative regions while preserving global coverage when attention is diffuse. Experiments on Video-MME, LongVideoBench, and MLVU demonstrate that EcoFrame achieves a better accuracy--efficiency trade-off across multiple VLM backbones. On Qwen2.5-VL, EcoFrame achieves an average accuracy of 64.4, surpassing BOLT at 63.5, while providing a $1.85\times$ speedup over AKS and BOLT. Compared with the agent-based A.I.R., EcoFrame maintains comparable accuracy with up to a $13.5\times$ inference speedup. Code will be available at https://github.com/AK-DREAM/EcoFrame.