Beyond Trial-and-Error: Agentic Optimization for Image-to-Video Adherence

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Aman Tyagi, Hemanth Boinpally, Jonathan Chen, Douglas Gebert, Steven Hickson
• Venue: arXiv: cs.CV
• Published: 2026-08-12
• Collected: 2026-08-13
• URL: http://arxiv.org/abs/2608.12290v1
• PDF: https://arxiv.org/pdf/2608.12290v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

Researchers have developed a new framework called 'Agentic Self-Improvement' to improve the performance of Image-to-Video (I2V) models, which are commonly used for automated content creation. The new framework aims to address the challenges caused by the stochastic nature of I2V models, such as their lack of fine-grained control and reliability in professional workflows. To achieve this, the framework introduces an iterative prompt optimization loop that uses a multimodal Large Language Model (mLLM) to refine input prompts and ensures semantic adherence and artifact detection through two automated evaluations: Davidsonian Scene Graph (DSG) queries and Common Mistake Questions (CMQ). The framework also uses Bayesian optimization for co-optimizing stochastic seeds and CFG scales. The results show that the framework significantly outperforms unguided search methods, with human preference studies indicating that videos generated using the agentic approach are preferred over baseline outputs in up to 69% of cases.

What the Document Says

• The paper introduces a new framework named 'Agentic Self-Improvement' for improving Image-to-Video (I2V) models.
• The primary goal is to address the limitations caused by the stochastic nature of I2V models, such as lack of fine-grained control and reliability.
• The framework uses an iterative prompt optimization loop with a multimodal Large Language Model (mLLM).
• Two automated evaluations are implemented: Davidsonian Scene Graph (DSG) queries for semantic adherence, and Common Mistake Questions (CMQ) for artifact detection.
• Bayesian optimization is used to co-optimize stochastic seeds and CFG scales.
• A suite of quality metrics is employed, including the Video-Text Adherence (VTA) score derived from DSG and CMQ evaluations.
• The framework outperforms unguided search methods in human preference studies.
• In these studies, videos generated via the agentic approach were preferred over baseline outputs, achieving win rates up to 69%.

Key Concepts Mentioned

• Image-to-Video models
• stochasticity
• Agentic Self-Improvement framework
• iterative prompt optimization loop
• multimodal Large Language Model (mLLM)
• Davidsonian Scene Graph (DSG) queries
• Common Mistake Questions (CMQ)
• Bayesian optimization
• quality metrics
• Video-Text Adherence (VTA) score

Claims Made by the Document

• The new framework significantly outperforms unguided search methods.
• In human preference studies, videos generated via the agentic approach were preferred over baseline outputs.

Evidence Used

• In human preference studies, videos generated via our agentic approach were strongly preferred over baseline outputs, achieving win rates up to 69%.

Methods / Approach

• Iterative prompt optimization loop
• Automated evaluations (DSG queries and CMQ)
• Bayesian optimization

Limitations Stated by the Authors

• UNKNOWN
• The paper does not discuss any limitations mentioned by the authors.

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The effectiveness of the framework in various real-world scenarios is unknown.
• It is unclear how the framework handles complex or ambiguous input prompts.
• The generalizability of the results to different I2V models is uncertain.

Open Questions

• How does the framework handle complex or ambiguous input prompts?
• To what extent can the framework be generalized to different Image-to-Video models?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The paper discusses the Agentic Self-Improvement framework for improving the performance of Image-to-Video models, but it does not delve into its limitations.
• Further research or analysis might be required to better understand the framework's effectiveness in various scenarios and generalizability.

Raw Abstract / Excerpt

Modern black-box Image-to-Video (I2V) models offer powerful capabilities in automated content creation, yet their lack of fine-grained control and reliability presents significant challenges in professional workflows. Their inherent stochasticity causes minor variations in textual prompts or hyperparameters to yield drastically different outputs often necessitating inefficient, brute-force trial-and-error processes. To address these limitations, we introduce the ``Agentic Self-Improvement' framework, which reframes video synthesis into a closed-loop, goal-directed optimization. Our framework systematically navigates the generation parameter space using a novel two-stage approach. In the first stage, an iterative prompt optimization loop uses a multimodal Large Language Model (mLLM) to refine the input prompt. This refinement implements two automated evaluations: Davidsonian Scene Graph (DSG) queries ensure semantic adherence, and Common Mistake Questions (CMQ) for artifact detection. At the second stage, we use Bayesian optimization to efficiently co-optimize stochastic seeds and CFG scales. This search is guided by a suite of quality metrics, including the novel Video-Text Adherence (VTA) score derived from the DSG and CMQ evaluations. Our framework significantly outperforms unguided search methods: in human preference studies, videos generated via our agentic approach were strongly preferred over baseline outputs, achieving win rates up to 69%. This work provides a practical and extensible methodology for enhancing the predictability and control of state-of-the-art video generation models, moving the field beyond speculative curiosities toward reliable, production-ready tools.