Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note2 min read

Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild

Source

arXiv

arxiv.org/abs/2607.13881v1

Read the source

Source

Plain-Language Brief

The paper presents a new framework called AgentHOI that uses multimodal large language models (MLLMs) to detect human-object interactions (HOIs) in complex scenes without the need for any HOI data during training. The framework employs two key mechanisms - Context-aware Multi-round Reasoning and Multifaceted Interaction Localization - to address challenges like incomplete interaction discovery and ambiguous localization.

What the Document Says

  • The paper discusses a new training-free framework called AgentHOI for HOI detection using MLLMs.
  • AgentHOI orchestrates complementary vision foundation modules for open-ended semantic reasoning and spatial grounding.
  • Two key mechanisms are introduced: Context-aware Multi-round Reasoning and Multifaceted Interaction Localization.
  • The framework demonstrates superior performance in real-world settings compared to state-of-the-art supervised and weakly supervised methods.

Key Concepts Mentioned

  • AgentHOI
  • MLLMs
  • Human-object interaction detection (HOID)
  • Context-aware Multi-round Reasoning
  • Multifaceted Interaction Localization

Claims Made by the Document

  • AgentHOI achieves superior performance over state-of-the-art methods in real-world settings.
  • The framework does not require any HOID data for training.

Evidence Used

  • Extensive experiments demonstrate that AgentHOI achieves superior performance... Despite requiring no HOID data for training.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Human-object interaction detection (HOID) has traditionally been formulated as a supervised detection problem over predefined interaction categories. While such paradigms achieve strong performance on closed-set benchmarks, they fundamentally entangle interaction understanding with dataset-specific supervision, limiting their ability to generalize to open-world and compositional scenarios. Recent HOI detectors attempt to leverage MLLMs through prompting strategies to transfer interaction-specific knowledge. However, such prompt-based approaches primarily focus on extracting discriminative representations from pretrained models, while underexploring their inherent multimodal reasoning capabilities. As a result, they struggle to provide informative contextual reasoning for ambiguous and open-world interaction scenarios. In this work, we present AgentHOI, a training-free, agentic framework that transfers the generalist multimodal reasoning capabilities of foundation models to HOI detection in the wild. Instead of learning interaction classifiers, AgentHOI modularly orchestrates complementary vision foundation modules to perform open-ended semantic reasoning and spatial grounding in a coordinated manner. To address the challenges of incomplete interaction discovery and ambiguous localization in complex scenes, we introduce two key mechanisms: (1) Context-aware Multi-round Reasoning, which progressively refines interaction hypotheses to ensure exhaustive and compositional HOI discovery, and (2) Multifaceted Interaction Localization, which enhances grounding precision by generating instance-specific descriptions that integrate semantic, spatial, and appearance cues. Extensive experiments demonstrate that AgentHOI achieves superior performance over state-of-the-art supervised and weakly supervised methods in real-world settings, despite requiring no HOID data for training.