Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild

Source

arXiv

arxiv.org/abs/2607.13881v1

Read the source

Source

Plain-Language Brief

The paper presents a new framework called AgentHOI that uses large language models to detect human-object interactions in complex scenes without needing any specific training data. It focuses on open-ended semantic reasoning and spatial grounding, with two key mechanisms for improving interaction discovery and localization.

What the Document Says

  • The paper introduces AgentHOI, a training-free framework that uses a large language model to detect human-object interactions (HOI) in complex scenes.
  • AgentHOI modularly orchestrates complementary vision foundation modules for open-ended semantic reasoning and spatial grounding.
  • Two key mechanisms are introduced: Context-aware Multi-round Reasoning and Multifaceted Interaction Localization, to improve interaction discovery and localization in complex scenes.
  • The paper claims that AgentHOI outperforms state-of-the-art methods in real-world settings without needing HOID training data.

Key Concepts Mentioned

  • Human-object interaction detection
  • Multimodal large language models
  • Training-free framework
  • Open-ended semantic reasoning
  • Spatial grounding
  • Context-aware Multi-round Reasoning
  • Multifaceted Interaction Localization

Claims Made by the Document

  • AgentHOI outperforms state-of-the-art methods in real-world settings without needing HOID training data.

Evidence Used

  • Extensive experiments demonstrate that AgentHOI achieves superior performance over state-of-the-art supervised and weakly supervised methods in real-world settings, despite requiring no HOID data for training.

Methods / Approach

  • AgentHOI modularly orchestrates complementary vision foundation modules
  • Context-aware Multi-round Reasoning
  • Multifaceted Interaction Localization

Limitations Stated by the Authors

  • UNKNOWN
  • Not stated in source

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The paper does not provide a clear comparison between AgentHOI and existing prompt-based approaches.
  • The performance of AgentHOI is only demonstrated on real-world settings, but its performance on other types of data or scenarios is unknown.

Open Questions

  • What specific vision foundation modules are used by AgentHOI?
  • How does AgentHOI handle interactions that are not compositional or fall under unseen categories?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The paper focuses on real-world settings, so it would be interesting to see how AgentHOI performs in other types of data or scenarios.

Raw Abstract / Excerpt

Human-object interaction detection (HOID) has traditionally been formulated as a supervised detection problem over predefined interaction categories. While such paradigms achieve strong performance on closed-set benchmarks, they fundamentally entangle interaction understanding with dataset-specific supervision, limiting their ability to generalize to open-world and compositional scenarios. Recent HOI detectors attempt to leverage MLLMs through prompting strategies to transfer interaction-specific knowledge. However, such prompt-based approaches primarily focus on extracting discriminative representations from pretrained models, while underexploring their inherent multimodal reasoning capabilities. As a result, they struggle to provide informative contextual reasoning for ambiguous and open-world interaction scenarios. In this work, we present AgentHOI, a training-free, agentic framework that transfers the generalist multimodal reasoning capabilities of foundation models to HOI detection in the wild. Instead of learning interaction classifiers, AgentHOI modularly orchestrates complementary vision foundation modules to perform open-ended semantic reasoning and spatial grounding in a coordinated manner. To address the challenges of incomplete interaction discovery and ambiguous localization in complex scenes, we introduce two key mechanisms: (1) Context-aware Multi-round Reasoning, which progressively refines interaction hypotheses to ensure exhaustive and compositional HOI discovery, and (2) Multifaceted Interaction Localization, which enhances grounding precision by generating instance-specific descriptions that integrate semantic, spatial, and appearance cues. Extensive experiments demonstrate that AgentHOI achieves superior performance over state-of-the-art supervised and weakly supervised methods in real-world settings, despite requiring no HOID data for training.