VocaDet: Sample-Driven Open-Vocabulary Object Detection and Segmentation via Visual Tokenization and Vector Database Retrieval

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: ZhiXin Sun
• Venue: arXiv: cs.CV
• Published: 2026-07-09
• Collected: 2026-07-11
• URL: http://arxiv.org/abs/2607.08541v1
• PDF: https://arxiv.org/pdf/2607.08541v1
• DOI: UNKNOWN

Plain-Language Brief

The text presents a new framework called VocaDet for open-vocabulary object detection and segmentation. This framework learns object concepts directly from user-provided positive and negative sample collections without the need for model retraining. The key idea is to transform continuous visual representations into discrete visual vocabularies and perform efficient retrieval-based recognition through a scalable vector database.

What the Document Says

• VocaDet is a new framework for open-vocabulary object detection and segmentation.
• It learns object concepts directly from user-provided positive and negative sample collections without model retraining.
• The framework transforms continuous visual representations into discrete visual vocabularies.
• Recognition is performed through a scalable vector database.
• DINOv3 is used as the visual feature extractor.
• Agglomerative clustering with adaptive clustering sensitivity is applied to generate multi-granularity visual tokens.
• These visual tokens, position-debiased representations, and spatial topology information are stored in a vector database as expandable object memories.
• During inference, query images are converted into visual tokens and matched against the stored object memories for object localization and segmentation.
• A background filtering mechanism is introduced to remove frequently occurring background patterns.
• Experiments on the UA-DETRAC dataset demonstrate effective open-vocabulary detection performance without conventional detector training, while supporting continuous expansion as more samples are accumulated.

Key Concepts Mentioned

• VocaDet
• open-vocabulary object detection and segmentation
• user-provided positive and negative sample collections
• visual vocabularies
• scalable vector database
• DINOv3
• agglomerative clustering
• position-debiased representations
• spatial topology information
• background filtering mechanism

Claims Made by the Document

• VocaDet can effectively perform open-vocabulary object detection and segmentation without conventional detector training.
• The framework supports continuous expansion as more samples are accumulated.

Evidence Used

• Experiments on the UA-DETRAC dataset

Methods / Approach

• Transforming visual representations into discrete visual vocabularies
• Storing expandable object memories in a vector database
• Retrieval-based recognition
• Background filtering mechanism

Limitations Stated by the Authors

The text does not mention any limitations according to the authors.

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The method requires user-provided positive and negative sample collections for learning object concepts.
• The effectiveness of the framework may depend on the quality and quantity of provided samples.

Open Questions

• What is the specific performance of VocaDet compared to other methods in terms of speed, accuracy, and scalability?
• How does the background filtering mechanism handle complex scenes with multiple objects and background patterns?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The text implies that VocaDet is efficient but does not provide specific performance metrics.
• The background filtering mechanism is mentioned but its details are not explained.

Raw Abstract / Excerpt

Open-vocabulary object detection and segmentation aim to recognize arbitrary objects beyond predefined categories. Although recent vision-language and reference-based approaches have significantly advanced this field, they often rely on text prompts, limited visual examples, or expensive feature matching procedures, making them difficult to scale to large and continuously expanding object repositories. In this work, we propose VocaDet, a sample-driven open-vocabulary object detection and segmentation framework that learns object concepts directly from user-provided positive and negative sample collections without model retraining. The key idea is to transform continuous visual representations into discrete visual vocabularies and perform efficient retrieval-based recognition through a scalable vector database. Specifically, we employ DINOv3 as the visual feature extractor and apply agglomerative clustering with adaptive clustering sensitivity to generate multi-granularity visual tokens. These visual tokens, together with position-debiased representations and spatial topology information, are stored as expandable object memories in a vector database. During inference, query images are converted into visual tokens and efficiently matched against the stored object memories for object localization and segmentation. Furthermore, a background filtering mechanism is introduced to remove frequently occurring background patterns and reduce redundant retrieval operations in practical fixed-camera scenarios. Experiments on the UA-DETRAC dataset demonstrate that VocaDet achieves effective open-vocabulary detection performance without conventional detector training, while supporting continuously expandable recognition capability as additional positive and negative samples are accumulated.