AutoRelAnnotator: Calibrated Model Cascades for Cost-Efficient Relevance Evaluation in Sponsored Search

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Md Omar Faruk Rokon, Shasvat Desai, Hong Yao, Kuang-chih Lee
• Venue: arXiv: cs.IR
• Published: 2026-06-24
• Collected: 2026-06-25
• URL: http://arxiv.org/abs/2606.25871v1
• PDF: https://arxiv.org/pdf/2606.25871v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers propose a method to generate high-quality relevance annotations at a lower cost without the need for human labeling. This is important for training data preparation, NDCG evaluation, and root cause analysis in search ranking systems. The method involves a calibrated model cascade that routes queries through progressively larger fine-tuned classifiers. The authors demonstrate the effectiveness of this system by processing over 150 million annotations across six offline use cases.

What the Document Says

• The paper presents a method for cost-efficient relevance annotation called calibrated model cascade.
• This approach is designed to improve the accuracy of search ranking systems without human annotation.
• Accuracy and cost are considered as separate optimization targets.
• Domain-specific fine-tuning improves accuracy, while cascading reduces compute cost.
• Per-class isotonic calibration is introduced as a component of the cascade, providing a small but significant gain.
• The system was tested in production across six offline use cases and processed over 150 million annotations.

Key Concepts Mentioned

• relevance annotation
• search ranking systems
• cost-efficient
• calibrated model cascade
• domain-specific fine-tuning
• isotonic calibration

Claims Made by the Document

• Fine-tuning contributes 20 accuracy points.
• Cascading is approximately accuracy-neutral but halves compute cost.
• Per-class isotonic calibration provides a small but statistically significant gain (+0.6 points).

Evidence Used

• The paper presents experimental results showing the performance of the proposed method.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

How can we generate high-quality relevance annotations at scale without the cost and delays of human labeling? Relevance annotations are the backbone of search ranking systems which is needed for training data preparation, NDCG evaluation, and root cause analysis. However, human annotation is slow and off-the-shelf LLMs suffer from accuracy on domain-specific tasks. We propose a calibrated model cascade, a systematic approach for cost-efficient offline relevance annotation by routing queries through progressively larger fine-tuned classifiers. Our central insight is that accuracy and cost are orthogonal optimizations: domain-specific fine-tuning drives accuracy, cascading drives cost, and per-class isotonic calibration adds a small but reliable gain on top. Our contribution is threefold: (a) we decompose the gains and show that fine-tuning contributes 20 accuracy points while cascading is approximately accuracy-neutral but halves compute cost, (b) we introduce per-class isotonic calibration as one component of the cascade, contributing a small but statistically significant gain (+0.6 points over the strongest calibration baseline), and (c) we validate the system in production across six offline use cases, processing 150M+ annotations and enabling faster experimentation cycles. Our work is a building block for scalable, high-quality offline annotation pipelines in search and advertising systems.