Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

BayesAME: Bayesian Active Model Evaluation

Source

arXiv

arxiv.org/abs/2607.27023v1

Read the source

Source

Plain-Language Brief

BayesAME is a new Bayesian framework designed to automatically determine the optimal size of a coreset (a subset of items for evaluating large generative models). This method improves upon existing ones by modeling performance as a random variable and using an information-gain criterion to select items. It also includes a multi-target extension that reduces the coreset size by considering performance correlations across multiple target models.

What the Document Says

  • BayesAME is a sequential Bayesian framework for automatically determining the coreset size
  • It models performance as a random variable with a latent ability for each group of items sharing similar historical model performances
  • The joint prior distribution encodes the belief that the target model behaves similarly to these historical models
  • The posterior distribution over these abilities is used to derive performance estimators and quantify performance uncertainty
  • Items are selected to add to the coreset via an information-gain criterion
  • The coreset is iteratively augmented until the performance estimate fluctuation and the performance uncertainty fall below user-defined thresholds
  • A multi-target extension captures performance correlations across multiple target models
  • Through extensive experiments, BayesAME outperforms sequential adaptations of existing methods
  • The method addresses recent skepticism in the literature, establishing that non-random coreset selection is advantageous over random selection
  • Using continuous response log-likelihoods enhances estimation accuracy

Key Concepts Mentioned

  • BayesAME
  • coreset
  • latent ability
  • joint prior distribution
  • performance estimators
  • performance uncertainty
  • information-gain criterion
  • multi-target extension

Claims Made by the Document

  • BayesAME outperforms sequential adaptations of existing methods
  • Non-random coreset selection is advantageous over random selection

Evidence Used

  • Through extensive experiments across diverse benchmarks, we demonstrate that BayesAME consistently outperforms sequential adaptations of existing methods.
  • Crucially, our comprehensive analysis addresses recent skepticism in the literature.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Evaluating large generative models across benchmarks is time-consuming and computationally expensive. This drives the need for methods that can estimate full benchmark performance by evaluating models on only a subset of items, known as a coreset. Current literature mostly requires the practitioner to input a coreset size. However, when reliable performance estimation takes priority over efficiency, an evaluation method should also be capable of automatically determining a coreset size that reflects this priority. We introduce BayesAME, a sequential Bayesian framework specifically targeting automatic determination of the coreset size. BayesAME models performance as a random variable by defining a latent ability for each group of items sharing the same historical model performances, with a joint prior distribution encoding the belief that the target model behaves similarly to these historical models. The posterior distribution over these abilities is used to derive performance estimators, quantify performance uncertainty, and select items to add to the coreset via an information-gain criterion. The coreset is iteratively augmented until the performance estimate fluctuation and the performance uncertainty fall below their respective user-defined thresholds. We propose a multi-target extension that captures performance correlations across multiple target models to further reduce the coreset size. Through extensive experiments across diverse benchmarks, we demonstrate that BayesAME consistently outperforms sequential adaptations of existing methods. Crucially, our comprehensive analysis addresses recent skepticism in the literature, establishing that non-random coreset selection is advantageous over random selection. Finally, we highlight that leveraging continuous response log-likelihoods over traditional binary scores significantly enhances estimation accuracy.