Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Sequential Fairness Auditing with Limited Output Access

Source

arXiv

arxiv.org/abs/2606.30338v1

Read the source

Source

Plain-Language Brief

This research proposes a new method for auditing AI systems for fairness in real-world scenarios where auditors have limited access to deployed models. The method is a tolerance-aware sequential hypothesis-testing problem that allows auditors to collect evidence over time and stop once enough support for compliance or violation has been obtained. The framework is applicable to decision-based fairness audits and can be extended to score- and logit-based proxy audits with richer observables. Results show that the method's efficiency varies based on the fairness metric and level of model access, with richer outputs reducing the number of queries required in some cases but offering limited gains in near-threshold cases.

What the Document Says

  • The paper discusses sequential fairness auditing for AI systems.
  • Auditors have limited access to deployed models and rely on query-based interactions.
  • Most existing methods assume static datasets and fixed-sample statistical tests, making them inadequate for real-world scenarios.
  • The proposed method formulates fairness auditing as a tolerance-aware sequential hypothesis-testing problem under limited model output access.
  • A sequential generalized likelihood-ratio framework is developed to accumulate evidence from a finite audit pool.
  • The framework is applicable to decision-based Statistical Parity and Equal Opportunity audits, and extended for score- and logit-based proxy audits.
  • Results indicate that both the fairness metric and level of model access affect audit efficiency.
  • Richer outputs can reduce the number of queries required in some cases but offer limited gains in near-threshold cases.

Key Concepts Mentioned

  • Fairness auditing
  • Sequential hypothesis testing
  • Limited model output access
  • Statistical Parity
  • Equal Opportunity
  • Score-based proxy audits
  • Logit-based proxy audits

Claims Made by the Document

  • The proposed method is practical for sequential fairness auditing under realistic deployment constraints.
  • Richer outputs can reduce the number of queries required for some fairness metrics and operating regimes.

Evidence Used

  • The paper presents a new statistical framework for sequential fairness auditing.
  • Results from the application of this framework are provided in the paper.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

External evaluations are becoming increasingly central to the governance of AI systems. In practice, however, independent auditors often have limited access to deployed models and must rely on query-based interactions. Most existing fairness evaluation methods assume static datasets and fixed-sample statistical tests, making them poorly suited to real-world auditing scenarios in which evidence must be collected sequentially under query constraints. In this work, we formulate fairness auditing as a tolerance-aware sequential hypothesis-testing problem under limited model output access. We develop a sequential generalized likelihood-ratio framework that allows auditors to accumulate evidence from a finite audit pool and stop once sufficient support for compliance or violation has been obtained. The framework is instantiated for decision-based Statistical Parity and Equal Opportunity audits, and extended to score- and logit-based proxy audits when richer observables are available. Our results show that both the fairness metric and the level of model access significantly affect audit efficiency, and that the benefits of richer output information are not uniform across auditing settings. In particular, richer outputs can substantially reduce the number of queries required for some fairness metrics and operating regimes, while offering limited gains in near-threshold cases. This work provides a practical statistical framework for sequential fairness auditing under realistic deployment constraints.