Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability

Source

arXiv

arxiv.org/abs/2606.24589v1

Read the source

Source

Plain-Language Brief

The paper introduces AdversaBench, a red-teaming pipeline for evaluating large language models. The pipeline generates hard inputs using five structured operators and confirms failures through a three-judge panel with a meta-judge tiebreaker. The experiments were conducted on 45 seeds across three categories: reasoning, instruction-following, and tool use. Every seed produced a confirmed failure. The findings include variations in operator effectiveness by category, hidden difficulty in binary failure rate, low judge agreement but high category-level disagreement rates, and the transfer of adversarial prompts between Llama models.

What the Document Says

  • AdversaBench is an end-to-end red-teaming pipeline for evaluating large language models.
  • The pipeline generates hard inputs using five structured operators (inject_distractor, insert_contradiction, ask_for_evidence, swap_entities, and rephrase_query).
  • A target model is queried after the input mutations.
  • Failures are confirmed through a three-judge panel with a meta-judge tiebreaker.
  • The experiments were conducted on 45 seeds across three categories: reasoning, instruction-following, and tool use.
  • Every seed produced a confirmed failure.
  • Operator effectiveness varies sharply by category.
  • Inject_distractor scores 0.00 mean reward on instruction-following seeds but 0.80-0.83 on reasoning and tool-use.
  • Binary failure rate hides difficulty; instruction-following seeds required 2.4 attacker iterations on average versus 1.1 for other categories.
  • Pairwise judge agreement is 80-87%, but near-zero Cohen's kappa due to label skew.
  • Category-level disagreement rates are more informative.
  • Adversarial prompts generated against Llama 3.1 8B do not transfer zero-shot to Llama 3.3 70B.

Key Concepts Mentioned

  • AdversaBench
  • red-teaming pipeline
  • large language models
  • structured operators
  • target model
  • three-judge panel
  • binary failure rate
  • Cohen's kappa
  • Llama

Claims Made by the Document

  • AdversaBench is effective in confirming failures of large language models.
  • Operator effectiveness varies by category.
  • Binary failure rate hides difficulty.
  • Pairwise judge agreement is high but Cohen's kappa is low due to label skew.
  • Category-level disagreement rates are more informative.
  • Adversarial prompts generated against one Llama model do not transfer zero-shot to another.

Evidence Used

  • The experiments produced confirmed failures for every seed.
  • The variations in operator effectiveness by category were observed.
  • The binary failure rate hid difficulty, as seen in the survival curves.

Methods / Approach

  • End-to-end red-teaming pipeline
  • Three-judge panel with meta-judge tiebreaker
  • Five structured operators
  • Queries to a target model

Limitations Stated by the Authors

  • Label skew leads to low Cohen's kappa, affecting the interpretation of pairwise judge agreement.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The paper does not discuss how the authors handled potential biases in the selection of seeds or judges.
  • The paper does not provide details on how the meta-judge tiebreaker works.

Open Questions

  • What are the specific biases that may have been present in the seed and judge selection process?
  • How exactly does the meta-judge tiebreaker work?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The authors did not discuss potential biases in the seed or judge selection process.

Raw Abstract / Excerpt

Scaling adversarial evaluation of large language models requires both a method for generating hard inputs and a reliable way to confirm that resulting failures are real. We present AdversaBench, an end-to-end red-teaming pipeline that mutates seed prompts with five structured operators, queries a target model, and confirms failures through a three-judge panel with a meta-judge tiebreaker. We report experiments on 45 seeds across three categories: reasoning, instruction-following, and tool use. Every seed produced a confirmed failure. Four findings stand out. First, operator effectiveness varies sharply by category: inject_distractor scores 0.00 mean reward on instruction-following seeds but 0.80-0.83 on reasoning and tool-use. Second, binary failure rate hides difficulty: instruction-following seeds required 2.4 attacker iterations on average versus 1.1 for other categories, a gap visible in survival curves. Third, pairwise judge agreement of 80-87% coexists with near-zero Cohen's kappa due to label skew; category-level disagreement rates are more informative. Fourth, adversarial prompts generated against Llama 3.1 8B transfer zero-shot to Llama 3.3 70B, suggesting the mutations exploit general behavioral patterns rather than model-specific weaknesses. Code, dataset, and analysis scripts are available at https://github.com/khanak0509/AdversaBench .