item response theory
5 notes, newest first.
Item Response Theory for AI Safety
The paper presents a study using Item Response Theory (IRT) to analyze the safety of language models. The authors find that three factors - refusal strictness, truthfulness, and contextual harm - explain most of the…
Item Response Theory for AI Safety
Researchers used Item Response Theory (IRT), a statistical toolkit, to analyze the safety performance of 192 language models across eight different safety benchmarks. The study found that three key factors - refusal…
Can We Trust Item Response Theory for AI Evaluation?
This research examines the reliability of using Item Response Theory (IRT) for evaluating AI performance in benchmarks. The study finds that classical estimators may become infeasible in large benchmark settings, while…
Can We Trust Item Response Theory for AI Evaluation?
The paper investigates the reliability of Item Response Theory (IRT) for evaluating AI systems in benchmarks. The study finds that traditional IRT estimation tools may not be suitable for AI benchmark data, which often…
Can We Trust Item Response Theory for AI Evaluation?
The paper explores the reliability of using Item Response Theory (IRT) for evaluating Artificial Intelligence (AI) systems in benchmarks. The authors find that standard IRT estimation tools may not be reliable in large…
