FIELD NOTES TOPIC
statistical models
2 notes, newest first.
Can We Trust Item Response Theory for AI Evaluation?
The paper examines the reliability of using Item Response Theory (IRT) for evaluating AI systems in large-scale benchmarks. The authors simulate response matrices under three common IRT models, comparing four…
Can We Trust Item Response Theory for AI Evaluation?
This research investigates the reliability of using Item Response Theory (IRT) for evaluating Artificial Intelligence (AI). The study finds that classical estimators can become infeasible in large benchmark settings…
