Field notes
2,343 notes from my AI research vault, mapped by topic. Pick a topic to see its notes, or search.
Click the web, then scroll to zoom
Every note is in the list below. The web of topics draws here once the page has loaded.
5 of 2,343 notes match.
- Question answered
- Source note
- Topic
- Shared notes
Notes on item response theory
5 notes
Item Response Theory for AI Safety
The paper presents a study using Item Response Theory (IRT) to analyze the safety of language models. The authors find that three factors - refusal strictness, truthfulness, and contextual harm - explain most of the…
Item Response Theory for AI Safety
Researchers used Item Response Theory (IRT), a statistical toolkit, to analyze the safety performance of 192 language models across eight different safety benchmarks. The study found that three key factors - refusal…
Can We Trust Item Response Theory for AI Evaluation?
This research examines the reliability of using Item Response Theory (IRT) for evaluating AI performance in benchmarks. The study finds that classical estimators may become infeasible in large benchmark settings, while…
Can We Trust Item Response Theory for AI Evaluation?
The paper investigates the reliability of Item Response Theory (IRT) for evaluating AI systems in benchmarks. The study finds that traditional IRT estimation tools may not be suitable for AI benchmark data, which often…
Can We Trust Item Response Theory for AI Evaluation?
The paper explores the reliability of using Item Response Theory (IRT) for evaluating Artificial Intelligence (AI) systems in benchmarks. The authors find that standard IRT estimation tools may not be reliable in large…
An AI research agent drafts these notes from public papers and posts, and each one keeps its source link. Only the folders I choose to publish leave my Obsidian vault.
