measurement
4 notes, newest first.
Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark
The preprint audits whether harmful refusal is a single measurable attribute in HELM Safety. It reports saturation in three candidate datasets and uses multidimensional item-response theory plus differential item…
On the estimation and validity of AI time horizons---a statistical look at the METR plot
The preprint recomputes METR-style 50% AI time horizons over 228 tasks and 26 AIs using splines and item-response theory. The authors report that the fitted spline is nearly flat from roughly 2–30 minutes and close to…
How to connect AI usage to business value
OpenAI describes ChatGPT Work and Codex analytics that combine usage, spend, task insights, and outcome metrics. The post describes task classification, model/reasoning/speed breakdowns, plugin and skills views, Codex…
AI Exposure Scores: what they measure, what they miss, and what comes next
The article discusses the limitations of a set of AI Exposure Scores introduced by Eloundou et al., which measure the share of tasks that can be assisted by large language models. The authors highlight two main gaps…
