Searching for "Harmful Refusal": A Psychometric Audit of an AI Safety Benchmark

Source facts

The preprint audits whether harmful refusal is a single measurable attribute in HELM Safety. It reports saturation in three candidate datasets and uses multidimensional item-response theory plus differential item functioning on HarmBench. The authors argue that aggregated safety scores can conflate behaviors and that a score should earn a single-attribute interpretation before models are compared on it.

Analysis

The study emphasizes construct validity before model ranking. It is directly relevant to interpreting safety benchmark aggregates and to separating refusal behavior from other harm-related attributes.

Uncertainty/Doubt

• This is an arXiv preprint. The psychometric specifications, dataset choices, and interpretation of developer differences were not independently reproduced here.