language model evaluation
7 notes, newest first.
From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch
Researchers present a new evaluation framework called 'Grip on LLMs' for Dutch governmental use of large language models. The framework focuses on six dimensions: factuality, honesty, social bias, energy consumption…
One Human, $N$ Agents: Audit-Budget Allocation for LLM Agent Fleets under Miscalibrated, Correlated Confidence
The paper discusses a scenario where a single human auditor has to evaluate multiple LLM (Language Learning Models) agents under a limited budget. The authors model this process as a noisy inspection over a two-level…
Reason-Mediated Behavioral Models for Auditing LLM Social Simulators
Researchers propose a new method to evaluate large language models used as social simulators, including synthetic survey respondents. They argue that existing evaluation methods are insufficient because these models…
RUMBA: Russian User Memory Benchmark
Researchers introduce a new benchmark called RUMBA for evaluating the long-term conversational memory abilities of Language Models (LLMs), particularly focusing on Russian dialogues. The benchmark provides a detailed…
Judge-dependent safety gains and model-specific helpfulness costs of evidence-sufficiency prompting in clinical LLMs
A study investigates the impact of a structured evidence-sufficiency prompt on reducing unsafe overconfident answers from four different language models in a medical context. The study finds that the prompt…
Validity of LLMs as data annotators: AMALIA on authority
A study evaluates AMALIA, a Portuguese language model, in measuring moral values and authority. The model shows agreement with human coders but fails to perform well when the test is broken down into smaller parts…
AU-Harness: An Open-Source Toolkit for Holistic Evaluation of AudioLLMs
The authors introduce AU-Harness, an efficient and comprehensive evaluation framework for Large Audio Language Models (LALMs). The system addresses limitations in existing toolkits by improving speed, providing…
