From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Laurens Samson, Iva Gornishka, Gossa Lô, Yuki M. Asano, Sennay Ghebreab
• Venue: arXiv: cs.CL
• Published: 2026-08-10
• Collected: 2026-08-11
• URL: http://arxiv.org/abs/2608.09925v1
• PDF: https://arxiv.org/pdf/2608.09925v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

The paper presents a new evaluation framework called 'Grip on LLMs' specifically designed for Dutch governmental use of large language models (LLMs). The framework is created in collaboration with domain experts and identifies six key evaluation dimensions for these models. These dimensions include factuality, honesty, social bias, energy consumption, cost, and training data transparency. The authors find that no single model excels across all dimensions and that trade-offs are inevitable, as higher quality often comes at a greater environmental impact and financial cost.

What the Document Says

• The 'Grip on LLMs' evaluation framework was developed for Dutch governmental use of large language models.
• The framework was created in collaboration with domain experts from a major Dutch municipal organization.
• Six evaluation dimensions were identified: factuality, honesty, social bias, energy consumption, cost, and training data transparency.
• A benchmark suite covering more than 30 multilingual and Dutch-specific models was developed to operationalise these dimensions.
• The results reveal that no single model excels across all evaluation dimensions.
• Trade-offs are inevitable, as higher quality often comes at a greater environmental impact and financial cost.
• Bias remains largely independent of both quality, environmental impact, and financial cost.

Key Concepts Mentioned

• large language models
• evaluation framework
• Dutch governmental use
• domain experts
• evaluation dimensions
• benchmark suite

Claims Made by the Document

• The 'Grip on LLMs' framework was developed for Dutch governmental use.
• No single model excels across all evaluation dimensions.
• Trade-offs are inevitable in large language models.
• Higher quality often comes at a greater environmental impact and financial cost.

Evidence Used

• The paper describes the development of the 'Grip on LLMs' framework.
• The paper presents the six evaluation dimensions and their operationalisation.

Methods / Approach

• Advisory board process, user research, and a survey of users of a civil-servant chatbot

Limitations Stated by the Authors

• The authors note that they only evaluated more than 30 models but do not specify the total number of models available.

Limitations Noted by the Agent

Agent interpretation — not from the source.

• The paper does not provide data or specific results to verify the claims about the trade-offs between quality, environmental impact, and financial cost.

Open Questions

• What is the total number of large language models available?
• What are the exact results of the evaluation that show the trade-offs between quality, environmental impact, and financial cost?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The paper does not mention any specific large language models by name.

Raw Abstract / Excerpt

Large language models are increasingly being deployed in governmental settings, yet few existing evaluation frameworks jointly reflect the values of public administration and the linguistic requirements of non-English contexts. We present the 'Grip on LLMs' framework, a systematic evaluation suite for Dutch governmental use developed in collaboration with domain experts from a major Dutch municipal organisation. Through an advisory board process, user research, and a survey of the users of a civil-servant chatbot, we identify six evaluation dimensions (factuality, honesty, social bias, energy consumption, cost, and training data transparency) and operationalise them into a benchmark suite covering more than 30 multilingual and Dutch-specific models. Our results reveal that no single model excels across all dimensions, and that trade-offs are unavoidable: higher quality consistently comes at greater environmental impact and financial cost, while bias remains largely independent of both. We further find that factuality (whether a model answers correctly) and honesty (whether a model acknowledges what it does not know) are governed by distinct properties, with high factuality not implying high honesty. To make these findings actionable for non-technical audiences, we release a publicly accessible, user-friendly model overview designed for the full range of stakeholders involved in governmental LLM selection, from engineers to policymakers.