Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Validity of LLMs as data annotators: AMALIA on authority

Source

arXiv

arxiv.org/abs/2607.08731v1

Read the source

Source

Plain-Language Brief

A study tests the validity of a large language model (LLM), AMALIA, in measuring moral foundation concepts for European Portuguese. The study finds that while AMALIA agrees with human coders on these concepts, it does not fully understand them as intended. The study suggests that future benchmark batteries for LLMs should test not only agreement with human coders but also the evidence-based reasoning behind those agreements.

What the Document Says

  • AMALIA is a Portuguese language model that agrees with trained human coders on the moral foundation of authority within six F1 points.
  • Agreement does not guarantee validity, as for theoretical constructs, it's unclear if AMALIA follows the construct's theory or uses correlated shortcuts.
  • The study tests this using the recovery gap: performance loss when a holistic prompt is broken down and recombined by the theory's rule.
  • Calibration closing the gap indicates some portability across models and languages, but AMALIA's calibration does not close the gap for one construct and corpus.
  • Error analysis suggests AMALIA relies on surface correlates, such as moral outrage near authority figures.
  • An open multilingual LLM closes the gap on the same Portuguese corpus under the same instructions, suggesting the corpus is not the main explanation.
  • The study concludes that while AMALIA can pre-code at scale, it cannot yet measure this construct well enough to stand alone.

Key Concepts Mentioned

  • Large Language Models (LLMs)
  • AMALIA
  • European Portuguese
  • moral foundation
  • authority
  • agreement
  • validity
  • recovery gap
  • calibration
  • portability

Claims Made by the Document

  • AMALIA agrees with human coders on the moral foundation of authority.
  • Agreement does not guarantee validity for theoretical constructs.
  • Calibration can indicate portability across models and languages.
  • AMALIA's calibration does not close the recovery gap for one construct and corpus.
  • Error analysis suggests AMALIA relies on surface correlates.

Evidence Used

  • AMALIA's agreement with human coders within six F1 points.
  • Performance loss when a holistic prompt is broken down and recombined by the theory's rule.
  • Error analysis results.

Methods / Approach

  • Recovery gap test
  • Calibration testing
  • Error analysis

Limitations Stated by the Authors

The study is a single counterexample, not a verdict on national models.

Limitations Noted by the Agent

Agent interpretation — not from the source.

The study only tests one construct and corpus, so the findings may not be applicable to other concepts or datasets.

Open Questions

  • What are the implications of these findings for other large language models?
  • How can we improve the validity of LLMs in measuring moral foundation concepts?

Notes for Future Q&A

May include interpretation. Labeled where it does.

The study discusses the importance of testing not only agreement with human coders but also the evidence-based reasoning behind those agreements.

Raw Abstract / Excerpt

A national language model offers a linguistic community its own instrument for measuring what its citizens say and value. Portugal's AMALIA, a publicly funded 9B-parameter model for European Portuguese, appears competitive on agreement alone: asked to code the moral foundation of authority, it agrees with trained human coders to within six F1 points of open models eight to thirteen times its size. Yet agreement is reliability, not validity. For theoretical constructs that must be inferred rather than read from surface features, the question is whether the model follows the construct's theory or reaches the right code by correlated shortcuts. We test this with the recovery gap: the loss in performance when a holistic prompt is decomposed into the codebook's atomic clauses and recombined by the theory's explicit rule. If calibration closes that gap, some portability should survive across models and languages; where it does not, the construct-model instrument is the likely locus of failure. We ask whether a calibrated English instrument transfers to AMALIA-9B and to European Portuguese. For one construct and one corpus, it does not. Decomposition recovers only about half of AMALIA's holistic performance, and error analysis suggests reliance on surface correlates, especially moral outrage near authority figures. An open multilingual LLM closes the gap on the same Portuguese corpus under the same instructions, pointing away from the corpus as the main explanation. AMALIA can still screen and pre-code at scale, but it cannot yet measure this construct well enough to stand alone. The study is a single counterexample, not a verdict on national models; it argues that sovereign-LLM benchmark batteries should test not only agreement with human coders, but the evidential route by which that agreement is warranted.