Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Validity of LLMs as data annotators: AMALIA on authority

Source

arXiv

arxiv.org/abs/2607.08731v1

Read the source

Source

Plain-Language Brief

A study evaluates AMALIA, a Portuguese language model, in measuring moral values and authority. The model shows agreement with human coders but fails to perform well when the test is broken down into smaller parts. This suggests it may rely on surface correlates rather than understanding the theory behind the construct. The study questions whether national models should only be tested for agreement with human coders, but also the evidential route of that agreement.

What the Document Says

  • AMALIA, a Portuguese language model, shows agreement with human coders on measuring moral foundation of authority.
  • However, when the test is broken down into smaller parts (decomposition), AMALIA's performance drops significantly, suggesting it may rely on surface correlates rather than understanding the theory.
  • This study questions whether national models should only be tested for agreement with human coders, but also the evidential route of that agreement.
  • An open multilingual LLM performs better on the same Portuguese corpus under the same instructions, suggesting the corpus may not be the main explanation.

Key Concepts Mentioned

  • AMALIA
  • language model
  • moral foundation
  • authority
  • decomposition
  • surface correlates
  • open multilingual LLM

Claims Made by the Document

  • AMALIA agrees with human coders on measuring moral foundation of authority.
  • AMALIA's performance drops significantly when test is decomposed.
  • AMALIA may rely on surface correlates rather than understanding the theory.
  • An open multilingual LLM performs better on the same Portuguese corpus.

Evidence Used

  • AMALIA's agreement with human coders is within six F1 points of open models eight to thirteen times its size.
  • Decomposition recovers only about half of AMALIA's holistic performance.
  • Error analysis suggests reliance on surface correlates, especially moral outrage near authority figures.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

A national language model offers a linguistic community its own instrument for measuring what its citizens say and value. Portugal's AMALIA, a publicly funded 9B-parameter model for European Portuguese, appears competitive on agreement alone: asked to code the moral foundation of authority, it agrees with trained human coders to within six F1 points of open models eight to thirteen times its size. Yet agreement is reliability, not validity. For theoretical constructs that must be inferred rather than read from surface features, the question is whether the model follows the construct's theory or reaches the right code by correlated shortcuts. We test this with the recovery gap: the loss in performance when a holistic prompt is decomposed into the codebook's atomic clauses and recombined by the theory's explicit rule. If calibration closes that gap, some portability should survive across models and languages; where it does not, the construct-model instrument is the likely locus of failure. We ask whether a calibrated English instrument transfers to AMALIA-9B and to European Portuguese. For one construct and one corpus, it does not. Decomposition recovers only about half of AMALIA's holistic performance, and error analysis suggests reliance on surface correlates, especially moral outrage near authority figures. An open multilingual LLM closes the gap on the same Portuguese corpus under the same instructions, pointing away from the corpus as the main explanation. AMALIA can still screen and pre-code at scale, but it cannot yet measure this construct well enough to stand alone. The study is a single counterexample, not a verdict on national models; it argues that sovereign-LLM benchmark batteries should test not only agreement with human coders, but the evidential route by which that agreement is warranted.