Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Oluwanifemi Bamgbose, Simon Rosen, Jash Shah, Lindsay Devon Brin, Hoang H Nguyen, Anke Koelzer, Rachel Hansen, Tara Bogavelli, Fanny Riols
• Venue: arXiv: cs.SD
• Published: 2026-08-10
• Collected: 2026-08-11
• URL: http://arxiv.org/abs/2608.09930v1
• PDF: https://arxiv.org/pdf/2608.09930v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

Researchers have created a new method for evaluating text-to-speech (TTS) systems by breaking down the concept of 'naturalness' into ten distinct linguistic dimensions. They found that current TTS evaluation methods do not fully capture these dimensions and often fail to detect various speech errors.

What the Document Says

• The study investigates automated Text-to-Speech (TTS) evaluation methods.
• These methods include Mean Opinion Score (MOS) predictors and Audio Large Language Models (Audio-LLM) judges.
• The researchers question whether these methods accurately reflect human perception of speech.
• They propose a linguistically grounded annotation schema with 10 distinct perceptual dimensions.
• This schema is used to create a dimension-level meta-evaluation benchmark for TTS, consisting of 860 utterances annotated by trained linguist raters.
• The results show that MOS predictors focus on acoustic signal quality but do not capture a broad range of linguistically structured speech errors.
• Audio-LLM judges demonstrate selective, prompt-dependent detection that does not generalize across all dimensions.
• Neither evaluation method reliably captures a breadth of linguistically structured speech errors.
• The dataset, annotation schema, and evaluation code are made publicly available for further TTS research.

Key Concepts Mentioned

• Automated Text-to-Speech (TTS) evaluation
• Mean Opinion Score (MOS) predictors
• Audio Large Language Models (Audio-LLM)
• linguistically grounded annotation schema
• perceptual dimensions
• speech errors

Claims Made by the Document

• Current TTS evaluation methods do not accurately reflect human perception of speech.
• The proposed linguistically grounded annotation schema can improve TTS evaluation.
• Neither MOS predictors nor Audio-LLM judges capture a broad range of linguistically structured speech errors.

Evidence Used

• Results from benchmarking four MOS predictors and four Audio-LLM judges
• Findings about the lack of generalization across all dimensions

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Models (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive. We deconstruct 'naturalness' into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters. Results from benchmarking four MOS predictors and four Audio-LLM judges reveal that MOS predictors collapse onto acoustic signal quality, while Audio-LLM judges show selective, prompt-dependent detection that does not generalise across all dimensions. Neither class reliably captures a breadth of linguistically structured speech errors. Our dataset, annotation schema, and evaluation code are publicly released to support more targeted and interpretable TTS evaluation.