SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models

Source

Source name: arXiv
Source kind: paper
Authors / organization: Thomas Thebaud, Yuzhe Wang, Hao Zhang, Sathvik Manikantan Napa Ugandhar, Ashish Hallur, Georgi Tinchev, Venkatesh Ravichandran, Laureano Moro-Velazquez
Venue: arXiv: cs.CL
Published: 2026-07-06
Collected: 2026-07-07
URL: http://arxiv.org/abs/2607.05365v1
PDF: https://arxiv.org/pdf/2607.05365v1
DOI: UNKNOWN

Plain-Language Brief

A new benchmark called SPEARBench is introduced to evaluate the naturalness of speech-to-speech language models during question-answer interactions. The benchmark covers aspects like response latency, interruptions, speech quality, ASR robustness, dialect consistency, emotional naturalness, interpersonal stance, and explainable baselines.

What the Document Says

Streaming speech-to-speech language models are evaluated using a new benchmark called SPEARBench.
SPEARBench is designed for question-answer interactions to assess the naturalness of these systems.
The benchmark constructs dialogue prompts from the Seamless Interaction corpus and evaluates generated answers using a multidimensional protocol.
The protocol covers response latency, interruptions, speech quality, ASR robustness, language and dialect consistency, emotional naturalness, interpersonal stance, and explainable distributional baselines.
SPEARBench includes original human answers as a reference condition.
Results are reported for several contemporary models.
The results show that current models can achieve high signal-level quality and low ASR error but still differ from human conversational behavior in latency, overlap, dialect preservation, emotional adaptation, and interpersonal stance dynamics.

Key Concepts Mentioned

speech-to-speech language models
SPEARBench
question-answer interactions
multidimensional evaluation protocol
Seamless Interaction corpus
response latency
interruptions
speech quality
ASR robustness
language and dialect consistency
emotional naturalness
interpersonal stance
explainable distributional baselines

Claims Made by the Document

Standard speech and text benchmarks do not capture the naturalness of conversational behavior in streaming speech-to-speech language models.
SPEARBench can evaluate naturalness in speech-to-speech language models from question-answer interactions.

Evidence Used

The authors' introduction and explanation of SPEARBench in the text.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Streaming speech-to-speech language models aim to answer spoken queries directly with synthetic speech. However, standard speech and text benchmarks do not capture whether these systems behave naturally in conversations, where timing, turn-taking, prosody, interpersonal stance, language and dialect consistency, and relationship-aware appropriateness jointly shape perceived quality. We introduce SPEARBench, a benchmark for evaluating naturalness in speech-to-speech language models from question-answer interactions. SPEARBench constructs controlled dialogue prompts from the Seamless Interaction corpus, runs inference across multiple models, and evaluates generated answers using a multidimensional protocol that covers response latency, interruptions, speech quality, ASR robustness, language and dialect consistency, emotional naturalness, interpersonal stance, and explainable distributional baselines. The benchmark includes original human answers as a reference condition and reports results for several contemporary models. Results show that current models can achieve high signal-level quality and low ASR error while still differing from human conversational behavior in latency, overlap, dialect preservation, emotional adaptation, and interpersonal stance dynamics.