Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models

Source

arXiv

arxiv.org/abs/2606.25990v1

Read the source

Source

Plain-Language Brief

The paper introduces SpeechEQ, a framework designed to evaluate the sociolinguistic reasoning of Speech-Language Models (SLMs) in multi-turn dialogues. The framework includes a dataset and evaluation protocol for measuring Emotional Quotient (EQ), which is used to assess human emotional intelligence. Experiments showed that current SLMs have limitations in understanding and applying paralinguistic cues, such as relying too much on text and being limited by safety traps and contextual amnesia.

What the Document Says

  • The paper discusses a critical bottleneck for natural human-AI communication: the ability of multimodal conversational systems to navigate paralinguistic social cues during spoken interaction. Existing evaluations of machine emotional intelligence only assess reasoning through isolated text or passive acoustic perception, overlooking complex cross-modal reasoning required for active, multi-turn dialogue.
  • The authors introduce SpeechEQ, a framework designed to evaluate the sociolinguistic reasoning of Speech-Language Models (SLMs).
  • SpeechEQ includes a validated dataset of 2,265 dialogues across 15 Emotional Quotient (EQ) subscales grounded in EQ-i 2.0 theory.
  • The framework also proposes a Spoken EQ (SEQ) score inspired by human EQ assessments for the multi-turn evaluation protocol.
  • Experiments show limitations in how both existing Speech Emotion Recognition and end-to-end Speech-Language Models understand and apply paralinguistic cues through speech.
  • End-to-end architectures outperform cascaded systems, but current multimodal models remain bottlenecked by a text-reliant 'modality shortcut', an alignment-induced 'safety trap', and 'contextual amnesia'.
  • The paper highlights the barriers to truly emotionally aware AI.
  • SpeechEQ can be accessed at https://huggingface.co/datasets/SpeechEQ/SpeechEQ and there's a demo page at https://binomial14.github.io/speecheq-demo/

Key Concepts Mentioned

  • multimodal conversational systems
  • paralinguistic social cues
  • Emotional Intelligence Quotient (EQ)
  • Speech-Language Models (SLMs)
  • Speech Emotion Recognition
  • end-to-end Speech-Language Models
  • modality shortcut
  • safety trap
  • contextual amnesia

Claims Made by the Document

  • Existing evaluations of machine emotional intelligence overlook complex cross-modal reasoning required for active, multi-turn dialogue.
  • SpeechEQ can evaluate the sociolinguistic reasoning of Speech-Language Models (SLMs).
  • End-to-end architectures outperform cascaded systems in understanding and applying paralinguistic cues through speech.
  • Current multimodal models remain bottlenecked by a text-reliant 'modality shortcut', an alignment-induced 'safety trap', and 'contextual amnesia'.

Evidence Used

  • The authors discuss the limitations of existing evaluations of machine emotional intelligence.
  • The authors describe SpeechEQ, its dataset, and evaluation protocol.
  • Experiments results are presented showing limitations in how both existing Speech Emotion Recognition and end-to-end Speech-Language Models understand and apply paralinguistic cues through speech.
  • The text mentions the outperformance of end-to-end architectures over cascaded systems.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

As multimodal conversational systems increasingly engage in spoken interaction, their ability to navigate paralinguistic social cues has become a critical bottleneck for natural human-AI communication. However, existing evaluations of machine emotional intelligence assess reasoning exclusively through isolated text or passive acoustic perception, overlooking the complex cross-modal reasoning required for active, multi-turn dialogue. We introduce \textsc{SpeechEQ}, a comprehensive framework designed to evaluate the sociolinguistic reasoning of Speech-Language Models (SLMs). The framework includes a validated dataset of 2,265 dialogues across 15 Emotional Quotient (EQ) subscales grounded in EQ-i 2.0 theory, along with a multi-turn evaluation protocol measured by our proposed Spoken EQ (SEQ) score inspired by human EQ assessments. Experiments show limitations in how both existing Speech Emotion Recognition and end-to-end Speech-Language Models understand and apply paralinguistic cues through speech. While end-to-end architectures outperform cascaded systems, \textsc{SpeechEQ} reveals that current multimodal models remain bottlenecked by a text-reliant modality shortcut,'' an alignment-induced safety trap,'' and ``contextual amnesia,'' highlighting the barriers to truly emotionally aware AI. Our benchmark can be accessed at https://huggingface.co/datasets/SpeechEQ/SpeechEQ and demo page at https://binomial14.github.io/speecheq-demo/