Whisper-Aware LLM: Self-Supervised Uncertainty Learning for Robust Whispered Speech Recognition

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Gaopeng Xu, Zhenyu Wang, Zheng Xue, Yinfeng Xia, Haitao Yao
• Venue: arXiv: cs.SD
• Published: 2026-08-11
• Collected: 2026-08-12
• URL: http://arxiv.org/abs/2608.10836v1
• PDF: https://arxiv.org/pdf/2608.10836v1
• DOI: UNKNOWN
• Registry group: academic_apis_and_preprints
• Evidence role: canonical_academic_index
• Language: en
• Country / region: UNKNOWN

Plain-Language Brief

The Whisper-Aware LLM is a new framework designed to improve the performance of ASR (Automatic Speech Recognition) systems for whispered speech. The model learns to perceive and react to the uncertainty associated with whispered speech by developing self-awareness through targeted self-supervised tasks. A Confidence-Fused Decoding mechanism is introduced to provide instructions and attention modulation to the LLM decoder. Experiments show that this approach reduces the CER (Character Error Rate) by 17% on the AISHELL6-Whisper dataset, and significantly lowers hallucination rates.

What the Document Says

• The paper introduces the Whisper-Aware LLM, a framework that aims to improve ASR systems for whispered speech.
• The model learns self-awareness through targeted self-supervised tasks and quantifies physical deficiencies of acoustic signals.
• The learned uncertainty is operationalized via a Confidence-Fused Decoding mechanism.
• Experiments confirm the effectiveness of this approach, with a 17% relative CER reduction on AISHELL6-Whisper dataset and lower hallucination rates.

Key Concepts Mentioned

• ASR systems
• whispered speech
• self-awareness
• targeted self-supervised tasks
• Confidence-Fused Decoding

Claims Made by the Document

• The Whisper-Aware LLM improves ASR systems for whispered speech.
• The model can learn self-awareness through targeted self-supervised tasks and quantify physical deficiencies of acoustic signals.

Evidence Used

• The paper describes the development of the Whisper-Aware LLM and its experiments.

Methods / Approach

• Learning self-awareness through targeted self-supervised tasks, Confidence-Fused Decoding mechanism

Limitations Stated by the Authors

Not stated in source

Limitations Noted by the Agent

Agent interpretation — not from the source.

The model's performance may be limited to the quality and quantity of data used for training.

Open Questions

• What are the specific targeted self-supervised tasks used?
• How is the Confidence-Fused Decoding mechanism implemented?

Notes for Future Q&A

May include interpretation. Labeled where it does.

• The paper focuses on improvements in ASR systems for whispered speech, but does not discuss other types of speech.
• The paper does not provide details about the specific implementation of the self-supervised tasks or the Confidence-Fused Decoding mechanism.

Raw Abstract / Excerpt

The signal ambiguity of whispered speech drives ASR systems toward two opposing failure modes: failing to capture whispered speech or hallucinatory transcription of noise. This paper introduces the Whisper-Aware LLM, a framework that teaches an Audio-LLM to perceive and react to this uncertainty. Our model develops an intrinsic self-awareness by learning to quantify the physical deficiencies of acoustic signals through targeted self-supervised tasks. This learned uncertainty is then operationalized via a novel Confidence-Fused Decoding mechanism, which provides both high-level instructions and frame-level attention modulation to the LLM decoder. Our experiments confirm the effectiveness of this approach. The model sets a new state-of-the-art on whispered speech with a 17% relative CER reduction on AISHELL6-Whisper. At the same time, it directly addresses the reliability trade-off, with hallucination rates dropping from over 25% to 4.5%.