Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model

Source

arXiv

arxiv.org/abs/2607.13013v1

Read the source

Source

Plain-Language Brief

Researchers train a discrete diffusion language model to transcribe speech instead of using autoregressive decoders that emit one token at a time. The model is based on a mixture-of-experts called DiffusionGemma, and it's trained with about 42 million parameters which is less than 0.2% of the backbone. A connectionist temporal classification loss is applied to overcome issues with gradient reaching the projector. The model achieves a word error rate of 6.6% on LibriSpeech test-clean and can transcribe utterances in approximately eight parallel steps.

What the Document Says

  • The study focuses on audio-native speech recognition using a discrete diffusion language model instead of autoregressive decoders.
  • DiffusionGemma, a 26B mixture-of-experts model is used for text generation through uniform, random-token discrete diffusion rather than the absorbing-mask scheme.
  • A frozen Whisper encoder provides acoustic features, and a lightweight projector maps them into the model embedding space.
  • Low-rank adapters let the frozen backbone attend to the new modality.
  • About 42M parameters are trained, which is 0.16 percent of the backbone.
  • The natural training objectives fail to ground the audio due to a deadlock issue, but this is overcome by applying a connectionist temporal classification loss through the frozen output head.
  • The model achieves a word error rate of 6.6% on LibriSpeech test-clean and can transcribe utterances in roughly eight parallel steps.
  • The study evaluates the model on English, Hindi, and Mandarin using a single adapter trained on six languages.

Key Concepts Mentioned

  • automatic speech recognition
  • discrete diffusion language model
  • DiffusionGemma
  • Whisper encoder
  • connectionist temporal classification loss

Claims Made by the Document

  • A discrete diffusion language model can transcribe speech instead of autoregressive decoders.
  • The proposed model reaches a word error rate of 6.6% on LibriSpeech test-clean and can transcribe utterances in approximately eight parallel steps.

Evidence Used

  • Training and testing the model as described in the text.
  • Evaluating the model on English, Hindi, and Mandarin using a single adapter trained on six languages.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Automatic speech recognition is dominated by autoregressive decoders that emit one token at a time. We ask whether a discrete diffusion language model can transcribe speech instead, refining a whole transcript in parallel over a small number of denoising steps. We train an audio-native interface for DiffusionGemma, a 26B mixture-of-experts model that generates text by uniform, random-token discrete diffusion rather than the absorbing-mask scheme common to recent diffusion language models. A frozen Whisper encoder supplies acoustic features, a lightweight projector maps them into the model embedding space, and low-rank adapters let the frozen backbone attend to the new modality. About 42M parameters are trained, which is 0.16 percent of the backbone. We find that the natural training objectives fail to ground the audio because their gradient reaches the projector only through attention that has already dismissed it. A connectionist temporal classification loss applied through the frozen output head breaks this deadlock. The resulting model reaches 6.6 percent word error rate on LibriSpeech test-clean, transcribes in roughly eight parallel steps regardless of utterance length, and uses a single adapter trained on six languages, which we evaluate here on English, Hindi, and Mandarin.