When Synthetic Speech Is All You Have: Better Call GRPO

Source

Source name: arXiv
Source kind: paper
Authors / organization: Shashi Kumar, Yanis Labrak, Hasindri Watawana, Sergio Burdisso, Esaú Villatoro-Tello, Kadri Hacioğlu, Petr Motlicek, Andreas Stolcke
Venue: arXiv: cs.CL
Published: 2026-07-09
Collected: 2026-07-10
URL: http://arxiv.org/abs/2607.08409v1
PDF: https://arxiv.org/pdf/2607.08409v1
DOI: UNKNOWN

Plain-Language Brief

The paper presents a new approach for adapting Large Language Model (LLM) based Automatic Speech Recognition (ASR) systems to regulated domains like banking using synthetic text-to-speech instead of real speech due to privacy concerns and cost. The proposed method, Group Relative Policy Optimization (GRPO), uses reinforcement learning instead of supervised fine-tuning (SFT). GRPO is shown to significantly outperform SFT in reducing Word Error Rate (WER) by up to 45%.

What the Document Says

The paper discusses the use of synthetic text-to-speech (TTS) for adapting LLM-based ASR systems in regulated domains due to privacy and cost concerns with real speech.
Synthetic speech is found to be acoustically mismatched with real recordings, and work has primarily been done within supervised fine-tuning (SFT).
The paper proposes using reinforcement learning instead, specifically Group Relative Policy Optimization (GRPO), which extracts more from the same synthetic speech than SFT.
Synthetic-only adaptation of the model with GRPO reduces WER by 40% relative to SFT (36.71%→22.09%).
An approach combining SFT-then-GRPO further reduces WER by an additional 5%.
The gains are attributed to behavior rather than representation: GRPO reduces insertion errors by improving stopping calibration and speech-to-text alignment by better anchoring attention to audio, leaving early-layer representations intact.
The paper suggests that when synthetic speech is the main resource, reinforcement learning should be preferred over supervised fine-tuning.

Key Concepts Mentioned

Automatic Speech Recognition (ASR)
Large Language Model (LLM)
Text-to-Speech (TTS)
Word Error Rate (WER)
Supervised Fine-Tuning (SFT)
Reinforcement Learning
Group Relative Policy Optimization (GRPO)

Claims Made by the Document

Synthetic speech is a viable substitute for real speech in LLM-based ASR systems.
GRPO outperforms SFT in reducing WER by up to 45%.

Evidence Used

Reduction of WER from 36.71% to 22.09% with GRPO (synthetic-only adaptation), and further reduction to 22.04% with SFT-then-GRPO combination.
Improvements in stopping calibration, speech-to-text alignment, insertion errors, and preservation of early-layer representations.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

LLM-based ASR adapted to regulated domains such as banking is bottlenecked by privacy: real speech is costly and legally constrained to collect, making synthetic text-to-speech (TTS) an attractive substitute. Yet synthetic speech stays acoustically mismatched with real recordings, and work on this gap has stayed within supervised fine-tuning (SFT). We instead turn to reinforcement learning, and show that Group Relative Policy Optimization (GRPO) extracts far more from the same synthetic speech than SFT. Synthetic-only adaptation of the model with GRPO, a critic-free method rewarding low-WER hypotheses, reduces WER by 40% relative to SFT (36.71%$\to$22.09%), and an SFT-then-GRPO combination pushes this further to 45%. We trace the gain to behavior rather than representation: GRPO reduces insertion errors by improving stopping calibration and speech-to-text alignment by better anchoring attention to audio, leaving early-layer representations intact. When synthetic speech is the main resource, reinforcement learning should be preferred over supervised fine-tuning.