Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing

Source

arXiv

arxiv.org/abs/2607.18934v1

Read the source

Source

Plain-Language Brief

The authors propose a method to improve the accuracy of automatic speech recognition (ASR) models by controlling the activation of 'verbatim' versus 'intended' transcription styles. They argue that current models, trained on heterogeneous data, treat this style as an uncontrolled latent variable, leading to instability and confounding in evaluation. The authors develop a method using coverage-aware decoder task tokens and supervised cross-attention fine-tuning to enhance the models' performance in verbatim accuracy, disfluency detection, and intended-mode quality across languages.

What the Document Says

  • Modern ASR models have an uncontrolled latent variable for transcription style (verbatim vs. intended), causing instability and evaluation confounding.
  • The authors show that models already encode both styles but the challenge is controlled activation.
  • Using coverage-aware decoder task tokens trained on parallel verbatim/intended transcript pairs, they improve German disfluency F1 from 10% to 79% zero-shot.
  • Full English-only fine-tuning surpasses all baselines in verbatim accuracy, disfluency detection, and intended-mode quality across languages.
  • The authors introduce supervised cross-attention fine-tuning that improves word-level timestamps on disfluent speech beyond forced-alignment baselines.
  • They propose 'verbatimize', a new task for creating and enriching speech corpora with high-quality canonical verbatim transcriptions.

Key Concepts Mentioned

  • ASR
  • transcription style
  • verbatim vs. intended
  • coverage-aware decoder task tokens
  • disfluency F1
  • supervised cross-attention fine-tuning
  • word-level timestamps
  • verbatimize

Claims Made by the Document

  • Current ASR models have an uncontrolled latent variable for transcription style.
  • The authors' method improves German disfluency F1 significantly.
  • Full English-only fine-tuning surpasses all baselines in various performance metrics.
  • Supervised cross-attention fine-tuning improves word-level timestamps on disfluent speech.

Evidence Used

  • The text mentions the instability and evaluation confounding caused by the uncontrolled latent variable in current ASR models.
  • Improvement of German disfluency F1 from 10% to 79% is stated as a result of using coverage-aware decoder task tokens.
  • The text states that full English-only fine-tuning surpasses all baselines in various performance metrics across languages.
  • Improvement of word-level timestamps on disfluent speech through supervised cross-attention fine-tuning is mentioned.

Methods / Approach

UNKNOWN

Limitations Stated by the Authors

  • The authors do not discuss potential limitations or challenges of their proposed method beyond the scope of the paper.

Limitations Noted by the Agent

Agent interpretation — not from the source.

  • The agent may have limitations in understanding the technical details and implications of the proposed method.
  • The agent may not be able to evaluate the practical applicability and impact of the proposed method in real-world scenarios.

Open Questions

  • What are the potential challenges or limitations in implementing 'verbatimize' for creating and enriching speech corpora?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • The text does not specify the exact nature of the instability caused by the uncontrolled latent variable in current ASR models.
  • It is unknown whether the method for improving German disfluency F1 can be applied to other languages.

Raw Abstract / Excerpt

Modern ASR models trained on heterogeneously annotated data treat transcription style (verbatim vs. intended) as an uncontrolled latent variable, causing measurable decoding instability, evaluation confounding (up to 60% of reported WER attributable to style mismatch), and unreliable word-level timing. We show that models already encode both styles; the challenge is controlled activation. Using coverage-aware decoder task tokens trained on parallel verbatim/intended transcript pairs, we raise German disfluency F1 from 10% to 79% zero-shot, despite English-only training. Full English-only fine-tuning surpasses all baselines in verbatim accuracy, disfluency detection, and intended-mode quality across both languages. We further introduce supervised cross-attention fine-tuning that improves word-level timestamps on disfluent speech beyond forced-alignment baselines. Finally, we propose verbatimize, a new task enabling scalable creation and enrichment of speech corpora with high-quality canonical verbatim transcriptions.