Adnan KhanAdnanProduct & AI
San Francisco Bay Area
Get in touch

Source note3 min read

Unified Audio Intelligence Without Regressing on Text Intelligence

Source

arXiv

arxiv.org/abs/2607.05196v1

Read the source

Source

  • Source name: arXiv
  • Source kind: paper
  • Authors / organization: Zhifeng Kong, Sang-gil Lee, Jaehyeon Kim, Boxin Wang, Zihan Liu, Sungwon Kim, Yang Chen, Arushi Goel, Rajarshi Roy, Wenliang Dai, Zhuolin Yang, Yangyi Chen, Dongfu Jiang, Sreyan Ghosh, Tuomas Rintamaki, Andrew Tao, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping
  • Venue: arXiv: cs.CL
  • Published: 2026-07-06
  • Collected: 2026-07-07
  • URL: http://arxiv.org/abs/2607.05196v1
  • PDF: https://arxiv.org/pdf/2607.05196v1
  • DOI: UNKNOWN

Plain-Language Brief

The researchers introduce a unified audio-text language model called Audex, which is built on a strong text-only model. Audex allows for strong fusion of audio and text, seamless multimodal generation, and compatibility with standard language model training and inference infrastructure. The model delivers state-of-the-art performance in various audio and speech tasks while preserving the reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only backbone.

What the Document Says

  • The researchers introduce a unified audio-text language model named Audex.
  • Audex is built on Nemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM.
  • Audex adopts a simple unified design with a single Transformer decoder.
  • Audio inputs are encoded and projected into the text embedding space in Audex.
  • Text tokens and quantized audio output tokens are treated uniformly during generation in Audex.
  • The architecture allows for strong audio-text fusion, seamless multimodal generation, and compatibility with standard LLM training and inference infrastructure.
  • The researchers curate audio-text datasets comprising 157.4B audio tokens and 320.5B text tokens.
  • They apply multi-stage supervised training on these datasets, followed by text-only Cascade RL and multi-domain on-policy distillation.
  • Audex delivers state-of-the-art performance in various audio and speech tasks.
  • The model preserves the reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only backbone.

Key Concepts Mentioned

  • audio intelligence
  • language model
  • Audex
  • Nemotron-Cascade-2-30B-A3B
  • Transformer decoder
  • audio tokens
  • text tokens

Claims Made by the Document

  • Audex is a unified audio-text language model.
  • Audex delivers state-of-the-art performance in various audio and speech tasks.
  • Audex preserves the reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only backbone.

Evidence Used

  • The text mentions that the researchers introduce a unified audio-text language model called Audex.
  • The text states that Audex delivers state-of-the-art performance in various audio and speech tasks.
  • The text implies that Audex preserves the reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only backbone by mentioning it maintains these capabilities while fusing audio and text.

Methods / Approach

  • meticulously curating audio-text datasets
  • multi-stage supervised training
  • text-only Cascade RL
  • multi-domain on-policy distillation

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

The model's performance may be limited by the quality and diversity of the audio-text datasets used for training.

Open Questions

  • What is MoE LLM?
  • What are Cascade RL and on-policy distillation?

Notes for Future Q&A

May include interpretation. Labeled where it does.

  • MoE (Multi-Output Entropy) is a method for scaling deep learning models. It allows each neuron to output a probability distribution over multiple outputs.
  • Cascade RL (Reinforcement Learning) and on-policy distillation are methods used in training reinforcement learning models.

Raw Abstract / Excerpt

Audio intelligence involves understanding, reasoning about, and generating both audio and speech. In this work, we introduce Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM built on Nemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM. Audex adopts a simple unified design with a single Transformer decoder: audio inputs are encoded and projected into the text embedding space, while text tokens and quantized audio output tokens are treated uniformly during generation. This architecture enables strong audio-text fusion, seamless multimodal generation, and compatibility with standard LLM training and inference infrastructure. For training, we meticulously curate audio-text datasets comprising 157.4B audio tokens and 320.5B text tokens. We apply multi-stage supervised training on these datasets, followed by text-only Cascade RL and multi-domain on-policy distillation. Audex delivers state-of-the-art audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech generation, while preserving very compelling reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only LLM backbone with marginal or no regression. We release the model checkpoints to facilitate open research.