Unified Audio Intelligence Without Regressing on Text Intelligence

Source

• Source name: arXiv
• Source kind: paper
• Authors / organization: Zhifeng Kong, Sang-gil Lee, Jaehyeon Kim, Boxin Wang, Zihan Liu, Sungwon Kim, Yang Chen, Arushi Goel, Rajarshi Roy, Wenliang Dai, Zhuolin Yang, Yangyi Chen, Dongfu Jiang, Sreyan Ghosh, Tuomas Rintamaki, Andrew Tao, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping
• Venue: arXiv: cs.CL
• Published: 2026-07-06
• Collected: 2026-07-07
• URL: http://arxiv.org/abs/2607.05196v1
• PDF: https://arxiv.org/pdf/2607.05196v1
• DOI: UNKNOWN

Plain-Language Brief

Researchers have developed a unified language model called Audex, which combines audio and text intelligence. The model is built on Nemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM (Language Model). Audex uses a single Transformer decoder for both audio inputs and text tokens, allowing for strong fusion of audio and text data. The model was trained on large datasets and delivers state-of-the-art performance in various tasks related to audio understanding, speech recognition, translation, text-to-speech, audio generation, and speech-to-speech generation. Audex also retains the reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only LLM backbone.

What the Document Says

• The paper introduces a unified audio-text language model named Nemotron-Labs-Audex-30B-A3B (Audex).
• Audex is built on Nemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM.
• Audex uses a single Transformer decoder for both audio inputs and text tokens.
• The model was trained on large datasets consisting of 157.4B audio tokens and 320.5B text tokens.
• Training involves multi-stage supervised training, followed by text-only Cascade RL (Reinforcement Learning) and multi-domain on-policy distillation.

Key Concepts Mentioned

• Audio Intelligence
• Language Model
• Transformer Decoder
• Nemotron-Cascade-2-30B-A3B
• Audex
• Multimodal Generation

Claims Made by the Document

• Audex delivers state-of-the-art performance in various tasks related to audio understanding, speech recognition, translation, text-to-speech, audio generation, and speech-to-speech generation.

Evidence Used

• The paper mentions that Audex achieves strong performance in multiple tasks.

Methods / Approach

• Multi-stage supervised training, Cascade RL, and multi-domain on-policy distillation

Limitations Stated by the Authors

UNKNOWN

Limitations Noted by the Agent

Agent interpretation — not from the source.

UNKNOWN

Open Questions

UNKNOWN

Notes for Future Q&A

May include interpretation. Labeled where it does.

UNKNOWN

Raw Abstract / Excerpt

Audio intelligence involves understanding, reasoning about, and generating both audio and speech. In this work, we introduce Nemotron-Labs-Audex-30B-A3B (Audex), a unified audio-text LLM built on Nemotron-Cascade-2-30B-A3B, a strong text-only MoE LLM. Audex adopts a simple unified design with a single Transformer decoder: audio inputs are encoded and projected into the text embedding space, while text tokens and quantized audio output tokens are treated uniformly during generation. This architecture enables strong audio-text fusion, seamless multimodal generation, and compatibility with standard LLM training and inference infrastructure. For training, we meticulously curate audio-text datasets comprising 157.4B audio tokens and 320.5B text tokens. We apply multi-stage supervised training on these datasets, followed by text-only Cascade RL and multi-domain on-policy distillation. Audex delivers state-of-the-art audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech generation, while preserving very compelling reasoning, alignment, knowledge, long-context, and agentic capabilities of its text-only LLM backbone with marginal or no regression. We release the model checkpoints to facilitate open research.