Z.AI Logo

GLM-TTS: Controllable & Emotion-Expressive Zero-shot TTS with Multi-Reward Reinforcement Learning

Mutimodal AI
GLM-TTS is a high-quality text-to-speech (TTS) synthesis system based on large language models, supporting zero-shot voice cloning and streaming inference. This system adopts a two-stage architecture: first, it uses LLM to generate speech token sequences, then uses Flow model to convert tokens into high-quality audio waveforms. By introducing a Multi-Reward Reinforcement Learning framework, GLM-TTS can generate more expressive and emotional speech, significantly improving the expressiveness of traditional TTS systems.
Features:
  • Zero-shot Voice Cloning: Clone any speaker's voice with just 3-10 seconds of prompt audio
  • RL-enhanced Emotion Control: Achieve more natural emotional expression and prosody control through multi-reward reinforcement learning framework
  • Streaming Inference: Support real-time streaming audio generation, suitable for interactive applications
  • High-quality Synthesis: Generate natural and expressive speech with quality comparable to commercial systems
  • Multi-language Support: Primarily supports Chinese, while also supporting English mixed text
  • Phoneme-level Modeling: Support phoneme-level text-to-speech conversion
  • Flexible Inference Methods: Support multiple sampling strategies and inference modes

System Architecture

GLM-TTS adopts a two-stage design: in the first stage, a large language model (LLM) based on Llama architecture converts input text into speech token sequences; in the second stage, the Flow Matching model converts these token sequences into high-quality mel-spectrogram, and finally generates audio waveforms through a vocoder. The system supports zero-shot voice cloning by extracting speaker features from prompt audio without fine-tuning for specific speakers.

Fine-grained Pronunciation Control (Phoneme-in)

For scenarios demanding high pronunciation accuracy, such as educational assessments and audiobooks, GLM-TTS introduces the Phoneme-in mechanism to address automatic pronunciation ambiguity in polyphones (e.g., "行" which can be read as xíng or háng) and rare characters. This mechanism supports "Hybrid Phoneme + Text" input, enabling precise, targeted control over specific vocabulary pronunciation.
  • Hybrid Training During training, random G2P (Grapheme-to-Phoneme) conversion is applied to parts of the text. This strategy compels the model to adapt to hybrid input sequences, preserving its ability to understand pure text while enhancing generalization for phoneme inputs.
  • Targeted Inference Inference follows a G2P -> Table Lookup Replacement -> Hybrid Input workflow:
    1. Global Conversion: Obtain the complete phoneme sequence for the input text.
    2. Dynamic Replacement: Using a "Dynamic Controllable Dictionary," automatically identify polyphones or rare characters and replace them with specified target phonemes.
    3. Hybrid Generation: Feed the combination of replaced phonemes and original text into GLM-TTS as a hybrid input. This ensures precise pronunciation control for specific words while maintaining natural prosody.

RL Alignment

To address the issue of flat emotional expression in traditional TTS, we introduce a multi-reward reinforcement learning framework. This framework comprehensively evaluates generated speech through multiple reward functions (including similarity reward, CER reward, emotion reward, laughter reward, etc.) and uses the GRPO (Group Relative Policy Optimization) algorithm to optimize the LLM's generation strategy. Specifically:
  1. Multi-reward Design: The system designs various reward functions to evaluate the quality of generated speech from different dimensions, including sound quality, similarity, emotional expression, etc.
  2. Reward Server: Computes multiple reward functions through a distributed reward server, supporting parallel processing
  3. Policy Optimization: Uses GRPO algorithm to optimize the LLM's generation strategy based on reward signals, enhancing the emotional expressiveness of speech
  4. Token-level Rewards: Supports fine-grained token-level reward allocation, providing more precise optimization signals
Through RL optimization, GLM-TTS_RL reduces the CER metric from 1.03 to 0.89 compared to the base model, while maintaining high similarity, achieving better sound quality and expressiveness.

Evaluation Results

Evaluated on seed-tts-eval zh testset. To maintain consistency with the original evaluation, inference was performed without the --phoneme flag.
CER: Character Error Rate (lower is better ↓) | SIM: Similarity (higher is better ↑)

Serve GLM-TTS Locally