Abstract. By incorporating large language models (LLMs) and non-autoregressive (NAR) decoders, existing zero-shot text-to-speech (TTS) systems are able to generate natural and human-like speech. However, these large systems are usually time-consuming, and the NAR part is rarely explored. In this paper, we investigate the design space of NAR models for two-stage TTS systems and propose a chunked flow matching model. Our NAR model 1) faithfully fulfills the acoustic characteristics (e.g., timbre, accent) for the LLM outputs; 2) supports streaming inference with constant costs for each inference chunk and minimal performance drop; 3) can be seamlessly adapted to the single-stage NAR-only TTS system. Experiments show that our model not only significantly improves the two-stage TTS system and achieves $\textless$ 150ms first-packet delay for streaming inference but also yields competitive performance on the single-stage NAR TTS system.

Contents

• Model Overview
• Expressive Zero-Shot TTS on Spk-5k
• Cross-lingual and Code-switch
• Dynamic Speed Control

Model Overview

Figure 1. An overview of the proposed flow-matching model Fox-TTS. The symbol "P" represents the mean pooling operation. The flow-based denoiser is subject to conditioning in two distinct yet complementary ways: First, it temporally interfaces with the phoneme sequences via a cross-attention mechanism, ensuring that the temporal dynamics of the input text are effectively captured. Second, it gets the global conditional signals through adaptive layer normalization (AdaLN), in which all the external conditions are fused.

Expressive Zero-Shot TTS on Spk-5k

Spk-5k: The first multi-speaker, multi-style benchmark tailored for expressive zero-shot TTS.

Cross-lingual and Code-switch

Dynamic Speed Control