In this episode, we discuss F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching by Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, Xie Chen. F5-TTS is a fully non-autoregressive text-to-speech system that utilizes flow matching with Diffusion Transformer (DiT) and addresses limitations of previous systems like E2 TTS by padding text inputs with filler tokens to match speech input lengths. It includes ConvNeXt for refining text representations and employs a new Sway Sampling strategy to enhance performance during inference without retraining. The system achieves a rapid inference real-time factor of 0.15 while providing high-quality speech synthesis, capable of zero-shot performance and code-switching, and is trained on a 100K-hour multilingual dataset with resources available for community use.
AI Breakdown