Plachtaa /VALL-E-X

An open source implementation of Microsoft's VALL-E X zero-shot TTS model. Demo is available in https://plachtaa.github.io/vallex/

★ 7,907⑂ 772👁 1⑃ 18● 158▤ 0
PythonCreated 2023-07-29
+0
Daily
+0
Weekly
-10
Monthly
7,907
stars
772
forks
1
watchers
18
prs
158
issues
0
releases

README

VALL-E X: Multilingual Text-to-Speech Synthesis and Voice Cloning 🔊


English | 中文

An open source implementation of Microsoft's VALL-E X zero-shot TTS model.
We release our trained model to the public for research or application usage.

!vallex-framework

VALL-E X is an amazing multilingual text-to-speech (TTS) model proposed by Microsoft. While Microsoft initially publish in their research paper, they did not release any code or pretrained models. Recognizing the potential and value of this technology, our team took on the challenge to reproduce the results and train our own model. We are glad to share our trained VALL-E X model with the community, allowing everyone to experience the power next-generation TTS! 🎧


More details about the model are presented in model card.

 📖 Quick Index
 🚀 Updates
 📢 Features
 💻 Installation
 🎧 Demos
 🐍 Usage
 ❓ FAQ
 🧠 TODO

 🚀 Updates
2023.09.10
- Added AR decoder batch decoding for more stable generation result.

2023.08.30
- Replaced EnCodec decoder with Vocos decoder, improved audio quality. (Thanks to @v0xie)

2023.08.23
- Added long text generation.

2023.08.20
- Added Chinese README.

2023.08.14
- Pretrained VALL-E X checkpoint is now released. Download it here

 💻 Installation
 Install with pip, Python 3.10, CUDA 11.7 ~ 12.0, PyTorch 2.0+
commandline
git clone https://github.com/Plachtaa/VALL-E-X.git
cd VALL-E-X
pip install -r requirements.txt


 Note: If you want to make prompt, you need to install ffmpeg and add its folder to the environment variable PATH.

When you run the program for the first time, it will automatically download the corresponding model. 

If the download fails and reports an error, please follow the steps below to manually download the model.

(Please pay attention to the capitalization of folders)

1. Check whether there is a checkpoints folder in the installation directory. 
If not, manually create a checkpoints folder (./checkpoints/) in the installation directory.

2. Check whether there is a vallex-checkpoint.pt file in the checkpoints folder. 
If not, please manually download the vallex-checkpoint.pt file from here and put it in the checkpoints folder.

3. Check whether there is a whisper folder in the installation directory. 
If not, manually create a whisper folder (./whisper/) in the installation directory.

4. Check whether there is a medium.pt file in the whisper folder. 
If not, please manually download the medium.pt file from here and put it in the whisper folder.

  🎧 Demos
Not ready to set up the environment on your local machine just yet? No problem! We've got you covered with our online demos. You can try out VALL-E X directly on Hugging Face or Google Colab, experiencing the model's capabilities hassle-free!





 📢 Features

VALL-E X comes packed with cutting-edge functionalities:

1. Multilingual TTS: Speak in three languages - English, Chinese, and Japanese - with natural and expressive speech synthesis.

2. Zero-shot Voice Cloning: Enroll a short 3~10 seconds 

Snapshot: 2026-11-04T14:00:00Z · Data license Apache-2.0 · JSON