A text-to-speech system built from scratch at EPFL-AI-TEAM. Every stage is our own model, with no pre-trained speech components anywhere in the chain. PyTorch for the models, FastAPI for the server, React for the web app.
Type a sentence, get a waveform back:
python synthesize.py --text "The quick brown fox jumps over the lazy dog"Four stages, each one a separate model, wired together in ml/pipeline.py.
text -> normalize + tokenize -> G2P -> acoustic model -> vocoder -> audio
phonemes mel spectrogram 22.05 kHz
Audio is 22.05 kHz throughout, with 80 mel bins and a hop length of 256 samples, so one mel frame is about 11.6 ms. Those numbers have to agree across all three models or nothing lines up.
wetext handles normalization, turning 5 into five and expanding abbreviations.
A segmenter then splits long input into chunks that fit the model's context, and a
character-level tokenizer encodes what's left: 4 special tokens, punctuation, and
lowercase ASCII.
Normalization runs before tokenization on purpose. Do it the other way round and digits never reach the G2P as words.
7.5M parameters. Encoder-decoder Transformer, 4 layers each side, d_model 256, 4 heads, d_ff 1024, pre-norm.
Maps characters to stressless ARPAbet (39 phonemes plus 4 specials). Written by
hand rather than pulled from a library: the attention, the blocks, the positional
encoding and both decoding strategies (greedy and beam search) are all in
ml/models/G2Ptransformer/.
The interesting part is that it phonemizes a whole sentence in one pass instead of word by word. That gives it the surrounding context it needs for heteronyms, so it can tell these apart:
| "I want to read a good book today" | R IY D |
| "he has read that book" | R EH D |
| "the desert is dry" | D EH Z ER T |
| "do not desert your post" | D IH Z ER T |
| "please close the door" | K L OW Z |
| "the store is very close to my house" | K L OW S |
Trained on flexthink/librig2p-nostress, with a mined set of homograph sentences
upsampled to roughly a quarter of the data to push on exactly this. Best validation
loss 0.0976 at epoch 128.
If the sentence-level pass returns a word count that doesn't match the input, the pipeline falls back to per-word G2P rather than emitting garbage.
30.3M parameters. FastSpeech-style, non-autoregressive.
Four pieces, in ml/models/acoustic_model/:
- Text encoder turns phoneme IDs into hidden states. 6 Transformer layers, d_model 384, 2 heads, d_ff 1536.
- Variance adapter predicts duration, pitch and energy from those hidden states, each with a 2-layer Conv1d stack. It's a mixture density network, so each head outputs mixture weights, means and log-sigmas rather than one number, which lets you sample prosody instead of always getting the average. Durations are predicted in the log domain and recovered as integer frame counts.
- Length regulator expands each phoneme's hidden state by its predicted duration, taking the sequence from phoneme rate to frame rate. This is what replaces autoregression, and it's why the whole utterance generates in one shot.
- Mel decoder is another 6-layer Transformer stack projecting to 80 bins, followed by a Tacotron-2 style 5-layer convolutional post-net that predicts a residual. L1 loss alone produces over-smoothed, blurry mels that sound metallic through a vocoder; the post-net sharpens formants and harmonics. Both the coarse and the refined mel are supervised.
Trained on LJSpeech with Montreal Forced Aligner durations, plus a multi-scale mel discriminator for adversarial sharpening. Best validation loss 1.1483 at epoch 89. Phoneme vocab is 45 tokens.
Prosody is controllable at inference through speed, pitch, energy and an MDN
sampling temperature. Speed divides the predicted durations, so lower is slower.
13.9M parameters. HiFi-GAN V1 generator.
Transposed convolutions upsample by 8, 8, 2, 2, which multiplies to 256 and has to equal the mel hop length exactly. Each stage runs multi-receptive-field fusion, blending residual blocks with kernels 3, 7 and 11 at dilations 1, 3 and 5.
Trained adversarially against multi-period and multi-scale discriminators with feature-matching and mel-reconstruction losses, out to 200k steps on LJSpeech.
One detail worth knowing: the acoustic model emits standardized mels, and the
vocoder was trained on dB-scale ones. stats.json next to the acoustic checkpoint
holds the mean and standard deviation that bridge the two. Skip that step and the
output is noise.
Checkpoints are gitignored and live on SharePoint. Put each best.pt into the
matching folder:
ml/models/G2Ptransformer/checkpoints/best.pt
ml/models/acoustic_model/checkpoints/best.pt + phoneme_vocab.json + stats.json
ml/models/hifigan/checkpoints/best.pt
The acoustic model won't start without both JSON sidecars. phoneme_vocab.json
pins the phoneme-to-ID mapping it was trained with, and stats.json holds the
denormalization stats described above.
Python 3.10+, Git, and Node 18+ for the frontend.
git clone git@github.com:EPFL-AI-Team/VOICES.git
cd VOICES
python3 -m venv venv
source venv/bin/activate # venv\Scripts\Activate.ps1 on WindowsInstall PyTorch first, since the right build depends on your hardware. Check with
nvidia-smi:
# NVIDIA GPU, CUDA 12.4
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu124
# CPU only
pip install torch torchaudio --index-url https://download.pytorch.org/whl/cpuThen pip install -r requirements.txt, and cd frontend && npm install if you need
the web app. On Apple silicon torch.cuda.is_available() reports False and that's
expected, the pipeline uses MPS.
Cloning over SSH assumes you have a key on GitHub already. If ls ~/.ssh comes up
empty, generate one with ssh-keygen -t ed25519 -C "you@example.com", add
~/.ssh/id_ed25519.pub under Settings, SSH and GPG keys, and confirm with
ssh -T git@github.com.
Synthesis. From the repo root. Output lands in synthesized_output/, and
leaving off --text gives you an interactive prompt.
python synthesize.py --text "The quick brown fox jumps over the lazy dog" --speed 1.05API. Also from the repo root, not from inside backend/, since the package uses
relative imports and uvicorn api.main:app will fail there.
python -m backend.run # wraps uvicorn backend.api.main:app --reload --port 8000Routes sit under /api: GET /api/health, POST /api/g2p for phonemes, and
POST /api/tts which returns a base64 WAV with its sample rate and duration.
Interactive docs at http://localhost:8000/docs.
Web app. cd frontend && npm run dev, then http://localhost:5173. Vite proxies
/api to port 8000, so start the backend first and it works as-is.
G2P on its own, from inside ml/:
python -m predict_g2p "he has read that book"
python -m demo_g2pml/
├── text_processing/ normalizer, segmenter, tokenizers
├── models/
│ ├── G2Ptransformer/ attention, blocks, the G2P model and trainer
│ ├── acoustic_model/ encoder, variance adapter, decoder, post-net
│ └── hifigan/ generator, discriminators, losses
├── preprocessing/ dataset loaders, mel/pitch/energy extraction
├── pipeline.py the four stages wired together
└── data/ corpora, gitignored
backend/
├── api/ routes and schemas
└── services/ wraps the ML pipeline
frontend/src/ React app
synthesize.py end-to-end CLI
The acoustic model imports its Transformer blocks from G2Ptransformer/, so changes
in there affect both models.
Branch off main, one branch per task, prefixed feature/, fix/, chore/ or
docs/. Commit messages use the same verbs (feat:, fix:, and so on). Open a PR
against main and get at least one approving review; direct pushes are blocked.
If main moves while you're working, git fetch origin && git rebase origin/main.
Don't commit checkpoints, audio or raw datasets, they're gitignored and belong on
SharePoint. Keep requirements.txt to direct dependencies only.
MIT. See LICENSE.