Toucan Questions

[MrEdwards007]

I have a few questions that I hope will not much of your time.

• Is there support for IPA or some other phonetic pronunciation for words that are incorrectly pronounced or that you have a specific pronunciation for?
• Is there a way to add specific lengths of silence e.g. [[slnc 2000]], where 2000 is 2000 milliseconds
• How does one add emphasis or emotion?
• Are there any scripts that can be used for fine tuning a single voice? Hopefully, a simple one with a GUI.
• Can you utilize TensorBoard to view the training logs and see what is likely the best checkpoint?
• Can you provide some guidance on what is different about my voice that no cloning software is effective (I've tried many)? The only exception is using post processing using RVC but again, that is post processing.
  My_Voice_mp3.zip

I'm ultimately looking for a clone close enough that I could fool myself. I get pretty close when I use RVC.

I tried the space on HuggingFace but the clone using any of my uploaded samples sounded like another guy.
I have approximately 1.5 hours of good quality audio, similar to the attached, so I can fine-tune if needed.
https://huggingface.co/spaces/Flux9665/MassivelyMultilingualTTS

[Flux9665]

Is there support for IPA or some other phonetic pronunciation for words that are incorrectly pronounced or that you have a specific pronunciation for?

Not directly, but that's a very good idea. I'll add it to my list to do. There are a few things that need to be considered to implement this, but it shouldn't be too difficult. For now, you could extend the following function that is applied to any English text to change the orthography of certain words automatically to trick the phonemizer into converting the word in a way that sounds more like what it's supposed to be:

IMS-Toucan/Preprocessing/TextFrontend.py

Line 1047 in c00f8a4

def english_text_expansion(text):

Is there a way to add specific lengths of silence e.g. [[slnc 2000]], where 2000 is 2000 milliseconds

Again, not directly. You can add ~ to the text to insert a pause at that point, but the model will decide how long the pause should be. It is possible to modify the length of the pause after the model has predicted it, but there's currently no simple way of doing this. I plan to implement a GUI where you can modify an utterance intuitively, but that seems pretty difficult, so I have put it off for a while already.

How does one add emphasis or emotion?

Same as above, it's possible by overwriting some intermediate results, but there's currently no simple way of doing this. In the future there will be, but it might take a while.

Are there any scripts that can be used for fine tuning a single voice? Hopefully, a simple one with a GUI.

There is one. It doesn't come with a GUI, but the changes you need to do to the code are pretty minimal. And again, I plan to improve this in the future and even add a GUI where you can just throw in some data. You just mentioned a lot of points that I plan for the future, but didn't get to yet, haha.

For finetuning, all you need to implement is a function that returns a dictionary of paths to audios that map to their respective transcripts. Each audio should be around 6-10 seconds long ideally, a bit longer or a bit shorter is no problem. You then plug this dictionary into the following function and run the run_training_pipeline.py script with "finetuning_example_simple" as command line argument.

IMS-Toucan/Recipes/finetuning_example_simple.py

Line 38 in c00f8a4

train_data = prepare_tts_corpus(transcript_dict=build_path_to_transcript_integration_test(),

Can you utilize TensorBoard to view the training logs and see what is likely the best checkpoint?

In generative tasks, the loss is usually not really indicative of the "best" checkpoint, so I would always just use the latest one. There is no TensorBoard integration, but there is Weights and Biases integration, which basically does the same thing. You can create a free account there, log in on the command line and then use the --wandb flag when starting a training run.

Can you provide some guidance on what is different about my voice that no cloning software is effective (I've tried many)?

Actually I think your voice should be quite simple to clone. The recording is good. For Toucan specifically, the system is conditioned on short audios of 6-10 seconds, so the length might be an issue, but voice cloning is not one of Toucans strengths anyway. To get a good clone out of Toucan, you would need to finetune the model to your voice. A few minutes of finetuning data is enough. It's surprising though that you say no other software is good at cloning it either, I would have thought that systems optimized for cloning should be able to do it quite well.

[MrEdwards007]

This is really good information and I am looking into it. A few questions and a problem.

Questions

• Roughly, how much audio data is required to get extremely (nearly indistinguishable) from the training data
• Can the number of checkpoints be more than 5? I really want to keep all of them so that I can see which ones are best for my use case. I dont want to have to train a model again because I overtrained it.
• Once you've create your fine tuned model, how do you reference it?
• Where are the base models? I've been looking through directories but havent found models to build upon.

Problem

I create data set and I have been following the directions for finetuning but have encountered an error that I've been unable to resolve. The huggingface file cant be found.

Here is my process

python run_training_pipeline.py --gpu_id 0 --model_save_dir /home/homer/Documents/Programs/IMS-Toucan/WEdwards-Checkpoints --wandb finetuning_example_simple

In the file "corpus_prepartion" in the method "prepare_tts_corpus"
Line 56
path_to_checkpoint=hf_hub_download(repo_id="Flux9665/ToucanTTS", filename="Aligner.pt"),
The file cant be found.

Here is the trace

---

/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/site-packages/_distutils_hack/init.py:31: UserWarning: Setuptools is replacing distutils. Support for replacing an already imported distutils is deprecated. In the future, this condition will fail. Register concerns at https://github.com/pypa/setuptools/issues/new?template=distutils-deprecation.yml
warnings.warn(
Loaded an Aligner dataset with 228 datapoints from Corpora/integration_test.
Traceback (most recent call last):
File "/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 406, in hf_raise_for_status
response.raise_for_status()
File "/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/site-packages/requests/models.py", line 1024, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/Flux9665/ToucanTTS/resolve/main/Aligner.pt

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/homer/Documents/Programs/IMS-Toucan/run_training_pipeline.py", line 114, in
pipeline_dict[args.pipeline](gpu_id=args.gpu_id,
File "/home/homer/Documents/Programs/IMS-Toucan/Recipes/finetuning_example_simple.py", line 40, in run
train_data = prepare_tts_corpus(transcript_dict=build_path_to_transcript(),
File "/home/homer/Documents/Programs/IMS-Toucan/Utility/corpus_preparation.py", line 57, in prepare_tts_corpus
path_to_checkpoint=hf_hub_download(repo_id="Flux9665/ToucanTTS", filename="Aligner.pt"),
File "/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py", line 101, in inner_f
return f(*args, **kwargs)
File "/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1232, in hf_hub_download
return _hf_hub_download_to_cache_dir(
File "/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1295, in _hf_hub_download_to_cache_dir
(url_to_download, etag, commit_hash, expected_size, head_call_error) = _get_metadata_or_catch_error(
File "/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1746, in _get_metadata_or_catch_error
metadata = get_hf_file_metadata(
File "/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
return fn(*args, **kwargs)
File "/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1666, in get_hf_file_metadata
r = _request_wrapper(
File "/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 364, in _request_wrapper
response = _request_wrapper(
File "/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 388, in _request_wrapper
hf_raise_for_status(response)
File "/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 417, in hf_raise_for_status
raise _format(EntryNotFoundError, message, response) from e
huggingface_hub.errors.EntryNotFoundError: 404 Client Error. (Request ID: Root=1-66fe1e1b-261d66b223e2d6206c89a64b;5d2d317c-734e-4328-aa03-bc331fa6ef03)

[Flux9665]

Re: Questions

Roughly, how much audio data is required to get extremely (nearly indistinguishable) from the training data

I never explored when exactly that threshold occurs, but considering some research of other people and some of the language-learning experiments I've done, I would guess around 30 minutes. The more data you have, the longer you can train for, so having more surely won't hurt. It will probably forget about some of its multilingual capabilities though, if you finetune on a single language, but it will get much better at cloning this voice.

Can the number of checkpoints be more than 5? I really want to keep all of them so that I can see which ones are best for my use case. I dont want to have to train a model again because I overtrained it.

Sure, that's just storage optimization. But for generative tasks, overtraining is a rare phenomenon. Basically, if you have enough data (for TTS I would say 1 or 2 hours is enough), a generative model can almost never overfit. Just delete this line:

IMS-Toucan/Modules/ToucanTTS/toucantts_train_loop.py

Line 195 in 07edf9c

delete_old_checkpoints(save_directory, keep=5)

And then you should also change the amount of training steps in the finetuning recipe, since the default is kept very low:

IMS-Toucan/Recipes/finetuning_example_simple.py

Line 64 in 07edf9c

steps=5000,

Just set it to some very large number and kill the process yourself once you're satisfied.

Once you've create your fine tuned model, how do you reference it?

You instanciate an InferenceInterfaces/ToucanTTSInterface and pass the path to your checkpoint as tts_model_path. Then you can use it in the same way as the pretrained model. If you want to use the GUI demo, you would add tts_model_path="path to your checkpoint goes here" to the following initialization of the GUI:

IMS-Toucan/run_GUI_demo.py

Line 96 in 07edf9c

TTSWebUI(gpu_id="cuda" if torch.cuda.is_available() else "cpu")

Where are the base models? I've been looking through directories but havent found models to build upon.

I am not sure what you mean by base model. Do you mean the architecture of the model? That's in Modules/ToucanTTS/ToucanTTS.py

Re: Problem

Ah yes, sorry about that, the huggingface integration is brand new, I didn't have time to properly test it yet and there was indeed a file missing. I uploaded it to the hub, it should be working now. If you encounter any more missing files, please let me know.

[MrEdwards007]

IMS-Toucan/Modules/ToucanTTS/toucantts_train_loop.py

Line 195 in 07edf9c
delete_old_checkpoints(save_directory, keep=5) #

I would consider this a bug. If there are existing checkpoints in the folder when you start, they are counted towards your 5 checkpoints. I didnt clear the existing checkpoints and so only the "best.pt" was being saved. It took a minute of watching the directory before I realized what was happening.

[MrEdwards007]

Questions

• Would you elaborate on how to use the fine-tuned model as the model or the model path is not apparent on how that is passed to "run_text_to_file_reader.py"

def read_texts(sentence, filename, model_id=None, device="cpu", language="eng", speaker_reference=None, duration_scaling_factor=1.0):

• Is there a specific locations that models are looked for
• Are there pre-trained voice models? I was looking through the directories to see if there was a 'Paul' or 'Nancy' or others and if so, where do you download them to and, how do yo reference them.

Problem
I created a fine-tuned model with 10 minutes of audio and it appeared to have zero effect on the sound output. So either I incorrectly connected the fine-tuned model, or there wasnt enough audio data. I used 27000 steps (yes, really as I didnt know a good place to stop), which probably caused the model collapse that I've read about. So, I'm going to start again with 2 hours of audio, so that I know that I have enough audio data to work with.

I dont know what I'm looking for, as I was accustomed to looking for the total loss and looking for the lowest point in the graph. Here, I dont know what to look for in training the graph or numbers.

Would you provide guidance on the learning rate and how it should or could be adjusted.

wandb: Run history:
wandb: duration_loss █▇▆▆▄▄▅▃▄▂▃▂▃▂▃▃▂▄▃▂▄▃▂▃▁▃▃▁▃▂▃▃▂▁▄▁▁▂▂▂
wandb: energy_loss ▅▇▇▅▇▆▆▅█▄▁▃▆▃█▆▁▅▇▄█▄▅█▃▆▄▃▅▅█▆▄▄▆▂▄▃▅▅
wandb: learning_rate ▄█████████▇▇▇▇▇▆▆▆▆▅▅▅▄▄▄▃▃▃▃▃▂▂▂▂▂▁▁▁▁▁
wandb: pitch_loss ▃▆▆▄▆▅▅▄▇▃▁▃▅▃▇▆▂▄▇▃█▃▅▇▃▆▃▃▅▅█▆▃▄▇▂▃▃▄▅
wandb: regression_loss █▅▄▃▃▃▃▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb: stochastic_loss ▅▆▆▂▆▅▆▅▆▃▁▃▅▃▇▅▁▄▆▄█▄▅▇▃▆▃▃▅▅▇▅▃▄▆▂▄▂▅▄
wandb:
wandb: Run summary:
wandb: duration_loss 2.99871
wandb: energy_loss 1.96445
wandb: learning_rate 0.0
wandb: pitch_loss 1.89659
wandb: regression_loss 0.27856
wandb: stochastic_loss 2.00286

[MrEdwards007]

When running the demo, I get this warning frequently that the audio may be clipped. I can attest that it is being clipped, as I can hear it and see it in Audacity. I don't know what the cause it. I tried utilizing a lower volume reference but that didn't resolve the issue.

Is there something that I can do to resolve this?

/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/site-packages/pyloudnorm/normalize.py:62: UserWarning: Possible clipped samples in output.
warnings.warn("Possible clipped samples in output.")

[Flux9665]

I would consider this a bug. If there are existing checkpoints in the folder when you start, they are counted towards your 5 checkpoints.

This is intended behaviour, but yes maybe a warning message telling you that the directory is not empty and the ´resume´ flag is not present would be a good idea. I recommend changing the name of the model directory for every individual run to describe what makes this run unique:

IMS-Toucan/Recipes/finetuning_example_simple.py

Line 36 in 07edf9c

save_dir = os.path.join(MODELS_DIR, "ToucanTTS_FinetuningExample") # RENAME TO SOMETHING MEANINGFUL FOR YOUR DATA

Would you elaborate on how to use the fine-tuned model as the model or the model path is not apparent on how that is passed to "run_text_to_file_reader.py"

You pass either the absolute or the relative path to any .pt checkpoint to the ´model_id´ parameter of that function. Alternatively, you can just specify the ID of the model, which is what comes after "ToucanTTS_" in the name of the directory you save the checkpoints to. It will then look in the model dir (see below) and find the ´best.pt´ from the ToucanTTS_ directory in the model dir.

IMS-Toucan/Utility/storage_config.py

Line 1 in 07edf9c

MODELS_DIR = "Models/"

Are there pre-trained voice models? I was looking through the directories to see if there was a 'Paul' or 'Nancy' or others and if so, where do you download them to and, how do yo reference them.

The pretrained model is the one that is automatically loaded from huggingface as the default. Any other models you would need to train yourself. This pretrained one is very universal, so I didn't see a point in providing more than this one.

Problem

Early stopping in TTS does not make much sense, the losses don't tell you much. 27k steps is reasonable, you can even do a lot more. Best thing to look out for is the progress plot. If it looks good, then the model is probably good. The only way to make sure is to generate audio with it and listen. If the model collapses, then you will definitely see this in the progress plot. There is unfortunately no good objective metric for TTS, it's a big problem of the field.

If you trained the model and it did not change much, then you should probably increase the learning rate. Try 1e-4 here:

IMS-Toucan/Recipes/finetuning_example_simple.py

Line 59 in 07edf9c

lr=1e-5, # if you have enough data (over ~1000 datapoints) you can increase this up to 1e-4 and it will still be stable, but learn quicker.

Clipped Samples

You can try to normalize it to a lower loudness in the following snippet. If that doesn't work, then that's unfortunately not fixable, since the vocoder directly produces those values. I also get the warning often, but I don't notice it when listening, only when inspecting the signal. Maybe some post-processing could be helpful.

IMS-Toucan/InferenceInterfaces/ToucanTTSInterface.py

Line 146 in 07edf9c

loudness_in_db=-24.0,

Soon I will hopefully release another vocoder checkpoint with higher quality that will be used if a GPU is available during inference (on CPU it's too slow, on GPU it doesn't make a difference). Maybe that one will have less issues with clipping.

[MrEdwards007]

I'll try and let you know how it turns out. I have at least 8 hours before this finetuning run is complete.

[MrEdwards007]

Question

I feel like I'm doing something wrong. I have two hours of high-quality audio data and I've approximately 10-11 hours of training time (subjective, as I know that completely depends on my equipment) and 42416 steps (objective, an absolute number) so far but this doesn't sound like my voice.

• Is this normal to take this many steps for two hours of audio data. I have increased my steps to 70k, I'll continue from my last checkpoint to see if the model starts sounding like me on the next iteration (which will stop before I get to 70k, based on the problem that I am experiencing).
• Is there something that can be done to accelerate the training?
• Is there a way to know if I'm doing this correctly.
• Would doubling my audio file (using it twice, splicing together and then splitting to the required training dataset format) to create 4 hours of audio data help? This would make it so that I don't have to sit at the microphone for X additional hours.

If I know that I can get an extremely high-quality voice based on my training data, I would used a rented GPU, since I know this would be a one-time event and one-time cost but I cant put money into renting a GPU until I know I can get get the desired outcome. Currently, this doesn't sound like me and I don't have a horizon to know when I can get there.

I can spend more time at the microphone or find other productions that I've done and combine for a longer session.

Problem

Oddly, the training run stopped at

1. checkpoint_21208.pt
2. checkpoint_42416.pt (exactly double).

I'm not sure what is happening but I am seeing a pattern with training failure. I thought I was going crazy but in the last three training runs, attempting to fine-tune a model against two hours of of my voice data, roughly every 5 -5.5 hours, the process will stop. After reading the documentation a few times, I saved my checkpoints and continued to train from a previous checkpoint, so that I do not need to start from zero (thank goodness). I do not know what this means or how to address it.

if use_wandb:
    wandb.init(
        name=f"{__name__.split('.')[-1]}_{time.strftime('%Y%m%d-%H%M%S')}" if wandb_resume_id is None else None,
        id=wandb_resume_id,  # this is None if not specified in the command line arguments.
        resume="must" if wandb_resume_id is not None else None)

print("Training model")
train_loop(net=model,
           datasets=[train_data],
           device=device,
           save_directory=save_dir,
           batch_size=20,  # YOU MIGHT GET OUT OF MEMORY ISSUES ON SMALL GPUs, IF SO, DECREASE THIS.
           eval_lang="eng",  # THE LANGUAGE YOUR PROGRESS PLOTS WILL BE MADE IN
           warmup_steps=500,
           lr=1e-4,  # if you have enough data (over ~1000 datapoints) you can increase this up to 1e-4 and it will still be stable, but learn quicker.
           # DOWNLOAD THESE INITIALIZATION MODELS FROM THE RELEASE PAGE OF THE GITHUB OR RUN THE DOWNLOADER SCRIPT TO GET THEM AUTOMATICALLY
           path_to_checkpoint=hf_hub_download(repo_id="Flux9665/ToucanTTS", filename="ToucanTTS.pt") if resume_checkpoint is None else resume_checkpoint,
           fine_tune=True if resume_checkpoint is None and not resume else finetune,
           resume=resume,
           steps=70000,
           use_wandb=use_wandb,
           train_samplers=[torch.utils.data.RandomSampler(train_data)],
           gpu_count=1)
if use_wandb:
    wandb.finish()

EPOCH COMPLETE

91%|██████████████████████████████████████████████████████████████████████████████▏ | 10/11 [00:07<00:00, 1.40it/s]
Epoch: 1925
Time elapsed: 321 Minutes
Reconstruction Loss: 0.198
Steps: 42383

selecting checkpoints...
loading model /home/homer/Documents/Programs/IMS-Toucan/WEdwards-Checkpoints/checkpoint_42383.pt
averaging...
saving model...
...done!
100%|██████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:09<00:00, 1.12it/s]

EPOCH COMPLETE

91%|██████████████████████████████████████████████████████████████████████████████▏ | 10/11 [00:07<00:00, 1.31it/s]
Epoch: 1926
Time elapsed: 321 Minutes
Reconstruction Loss: 0.198
Steps: 42394

selecting checkpoints...
loading model /home/homer/Documents/Programs/IMS-Toucan/WEdwards-Checkpoints/checkpoint_42394.pt
averaging...
saving model...
...done!
100%|██████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:09<00:00, 1.10it/s]

EPOCH COMPLETE

91%|██████████████████████████████████████████████████████████████████████████████▏ | 10/11 [00:07<00:00, 1.23it/s]
Epoch: 1927
Time elapsed: 321 Minutes
Reconstruction Loss: 0.198
Steps: 42405

selecting checkpoints...
loading model /home/homer/Documents/Programs/IMS-Toucan/WEdwards-Checkpoints/checkpoint_42405.pt
averaging...
saving model...
...done!
100%|██████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:09<00:00, 1.16it/s]

EPOCH COMPLETE

91%|██████████████████████████████████████████████████████████████████████████████▏ | 10/11 [00:07<00:00, 1.30it/s]
Epoch: 1928
Time elapsed: 322 Minutes
Reconstruction Loss: 0.198
Steps: 42416

91%|██████████████████████████████████████████████████████████████████████████████▏ | 10/11 [00:09<00:00, 1.11it/s]
Traceback (most recent call last):
File "/home/homer/Documents/Programs/IMS-Toucan/run_training_pipeline.py", line 114, in
pipeline_dict[args.pipeline](gpu_id=args.gpu_id,
File "/home/homer/Documents/Programs/IMS-Toucan/Recipes/finetuning_example_simple.py", line 57, in run
train_loop(net=model,
File "/home/homer/Documents/Programs/IMS-Toucan/Modules/ToucanTTS/toucantts_train_loop_arbiter.py", line 55, in train_loop
mono_language_loop(net=net,
File "/home/homer/Documents/Programs/IMS-Toucan/Modules/ToucanTTS/toucantts_train_loop.py", line 217, in train_loop
path_to_most_recent_plot = plot_progress_spec_toucantts(model,
File "/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/homer/Documents/Programs/IMS-Toucan/Utility/utils.py", line 71, in plot_progress_spec_toucantts
tf = ArticulatoryCombinedTextFrontend(language=lang)
File "/home/homer/Documents/Programs/IMS-Toucan/Preprocessing/TextFrontend.py", line 594, in init
self.phonemizer_backend = EspeakBackend(language=self.g2p_lang,
File "/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/site-packages/phonemizer/backend/espeak/espeak.py", line 45, in init
super().init(
File "/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/site-packages/phonemizer/backend/espeak/base.py", line 45, in init
self._espeak = EspeakWrapper()
File "/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/site-packages/phonemizer/backend/espeak/wrapper.py", line 60, in init
self._espeak = EspeakAPI(self.library())
File "/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/site-packages/phonemizer/backend/espeak/api.py", line 84, in init
self._library = ctypes.cdll.LoadLibrary(str(espeak_copy))
File "/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/ctypes/init.py", line 452, in LoadLibrary
return self._dlltype(name)
File "/home/homer/anaconda3/envs/toucan-tts/lib/python3.10/ctypes/init.py", line 374, in init
self._handle = _dlopen(self._name, mode)
OSError: /tmp/tmpwjn_1le0/libespeak-ng.so.1.1.49: failed to map segment from shared object
wandb: Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing.
wandb:
wandb: Run history:
wandb: duration_loss ▃▃▅▃▄▅▆▅▄▃▅▂▆▄▆▃▆▆▇▃▆▄▃▁▁▄▁▄█▃▄▄▅▆▅▇▄▂▂▄
wandb: energy_loss ▃▁▆▄▄▇▇▆▅▄▄▃▅▅▇▃▇▅▇▂▆▃▄▁▃▃▁▅█▄▅▅▅▇▅▇▄▂▂▅
wandb: learning_rate █▇▇▆▆▆▅▅▄▄▄▃▃▃▃▃▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb: pitch_loss ▄▁▆▄▄▆▇▆▄▄▄▂▅▅▆▃▆▆▆▃▅▄▃▁▂▄▁▅█▄▅▅▆▇▆▇▄▃▂▅
wandb: regression_loss ▆█▅▅▅▅▄▃▃▃▃▄▃▃▃▂▂▂▂▂▂▂▃▃▂▁▃▂▁▁▂▂▂▁▂▁▁▁▂▂
wandb: stochastic_loss ▃▂▅▄▃▆▆▅▃▄▃▁▅▅▆▃▅▅▇▄▄▃▃▁▁▄▁▅▇▄▄▄▅█▅▆▃▂▂▅
wandb:
wandb: Run summary:
wandb: duration_loss 2.46269
wandb: energy_loss 1.948
wandb: learning_rate 0.0
wandb: pitch_loss 1.91329
wandb: regression_loss 0.19839
wandb: stochastic_loss 2.05463

[Flux9665]

2 hours of training data is more than enough, you don't need to record additional data or change it in any way. The error you encountered is related to espeak, it usually occurs when your system goes to sleep mode or you log out.

70k steps should definitely be enough. Do you notice any change in the voice between checkpoints? If not, the model might not be loaded correctly. Can you double check at which location in the code you select the checkpoint?

Also, do the loss curves change in wandb? The regression loss should definitely go down.

[MrEdwards007]

I havent heard the difference between the checkpoints.
I considered that I may not be loading the model correctly and modified existing code, so I could hear the difference.
The same code, referencing two different checkpoints

import os
import torch
from InferenceInterfaces.ToucanTTSInterface import ToucanTTSInterface


def read_texts(sentence, filename, model_id=None, device="cpu", language="eng", speaker_reference=None, duration_scaling_factor=1.0):
    tts = ToucanTTSInterface(device=device, tts_model_path=model_id)
    tts.set_language(language)
    print("speaker_reference", speaker_reference)
    if speaker_reference is not None:
        tts.set_utterance_embedding(speaker_reference)
    if type(sentence) == str:
        sentence = [sentence]
    tts.read_to_file(text_list=sentence, file_location=filename, duration_scaling_factor=duration_scaling_factor, prosody_creativity=0.0)
    del tts

if __name__ == '__main__':
    exec_device = "cuda" if torch.cuda.is_available() else "cpu"

    # quick, easy sentences.

    sentence=[]
    sentence += ["Cybersecurity is the practice of protecting systems, networks, and programs from digital attacks aimed at accessing, changing, or destroying sensitive information."]
    sentence += ["A strong cybersecurity framework involves implementing multiple layers of defense across technology, processes, and people to reduce the risk of breaches."]
    sentence += ["Continuous monitoring and timely updates are essential in cybersecurity to address evolving threats and vulnerabilities within any organization’s infrastructure."]	    
    
    checkpoint  = "checkpoint_21208"
    model_path  = f"/home/homer/Documents/Programs/IMS-Toucan/Experiment/{checkpoint}.pt"
    output_file = f"output_audio_{checkpoint}.wav"
    speaker_reference = None

    read_texts(sentence=sentence, 
               filename=output_file, 
               model_id=model_path, 
               device=exec_device, 
               language="eng", speaker_reference=speaker_reference)
               
    checkpoint  = "checkpoint_44968"
    model_path  = f"/home/homer/Documents/Programs/IMS-Toucan/Experiment/{checkpoint}.pt"
    output_file = f"output_audio_{checkpoint}.wav"

    read_texts(sentence=sentence, 
               filename=output_file, 
               model_id=model_path, 
               device=exec_device, 
               language="eng", speaker_reference=speaker_reference)
   

In terms of wandb, I dont know what I'm looking at. I was seeking to learn what to look for. I understood with TensorBoard. Using wandb is new to me. I dont know what loss to look for here.

[Flux9665]

That looks absolutely correct. I would not expect a big change between step 22k and step 45k, because at 22k the model should already be pretty close to done and the changes after that are getting smaller and smaller. I think most interesting would be the changes within the first few hundred steps. But either way, it should definitely sound pretty close to you at this point.

Does the regression loss curve go down? Maybe try an even higher learning rate, maybe 1e-3. That shouldn't make a difference, but it might be worth to try.

Another option would be to train from scratch without finetuning. 2 hours should be enough for that as well.

[MrEdwards007]

I think I know what you are asking for, after thinking about it a bit more. Knowing that I should be looking for the regression loss was excellent information. I changed the smoothing, so I can see if it was decreasing and it remains flat, so I think it is not benefiting me to continue to train additional steps, if I am understanding this correctly.

It really does not sound like me. The only parallel is that it is a male voice and it is reasonable deep in comparison but the prosody sounds like a different person.

Maybe try an even higher learning rate, maybe 1e-3

I will give that a try.

Another option would be to train from scratch without finetuning.

That will be my next thing I try after changing the learning rate.

[Flux9665]

I just recorded 100 sentences of myself and I will see if I can get a model to sound like myself with that.

[Flux9665]

This is the regression loss of a run I just started. I am using 10 minutes of data I just recorded and after 8 minutes of training it looks pretty good already. I didn't change any of the defaults in the finetuning_simple recipe.

https://api.wandb.ai/links/flux9665/uldmaz9k