Synthetic dataset

[skirdey]

Hi folks! Love this package, lots of useful features!

I wanted to propose a synthetic speech dataset that would include generated speech data of healthy controls - meaning speech that lacks any specific biomarkers of physical or mental health conditions.
The idea is that synthetic speech, sampled from models like XTTS2 or cloned from healthy individuals can serve as a great baseline for many experiments in voice biomarkers and digital phenotyping.

I see the package already integrates with huggingface, it could be a perfect place to host synthetic speech dataset that could also serve power the unit tests.

At Amplifier, we can provide GPU compute to get the first iteration of data generated and analyzed.

The composition of the dataset is still up in the air, I was thinking something balanced across 4 age groups, biological sexes such as male, female and intersex, and languages starting with English and Spanish.

We can provide analysis from our foundational biomarker model to gain confidence that synthetic speech lacks particular health condition signals, and I am sure you have pre-trained models that detect specific impairment that can also be used to eliminate samples that are on the border between healthy controls and tests.

The value of synthetic speech dataset can be varied and helpful for researchers and startups tackling voice as vital sign problem or speech accessibility problems.

Let me know what you folks think!

[fabiocat93]

Hi @skirdey, that’s a great idea! Thanks for sharing it!

Here are some points to think about:

• What Does "Healthy" Mean? In a general sense, what does it mean for speech to be "healthy"? In some studies targeting a specific pathology, it's easy to classify anything, not that pathology as the healthy control group. But here, since we don't have specific disorders or diseases in mind, are we talking about the absence of any physical or mental health indicators or something more specific? It may be important to define this clearly.

• People might want the healthy controls to match in terms of sentences, emotions, and other variables. Should we create a standard dataset with pre-defined samples or a flexible method/pipeline that allows users to generate a healthy control dataset on demand? Each approach has its pros and cons. What do you think?

• I see how the dataset can be validated by checking for some specific biomarkers. However, I would argue that there are several more aspects to consider. For instance,
  -- Do we want to verify the identity of the pseudo-speakers to ensure they don't resemble real individuals too closely and that utterances by the same pseudo-speaker can be consistently assigned to the same pseudo-speaker?
  -- Should we test the intelligibility of the synthetic speech to make sure it's clear and understandable?
  -- Do we want to check how the synthetic speech accurately conveys emotions?

I hope we can start shaping the direction of our project from yours and these points!

[fabiocat93]

For assessing speech quality and intelligibility, we can use quantitative metrics such as:

• Wideband Perceptual Estimation of Speech Quality (PESQ)
• Short-Time Objective Intelligibility (STOI)
• Scale-Invariant Signal-to-Distortion Ratio (SI-SDR)

and qualitative metrics as:

• subjective Mean Opinion Score (MOS)

They are all implemented in Torchaudio-Squim, and we can easily integrate in Senselab

[skirdey]

These are great.

For synthetic speech generation, to make sure we are conform to Apache or MIT license, here are some projects that produce natural voices:

https://github.com/huggingface/parler-tts - Apache 2.0 License - great generic voices
https://github.com/metavoiceio/metavoice-src - Apache 2.0 License, supports voice cloning and quality is quite good
https://github.com/myshell-ai/OpenVoice - MIT License
https://huggingface.co/suno/bark - MIT License
https://github.com/RVC-Boss/GPT-SoVITS - MIT License, supports voice cloning

[skirdey]

@fabiocat93 what do you think about age groups?

I thought to use approximately this mapping:
Child: 4-12 years
Adolescent: 13-19 years
Adult: 20-64 years
Senior: 65 years and older

[fabiocat93]

for TTS, I also like tortoise-tts under Apache 2.0 License, which supports voice cloning

[fabiocat93]

@fabiocat93 what do you think about age groups?

also, children tend to undergo rapid changes in their voices due to their physical growth. Hence, it would be beneficial to have smaller age segments. But I see how we have to start somewhere, hence the group ages you suggested are ok for me

[satra]

Is the plan here to essentially use cloning of actual voices to create this dataset? or a generative model to create a dataset whose estimable parameters fall into the different groups under consideration?

[satra]

in that case, we absolutely need to dig into the rights and reuse of any voice that is cloned, and keep the ethics forefront in terms of potential uses of that voice.

[skirdey]

totally, there are quite a few datasets that provide data with established healthy controls and under MIT / Apache / Creative Commons licenses - I think it will be essential to keep the trace of where cloned voice came from and under what license in a form of a document as part of the dataset

[fabiocat93]

@skirdey just to clarify, when you say voice cloning, you mean a text-to-speech model able to emulate a specific target voice, don't you? (and not a speech-to-speech model, as for example FREEVC)

[skirdey]

I would say voice to voice like FREEVC as it often captures breathing and some other features unlike pure TTS, as capturing realistic patterns from original voice could provide stronger healthy control signal.

The closest would be a system like VoiceCraft that accepts text + original voice sample

[satra]

i just played with voicecraft and it worked really nicely.

[skirdey]

I am going to do a short pass at a few examples using parler-tts and share a google drive. parler-tts seem to give consistently natural voices that are also fully synthetic and apache 2.0 licensed, but they are missing the air gasps and inhalations,, which is common miss in many other TTS systems.

For the content of audios - we can have a similar approach of having tts read out Rainbow and North Wind And The Sun passage, and then LLM to act as a non-prompted text generator to get the unprompted speech.

• https://www.york.ac.uk/media/languageandlinguistics/documents/currentstudents/linguisticsresources/Standardised-reading.pdf
• https://read.gov/aesop/143.html

[satra]

are there any evaluation metrics for naturalness in different languages? this example sounded australian to me, but would it sound natural to an australian? are there distributional metrics on phoneme durations, etc.,. per language we could use, or would we have to do ethnographic evaluations?

[skirdey]

Thanks @skirdey. I find this audio better than the others but I still have some concerns in terms of how natural it sounds.

One thought: do you think it's helpful to include single words and/or nonwords in the dataset? I say that because I suspect models may sound more natural to the human ear when they produce 1 word only. What do you think?

This I don't really know. I haven't seen any studies on how much predictive power within a single word. Using something like VAD the sample can be split into individual words as well.

[skirdey]

are there any evaluation metrics for naturalness in different languages? this example sounded australian to me, but would it sound natural to an australian? are there distributional metrics on phoneme durations, etc.,. per language we could use, or would we have to do ethnographic evaluations?

good point, I would imagine that will require humans to evaluate global or regional accents. haven't seen a metric for this sort of eval.

[skirdey]

took another pass on generating a few more examples, using all healthy controls from the Parkinson's dataset - https://drive.google.com/file/d/1uvIgc5zVDtDa4RwZ9B37tHxdljm8nEhV/view?usp=sharing

It was generated sentence by sentence using cloning built-in into metavoice-1b. I like a lot about these samples, but there are still some artifacts that sound a bit un-natural. As next step, I am going to fine-tune metavoice on a few speakers and see if it helps reduce the artifacts. LMK what you think about structure and quality.

[fabiocat93]

Oh wow! I think this one is pretty accurate already. But I agree that it sounds noisy and with noisy I mean as if it was recorded through an old-school microphone

[fabiocat93]

hey @skirdey , do you know https://github.com/Camb-ai/MARS5-TTS? it sounds very clear, and I like the prosody. I think we should take it into account. It's under GNU AGPL 3.0 license, but they say we can request to use it under a different license by emailing them.

[skirdey]

MARS5 sounds really impressive, I am going to try it out on the same controls and share the results

[skirdey]

@fabiocat93 I've tried it on a few samples, and it seems to have a bug at the moment, when initial few seconds of speech are impressive, but then performance degrades quite a lot. Here is an example. It seems artifacts start at 12 seconds https://drive.google.com/file/d/1sfDT31to_SxOvC8rq5l6RA_DUmwxwrWQ/view?usp=drive_link

[skirdey]

after some tweaks and limiting generation to 12 seconds, I really like the result, we can totally use Mars. @fabiocat93 do you want to email their team and ask about the license?

Here is an example of north sun passage split by sentence from a healthy control dataset clone - https://drive.google.com/file/d/1AbTr9BwPyKJdpkVv1K_lQ_6xODTGkloZ/view?usp=drive_link

[skirdey]

It maintains very natural breathing patterns and also produces audio in 24khz which is a nice bonus

[fabiocat93]

I am happy you liked it. I will email them and keep you all posted on what they say. Fingers crossed!

[fabiocat93]

Also, here is an interesting paper that we should take into account while designing our dataset: https://dl.acm.org/doi/pdf/10.1145/3593013.3594049