Finetunining for better Results?

[Maki9009]

So I assume, the collab isn't actually finetuning. This makes sense because for some voices I can get close enough for it sounds good. But like let's say my voice. I have no idea why but it keeps making me sound like a Posh British man.

Any way to improve the cloning of somevoices?

[stargan2vc]

The scripts won't be released sadly

[indieshack]

+1 for this. I have an actor that I'm good friends, he in his 90's now, who recorded narration for a documentary project of mine some years ago. The project content has changed somewhat and I'd like to clone his voice (with his permission of course, he's still very mentally alert and finds the tortoise project fascinating) as it was when he recorded the original narration. I get the concern to not allow high accuracy cloning for various reasons, I think the need outweighs the downsides - I'm quite sure technology like this is already being used by state agencies (probably including our own).

[neonbjb]

I partnered with a company to make this fine-tuning technology available for exactly what you are describing, @indieshack: https://play.ht/pricing/

It's not currently cheap to clone voices, but that's because it is quite costly to rent the amount of compute required to perform the initial fine-tuning process. I know play.ht is really interested in supporting use cases like yours, so you might consider contacting them directly and maybe they will set up some sort of one-time fee for you.

[deviandice]

I'm confused. You've stated many times you can't release the training & fine tuning code because it's unethical, but it was an ethical choice to release the training code to a specific company who makes money from it and gives you a cut?

[Randy-H0]

That's what happens with greed. I was hoping this for to be open sourced and stuff. This just doesn't make sense. Creator is worried about unethical purposes yet they sell it all to a company!? Like come on, "supporting use cases like yours" or more interested in the money you provide? I feel like it's the latter one.

You can't just come in here, then tell us you won't be sharing the code, and then sell it to a company and advertise said company here

[Randy-H0]

If you really felt like you wanted to support use cases like indieshack provided, then support them instead of slapping a paywall behind everything

[indieshack]

I partnered with a company to make this fine-tuning technology available for exactly what you are describing, @indieshack: https://play.ht/pricing/

It's not currently cheap to clone voices, but that's because it is quite costly to rent the amount of compute required to perform the initial fine-tuning process. I know play.ht is really interested in supporting use cases like yours, so you might consider contacting them directly and maybe they will set up some sort of one-time fee for you.

Thanks - took a look, all I can see are pre-cooked voices, I want to actually clone my friend's voice.
BTW, in respect to other comments my view is that it's perfectly OK to make a few $$ out of programming. I think it's a great project with lots of potential.

[altryne]

play.ht is actually a cool service, and I assume @neonbjb chose them as they have a strong "personal" verification (they require you to record a consent using your own voice before cloning)

I'd also love for someone like @UberDuck to have access to this, @neonbjb did you consider licensing this to other places?

[Randy-H0]

play.ht is actually a cool service, and I assume @neonbjb chose them as they have a strong "personal" verification (they require you to record a consent using your own voice before cloning)

I'd also love for someone like @UberDuck to have access to this, @neonbjb did you consider licensing this to other places?

Uberduck uses open source stuff, 99.99% of models are made by the community and it's all made using open source techniques, switching to tortoise would mean that the training code would need to be open sourced too, since The community, responsible of 99.99% of models, uses colab notebooks

[neonbjb]

I've withheld posting here for a bit to try and temper my anger at the community.

but it was an ethical choice to release the training code to a specific company who makes money from it and gives you a cut?

Yes. When people in my culture want to do something that is ethically risky and requires more work than any one person can invest, they generally do so behind a corporation. It's a mechanism to allow people to work together for shared profits and to shield individuals from personal risk. Ack that not all cultures and value systems work this way.

The people who run play.ht genuinely care about their users and want to make this kind of TTS available to everyone. If you truly believe that it is my moral imperative to releasing fine-tuning to the world, then I think this is the correct way to do it. Most people who would actually use such a system do not have access to the GPUs or knowhow required to fine-tune these models. By partnering with a company who can offer this as a service, I am able to allow considerably more people access to this feature than if I simply open sourced it. The exception to the former statement is that if, after open sourcing it, someone else started a company that did the same thing play.ht is doing.

That's what happens with greed. I was hoping this for to be open sourced and stuff. This just doesn't make sense. Creator is worried about unethical purposes yet they sell it all to a company!? Like come on, "supporting use cases like yours" or more interested in the money you provide? I feel like it's the latter one.

Give me a break, man. I didn't "sell it all to a company". I have made very little money from my arrangement with play.ht; basically just contract labor rates. I did get a small amount of stock, which is how I get to share in their success. Regardless, this isn't about money for me at all:

• I built Tortoise because I'm a nerd who likes to program and I have a particular interest in machine learning.
• I open sourced it (despite the obvious commercial potential) because I have no interest in running a business and because money isn't really a huge motivating factor for me.
• I withdrew from active development on it because I am extremely concerned with how it could be abused, and I do not want my name to be associated with the abuse that will happen if I fully release and document fine-tuning.
• I partnered with play.ht because the community (here and through personal communication) convinced me that there was a need for this and partnering with a company that could control the technology seemed like a good avenue for making this happen.

Your response to this is totally out of line. I am a long time contributor to open source (and open content in general on the web), and will continue to contribute. What have you contributed? How would you feel if I commented on your projects saying "you didn't give enough away, please give more"? Or "your ethical views are wrong and you don't deserve to make decisions about your creations"?

[Randy-H0]

I've withheld posting here for a bit to try and temper my anger at the community.

but it was an ethical choice to release the training code to a specific company who makes money from it and gives you a cut?

Yes. When people in my culture want to do something that is ethically risky and requires more work than any one person can invest, they generally do so behind a corporation. It's a mechanism to allow people to work together for shared profits and to shield individuals from personal risk. Ack that not all cultures and value systems work this way.

The people who run play.ht genuinely care about their users and want to make this kind of TTS available to everyone. If you truly believe that it is my moral imperative to releasing fine-tuning to the world, then I think this is the correct way to do it. Most people who would actually use such a system do not have access to the GPUs or knowhow required to fine-tune these models. By partnering with a company who can offer this as a service, I am able to allow considerably more people access to this feature than if I simply open sourced it. The exception to the former statement is that if, after open sourcing it, someone else started a company that did the same thing play.ht is doing.

That's what happens with greed. I was hoping this for to be open sourced and stuff. This just doesn't make sense. Creator is worried about unethical purposes yet they sell it all to a company!? Like come on, "supporting use cases like yours" or more interested in the money you provide? I feel like it's the latter one.

Give me a break, man. I didn't "sell it all to a company". I have made very little money from my arrangement with play.ht; basically just contract labor rates. I did get a small amount of stock, which is how I get to share in their success. Regardless, this isn't about money for me at all:

• I built Tortoise because I'm a nerd who likes to program and I have a particular interest in machine learning.
• I open sourced it (despite the obvious commercial potential) because I have no interest in running a business and because money isn't really a huge motivating factor for me.
• I withdrew from active development on it because I am extremely concerned with how it could be abused, and I do not want my name to be associated with the abuse that will happen if I fully release and document fine-tuning.
• I partnered with play.ht because the community (here and through personal communication) convinced me that there was a need for this and partnering with a company that could control the technology seemed like a good avenue for making this happen.

Your response to this is totally out of line. I am a long time contributor to open source (and open content in general on the web), and will continue to contribute. What have you contributed? How would you feel if I commented on your projects saying "you didn't give enough away, please give more"? Or "your ethical views are wrong and you don't deserve to make decisions about your creations"?

The reason of open sourcing is that other people will be able to use your creations for themselves without having them to do it themselves. They get a gift from you and they're happy you gave it to them. The version of Tortoise that is open source is not bad but not usable either. Look at Nvidia, they made tacotron2 open source, stability made stable diffusion open source, OpenAi made jukebox open source, the world is improving in front of us. But OpenAi out gpt-3 behind a paywall, and they also put Dalle behind a paywall, did people like that even though there's literally a free trail? No! They gave backlash and the community also then invented other things like stability. This will happen to tortoise, not to spite you. Tortoise could've been used by a lot more people but when it's put behind a paywall of 45 USD per month, then no one's going to pay that really.

I haven't contributed much to the open source community because I can't code, I don't have a good GPU to train stuff and I also don't have the time to do so. I've tried my best to make a model for uberduck called CRUST, it's a multispeaker model trained on 20 hours and 168 speakers. It works good enough for me and I open sourced it for everyone to ever find it. People who can use it can train on as low as 30 seconds of data with reasonable results if the voice is "generic". Everyone can use it and that makes me happy. That also sparked another type of this model that's also open sourced.

The point is, make people happy without a paywall, way more people can enjoy it then

[iamkhalidbashir]

Subscribing just for the drama… ;)

[iamkhalidbashir]

@neonbjb regardless of your choice, we love you to opensource the model :)
But It would be waaaaay better if you open-sourced the training code instead of model 😁
Anyways thanks.

[iamkhalidbashir]

Also I think VALL-E from microsoft uses the same concept that this repo uses ? Lets hope they release the codebase, I heard it will be at end of jan

[Randy-H0]

Also I think VALL-E from microsoft uses the same concept that this repo uses ? Lets hope they release the codebase, I heard it will be at end of jan

Finally open source realistic voice cloning