Skip to content

discrete token for audio resynthesis #423

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
South-Twilight wants to merge 27 commits into
base: master
Choose a base branch
from
Open

discrete token for audio resynthesis #423

South-Twilight wants to merge 27 commits into from

Conversation

@South-Twilight
Copy link

@South-Twilight South-Twilight commented Feb 2, 2024 •
edited
Loading

Here is the PR for audio resynthesis in discrete token:

  1. We extend hubert_voc1 to token_voc1 and it can handle more models token;
  2. We add f0 for training and inference when finding poor prounciation in singing;
  3. We add multi-stream method including residual cluster and weight sum;
  4. Using embedding feature of models is also allowed.

The following models have been validated in opencpop recipe: HuBERT, XLS-R, WavLM, MERT, Encodec.

@South-Twilight
update hubert and mert in opencpop
85e1152
@South-Twilight
update opencpop hubert(in progress)
be6e500
@South-Twilight
add f0 in hubert(WIP)
121a151
@South-Twilight
update gitigonre
95160c2
@South-Twilight
discrete unit: add process of mfcc
d68da20
@South-Twilight
merge: update merge
2b4db7e
@South-Twilight
to(1): update base info
c39020d
@South-Twilight
sync hubert/mert/mfcc
5e0e580
@South-Twilight
sync with remote(2023.9.6)
6c27921
@South-Twilight
feat: add f0
8a8dc8f
@South-Twilight
fix merge conflict: merge add_f0
f601448
@South-Twilight
update run.sh
bd56257
@South-Twilight
feat: use pretrain feature embedding as input(for top-line)
a10a575
@South-Twilight
feat: merge f0 and use pretrain feature
94add27
@South-Twilight
feat: use different layer of pretrain model
1c0f192
@South-Twilight
feat(egs/opencpop/hubert_voc1, hifigan.py):
42f8aac 
1) add f0
2) use embedding feat as input (test topline)
3) add weight sum token
@South-Twilight
fix(hifigan.py : DiscreteSymbolF0Generator)
8a9b948 
1) separate single layer config: hifigan_hubert_16k_nodp_f0.v1.yaml
2) add annotation to DiscreteSymbolF0Generator.infer
@South-Twilight
feat(hubert_voc): add evaluate MCD
56ecd19 
1) add stage 4 of run.sh -- "Scoring"
@South-Twilight
fix(hubert_voc/conf):
c4c3478 
1) update training steps from 25w to 40w
@South-Twilight
Merge remote-tracking branch 'origin/master'
9850df5
@South-Twilight
fix(hubert_voc1/run.sh):
f805522 
1) add f0 rmse,semitone acc,uvu acc evaluation indicators
@South-Twilight
Merge remote-tracking branch 'origin/master'
b27b05b
@South-Twilight
fix(hubert/voc1): update the way to calculate f0
e3359da 
1) add continuous f0
2) add yaml for 48khz wav
@South-Twilight
Merge branch 'master' of https://github.com/South-Twilight/ParallelWa…
f7e9c96 
…veGAN
@South-Twilight
feat(egs/opencpop/hubert_voc1): add multi-stream layer and fix some b…
7427151 
…ugs:

1) add multi-stream RVQ cluster
2) add 48kHz encodec token
3) update some annotations
4) remove git some useless tracks
@South-Twilight
refactor(egs/*/hubert_voc1->egs/*/token_voc1): refactor hubert_voc to…
f7249ed 
… token_voc1 for PR

1) refactor conf
2) add annotations
@South-Twilight
Merge branch 'kan-bayashi:master' into master
6107aaf
@kan-bayashi kan-bayashi self-requested a review February 5, 2024 08:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kan-bayashi kan-bayashi Awaiting requested review from kan-bayashi

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@South-Twilight