This repository was archived by the owner on Jun 9, 2025. It is now read-only.
This repository was archived by the owner on Jun 9, 2025. It is now read-only.
training never start #190
Description
hello, im following this video with wsl docker environment.
i was successfully create short 4min of dataset.
here is my train configures
{
"epochs": 20,
"learning_rate": 0.0001,
"mel_lr_weight": 1,
"text_lr_weight": 1,
"learning_rate_scheme": "Cos. Annealing",
"learning_rate_schedule": "",
"learning_rate_restarts": 4,
"batch_size": 34,
"gradient_accumulation_size": 4,
"save_rate": 5,
"validation_rate": 5,
"half_p": false,
"bitsandbytes": true,
"validation_enabled": false,
"workers": 2,
"gpus": 1,
"source_model": "./models/tortoise/autoregressive.pth",
"resume_state": "",
"voice": "mio"
}
when i start training, below errors are occur.
Loaded vocoder model
Loaded TTS, ready for generation.
DEBUG:matplotlib.pyplot:Loaded backend agg version v2.2.
Unloaded TTS
Spawning process: ./train.sh ./training/mio/train.yaml
[Training] [2025-05-05T22:09:21.042841] ./train.sh: line 2: ./venv/bin/activate: No such file or directory
[Training] [2025-05-05T22:09:25.524093] /home/user/ai-voice-cloning/modules/dlas/dlas/trainer/base_model.py:6: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead.
[Training] [2025-05-05T22:09:25.525247] from torch.distributed.optim import ZeroRedundancyOptimizer
[Training] [2025-05-05T22:09:26.009507] 25-05-05 22:09:26.009 - INFO: name: mio
[Training] [2025-05-05T22:09:26.010469] model: extensibletrainer
[Training] [2025-05-05T22:09:26.011056] scale: 1
[Training] [2025-05-05T22:09:26.011621] gpu_ids: [0]
[Training] [2025-05-05T22:09:26.012193] start_step: 0
[Training] [2025-05-05T22:09:26.012703] checkpointing_enabled: True
[Training] [2025-05-05T22:09:26.013245] fp16: False
[Training] [2025-05-05T22:09:26.013807] bitsandbytes: True
[Training] [2025-05-05T22:09:26.014319] gpus: 1
[Training] [2025-05-05T22:09:26.014857] datasets:[
[Training] [2025-05-05T22:09:26.015340] train:[
[Training] [2025-05-05T22:09:26.015843] name: training
[Training] [2025-05-05T22:09:26.016337] n_workers: 2
[Training] [2025-05-05T22:09:26.016886] batch_size: 34
[Training] [2025-05-05T22:09:26.017502] mode: paired_voice_audio
[Training] [2025-05-05T22:09:26.018034] path: ./training/mio/train.txt
[Training] [2025-05-05T22:09:26.018558] fetcher_mode: ['lj']
[Training] [2025-05-05T22:09:26.019103] phase: train
[Training] [2025-05-05T22:09:26.019645] max_wav_length: 255995
[Training] [2025-05-05T22:09:26.020174] max_text_length: 200
[Training] [2025-05-05T22:09:26.020673] sample_rate: 22050
[Training] [2025-05-05T22:09:26.021182] load_conditioning: True
[Training] [2025-05-05T22:09:26.021743] num_conditioning_candidates: 2
[Training] [2025-05-05T22:09:26.022290] conditioning_length: 44000
[Training] [2025-05-05T22:09:26.022820] use_bpe_tokenizer: True
[Training] [2025-05-05T22:09:26.023358] tokenizer_vocab: ./models/tokenizers/ja_tokenizer.json
[Training] [2025-05-05T22:09:26.024144] load_aligned_codes: False
[Training] [2025-05-05T22:09:26.024680] data_type: img
[Training] [2025-05-05T22:09:26.025248] ]
[Training] [2025-05-05T22:09:26.025885] val:[
[Training] [2025-05-05T22:09:26.026528] name: validation
[Training] [2025-05-05T22:09:26.027093] n_workers: 2
[Training] [2025-05-05T22:09:26.027727] batch_size: 8
[Training] [2025-05-05T22:09:26.028249] mode: paired_voice_audio
[Training] [2025-05-05T22:09:26.028762] path: ./training/mio/validation.txt
[Training] [2025-05-05T22:09:26.029275] fetcher_mode: ['lj']
[Training] [2025-05-05T22:09:26.029823] phase: val
[Training] [2025-05-05T22:09:26.030433] max_wav_length: 255995
[Training] [2025-05-05T22:09:26.030964] max_text_length: 200
[Training] [2025-05-05T22:09:26.031594] sample_rate: 22050
[Training] [2025-05-05T22:09:26.032429] load_conditioning: True
[Training] [2025-05-05T22:09:26.032951] num_conditioning_candidates: 2
[Training] [2025-05-05T22:09:26.033486] conditioning_length: 44000
[Training] [2025-05-05T22:09:26.033985] use_bpe_tokenizer: True
[Training] [2025-05-05T22:09:26.034509] tokenizer_vocab: ./models/tokenizers/ja_tokenizer.json
[Training] [2025-05-05T22:09:26.035011] load_aligned_codes: False
[Training] [2025-05-05T22:09:26.035513] data_type: img
[Training] [2025-05-05T22:09:26.036045] ]
[Training] [2025-05-05T22:09:26.036676] ]
[Training] [2025-05-05T22:09:26.037170] steps:[
[Training] [2025-05-05T22:09:26.037708] gpt_train:[
[Training] [2025-05-05T22:09:26.038227] training: gpt
[Training] [2025-05-05T22:09:26.038778] loss_log_buffer: 500
[Training] [2025-05-05T22:09:26.039418] optimizer: adamw
[Training] [2025-05-05T22:09:26.040558] optimizer_params:[
[Training] [2025-05-05T22:09:26.042322] lr: 0.0001
[Training] [2025-05-05T22:09:26.044064] weight_decay: 0.01
[Training] [2025-05-05T22:09:26.045781] beta1: 0.9
[Training] [2025-05-05T22:09:26.047401] beta2: 0.96
[Training] [2025-05-05T22:09:26.049267] ]
[Training] [2025-05-05T22:09:26.050978] clip_grad_eps: 4
[Training] [2025-05-05T22:09:26.052570] injectors:[
[Training] [2025-05-05T22:09:26.053655] paired_to_mel:[
[Training] [2025-05-05T22:09:26.055096] type: torch_mel_spectrogram
[Training] [2025-05-05T22:09:26.056463] mel_norm_file: ./modules/tortoise-tts/tortoise/data/mel_norms.pth
[Training] [2025-05-05T22:09:26.057798] in: wav
[Training] [2025-05-05T22:09:26.058976] out: paired_mel
[Training] [2025-05-05T22:09:26.061151] ]
[Training] [2025-05-05T22:09:26.063005] paired_cond_to_mel:[
[Training] [2025-05-05T22:09:26.064905] type: for_each
[Training] [2025-05-05T22:09:26.066561] subtype: torch_mel_spectrogram
[Training] [2025-05-05T22:09:26.067889] mel_norm_file: ./modules/tortoise-tts/tortoise/data/mel_norms.pth
[Training] [2025-05-05T22:09:26.069644] in: conditioning
[Training] [2025-05-05T22:09:26.070862] out: paired_conditioning_mel
[Training] [2025-05-05T22:09:26.072906] ]
[Training] [2025-05-05T22:09:26.074727] to_codes:[
[Training] [2025-05-05T22:09:26.077130] type: discrete_token
[Training] [2025-05-05T22:09:26.079691] in: paired_mel
[Training] [2025-05-05T22:09:26.081598] out: paired_mel_codes
[Training] [2025-05-05T22:09:26.083514] dvae_config: ./models/tortoise/train_diffusion_vocoder_22k_level.yml
[Training] [2025-05-05T22:09:26.084737] ]
[Training] [2025-05-05T22:09:26.086301] paired_fwd_text:[
[Training] [2025-05-05T22:09:26.088345] type: generator
[Training] [2025-05-05T22:09:26.090853] generator: gpt
[Training] [2025-05-05T22:09:26.092852] in: ['paired_conditioning_mel', 'padded_text', 'text_lengths', 'paired_mel_codes', 'wav_lengths']
[Training] [2025-05-05T22:09:26.094197] out: ['loss_text_ce', 'loss_mel_ce', 'logits']
[Training] [2025-05-05T22:09:26.095092] ]
[Training] [2025-05-05T22:09:26.095631] ]
[Training] [2025-05-05T22:09:26.096156] losses:[
[Training] [2025-05-05T22:09:26.096990] text_ce:[
[Training] [2025-05-05T22:09:26.097696] type: direct
[Training] [2025-05-05T22:09:26.098313] weight: 1
[Training] [2025-05-05T22:09:26.098884] key: loss_text_ce
[Training] [2025-05-05T22:09:26.099378] ]
[Training] [2025-05-05T22:09:26.099984] mel_ce:[
[Training] [2025-05-05T22:09:26.100548] type: direct
[Training] [2025-05-05T22:09:26.101110] weight: 1
[Training] [2025-05-05T22:09:26.101676] key: loss_mel_ce
[Training] [2025-05-05T22:09:26.102202] ]
[Training] [2025-05-05T22:09:26.102726] ]
[Training] [2025-05-05T22:09:26.103240] ]
[Training] [2025-05-05T22:09:26.103757] ]
[Training] [2025-05-05T22:09:26.104314] networks:[
[Training] [2025-05-05T22:09:26.104987] gpt:[
[Training] [2025-05-05T22:09:26.105535] type: generator
[Training] [2025-05-05T22:09:26.106180] which_model_G: unified_voice2
[Training] [2025-05-05T22:09:26.106881] kwargs:[
[Training] [2025-05-05T22:09:26.107605] layers: 30
[Training] [2025-05-05T22:09:26.108303] model_dim: 1024
[Training] [2025-05-05T22:09:26.109244] heads: 16
[Training] [2025-05-05T22:09:26.109920] max_text_tokens: 402
[Training] [2025-05-05T22:09:26.110699] max_mel_tokens: 604
[Training] [2025-05-05T22:09:26.111477] max_conditioning_inputs: 2
[Training] [2025-05-05T22:09:26.112887] mel_length_compression: 1024
[Training] [2025-05-05T22:09:26.114024] number_text_tokens: 256
[Training] [2025-05-05T22:09:26.114986] number_mel_codes: 8194
[Training] [2025-05-05T22:09:26.115469] start_mel_token: 8192
[Training] [2025-05-05T22:09:26.116186] stop_mel_token: 8193
[Training] [2025-05-05T22:09:26.119310] start_text_token: 255
[Training] [2025-05-05T22:09:26.122739] train_solo_embeddings: False
[Training] [2025-05-05T22:09:26.123438] use_mel_codes_as_input: True
[Training] [2025-05-05T22:09:26.124507] checkpointing: True
[Training] [2025-05-05T22:09:26.125653] tortoise_compat: True
[Training] [2025-05-05T22:09:26.126341] ]
[Training] [2025-05-05T22:09:26.127035] ]
[Training] [2025-05-05T22:09:26.127664] ]
[Training] [2025-05-05T22:09:26.134083] path:[
[Training] [2025-05-05T22:09:26.139816] strict_load: True
[Training] [2025-05-05T22:09:26.141103] pretrain_model_gpt: ./models/tortoise/autoregressive.pth
[Training] [2025-05-05T22:09:26.141691] root: ./
[Training] [2025-05-05T22:09:26.142346] experiments_root: ./training/mio/finetune
[Training] [2025-05-05T22:09:26.142894] models: ./training/mio/finetune/models
[Training] [2025-05-05T22:09:26.143839] training_state: ./training/mio/finetune/training_state
[Training] [2025-05-05T22:09:26.144367] log: ./training/mio/finetune
[Training] [2025-05-05T22:09:26.146233] val_images: ./training/mio/finetune/val_images
[Training] [2025-05-05T22:09:26.149996] ]
[Training] [2025-05-05T22:09:26.152430] train:[
[Training] [2025-05-05T22:09:26.156041] niter: 20
[Training] [2025-05-05T22:09:26.162884] warmup_iter: -1
[Training] [2025-05-05T22:09:26.167282] mega_batch_factor: 4
[Training] [2025-05-05T22:09:26.180670] val_freq: 5
[Training] [2025-05-05T22:09:26.182713] ema_enabled: False
[Training] [2025-05-05T22:09:26.185135] default_lr_scheme: CosineAnnealingLR_Restart
[Training] [2025-05-05T22:09:26.186587] T_period: [5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5]
[Training] [2025-05-05T22:09:26.188407] warmup: 0
[Training] [2025-05-05T22:09:26.190143] eta_min: 1e-08
[Training] [2025-05-05T22:09:26.192035] restarts: [5, 10, 15, 20]
[Training] [2025-05-05T22:09:26.194478] restart_weights: [0.75, 0.5, 0.25, 0.125]
[Training] [2025-05-05T22:09:26.196446] ]
[Training] [2025-05-05T22:09:26.198473] eval:[
[Training] [2025-05-05T22:09:26.201798] pure: False
[Training] [2025-05-05T22:09:26.203363] output_state: gen
[Training] [2025-05-05T22:09:26.204991] ]
[Training] [2025-05-05T22:09:26.206398] logger:[
[Training] [2025-05-05T22:09:26.208080] save_checkpoint_freq: 5
[Training] [2025-05-05T22:09:26.210480] visuals: ['gen', 'mel']
[Training] [2025-05-05T22:09:26.212287] visual_debug_rate: 5
[Training] [2025-05-05T22:09:26.213198] is_mel_spectrogram: True
[Training] [2025-05-05T22:09:26.213869] ]
[Training] [2025-05-05T22:09:26.214540] is_train: True
[Training] [2025-05-05T22:09:26.215220] dist: False
[Training] [2025-05-05T22:09:26.215788]
[Training] [2025-05-05T22:09:26.216340] 25-05-05 22:09:26.009 - INFO: Random seed: 8123
[Training] [2025-05-05T22:09:29.525536] 25-05-05 22:09:29.525 - INFO: Number of training data elements: 34, iters: 1
[Training] [2025-05-05T22:09:29.526385] 25-05-05 22:09:29.525 - INFO: Total epochs needed: 20 for iters 20
[Training] [2025-05-05T22:09:29.542500] /home/user/ai-voice-cloning/modules/dlas/dlas/models/lucidrains/vq.py:188: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
[Training] [2025-05-05T22:09:29.543324] @autocast(enabled=False)
[Training] [2025-05-05T22:09:29.544003] /home/user/ai-voice-cloning/modules/dlas/dlas/models/lucidrains/vq.py:305: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
[Training] [2025-05-05T22:09:29.544614] @autocast(enabled=False)
[Training] [2025-05-05T22:09:30.129661] /home/user/miniconda/lib/python3.11/site-packages/rotary_embedding_torch/rotary_embedding_torch.py:33: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
[Training] [2025-05-05T22:09:30.130550] @autocast(enabled = False)
[Training] [2025-05-05T22:09:30.137779] /home/user/miniconda/lib/python3.11/site-packages/rotary_embedding_torch/rotary_embedding_torch.py:231: FutureWarning: `torch.cuda.amp.autocast(args...)` is deprecated. Please use `torch.amp.autocast('cuda', args...)` instead.
[Training] [2025-05-05T22:09:30.138419] @autocast(enabled = False)
[Training] [2025-05-05T22:09:30.726224] /home/user/miniconda/lib/python3.11/site-packages/transformers/configuration_utils.py:380: UserWarning: Passing `gradient_checkpointing` to a config initialization is deprecated and will be removed in v5 Transformers. Using `model.gradient_checkpointing_enable()` instead, or if you are using the `Trainer` API, pass `gradient_checkpointing=True` in your `TrainingArguments`.
[Training] [2025-05-05T22:09:30.727238] warnings.warn(
[Training] [2025-05-05T22:09:37.112657] /home/user/ai-voice-cloning/modules/dlas/dlas/trainer/steps.py:31: FutureWarning: `torch.cuda.amp.GradScaler(args...)` is deprecated. Please use `torch.amp.GradScaler('cuda', args...)` instead.
[Training] [2025-05-05T22:09:37.118404] self.scaler = GradScaler(enabled=self.opt['fp16'] or opt_get(
[Training] [2025-05-05T22:09:37.185316] /home/user/ai-voice-cloning/modules/dlas/dlas/trainer/injectors/audio_injectors.py:83: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
[Training] [2025-05-05T22:09:37.198026] self.mel_norms = torch.load(self.mel_norm_file)
[Training] [2025-05-05T22:09:37.784556] /home/user/ai-voice-cloning/modules/dlas/dlas/utils/util.py:529: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
[Training] [2025-05-05T22:09:37.786439] sd = torch.load(load_path, map_location=device)
[Training] [2025-05-05T22:09:38.718740] 25-05-05 22:09:38.718 - INFO: Loading model for [./models/tortoise/autoregressive.pth]
[Training] [2025-05-05T22:09:38.719811] INFO:base:Loading model for [./models/tortoise/autoregressive.pth]
[Training] [2025-05-05T22:09:38.720432] /home/user/ai-voice-cloning/modules/dlas/dlas/trainer/base_model.py:117: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
[Training] [2025-05-05T22:09:38.721017] load_net = torch.load(
[Training] [2025-05-05T22:09:45.587190] 25-05-05 22:09:45.575 - INFO: Start training from epoch: 0, iter: 0
[Training] [2025-05-05T22:09:45.587259] INFO:base:Start training from epoch: 0, iter: 0
[Training] [2025-05-05T22:09:45.587290] Disabled distributed training.
[Training] [2025-05-05T22:09:45.587308] Path already exists. Rename it to [./training/mio/finetune_archived_250505-220925]
[Training] [2025-05-05T22:09:45.587324] Loading from ./models/tortoise/dvae.pth
[Training] [2025-05-05T22:09:46.588699] Traceback (most recent call last):
[Training] [2025-05-05T22:09:46.588749] File "/home/user/ai-voice-cloning/./src/train.py", line 72, in <module>
[Training] [2025-05-05T22:09:46.589344] train(config_path, args.launcher)
[Training] [2025-05-05T22:09:46.589373] File "/home/user/ai-voice-cloning/./src/train.py", line 39, in train
[Training] [2025-05-05T22:09:46.589442] trainer.do_training()
[Training] [2025-05-05T22:09:46.589474] File "/home/user/ai-voice-cloning/modules/dlas/dlas/train.py", line 406, in do_training
[Training] [2025-05-05T22:09:46.589997] for train_data in tq_ldr:
[Training] [2025-05-05T22:09:46.590026] File "/home/user/miniconda/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 701, in __next__
[Training] [2025-05-05T22:09:46.590152] data = self._next_data()
[Training] [2025-05-05T22:09:46.590201] ^^^^^^^^^^^^^^^^^
[Training] [2025-05-05T22:09:46.590229] File "/home/user/miniconda/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 1465, in _next_data
[Training] [2025-05-05T22:09:46.590432] return self._process_data(data)
[Training] [2025-05-05T22:09:46.590467] ^^^^^^^^^^^^^^^^^^^^^^^^
[Training] [2025-05-05T22:09:46.590486] File "/home/user/miniconda/lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 1491, in _process_data
[Training] [2025-05-05T22:09:46.590680] data.reraise()
[Training] [2025-05-05T22:09:46.590718] File "/home/user/miniconda/lib/python3.11/site-packages/torch/_utils.py", line 715, in reraise
[Training] [2025-05-05T22:09:46.591533] raise exception
[Training] [2025-05-05T22:09:46.591646] AssertionError: Caught AssertionError in DataLoader worker process 0.
[Training] [2025-05-05T22:09:46.591675] Original Traceback (most recent call last):
[Training] [2025-05-05T22:09:46.591699] File "/home/user/ai-voice-cloning/modules/dlas/dlas/data/audio/paired_voice_audio_dataset.py", line 218, in __getitem__
[Training] [2025-05-05T22:09:46.591724] tseq, wav, text, path, type = self.get_wav_text_pair(
[Training] [2025-05-05T22:09:46.591749] ^^^^^^^^^^^^^^^^^^^^^^^
[Training] [2025-05-05T22:09:46.591775] File "/home/user/ai-voice-cloning/modules/dlas/dlas/data/audio/paired_voice_audio_dataset.py", line 201, in get_wav_text_pair
[Training] [2025-05-05T22:09:46.591803] text_seq = self.get_text(text)
[Training] [2025-05-05T22:09:46.591826] ^^^^^^^^^^^^^^^^^^^
[Training] [2025-05-05T22:09:46.591841] File "/home/user/ai-voice-cloning/modules/dlas/dlas/data/audio/paired_voice_audio_dataset.py", line 210, in get_text
[Training] [2025-05-05T22:09:46.591851] assert not torch.any(tokens == 1)
# ... skipped duplicated lines...
[Training] [2025-05-05T22:09:46.615064] AssertionError
[Training] [2025-05-05T22:09:46.615088]
[Training] [2025-05-05T22:09:46.615114] During handling of the above exception, another exception occurred:
[Training] [2025-05-05T22:09:46.615126]
[Training] [2025-05-05T22:09:46.615132] Traceback (most recent call last):
[Training] [2025-05-05T22:09:46.615156] File "/home/user/ai-voice-cloning/modules/dlas/dlas/data/audio/paired_voice_audio_dataset.py", line 218, in __getitem__
[Training] [2025-05-05T22:09:46.615181] tseq, wav, text, path, type = self.get_wav_text_pair(
[Training] [2025-05-05T22:09:46.615205] ^^^^^^^^^^^^^^^^^^^^^^^
[Training] [2025-05-05T22:09:46.615228] File "/home/user/ai-voice-cloning/modules/dlas/dlas/data/audio/paired_voice_audio_dataset.py", line 201, in get_wav_text_pair
[Training] [2025-05-05T22:09:46.615281] text_seq = self.get_text(text)
[Training] [2025-05-05T22:09:46.615306] ^^^^^^^^^^^^^^^^^^^
[Training] [2025-05-05T22:09:46.615330] File "/home/user/ai-voice-cloning/modules/dlas/dlas/data/audio/paired_voice_audio_dataset.py", line 210, in get_text
[Training] [2025-05-05T22:09:46.615354] assert not torch.any(tokens == 1)
[Training] [2025-05-05T22:09:46.615380] AssertionError
[Training] [2025-05-05T22:09:46.615394]
[Training] [2025-05-05T22:09:46.615403] During handling of the above exception, another exception occurred:
[Training] [2025-05-05T22:09:46.615414]
[Training] [2025-05-05T22:09:46.615426] Traceback (most recent call last):
[Training] [2025-05-05T22:09:46.615449] File "/home/user/miniconda/lib/python3.11/site-packages/torch/utils/data/_utils/worker.py", line 351, in _worker_loop
[Training] [2025-05-05T22:09:46.615495] data = fetcher.fetch(index) # type: ignore[possibly-undefined]
[Training] [2025-05-05T22:09:46.615516] ^^^^^^^^^^^^^^^^^^^^
[Training] [2025-05-05T22:09:46.615538] File "/home/user/miniconda/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
[Training] [2025-05-05T22:09:46.615558] data = [self.dataset[idx] for idx in possibly_batched_index]
[Training] [2025-05-05T22:09:46.615577] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Training] [2025-05-05T22:09:46.615597] File "/home/user/miniconda/lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 52, in <listcomp>
[Training] [2025-05-05T22:09:46.615605] data = [self.dataset[idx] for idx in possibly_batched_index]
[Training] [2025-05-05T22:09:46.615611] ~~~~~~~~~~~~^^^^^
[Training] [2025-05-05T22:09:46.615617] File "/home/user/ai-voice-cloning/modules/dlas/dlas/data/audio/paired_voice_audio_dataset.py", line 233, in __getitem__
[Training] [2025-05-05T22:09:46.615624] return self[(index+1) % len(self)]
[Training] [2025-05-05T22:09:46.615629] ~~~~^^^^^^^^^^^^^^^^^^^^^^^
[Training] [2025-05-05T22:09:46.615635] File "/home/user/ai-voice-cloning/modules/dlas/dlas/data/audio/paired_voice_audio_dataset.py", line 233, in __getitem__
[Training] [2025-05-05T22:09:46.615641] return self[(index+1) % len(self)]
[Training] [2025-05-05T22:09:46.615647] ~~~~^^^^^^^^^^^^^^^^^^^^^^^
[Training] [2025-05-05T22:09:46.615666] File "/home/user/ai-voice-cloning/modules/dlas/dlas/data/audio/paired_voice_audio_dataset.py", line 233, in __getitem__
[Training] [2025-05-05T22:09:46.615685] return self[(index+1) % len(self)]
[Training] [2025-05-05T22:09:46.615715] ~~~~^^^^^^^^^^^^^^^^^^^^^^^
[Training] [2025-05-05T22:09:46.615734] [Previous line repeated 97 more times]
[Training] [2025-05-05T22:09:46.615753] File "/home/user/ai-voice-cloning/modules/dlas/dlas/data/audio/paired_voice_audio_dataset.py", line 218, in __getitem__
[Training] [2025-05-05T22:09:46.615770] tseq, wav, text, path, type = self.get_wav_text_pair(
[Training] [2025-05-05T22:09:46.615789] ^^^^^^^^^^^^^^^^^^^^^^^
[Training] [2025-05-05T22:09:46.615797] File "/home/user/ai-voice-cloning/modules/dlas/dlas/data/audio/paired_voice_audio_dataset.py", line 201, in get_wav_text_pair
[Training] [2025-05-05T22:09:46.615816] text_seq = self.get_text(text)
[Training] [2025-05-05T22:09:46.615823] ^^^^^^^^^^^^^^^^^^^
[Training] [2025-05-05T22:09:46.615841] File "/home/user/ai-voice-cloning/modules/dlas/dlas/data/audio/paired_voice_audio_dataset.py", line 210, in get_text
[Training] [2025-05-05T22:09:46.615860] assert not torch.any(tokens == 1)
[Training] [2025-05-05T22:09:46.615878] AssertionError
[Training] [2025-05-05T22:09:46.615897]
[Training] [2025-05-05T22:09:58.374239] DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
i did before post
- pull again (obviously nothing happened, i did not touch any source files)
- build docker image again with
./setup-docker.sh
any help would be appreciate!