SpeechT5 TTS pipeline breaks by processor.

[jiqing-feng]

System Info

latest transformers

Who can help?

@ebezzam @vasqu @Rocketknight1

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

import torch
from transformers import pipeline

synthesiser = pipeline("text-to-speech", "microsoft/speecht5_tts")
speaker_embedding = (torch.rand(1, 512) * 0.2 - 0.1).to(torch.float16)

speech = synthesiser("Hello, my dog is cooler than you!", forward_params={"speaker_embeddings": speaker_embedding})

After PR #42326 , it got error:

Traceback (most recent call last):
  File "/home/jiqingfe/transformers/test_speecht5.py", line 7, in <module>
    speech = synthesiser("Hello, my dog is cooler than you!", forward_params={"speaker_embeddings": speaker_embedding})
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqingfe/transformers/src/transformers/pipelines/text_to_audio.py", line 252, in __call__
    return super().__call__(text_inputs, **forward_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqingfe/transformers/src/transformers/pipelines/base.py", line 1286, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqingfe/transformers/src/transformers/pipelines/base.py", line 1292, in run_single
    model_inputs = self.preprocess(inputs, **preprocess_params)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqingfe/transformers/src/transformers/pipelines/text_to_audio.py", line 175, in preprocess
    output = preprocessor(text, **kwargs, return_tensors="pt")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqingfe/transformers/src/transformers/models/speecht5/processing_speecht5.py", line 76, in __call__
    raise ValueError(
ValueError: You need to specify either an `audio`, `audio_target`, `text`, or `text_target` input to process.

Expected behavior

Previous behavior use tokenizer instead of processor.

[ebezzam]

Hi @jiqing-feng thanks for raising this issue. Indeed #42326 is the likely cause. microsoft/speecht5_tts was not in the unit tests so it got missed in ensuring backwards compatibility.

Could you open a PR to:

• Add speecht5_tts to the list of models that are expected to use the tokenizer here
• Add an integration test for microsoft/speecht5_tts similar to that of musicgen

Hope that's clear, but let me know if you have questions!

[jiqing-feng]

Hi @ebezzam . It's very clear. Please review #42830 . Thanks!