Skip to content

Add GenAI ASRPipeline evaluator#4044

Open
as-suvorov wants to merge 6 commits into
openvinotoolkit:masterfrom
as-suvorov:as/genai_asr_pipeline
Open

Add GenAI ASRPipeline evaluator#4044
as-suvorov wants to merge 6 commits into
openvinotoolkit:masterfrom
as-suvorov:as/genai_asr_pipeline

Conversation

@as-suvorov

@as-suvorov as-suvorov commented Jul 1, 2026

Copy link
Copy Markdown
Contributor
  • Add GenAI ASRPipeline evaluator
  • Add Qwen3ASROptimumPipeline evaluator
  • Fix Whisper config for optimum and hf - { num_beams: 1 }. Which aligned with GenAI configuration.

Qwen3-ASR deps requirements:
pip install --no-deps git+https://github.com/huggingface/optimum-intel.git@f48d93fddff8c91e198389c47a6d5974789b67f4
pip install qwen-asr (requires transformers==4.57.6)

Example config.json:

evaluations:
  - name: qwen3-asr-0.6b-genai
    module: custom_evaluators.asr_pipeline_evaluator.ASRPipelineEvaluator
    module_config:
      pipeline_class: GenAIASRPipeline
      launchers:
        - framework: openvino
          device: CPU
      datasets:
        - name: LibriSpeech_test_clean_flac_max_duration_15
          data_source: /home/asuvorov/projects/open_model_zoo/.venv/data/librispeech/test/LibriSpeech/test-clean
          annotation_conversion:
            converter: librispeech
            data_dir: /home/asuvorov/projects/open_model_zoo/.venv/data/librispeech/test/LibriSpeech/test-clean
            annotation_file: /home/asuvorov/projects/open_model_zoo/librispeech_test_clean_max_duration_15_10.jsonl
            use_flac: True
            max_duration: 15
            top_n: 10
          reader:
            type: flac_reader
            mono: True
            dtype: float64
          metrics:
            - type: wer

  - name: qwen3-asr-0.6b-optimum
    module: custom_evaluators.asr_pipeline_evaluator.ASRPipelineEvaluator
    module_config:
      pipeline_class: Qwen3ASROptimumPipeline
      max_new_tokens: 1000
      launchers:
        - framework: openvino
          device: CPU
      datasets:
        - name: LibriSpeech_test_clean_flac_max_duration_15
          data_source: /home/asuvorov/projects/open_model_zoo/.venv/data/librispeech/test/LibriSpeech/test-clean
          annotation_conversion:
            converter: librispeech
            data_dir: /home/asuvorov/projects/open_model_zoo/.venv/data/librispeech/test/LibriSpeech/test-clean
            annotation_file: /home/asuvorov/projects/open_model_zoo/librispeech_test_clean_max_duration_15_10.jsonl
            use_flac: True
            max_duration: 15
            top_n: 10
          reader:
            type: flac_reader
            mono: True
            dtype: float64
          metrics:
            - type: wer
accuracy_check -c checker_config_qwen3_asr_local.yaml -m /home/asuvorov/projects/openvino.genai/.vscode/models/Qwen/Qwen3-ASR-0.6B

Processing info:
model: qwen3-asr-0.6b-genai
launcher: openvino
device: CPU
dataset: LibriSpeech_test_clean_flac_max_duration_15
OpenCV version: 4.13.0
Annotation conversion for LibriSpeech_test_clean_flac_max_duration_15 dataset has been started
Parameters to be used for conversion:
converter: librispeech
data_dir: /home/asuvorov/projects/open_model_zoo/.venv/data/librispeech/test/LibriSpeech/test-clean
annotation_file: /home/asuvorov/projects/open_model_zoo/librispeech_test_clean_max_duration_15_10.jsonl
use_flac: True
max_duration: 15
top_n: 10
Annotation conversion for LibriSpeech_test_clean_flac_max_duration_15 dataset has been finished
10 objects processed in 13.674 seconds                                                                                                                                                                                             
wer: 3.57%
Processing info:
model: qwen3-asr-0.6b-optimum
launcher: openvino
device: CPU
dataset: LibriSpeech_test_clean_flac_max_duration_15
OpenCV version: 4.13.0
The tokenizer you are loading from '/home/asuvorov/projects/openvino.genai/.vscode/models/Qwen/Qwen3-ASR-0.6B' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
Annotation conversion for LibriSpeech_test_clean_flac_max_duration_15 dataset has been started
Parameters to be used for conversion:
converter: librispeech
data_dir: /home/asuvorov/projects/open_model_zoo/.venv/data/librispeech/test/LibriSpeech/test-clean
annotation_file: /home/asuvorov/projects/open_model_zoo/librispeech_test_clean_max_duration_15_10.jsonl
use_flac: True
max_duration: 15
top_n: 10
Annotation conversion for LibriSpeech_test_clean_flac_max_duration_15 dataset has been finished
  0%|                                                                                                                                                                                                              | 0/10 [00:00<?]The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
The attention mask is not set and cannot be inferred from input because pad token is same as eos token. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
 10%|████████████████████▏                                                                                                                                                                                     | 1/10 [00:03<00:32]The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
 20%|████████████████████████████████████████▍                                                                                                                                                                 | 2/10 [00:04<00:16]The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
 30%|████████████████████████████████████████████████████████████▌                                                                                                                                             | 3/10 [00:06<00:12]The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
 40%|████████████████████████████████████████████████████████████████████████████████▊                                                                                                                         | 4/10 [00:06<00:08]The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
 50%|█████████████████████████████████████████████████████████████████████████████████████████████████████                                                                                                     | 5/10 [00:08<00:06]The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
 60%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏                                                                                | 6/10 [00:10<00:06]The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
 70%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍                                                            | 7/10 [00:12<00:05]The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
 80%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌                                        | 8/10 [00:12<00:02]The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
 90%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊                    | 9/10 [00:14<00:01]The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input's `attention_mask` to obtain reliable results.
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
10 objects processed in 16.451 seconds                                                                                                                                                                                             
wer: 3.57%

@as-suvorov as-suvorov requested a review from pwolnows July 1, 2026 10:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant