Feat: MVP ASR Benchmarking using CV22

[Yip-Jia-Qi]

Approach

For ASR, simple WER and CER will be the main metric

Things to consider

• Jan User Profiles (languages)

Rank	Language	Count	Percentage
1	English	92,537	61.55%
2	Russian	11,014	7.33%
3	Spanish	8,522	5.67%
4	French	5,959	3.96%
5	German	5,886	3.92%
6	Chinese	4,582	3.05%
7	Portuguese	3,353	2.23%
8	Polish	2,587	1.72%
9	Italian	2,081	1.38%
10	Japanese	1,911	1.27%

Out of scope

• We will not benchmark model efficiency, just model size and ASR performance
• Running the benchmark. Once we have selected the benchmark then we will figure out how to download the data and run it

Stage plans

1. Lit Review
Select a panel of relevant benchmarks
2. Experiments
Download and run the benchmarks on a few ASR models
3. Evaluate
Complie the results and share with the team, to make a decision on the benchmark panel
4. Translation
Incorporate the benchmark into a replicable script so we can always quickly evaluate new ASR models that come out

[thinhlpg]

250825

Phase 1: Literature review

• What is current Jan user profile languages

• How does open asr leader board calculate wer (which framework? do they apply any normalization to the audio or to the text before calculate?

  • normalize text for both label and pred
  • cast audio to 16khz

• Understand the 8 benchmark used in open asr leaderboard english (this step is important since English is the majority of our user profile)

  ESB is a benchmark for evaluating the performance of a single automatic speech recognition (ASR) system across a broad set of speech datasets. It comprises eight English speech recognition datasets, capturing a broad range of domains, acoustic conditions, speaker styles, and transcription requirements. As such, it gives a better indication of how a model is likely to perform on downstream ASR compared to evaluating it on one dataset alone.

Dataset	Domain	Speaking Style	Train (h)	Dev (h)	Test (h)	Transcriptions	License
LibriSpeech	Audiobook	Narrated	960	11	11	Normalised	CC-BY-4.0
VoxPopuli	European Parliament	Oratory	523	5	5	Punctuated	CC0
TED-LIUM	TED talks	Oratory	454	2	3	Normalised	CC-BY-NC-ND 3.0
GigaSpeech	Audiobook, podcast, YouTube	Narrated, spontaneous	2500	12	40	Punctuated	apache-2.0
SPGISpeech	Financial meetings	Oratory, spontaneous	4900	100	100	Punctuated & Cased	User Agreement
Earnings-22	Financial meetings	Oratory, spontaneous	105	5	5	Punctuated & Cased	CC-BY-SA-4.0
AMI	Meetings	Spontaneous	78	9	9	Punctuated & Cased	CC-BY-4.0

(The table is from Open ASR Leaderboard / ESB paepr)

• How does open asr leader board calculate wer (which framework? do they apply any normalization to the audio or to the text before calculate?
  • normalize text for both label and pred
  • cast audio to 16khz
• see if we can find language-specific benchmarks for our top 10 lanaguages -> first, go with common voice 17, they have all the languages

Phase 2: Experiment

• fork open asr leaderboard
• convert datasets to openasr format and save under menlo's hf hub
• run with whisper (yes I think whisper is still the best baseline)

What is current Jan user profile languages? (got this from Emre)

Rank	Language	Count	Percentage
1	English	92,537	61.55%
2	Russian	11,014	7.33%
3	Spanish	8,522	5.67%
4	French	5,959	3.96%
5	German	5,886	3.92%
6	Chinese	4,582	3.05%
7	Portuguese	3,353	2.23%
8	Polish	2,587	1.72%
9	Italian	2,081	1.38%
10	Japanese	1,911	1.27%

✅ Lit Review - Select a panel of relevant benchmarks -> for english just use open_asr_leaderboard

• (not bench speed, just size and ASR performance)
• According to open_asr_leaderboard, there are the benchmark to go for (We report the Average WER (⬇️ lower the better) and RTFx (⬆️ higher the better))
  • AMI
    - Earnings22
    - Gigaspeech
  • LS Clean
  • LS Other
  • SPGISpeech
  • Tedlium
  • Voxpopuli

Size:

How open asr leaderboard do WER

Giving a WER of 0.33, or 33%. For a fair comparison, we calculate zero-shot (i.e. pre-trained models only) normalised WER for all the model checkpoints, meaning punctuation and casing is removed from the references and predictions. You can find the evaluation code on our Github repository. To read more about how the WER is computed, refer to the Audio Transformers Course.

https://github.com/huggingface/open_asr_leaderboard

# audio

def prepare_data(dataset):
    # Re-sample to 16kHz and normalise transcriptions
    dataset = dataset.cast_column("audio", Audio(sampling_rate=16000))
    dataset = dataset.map(normalize)
    dataset = dataset.filter(is_target_text_in_range, input_columns=["norm_text"])

    return dataset


# https://github.com/huggingface/open_asr_leaderboard/blob/main/nemo_asr/run_eval.py

import evaluate
wer_metric = evaluate.load("wer")

# https://github.com/huggingface/open_asr_leaderboard/blob/main/normalizer/data_utils.py

class EnglishTextNormalizer:
    def __init__(self, english_spelling_mapping=english_spelling_normalizer):
        self.ignore_patterns = r"\b(hmm|mm|mhm|mmm|uh|um)\b"
        self.replacers = {
            # common contractions
            r"\bwon't\b": "will not",
            r"\bcan't\b": "can not",
            r"\blet's\b": "let us",
            r"\bain't\b": "aint",
            r"\by'all\b": "you all",
            r"\bwanna\b": "want to",
            r"\bgotta\b": "got to",
            r"\bgonna\b": "going to",
            r"\bi'ma\b": "i am going to",
            r"\bimma\b": "i am going to",
            r"\bwoulda\b": "would have",
            r"\bcoulda\b": "could have",
            r"\bshoulda\b": "should have",
            r"\bma'am\b": "madam",
            # contractions in titles/prefixes
            r"\bmr\b": "mister ",
            r"\bmrs\b": "missus ",
            r"\bst\b": "saint ",
            r"\bdr\b": "doctor ",
            r"\bprof\b": "professor ",
            r"\bcapt\b": "captain ",
            r"\bgov\b": "governor ",
            r"\bald\b": "alderman ",
            r"\bgen\b": "general ",
            r"\bsen\b": "senator ",
            r"\brep\b": "representative ",
            r"\bpres\b": "president ",
            r"\brev\b": "reverend ",
            r"\bhon\b": "honorable ",
            r"\basst\b": "assistant ",
            r"\bassoc\b": "associate ",
            r"\blt\b": "lieutenant ",
            r"\bcol\b": "colonel ",
            r"\bjr\b": "junior ",
            r"\bsr\b": "senior ",
            r"\besq\b": "esquire ",
            # prefect tenses, ideally it should be any past participles, but it's harder..
            r"'d been\b": " had been",
            r"'s been\b": " has been",
            r"'d gone\b": " had gone",
            r"'s gone\b": " has gone",
            r"'d done\b": " had done",  # "'s done" is ambiguous
            r"'s got\b": " has got",
            # general contractions
            r"n't\b": " not",
            r"'re\b": " are",
            r"'s\b": " is",
            r"'d\b": " would",
            r"'ll\b": " will",
            r"'t\b": " not",
            r"'ve\b": " have",
            r"'m\b": " am",
        }
        self.standardize_numbers = EnglishNumberNormalizer()
        self.standardize_spellings = EnglishSpellingNormalizer(english_spelling_mapping)

    def __call__(self, s: str):
        s = s.lower()

        s = re.sub(r"[<\[][^>\]]*[>\]]", "", s)  # remove words between brackets
        s = re.sub(r"\(([^)]+?)\)", "", s)  # remove words between parenthesis
        s = re.sub(self.ignore_patterns, "", s)
        s = re.sub(r"\s+'", "'", s)  # standardize when there's a space before an apostrophe

        for pattern, replacement in self.replacers.items():
            s = re.sub(pattern, replacement, s)

        s = re.sub(r"(\d),(\d)", r"\1\2", s)  # remove commas between digits
        s = re.sub(r"\.([^0-9]|$)", r" \1", s)  # remove periods not followed by numbers
        s = remove_symbols_and_diacritics(s, keep=".%$¢€£")  # keep some symbols for numerics

        s = self.standardize_numbers(s)
        s = self.standardize_spellings(s)

        # now remove prefix/suffix symbols that are not preceded/followed by numbers
        s = re.sub(r"[.$¢€£]([^0-9])", r" \1", s)
        s = re.sub(r"([^0-9])%", r"\1 ", s)

        s = re.sub(r"\s+", " ", s)  # replace any successive whitespace characters with a space

        return s

normalizer = EnglishTextNormalizer()
def normalize(batch):
    batch["original_text"] = get_text(batch)
    batch["norm_text"] = normalizer(batch["original_text"])
    return batch


# normalize transcriptions with English normalizer
predictions = [data_utils.normalizer(pred.text) for pred in transcriptions]


wer = wer_metric.compute(references=all_data['references'], predictions=predictions)
wer = round(100 * wer, 2)

Try running open asr leader board

• https://github.com/[huggingface/open_asr_leaderboard](https://github.com/huggingface/open_asr_leaderboard)

• Language: English

This dataset contains 125 files totalling roughly 119 hours of English language earnings calls from global countries. This dataset provides the full audios, transcripts, and accompanying metadata such as ticker symbol, headquarters country, and our defined "Language Region".
The audio is in English, with speakers from seven different langauge regions and a total of 27 unique countries. As such, there is large diversity in the speakers and accents.

References

• https://huggingface.co/spaces/hf-audio/open_asr_leaderboard
• https://arxiv.org/abs/2210.13352

[thinhlpg]

(later)
https://semantichearing.cs.washington.edu/ - how they combine the noise

https://arxiv.org/abs/1910.10279
can going to cafes with a microphone then record noisy data

[Yip-Jia-Qi]

@thinhlpg
Some Benchmarks to look at for ASR are:
Earnings
Gigaspeech

see if they can cover all our languages
then see if we can find language-specific benchmarks for our top few languages
type of speech matters also, whether it’s natural conversation or formal or audiobook

[Yip-Jia-Qi]

Data from Common Voice Corpus 22.0
https://commonvoice.mozilla.org/en/datasets
https://github.com/common-voice/cv-dataset

Rank	Language	Validated Hours	Size (GB)
1	English	2,724	86,53
2	Russian	248	6.47
3	Spanish	587	47.81
4	French	1,067	27.83
5	German	1,369	34.38
6	Chinese	238	21.25
7	Portuguese	185	4.74
8	Polish	175	4.42
9	Italian	362	9.55
10	Japanese	225	11.78

@thinhlpg let's just use this

We should look for the test.tsv and see how many lines of test we get

[Yip-Jia-Qi]

Files have been downloaded from https://commonvoice.mozilla.org/rup/datasets

and stored in /mnt/nas/data/common_voice_22