Skip to content

README.md

Latest commit

 

History

History
136 lines (107 loc) · 6.75 KB

README.md

File metadata and controls

136 lines (107 loc) · 6.75 KB
Raw

Whisper ASR testing & fine-tuning

Environment for testing and fine-tuning OpenAI Whisper models on African languages, starting with English (baseline/control) and Kiswahili. Chichewa will be added the same way once that phase starts (see Adding a new language).

Layout

whisper/
├── lib/            # shared code - config loading, dataset I/O, collator, metrics, model loading
├── scripts/        # 4 CLI entry points, parameterized by --language
│   ├── prepare_dataset.py   # pull a dataset subset -> manifest + wav files
│   ├── run_baseline.py      # WER + confidence report for a (pretrained or fine-tuned) model
│   ├── finetune.py          # fine-tuning loop
│   └── export_model.py      # sanity check + CTranslate2 export
├── colab_finetune.ipynb     # runs finetune.py on a Colab GPU with real hyperparameters
├── english/
│   ├── config.yaml   # language + dataset config, CPU-smoke-test hyperparameter overrides
│   ├── data/          # generated by prepare_dataset.py (gitignored)
│   ├── results/       # generated by run_baseline.py (gitignored)
│   └── models/        # generated by finetune.py / export_model.py (gitignored)
└── kiswahili/          # same structure

All pipeline logic lives in lib/ and scripts/; language folders hold only a config file plus generated artifacts, so the four scripts behave identically for every language - only config.yaml differs.

Setup

python -m venv .venv
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass   # only if activation is blocked
.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip

# CPU box (this machine):
pip install torch --index-url https://download.pytorch.org/whl/cpu
# GPU box later: pip install torch  (or the CUDA-specific index URL from pytorch.org/get-started/locally)

pip install -r requirements.txt

ffmpeg is not required for the default dataset (WAV audio, decoded via soundfile). It would be needed if a source with MP3/OGG audio is added later (winget install --id Gyan.FFmpeg).

External access requirements

  • Internet is needed at pip install time and at dataset-download time.
  • No Hugging Face account or token is needed for the default dataset (google/fleurs, ungated, CC-BY-4.0).
  • Mozilla Common Voice is not used as the default source: as of October 2025 it's gated behind a separate Mozilla Data Collective account rather than being directly downloadable via datasets. prepare_dataset.py is dataset-source-agnostic (config-driven dataset_id/dataset_config), so Common Voice (or any other corpus, e.g. real Ushahidi audio) can be swapped in later purely via config.yaml, with no code changes.

Running the pipeline

All commands below work identically for --language english and --language kiswahili.

# 1. Pull a small dataset subset (defaults to data.max_samples[<split>] in config.yaml)
python whisper/scripts/prepare_dataset.py --language english --split train
python whisper/scripts/prepare_dataset.py --language english --split validation
python whisper/scripts/prepare_dataset.py --language english --split test

# 2. Baseline accuracy of the pretrained model (WER + per-sample confidence)
python whisper/scripts/run_baseline.py --language english --model openai/whisper-tiny

# 3. Fine-tune (CPU smoke test - see below)
python whisper/scripts/finetune.py --language english --model openai/whisper-tiny

# 4. Sanity-check + export the fine-tuned model
python whisper/scripts/export_model.py --language english \
  --model-dir whisper/english/models/whisper-tiny-finetuned

Results land in whisper/<language>/results/baseline_<model>_<split>.{json,csv} - per-sample reference/hypothesis/WER/confidence plus an aggregate WER and mean confidence. Confidence is currently a sequence-level proxy (exp(average log-prob) of the generated sequence, via model.generate(output_scores=True)); a per-word confidence flag would use model.compute_transition_scores aligned to word boundaries - not implemented yet, noted here as the next step if that granularity is needed.

Fine-tuning: CPU smoke test vs. real GPU run on Colab

This machine has no GPU. Locally, finetune.py runs with each language's config.yaml overrides - a handful of steps, batch size 2, fp16 forced off - just to prove the training loop executes, loss trends down over those steps, and a checkpoint saves and reloads correctly. It does not produce a usable model.

Real fine-tuning happens on a GPU, e.g. Google Colab (free T4 GPU tier; Colab Pro for longer/bigger runs): open whisper/colab_finetune.ipynb in Colab, which clones this repo, installs dependencies, pulls a much larger dataset slice, and runs the same finetune.py pointed at whisper/lib/default_config.yaml instead of a language's CPU overrides - using the reference hyperparameters from Hugging Face's "Fine-Tune Whisper for Multilingual ASR" guide (lr=1e-5, warmup_steps=500, max_steps=5000, batch_size=16, fp16=True). Nothing in the scripts changes between the two runs - only the config and the machine.

Colab's free tier disconnects after idle periods and caps sessions at ~12 hours. Training checkpoints every save_steps regardless of environment, so a disconnect loses at most that many steps - resume with:

python whisper/scripts/finetune.py --language kiswahili --config whisper/lib/default_config.yaml \
  --resume-from-checkpoint whisper/kiswahili/models/<checkpoint-dir>/checkpoint-<step>

Optional: pass --use-peft to either finetune.py invocation to wrap the model in a LoRA adapter (fewer trainable parameters - faster on CPU, lighter on GPU).

Export

export_model.py reloads the fine-tuned model as a sanity check (one sample transcription), then runs ct2-transformers-converter to produce an int8-quantized CTranslate2 export for fast CPU inference via faster-whisper. GGUF/whisper.cpp export is intentionally not implemented yet - its converter has known compatibility gaps with safetensors-only checkpoints; revisit if whisper.cpp deployment becomes a concrete need.

Adding a new language

  1. Create whisper/<language>/config.yaml with language_name (must match Whisper's tokenizer language name, e.g. swahili not sw), language_code, dataset_id, dataset_config, and any CPU-smoke-test train: overrides.
  2. Rerun the four scripts above with --language <language>.

For Chichewa, google/fleurs already has a config (ny_mw, Nyanja/Chichewa - Malawi) - confirm it's still current before use, since dataset catalogs change.