Skip to content

localai-org/italian-asr

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1 Commit
 
 
 
 

Repository files navigation

LocalAI

Italian ASR

Open-source models that turn Italian audio into text, locally, on your own hardware, with optional speaker diarization. Brought to you by the LocalAI team.

🇮🇹 In italiano · leggi qui

Una famiglia di modelli open-source per trascrivere audio in italiano, da eseguire in locale sul proprio hardware, con diarizzazione (etichette parlante) opzionale. Realizzata dal team di LocalAI.

Perché esistono questi modelli. La maggior parte dei modelli Whisper italiani disponibili online sono addestrati solo su Common Voice (persone che leggono frasi al microfono di casa). Vanno benissimo nei benchmark di lettura, ma collassano sull'audio reale: riunioni, telefonate, podcast tornano pieni di ripetizioni allucinate e frasi inventate. Abbiamo addestrato una famiglia completa pensata anche per il parlato spontaneo.

Quale modello scegliere:

Scenario Modello consigliato
🎙️ Riunioni, chiamate, podcast, audio reale whisper-large-v3-turbo-it-yodas-only
📚 Audiolibri, notizie, atti parlamentari whisper-large-v3-it-multi
🗣️ Servono le etichette parlante (diarizzazione) vibevoice-asr-it-yodas-only

Esempio rapido (Python):

pip install faster-whisper
from faster_whisper import WhisperModel

model = WhisperModel("LocalAI-io/whisper-large-v3-turbo-it-yodas-only-ct2-int8",
                    device="cpu", compute_type="int8")  # oppure device="cuda"
segments, _ = model.transcribe("audio.mp3", language="it")
for s in segments:
    print(f"[{s.start:5.1f}s] {s.text}")

Pronti per qualunque runtime. Ogni modello è pubblicato in tutti i formati di quantizzazione che la community usa davvero, non devi convertire niente a mano:

  • safetensors per 🤗 Transformers (precisione piena, PyTorch).
  • CTranslate2 INT8 per faster-whisper / WhisperX / LocalAI, circa 4× più piccolo, ottimo su CPU.
  • whisper.cpp GGML (f16/q4_0/q5_0/q8_0) per whisper.cpp e LocalAI, perfetto per edge / Raspberry Pi / mobile.
  • GGUF per la famiglia VibeVoice via vibevoice.cpp.

Tutti i 75 repo HuggingFace sono qui: huggingface.co/LocalAI-io. Tutto sotto licenza MIT, nessuna chiave API, nessuna telemetria, gira completamente in locale.

Per la documentazione completa (tutte le varianti, i benchmark, i dettagli tecnici), continua a leggere in inglese qui sotto. 👇


Quick start

Pick the path that matches your stack. All models live on the LocalAI-io HuggingFace organization and run fully offline.

Python (faster-whisper) is recommended for most users:

pip install faster-whisper
from faster_whisper import WhisperModel

model = WhisperModel("LocalAI-io/whisper-large-v3-turbo-it-yodas-only-ct2-int8",
                    device="cpu", compute_type="int8")  # or device="cuda"
segments, _ = model.transcribe("audio.mp3", language="it")
for s in segments:
    print(f"[{s.start:5.1f}s] {s.text}")

Command line (whisper.cpp), single-binary, CPU or CUDA:

hf download LocalAI-io/whisper-large-v3-turbo-it-yodas-only-ggml \
  ggml-model-q5_0.bin --local-dir .
./whisper-cli -m ggml-model-q5_0.bin -l it -f audio.wav

LocalAI server: OpenAI-compatible HTTP endpoint. See the LocalAI section below for ready-made YAML configs and curl example.

Need speaker labels? Use the VibeVoice variant; it produces [Speaker 0/1/2…] text segments natively.


What's in this repo

A family of Italian-tuned models in two architecture families and three deployment formats:

  • Whisper family, every model size (tiny → large-v3-turbo) × four training recipes (CV-only, Multi, Multi+YODAS, YODAS-only). Best raw throughput on GPU.
  • VibeVoice ASR family, a Qwen2-based architecture with native speaker diarization for conversational audio. Best when you need speaker labels or fewer hallucinated repetitions.

Pre-quantized for every runtime

Half the reason this catalog is large is that we ship each fine-tune in every quantization format the community actually uses, so you don't have to download a PyTorch checkpoint and convert it yourself. Pick the runtime that fits your hardware:

  • safetensors (full precision), for 🤗 Transformers, research, and PyTorch fine-tuning.
  • CTranslate2 INT8 (~4× smaller, CPU-friendly), for faster-whisper / WhisperX / LocalAI's whisperx backend. Best on CPU and the default for production transcription.
  • whisper.cpp GGML in four bit-widths (f16, q4_0, q5_0, q8_0), for whisper.cpp, pywhispercpp, LocalAI's whisper backend, mobile / edge / Raspberry-Pi-style deployments. q5_0 is the usual sweet spot.
  • GGUF (q4_0, q5_0, q8_0) for vibevoice.cpp, covers the VibeVoice family on the same C++/CUDA runtime.

That's a total of 75 repos on HuggingFace, all maintained in lock-step so the q5_0 GGML matches the int8 CTranslate2 build matches the safetensors source.

Training Recipes

Why the training data matters. A speech-to-text model is only good at the kind of speech it has heard during training. A model trained on people carefully reading sentences out loud will look fantastic on benchmark numbers and then make up entire sentences when you hand it a real meeting where two humans talk over each other. So we trained four versions of each model on four different "diets" of audio, ranging from clean read speech to messy real-world conversations.

The four datasets we mix and match:

  • Common Voice (CV), Mozilla's crowdsourced corpus. People reading sentences on a phone or laptop at home. Clean, varied speakers, but it's read speech, not conversational.
  • MLS (Multilingual LibriSpeech), long-form audiobook readings. Professional narrators, no overlap, no hesitations.
  • VoxPopuli (VoxPopuli), European Parliament recordings. Formal speech, but real microphones in a real room.
  • YODAS-Granary (YODAS), YouTube clips, pseudo-labeled. Real people talking spontaneously: meetings, podcasts, vlogs. Italian as it's actually spoken.

The four mixes:

Variant What it's trained on (plain-English) Best for Samples
CV-only (*-it) Just Common Voice. People reading sentences at home. CV-style benchmark WER. Falls apart on real conversations. 173k
Multi (*-it-multi) Common Voice + audiobooks + parliament. All read or formal speech. Audiobooks, news, parliamentary transcripts. Stable across read-speech domains. 255k
Multi + YODAS (*-it-multi-yodas-asr_only-nocv) Audiobooks + parliament + YouTube spontaneous Italian (~87k clips). Best when you don't know what kind of audio you'll get. 164k
YODAS-only (*-it-yodas-only) YouTube spontaneous Italian only (240k clips). 100% real conversational audio. Real-world meetings, calls, podcasts. Read-speech benchmark numbers regress (no audiobooks in training). 240k

Recommended pick:

Pick by use case: conversational → whisper-large-v3-turbo-it-yodas-only · read speech → whisper-large-v3-it-multi · multi-speaker / diarization → vibevoice-asr-it-yodas-only

  • 🎙️ Meetings, calls, podcasts, real-world Italian: *-it-yodas-only (large-v3-turbo is the sweet spot for production)
  • 📚 Audiobooks, news, parliamentary: *-it-multi
  • 🎯 CV-style benchmarks only: *-it
  • 🔀 Mixed/unknown audio: *-it-multi-yodas-asr_only-nocv or *-it-multi
  • 🗣️ Multi-speaker / need diarization: vibevoice-asr-it-yodas-only (see VibeVoice section)

Models

Italian ASR catalog: 75 HF repos across 6 model sizes, 4 training recipes, and 3 deployment formats, plus a VibeVoice fine-tune with native speaker diarization.

Each link below points to a HuggingFace repo. The ggml repos contain four quantizations (f16, q4_0, q5_0, q8_0), pick by your size/quality budget. The newer YODAS-trained ggml repos ship q4_0, q5_0, q8_0 (no f16, it consistently failed to upload via the HF Xet protocol; the q8_0 is near-lossless in practice).

CV-only and Multi (all sizes)

Base Params CV-only (*-it) Multi (*-it-multi)
openai/whisper-tiny 39M safetensors · ct2-int8 · ggml safetensors · ct2-int8 · ggml
openai/whisper-base 74M safetensors · ct2-int8 · ggml safetensors · ct2-int8 · ggml
openai/whisper-small 244M safetensors · ct2-int8 · ggml safetensors · ct2-int8 · ggml
openai/whisper-medium 769M safetensors · ct2-int8 · ggml safetensors · ct2-int8 · ggml
openai/whisper-large-v3 1.55B safetensors · ct2-int8 · ggml safetensors · ct2-int8 · ggml
openai/whisper-large-v3-turbo 809M safetensors · ct2-int8 · ggml safetensors · ct2-int8 · ggml

Multi + YODAS and YODAS-only (conversational coverage)

Base Params Multi + YODAS (*-it-multi-yodas-asr_only-nocv) YODAS-only (*-it-yodas-only)
openai/whisper-tiny 39M safetensors · ct2-int8 · ggml safetensors · ct2-int8 · ggml
openai/whisper-base 74M safetensors · ct2-int8 · ggml safetensors · ct2-int8 · ggml
openai/whisper-small 244M safetensors · ct2-int8 · ggml safetensors · ct2-int8 · ggml
openai/whisper-medium 769M safetensors · ct2-int8 · ggml safetensors · ct2-int8 · ggml
openai/whisper-large-v3 1.55B safetensors · ct2-int8 · ggml safetensors · ct2-int8 · ggml
openai/whisper-large-v3-turbo 809M safetensors · ct2-int8 · ggml safetensors · ct2-int8 · ggml

Note: smaller *-yodas-only sizes (tiny/base/small/medium) are inherently loop-prone on conversational audio compared to large-v3/large-v3-turbo. They're shipped for completeness and edge / CPU-only deployments, for the cleanest output on real-world Italian speech, prefer large-v3-turbo-it-yodas-only.

Results

Word Error Rate (%) on the Common Voice 25.0 Italian test set (15,184 samples). Lower is better.

Base Params CV-only Multi
whisper-tiny 39M 27.1% 29.4%
whisper-base 74M 19.2% 21.4%
whisper-small 244M 13.0% 15.6%
whisper-medium 769M 10.5% 12.4%
whisper-large-v3 1.55B , ,
whisper-large-v3-turbo 809M , ,

Large-v3 and large-v3-turbo results: these variants were trained with evaluation disabled (to avoid VRAM fragmentation during generation on very long sequences). Benchmarks will be added once we finish running the standalone eval pass, they are expected to improve on the whisper-medium numbers above.

Note on the Multi column: the *-multi models are evaluated on the CV-only test set here for an apples-to-apples comparison with the *-it variants. They were trained on a more diverse mix (CV + MLS + VoxPopuli), so CV-only WER slightly under-represents their real-world robustness on non-Common-Voice audio. If your deployment sees audiobooks, parliamentary recordings, or other non-crowdsourced speech, prefer the *-multi variant.

Training details: 10,000 steps, learning rate 1e-5, 500 warmup steps, bf16 on an NVIDIA GB10 (128GB unified memory). Common Voice 25.0 Italian was published 2026-03-09 and is the latest release at the time of training.

Real-world conversational benchmark

Italian conversational ASR real-meeting benchmark: whisper-large-v3-yodas 65s / 7 loops, whisper-large-v3-turbo-yodas 73s / 62 loops, vibevoice-asr-it-yodas-only 10min / 5 loops + diarization

To complement the read-speech CV numbers above, we also benchmarked the YODAS-trained variants on an internal ~17-minute Italian conversational recording held back as a private real-world test set. The audio itself is not published, only the head-to-head numbers below. All runs on a single NVIDIA GB10 with CUDA: Whisper at q5_0 via whisper.cpp (CUDA backend), VibeVoice at q4_0 via vibevoice.cpp master (resident-KV + flash-attention).

Model Wall time RTF (lower is faster) 4+ word loop runs Native diarization
whisper-large-v3-it-yodas-only (q5_0) 64.8 s 0.065 7 ,
whisper-large-v3-turbo-it-yodas-only (q5_0) 73.2 s 0.073 62 ,
vibevoice-asr-it-yodas-only (q4_0) 619.9 s 0.622 5

A 4+ word loop run counts each position where the same lowercased token appears four times in a row, a crude but consistent proxy for hallucinated repetitions on tough audio. Lower is better.

Takeaways:

  • For raw throughput, whisper-large-v3-it-yodas-only is unbeatable on GPU (~15× realtime).
  • For loop-free transcripts and free speaker diarization, vibevoice-asr-it-yodas-only wins, at the cost of being only ~1.6× realtime.
  • Avoid whisper-large-v3-turbo-it-yodas-only on this kind of audio, the smaller decoder degrades into long repetition runs (62 vs 7 for non-turbo), and is also slightly slower than non-turbo on GPU because of the extra fallback decode steps. Use it for read-speech where decoder length matters less.

Usage

LocalAI

LocalAI supports these models natively through the whisperx backend. Pick the variant that matches your accuracy / latency budget, drop the corresponding YAML into your LocalAI models/ directory, and transcribe via the OpenAI-compatible /v1/audio/transcriptions endpoint.

Quick hardware guide:

  • tiny / base, CPU-only edge devices, real-time on a Raspberry Pi.
  • small / medium, most servers, great accuracy / speed tradeoff.
  • large-v3-turbo, large-v3 quality at roughly half the latency.
  • large-v3, highest accuracy, when it matters most.

After setting up a model below, transcribe audio with:

curl http://localhost:8080/v1/audio/transcriptions \
  -H "Content-Type: multipart/form-data" \
  -F file="@audio.mp3" \
  -F model="<the-name-from-the-yaml-below>"

tiny (39M)

whisperx-tiny-it, Common Voice only
name: whisperx-tiny-it
backend: whisperx
known_usecases:
  - transcript
parameters:
  model: LocalAI-io/whisper-tiny-it-ct2-int8
  language: it
whisperx-tiny-it-multi, multi-dataset
name: whisperx-tiny-it-multi
backend: whisperx
known_usecases:
  - transcript
parameters:
  model: LocalAI-io/whisper-tiny-it-multi-ct2-int8
  language: it
whisperx-tiny-it-multi-yodas, multi + YODAS conversational data
name: whisperx-tiny-it-multi-yodas
backend: whisperx
known_usecases:
  - transcript
parameters:
  model: LocalAI-io/whisper-tiny-it-multi-yodas-asr_only-nocv-ct2-int8
  language: it

base (74M)

whisperx-base-it, Common Voice only
name: whisperx-base-it
backend: whisperx
known_usecases:
  - transcript
parameters:
  model: LocalAI-io/whisper-base-it-ct2-int8
  language: it
whisperx-base-it-multi, multi-dataset
name: whisperx-base-it-multi
backend: whisperx
known_usecases:
  - transcript
parameters:
  model: LocalAI-io/whisper-base-it-multi-ct2-int8
  language: it
whisperx-base-it-multi-yodas, multi + YODAS conversational data
name: whisperx-base-it-multi-yodas
backend: whisperx
known_usecases:
  - transcript
parameters:
  model: LocalAI-io/whisper-base-it-multi-yodas-asr_only-nocv-ct2-int8
  language: it

small (244M)

whisperx-small-it, Common Voice only
name: whisperx-small-it
backend: whisperx
known_usecases:
  - transcript
parameters:
  model: LocalAI-io/whisper-small-it-ct2-int8
  language: it
whisperx-small-it-multi, multi-dataset
name: whisperx-small-it-multi
backend: whisperx
known_usecases:
  - transcript
parameters:
  model: LocalAI-io/whisper-small-it-multi-ct2-int8
  language: it
whisperx-small-it-multi-yodas, multi + YODAS conversational data
name: whisperx-small-it-multi-yodas
backend: whisperx
known_usecases:
  - transcript
parameters:
  model: LocalAI-io/whisper-small-it-multi-yodas-asr_only-nocv-ct2-int8
  language: it

medium (769M)

whisperx-medium-it, Common Voice only
name: whisperx-medium-it
backend: whisperx
known_usecases:
  - transcript
parameters:
  model: LocalAI-io/whisper-medium-it-ct2-int8
  language: it
whisperx-medium-it-multi, multi-dataset
name: whisperx-medium-it-multi
backend: whisperx
known_usecases:
  - transcript
parameters:
  model: LocalAI-io/whisper-medium-it-multi-ct2-int8
  language: it
whisperx-medium-it-multi-yodas, multi + YODAS conversational data
name: whisperx-medium-it-multi-yodas
backend: whisperx
known_usecases:
  - transcript
parameters:
  model: LocalAI-io/whisper-medium-it-multi-yodas-asr_only-nocv-ct2-int8
  language: it

large-v3 (1.55B)

whisperx-large-v3-it, Common Voice only
name: whisperx-large-v3-it
backend: whisperx
known_usecases:
  - transcript
parameters:
  model: LocalAI-io/whisper-large-v3-it-ct2-int8
  language: it
whisperx-large-v3-it-multi, multi-dataset
name: whisperx-large-v3-it-multi
backend: whisperx
known_usecases:
  - transcript
parameters:
  model: LocalAI-io/whisper-large-v3-it-multi-ct2-int8
  language: it
whisperx-large-v3-it-multi-yodas, multi + YODAS conversational data
name: whisperx-large-v3-it-multi-yodas
backend: whisperx
known_usecases:
  - transcript
parameters:
  model: LocalAI-io/whisper-large-v3-it-multi-yodas-asr_only-nocv-ct2-int8
  language: it
whisperx-large-v3-it-yodas-only, pure YODAS (best on conversational)
name: whisperx-large-v3-it-yodas-only
backend: whisperx
known_usecases:
  - transcript
parameters:
  model: LocalAI-io/whisper-large-v3-it-yodas-only-ct2-int8
  language: it

large-v3-turbo (809M)

whisperx-large-v3-turbo-it, Common Voice only
name: whisperx-large-v3-turbo-it
backend: whisperx
known_usecases:
  - transcript
parameters:
  model: LocalAI-io/whisper-large-v3-turbo-it-ct2-int8
  language: it
whisperx-large-v3-turbo-it-multi, multi-dataset
name: whisperx-large-v3-turbo-it-multi
backend: whisperx
known_usecases:
  - transcript
parameters:
  model: LocalAI-io/whisper-large-v3-turbo-it-multi-ct2-int8
  language: it
whisperx-large-v3-turbo-it-yodas-only, pure YODAS (recommended for conversational)
name: whisperx-large-v3-turbo-it-yodas-only
backend: whisperx
known_usecases:
  - transcript
parameters:
  model: LocalAI-io/whisper-large-v3-turbo-it-yodas-only-ct2-int8
  language: it

Transformers

from transformers import pipeline

pipe = pipeline(
    "automatic-speech-recognition",
    model="LocalAI-io/whisper-medium-it-multi",
)
result = pipe("audio.mp3", generate_kwargs={"language": "it", "task": "transcribe"})
print(result["text"])

faster-whisper / WhisperX

The -ct2-int8 repos are drop-in replacements for the upstream CT2 Whisper models:

from faster_whisper import WhisperModel

model = WhisperModel(
    "LocalAI-io/whisper-medium-it-multi-ct2-int8",
    device="cpu",
    compute_type="int8",
)
segments, info = model.transcribe("audio.mp3", language="it")
for segment in segments:
    print(f"[{segment.start:.1f}s - {segment.end:.1f}s] {segment.text}")

whisper.cpp

Each -ggml repo contains four quantizations: ggml-model-f16.bin, ggml-model-q8_0.bin, ggml-model-q5_0.bin, ggml-model-q4_0.bin. Pick the one that fits your size / quality budget, q5_0 is the usual sweet spot, q8_0 is near-lossless, q4_0 is smallest.

huggingface-cli download LocalAI-io/whisper-medium-it-multi-ggml ggml-model-q5_0.bin --local-dir .
./whisper-cli -m ggml-model-q5_0.bin -f audio.wav -l it

These models also work with pywhispercpp and any other whisper.cpp-based runtime.

VibeVoice ASR (Italian)

In addition to the Whisper variants above, we ship a VibeVoice ASR fine-tune trained on the same YODAS-only Italian mix (LoRA, ~10k steps on the GB10). VibeVoice is a different architecture (Qwen2-based with native diarization-aware decoding), useful when you want speaker-segmented output, or fewer hallucinated loops on tough conversational audio. On our private real-world conversational test set it produced fewer 4+ word repetition runs than Whisper large-v3-yodas (5 vs 7), with native speaker diarization in the output.

Variant Repo Notes
Merged model LocalAI-io/vibevoice-asr-it-yodas-only Drop-in replacement for microsoft/VibeVoice-ASR.
LoRA adapter LocalAI-io/vibevoice-asr-it-yodas-only-lora Apply on top of the upstream base; ~tens of MB.
GGUF (vibevoice.cpp) LocalAI-io/vibevoice-asr-it-yodas-only-ggml q4_0, q5_0, q8_0 quantizations for vibevoice.cpp.

Transformers

from vibevoice.modular.modeling_vibevoice_asr import VibeVoiceASRForConditionalGeneration
from vibevoice.processor.vibevoice_asr_processor import VibeVoiceASRProcessor
import torch

processor = VibeVoiceASRProcessor.from_pretrained("LocalAI-io/vibevoice-asr-it-yodas-only")
model = VibeVoiceASRForConditionalGeneration.from_pretrained(
    "LocalAI-io/vibevoice-asr-it-yodas-only",
    dtype=torch.bfloat16,
    attn_implementation="sdpa",  # or "flash_attention_2" if installed
)

vibevoice.cpp (GGUF)

Build vibevoice.cpp once (use -DVIBEVOICE_GGML_CUDA=ON for GPU), download a GGUF + the Qwen2.5 tokenizer GGUF, and run:

hf download LocalAI-io/vibevoice-asr-it-yodas-only-ggml \
  vibevoice-asr-it-yodas-only-q5_0.gguf --local-dir models/

./build/bin/vibevoice-cli asr \
  --model     models/vibevoice-asr-it-yodas-only-q5_0.gguf \
  --tokenizer models/tokenizer.gguf \
  --audio     audio.wav \
  --max-new-tokens 8000 --verbose

The output is a JSON array of diarized segments (Start, End, Speaker, Content).

Datasets

  • Common Voice 25.0 Italian, crowdsourced read speech, the backbone of the CV-only and Multi training mixes.
  • Multilingual LibriSpeech (MLS) Italian, audiobook readings, adds long-form narration to the Multi mix.
  • VoxPopuli Italian, European Parliament recordings, adds parliamentary speech to the Multi mix.
  • YODAS-Granary Italian, YouTube-sourced spontaneous speech, pseudo-labeled with faster-whisper-large-v3 and cleaned with Qwen2.5-7B-Instruct. Provides the conversational coverage that the Multi mix alone is missing. Used in two slices: asr_only (~87k clips) for the Multi+YODAS variants, and asr_only + ast (200k cap) for the YODAS-only variants.

License

Models and code are released under the MIT license, consistent with the upstream OpenAI Whisper release.

Citation

If you use these models in academic work or want to cite this release, please use the following BibTeX entry:

@software{italian_asr_2026,
  author    = {Di Giacinto, Ettore and Palethorpe, Richard},
  title     = {{Italian ASR}: An open family of Italian speech-to-text models
               (Whisper and VibeVoice fine-tunes) with a C++ inference engine
               for VibeVoice},
  year      = {2026},
  publisher = {LocalAI},
  url       = {https://github.com/mudler/italian-asr},
  note      = {Models on HuggingFace at \url{https://huggingface.co/LocalAI-io};
               vibevoice.cpp engine at \url{https://github.com/mudler/vibevoice.cpp}}
}

Plain-text:

Di Giacinto, E. and Palethorpe, R. (2026). Italian ASR: An open family of Italian speech-to-text models (Whisper and VibeVoice fine-tunes) with a C++ inference engine for VibeVoice. LocalAI. https://github.com/mudler/italian-asr

Acknowledgements

Built by Ettore Di Giacinto and Richard Palethorpe of the LocalAI team. If you find these models useful, please star LocalAI on GitHub, it genuinely helps the project.

About

Open-source models that turn Italian audio into text, locally, on your own hardware, with optional speaker diarization. Brought to you by the LocalAI team.

Topics

Resources

Stars

4 stars

Watchers

0 watching

Forks

Releases

No releases published

Packages

 
 
 

Contributors