From 8f855b5e8d1d01504426a155e73a0c3a9c856b6f Mon Sep 17 00:00:00 2001 From: Abhishek Mishra Date: Wed, 17 Jun 2026 11:15:34 +0530 Subject: [PATCH 1/3] feat(speech-to-text): add Pulse STT benchmarking scripts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two self-contained reference scripts for measuring Pulse STT accuracy (WER) + latency against a user-supplied evaluation dataset: ping_pulse_offline.py — POST /waves/v1/stt/ (pre-recorded) reports WER, per-clip latency, RTF (p50/p90/p95) ping_pulse_streaming.py — WSS /waves/v1/stt/live (real-time WS) reports WER, tail latency (p50/p90/p95) Both scripts: - Single-file, no repo-internal imports — copy anywhere and run - Accept a folder layout: DATA_DIR/audio/ + DATA_DIR/metadata.csv - --language en | hi (the two with mature WER normalisation pipelines) - Print JSON aggregate at end (pipe-able to jq) - Use stdlib urllib for HTTP, websockets package for streaming Linked from the Pulse STT 'Measuring Latency' docs page as the canonical reference implementation of the methodology described there. --- speech-to-text/benchmarks/README.md | 98 +++++ .../benchmarks/ping_pulse_offline.py | 354 +++++++++++++++++ .../benchmarks/ping_pulse_streaming.py | 376 ++++++++++++++++++ 3 files changed, 828 insertions(+) create mode 100644 speech-to-text/benchmarks/README.md create mode 100644 speech-to-text/benchmarks/ping_pulse_offline.py create mode 100644 speech-to-text/benchmarks/ping_pulse_streaming.py diff --git a/speech-to-text/benchmarks/README.md b/speech-to-text/benchmarks/README.md new file mode 100644 index 0000000..90ffa5f --- /dev/null +++ b/speech-to-text/benchmarks/README.md @@ -0,0 +1,98 @@ +# Pulse STT — Benchmarking Scripts + +Self-contained reference scripts for measuring Pulse STT **accuracy** (Word Error Rate) and **latency** against your own evaluation dataset. Use these as a starting point for vendor comparisons or regression testing on a model upgrade. + +Both scripts are single-file, no imports from this repo. Copy either one anywhere and run. + +## Scripts + +| Script | Endpoint | Reports | +|---|---|---| +| [`ping_pulse_offline.py`](./ping_pulse_offline.py) | `POST /waves/v1/stt/` (pre-recorded HTTP) | WER (corpus-level), per-clip latency, RTF (real-time factor) at p50 / p90 / p95 | +| [`ping_pulse_streaming.py`](./ping_pulse_streaming.py) | `WSS /waves/v1/stt/live` (real-time WebSocket) | WER (corpus-level), tail latency at p50 / p90 / p95 | + +For the methodology behind each metric — what tail latency means, how RTF is computed, which transcripts to sample from — see the [Measuring Latency docs page](https://docs.smallest.ai/waves/documentation/speech-to-text-pulse/benchmarks/measuring-latency). + +## Quick start + +```bash +# 1. Install dependencies (one time) +pip install jiwer tqdm numpy librosa websockets +# If you'll evaluate English audio: +pip install whisper-normalizer + +# 2. Set your API key +export SMALLEST_API_KEY=sk_your_key_here + +# 3. Prepare your eval folder (see "Dataset format" below) + +# 4. Run +python ping_pulse_offline.py /path/to/data_dir --language en --model pulse-pro +python ping_pulse_streaming.py /path/to/data_dir --language en +``` + +## Dataset format + +Both scripts expect the same input layout. `DATA_DIR` is whatever path you pass on the command line: + +``` +DATA_DIR/ +├── audio/ # your audio files (.wav, .mp3, .flac, .ogg, .m4a, .webm) +│ ├── clip_001.wav +│ ├── clip_002.mp3 +│ └── ... +└── metadata.csv # ground-truth transcripts +``` + +`metadata.csv` is comma-separated with one row per clip: + +```csv +filename,reference +clip_001.wav,Hello world this is a test. +clip_002.mp3,The quick brown fox jumps over the lazy dog. +``` + +The `filename` column matches a file in `audio/`. The `reference` column is the ground-truth transcript used for WER scoring. + +## Language support + +Both scripts currently restrict `--language` to `en` (English) and `hi` (Hindi) — the two languages with mature normalisation pipelines for WER scoring. To benchmark another language, edit the `choices=[...]` list in the script and provide pre-normalised reference text. + +`ping_pulse_offline.py` supports both `--model pulse` (multilingual) and `--model pulse-pro` (English-only, leaderboard-ranked). `ping_pulse_streaming.py` only supports `pulse`; Pulse Pro is HTTP-only. + +## Output + +Both scripts print a JSON aggregate at the end: + +```json +{ + "aggregate": { + "mode": "batch", + "language": "en", + "model": "pulse-pro", + "num_clips": 250, + "corpus_wer_pct": 4.55, + "latency_p50_s": 0.42, + "latency_p95_s": 0.87, + "rtf_p50": 0.012, + "rtf_p95": 0.024 + }, + "clips": [ ... ] +} +``` + +Pipe to `jq` to extract just the headline numbers: + +```bash +python ping_pulse_offline.py /path/to/data --language en | jq '.aggregate' +``` + +## What this measures vs. what it doesn't + +These scripts measure **wall-clock client-observed latency**, which includes network transit + server processing + client overhead. They do **not** attribute latency by component. For component attribution (network vs model vs client), see the [docs page](https://docs.smallest.ai/waves/documentation/speech-to-text-pulse/benchmarks/measuring-latency#component-breakdown). + +For streaming, the script reports **tail latency** — the time from the last audio chunk being sent to the final transcript landing. It does **not** report transcript latency (the running gap during the session) or end-of-utterance latency (silence → final). Both of those require client-side instrumentation; see the docs page for measurement patterns. + +## Questions / issues + +Open an issue on this repo, ping us on [Discord](https://discord.gg/9WtSXv26WE), or email [support@smallest.ai](mailto:support@smallest.ai). diff --git a/speech-to-text/benchmarks/ping_pulse_offline.py b/speech-to-text/benchmarks/ping_pulse_offline.py new file mode 100644 index 0000000..ea6631f --- /dev/null +++ b/speech-to-text/benchmarks/ping_pulse_offline.py @@ -0,0 +1,354 @@ +#!/usr/bin/env python3 +"""Ping prod Pulse BATCH (offline/pre-recorded STT) and report accuracy + latency. + +================================================================================ +WHAT THIS SCRIPT DOES +================================================================================ +It takes a folder of audio files plus their correct transcripts, sends each +audio file to Smallest AI's *pre-recorded* (offline) Pulse speech-to-text +service — one whole file per HTTP request — and prints how ACCURATE and how +FAST the service was: + + * Accuracy -> Word Error Rate (WER), as a percentage. Lower is better. + * Speed -> request latency and RTF (explained under "METRICS" below). + +It calls the model-tiered endpoint, so it can hit either the multilingual +"pulse" model or the English-only, highest-accuracy "pulse-pro" model: + + POST https://api.smallest.ai/waves/v1/stt/?model={pulse|pulse-pro}&language= + +This is completely self-contained: it imports nothing from this repo's code. +You can copy this single file anywhere and run it. + +================================================================================ +STEP 1 — INSTALL THE DEPENDENCIES (one time) +================================================================================ +This script needs these Python packages (scoring + progress bar): + + pip install jiwer tqdm + +If (and only if) you will transcribe ENGLISH audio, also install: + + pip install whisper-normalizer + +There is NO audio library to install: the HTTP call uses Python's built-in +urllib, and audio duration is read from the service's own response. +(Inside this repo's conda env these are already installed — you can skip this.) + +================================================================================ +STEP 2 — SET YOUR API KEY +================================================================================ +You need a Smallest AI API key, provided as an environment variable named +SMALLEST_API_KEY. Two ways: + + (a) In this repo, keep the key in the .env file and prefix commands with + `dotenv run` (this loads .env for you): + + dotenv run python scripts/ping_pulse_batch.py ... + + (b) Anywhere else, export it yourself first: + + export SMALLEST_API_KEY=sk_your_key_here + python ping_pulse_batch.py ... + +================================================================================ +STEP 3 — PREPARE YOUR INPUT FOLDER +================================================================================ +Point the script at a folder (call it DATA_DIR) that contains: + + DATA_DIR/ + ├── audio/ <- put all your audio files in here (wav, mp3, ...) + │ ├── clip_001.wav + │ ├── clip_002.wav + │ └── ... + └── metadata.csv <- a CSV with EXACTLY these two column headers: + audio_filename,transcript + +`metadata.csv` example (the header row is required, spelled exactly like this): + + audio_filename,transcript + clip_001.wav,hello how are you + clip_002.wav,the quick brown fox + + * audio_filename = the file's name inside the audio/ folder (NOT a full path). + * transcript = the ground-truth / correct text for that audio. + +================================================================================ +STEP 4 — RUN IT +================================================================================ + # Hindi audio, multilingual "pulse" model: + dotenv run python scripts/ping_pulse_batch.py DATA_DIR --language hi --model pulse + + # English audio, highest-accuracy "pulse-pro" model: + dotenv run python scripts/ping_pulse_batch.py DATA_DIR --language en --model pulse-pro + + * --model defaults to "pulse" if omitted. + * "pulse-pro" is English-only; the script refuses --language hi with it. + +Results are printed to the screen AND saved to a JSON file inside DATA_DIR +named results_batch__.json (per-clip details + an aggregate block). + +================================================================================ +METRICS — HOW TO READ THE OUTPUT +================================================================================ + * WER (%) Word Error Rate. 0% = perfect. Computed with jiwer after + text normalization (English: Whisper's EnglishTextNormalizer; + Hindi: strip punctuation + lowercase). Corpus WER pools every + clip's errors: sum(substitutions+deletions+insertions)/sum(ref words). + + * latency (s) Wall-clock time of the HTTP request as measured by THIS client + (network + server processing). Lower is better. + + * RTF "Real-Time Factor", the standard definition: + RTF = processing_time / audio_duration + where processing_time here is THIS client's round-trip latency + (so it includes network). RTF of 0.1 means it took one-tenth of + the audio's length (10x faster than real time); below 1.0 is + faster than real time. Reported as p50/p90/p95 across clips + (p90/p95 are the slow tail). + + * server_rtfx / When the service returns its own timing (pulse-pro does), these + server_processing_time_ms are surfaced too. server_rtfx is the INVERSE of RTF + measured server-side only (audio_duration / server_processing_time), + e.g. 50 = 50x faster than real time, excluding network. + +Clips are processed one at a time (sequentially) so latency is never inflated by +running several requests in parallel. + +================================================================================ +WHICH PROD MODES THIS COVERS +================================================================================ + Hindi — Pulse Batch : --language hi --model pulse + English — Pulse Pro Batch : --language en --model pulse-pro +(The real-time/streaming modes live in the sibling script ping_pulse_streaming.py.) +""" + +import argparse +import csv +import json +import os +import re +import time +import unicodedata +import urllib.error +import urllib.request +from pathlib import Path +from urllib.parse import urlencode + +from jiwer import process_words +from tqdm import tqdm + +PULSE_BATCH_URL = os.environ.get("PULSE_BATCH_URL", "https://api.smallest.ai/waves/v1/stt/") + + +# ── Normalization ─────────────────────────────────────────────────────────── + + +def strip_punctuation(text: str) -> str: + """Drop Unicode punctuation/symbols, collapse whitespace, lowercase.""" + + cleaned = "".join(character if unicodedata.category(character)[0] not in ("P", "S") else " " for character in text) + return re.sub(r"\s+", " ", cleaned).strip().lower() + + +def make_normalizer(language: str): + """Return a text normalizer for the language. + + English uses Whisper's EnglishTextNormalizer (lazy import — only here). + Hindi (and anything else) uses the simple strip-punctuation normalizer; + Whisper's Hindi normalizer crashes on native Devanagari digits. + """ + + if language == "en": + from whisper_normalizer.english import EnglishTextNormalizer + + english_normalizer = EnglishTextNormalizer() + return lambda text: english_normalizer(text) + return strip_punctuation + + +# ── Transcribing a single clip ──────────────────────────────────────────────── + + +def transcribe_clip(audio_path: Path, language: str, model: str, api_key: str) -> dict[str, object]: + """POST raw audio bytes; return transcript + timing fields.""" + + url = f"{PULSE_BATCH_URL}?{urlencode({'model': model, 'language': language})}" + request = urllib.request.Request( + url, + data=audio_path.read_bytes(), + method="POST", + headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/octet-stream"}, + ) + + start = time.perf_counter() + try: + with urllib.request.urlopen(request, timeout=600) as response: + payload = json.loads(response.read().decode("utf-8")) + except urllib.error.HTTPError as error: + body = error.read().decode("utf-8", "replace") + raise RuntimeError(f"HTTP {error.code} from Pulse batch: {body}") from error + latency = time.perf_counter() - start + + transcript = payload.get("transcription") + if not isinstance(transcript, str): + raise RuntimeError(f"response missing string 'transcription': {payload!r}") + + metadata = payload.get("metadata") or {} + return { + "hypothesis": transcript, + "latency": latency, + "audio_seconds": metadata.get("duration"), + "processing_time_ms": metadata.get("processing_time_ms"), + "rtfx": metadata.get("rtfx"), + } + + +# ── Aggregation helpers ─────────────────────────────────────────────────────── + + +def percentile(values: list[float], fraction: float) -> float: + """Nearest-rank percentile of a list (fraction in [0, 1]).""" + + finite = sorted(value for value in values if value == value) # drop NaN + if not finite: + return float("nan") + index = min(len(finite) - 1, max(0, round(fraction * (len(finite) - 1)))) + return finite[index] + + +def read_metadata(data_dir: Path) -> list[tuple[str, str]]: + """Read metadata.csv -> list of (audio_filename, transcript).""" + + rows: list[tuple[str, str]] = [] + with (data_dir / "metadata.csv").open(newline="", encoding="utf-8") as handle: + reader = csv.DictReader(handle) + if reader.fieldnames is None or "audio_filename" not in reader.fieldnames or "transcript" not in reader.fieldnames: + raise SystemExit(f"metadata.csv must have headers 'audio_filename,transcript' (got {reader.fieldnames})") + for row in reader: + rows.append((row["audio_filename"].strip(), row["transcript"])) + return rows + + +# ── Main ────────────────────────────────────────────────────────────────────── + + +def run(data_dir: Path, language: str, model: str, api_key: str) -> dict[str, object]: + + normalize = make_normalizer(language) + rows = read_metadata(data_dir) + + per_clip: list[dict[str, object]] = [] + total_subs = total_dels = total_ins = total_ref_words = 0 + + for audio_filename, reference in tqdm(rows, desc="batch", unit="clip"): + audio_path = data_dir / "audio" / audio_filename + if not audio_path.exists(): + tqdm.write(f" [skip] missing audio: {audio_path}") + continue + + result = transcribe_clip(audio_path, language, model, api_key) + hypothesis = str(result["hypothesis"]) + + norm_ref = normalize(reference) + norm_hyp = normalize(hypothesis) + measured = process_words(norm_ref, norm_hyp) if norm_ref.strip() else None + + if measured is not None: + subs, dels, ins = measured.substitutions, measured.deletions, measured.insertions + ref_words = len(measured.references[0]) + clip_wer = measured.wer * 100.0 + total_subs += subs + total_dels += dels + total_ins += ins + total_ref_words += ref_words + else: + subs = dels = ins = ref_words = 0 + clip_wer = float("nan") + + latency = float(result["latency"]) + audio_seconds = result["audio_seconds"] + rtf = (latency / audio_seconds) if isinstance(audio_seconds, (int, float)) and audio_seconds else float("nan") + + clip = { + "audio_filename": audio_filename, + "reference": reference, + "hypothesis": hypothesis, + "wer_pct": round(clip_wer, 2), + "subs": subs, + "dels": dels, + "ins": ins, + "ref_words": ref_words, + "audio_seconds": round(audio_seconds, 2) if isinstance(audio_seconds, (int, float)) else None, + "latency_s": round(latency, 3), + "rtf": round(rtf, 4), + "server_processing_time_ms": result["processing_time_ms"], + "server_rtfx": result["rtfx"], + "norm_reference": norm_ref, + "norm_hypothesis": norm_hyp, + } + per_clip.append(clip) + + # ref/hyp shown here are the NORMALIZED strings actually scored. + tqdm.write( + f" {audio_filename}: WER={clip['wer_pct']}% " + f"latency={clip['latency_s']}s RTF={clip['rtf']}" + + (f" server_rtfx={clip['server_rtfx']}" if clip["server_rtfx"] is not None else "") + + f"\n ref: {norm_ref}\n hyp: {norm_hyp}" + ) + + corpus_wer = (100.0 * (total_subs + total_dels + total_ins) / total_ref_words) if total_ref_words else float("nan") + latencies = [float(clip["latency_s"]) for clip in per_clip] + rtfs = [float(clip["rtf"]) for clip in per_clip] + + aggregate = { + "mode": "batch", + "language": language, + "model": model, + "endpoint": PULSE_BATCH_URL, + "num_clips": len(per_clip), + "corpus_wer_pct": round(corpus_wer, 2), + "latency_p50_s": round(percentile(latencies, 0.50), 3), + "latency_p90_s": round(percentile(latencies, 0.90), 3), + "latency_p95_s": round(percentile(latencies, 0.95), 3), + "rtf_p50": round(percentile(rtfs, 0.50), 4), + "rtf_p90": round(percentile(rtfs, 0.90), 4), + "rtf_p95": round(percentile(rtfs, 0.95), 4), + } + return {"aggregate": aggregate, "clips": per_clip} + + +def main(): + + parser = argparse.ArgumentParser(description="Ping prod Pulse batch (offline); report WER + latency + RTF.") + parser.add_argument("data_dir", type=Path, help="Directory with audio/ and metadata.csv") + parser.add_argument("--language", required=True, choices=["en", "hi"], help="Pulse language code") + parser.add_argument("--model", default="pulse", choices=["pulse", "pulse-pro"], help="Model tier (pulse-pro is English-only)") + args = parser.parse_args() + + if args.model == "pulse-pro" and args.language != "en": + raise SystemExit("pulse-pro is English-only; use --language en.") + + api_key = os.environ.get("SMALLEST_API_KEY") + if not api_key: + raise SystemExit("SMALLEST_API_KEY is not set (run via `dotenv run`).") + if not (args.data_dir / "metadata.csv").exists(): + raise SystemExit(f"metadata.csv not found in {args.data_dir}") + + print(f"Pulse BATCH model={args.model} lang={args.language} endpoint={PULSE_BATCH_URL}\n") + output = run(args.data_dir, args.language, args.model, api_key) + + aggregate = output["aggregate"] + print("\n── Aggregate ──") + print(f"clips\t\t\t{aggregate['num_clips']}") + print(f"corpus WER\t\t{aggregate['corpus_wer_pct']}%") + print(f"latency p50/p90/p95 (s)\t{aggregate['latency_p50_s']}\t{aggregate['latency_p90_s']}\t{aggregate['latency_p95_s']}") + print(f"RTF p50/p90/p95\t\t{aggregate['rtf_p50']}\t{aggregate['rtf_p90']}\t{aggregate['rtf_p95']}") + + results_path = args.data_dir / f"results_batch_{args.model}_{args.language}.json" + results_path.write_text(json.dumps(output, ensure_ascii=False, indent=2), encoding="utf-8") + print(f"\nWrote {results_path}") + + +if __name__ == "__main__": + main() diff --git a/speech-to-text/benchmarks/ping_pulse_streaming.py b/speech-to-text/benchmarks/ping_pulse_streaming.py new file mode 100644 index 0000000..98312c5 --- /dev/null +++ b/speech-to-text/benchmarks/ping_pulse_streaming.py @@ -0,0 +1,376 @@ +#!/usr/bin/env python3 +"""Ping prod Pulse STREAMING (real-time WebSocket STT) and report accuracy + latency. + +================================================================================ +WHAT THIS SCRIPT DOES +================================================================================ +It takes a folder of audio files plus their correct transcripts, sends each +audio file to Smallest AI's *streaming* Pulse speech-to-text service exactly the +way a live application would (audio streamed in small chunks, in real time), and +then prints how ACCURATE and how FAST the service was: + + * Accuracy -> Word Error Rate (WER), as a percentage. Lower is better. + * Speed -> tail latency (explained under "METRICS" below). + +This is completely self-contained: it imports nothing from this repo's code. +You can copy this single file anywhere and run it. + +================================================================================ +STEP 1 — INSTALL THE DEPENDENCIES (one time) +================================================================================ +This script needs these Python packages installed in your environment: + + pip install numpy librosa websockets jiwer tqdm + +If (and only if) you will transcribe ENGLISH audio, also install: + + pip install whisper-normalizer + +(Inside this repo's conda env these are already installed — you can skip this.) + +================================================================================ +STEP 2 — SET YOUR API KEY +================================================================================ +You need a Smallest AI API key, provided as an environment variable named +SMALLEST_API_KEY. Two ways: + + (a) In this repo, keep the key in the .env file and prefix commands with + `dotenv run` (this loads .env for you): + + dotenv run python scripts/ping_pulse_streaming.py ... + + (b) Anywhere else, export it yourself first: + + export SMALLEST_API_KEY=sk_your_key_here + python ping_pulse_streaming.py ... + +================================================================================ +STEP 3 — PREPARE YOUR INPUT FOLDER +================================================================================ +Point the script at a folder (call it DATA_DIR) that contains: + + DATA_DIR/ + ├── audio/ <- put all your audio files in here (wav, mp3, ...) + │ ├── clip_001.wav + │ ├── clip_002.wav + │ └── ... + └── metadata.csv <- a CSV with EXACTLY these two column headers: + audio_filename,transcript + +`metadata.csv` example (the header row is required, spelled exactly like this): + + audio_filename,transcript + clip_001.wav,hello how are you + clip_002.wav,the quick brown fox + + * audio_filename = the file's name inside the audio/ folder (NOT a full path). + * transcript = the ground-truth / correct text for that audio. + +================================================================================ +STEP 4 — RUN IT +================================================================================ + # English audio: + dotenv run python scripts/ping_pulse_streaming.py DATA_DIR --language en + + # Hindi audio: + dotenv run python scripts/ping_pulse_streaming.py DATA_DIR --language hi + +Results are printed to the screen AND saved to a JSON file inside DATA_DIR +named results_streaming_.json (per-clip details + an aggregate block). + +================================================================================ +METRICS — HOW TO READ THE OUTPUT +================================================================================ + * WER (%) Word Error Rate. 0% = perfect. Computed with jiwer after + text normalization (English: Whisper's EnglishTextNormalizer; + Hindi: strip punctuation + lowercase). Corpus WER pools every + clip's errors: sum(substitutions+deletions+insertions)/sum(ref words). + + * tail latency Audio is streamed at real-time speed; this is the gap from + sending the LAST piece of audio to receiving the FINAL + transcript — i.e. how long after you stop talking the result + is fully done. + +Clips are processed one at a time (sequentially) so these latency numbers are +never inflated by running several streams in parallel. + +================================================================================ +WHICH PROD MODES THIS COVERS +================================================================================ + Hindi — Pulse Streaming : --language hi + English — Pulse Streaming : --language en +(The offline/batch modes, including English "Pulse Pro", live in the sibling +script ping_pulse_batch.py.) +""" + +import argparse +import asyncio +import csv +import json +import os +import re +import time +import unicodedata +from pathlib import Path + +import numpy as np +import websockets + +from jiwer import process_words +from tqdm import tqdm + +PULSE_WS_URL = os.environ.get("PULSE_WS_URL", "wss://api.smallest.ai/waves/v1/pulse/get_text") + +SAMPLE_RATE = 16000 +ENCODING = "linear16" # 16-bit signed little-endian PCM — what Pulse expects on the wire +CHUNK_SECONDS = 0.160 +CHUNK_SAMPLES = int(CHUNK_SECONDS * SAMPLE_RATE) + + +# ── Normalization ─────────────────────────────────────────────────────────── + + +def strip_punctuation(text: str) -> str: + """Drop Unicode punctuation/symbols, collapse whitespace, lowercase.""" + + cleaned = "".join(character if unicodedata.category(character)[0] not in ("P", "S") else " " for character in text) + return re.sub(r"\s+", " ", cleaned).strip().lower() + + +def make_normalizer(language: str): + """Return a text normalizer for the language. + + English uses Whisper's EnglishTextNormalizer (lazy import — only here). + Hindi (and anything else) uses the simple strip-punctuation normalizer; + Whisper's Hindi normalizer crashes on native Devanagari digits. + """ + + if language == "en": + from whisper_normalizer.english import EnglishTextNormalizer + + english_normalizer = EnglishTextNormalizer() + return lambda text: english_normalizer(text) + return strip_punctuation + + +# ── Streaming a single clip ─────────────────────────────────────────────────── + + +async def stream_clip(audio_path: Path, language: str, api_key: str) -> dict[str, float | str]: + """Stream one clip in real time; return transcript + timing fields.""" + + import librosa + + audio, _ = librosa.load(audio_path.as_posix(), sr=SAMPLE_RATE, mono=True) + audio_seconds = len(audio) / SAMPLE_RATE + + params = f"language={language}&encoding={ENCODING}&sample_rate={SAMPLE_RATE}" + ws_url = f"{PULSE_WS_URL}?{params}" + + segments: list[str] = [] + received_is_last = False + + # Timing markers (perf_counter seconds). + timing: dict[str, float | None] = {"audio_end": None, "final": None, "last_message": None} + + async with websockets.connect( + ws_url, + additional_headers={"Authorization": f"Bearer {api_key}"}, + open_timeout=15, + ping_interval=None, + ) as websocket: + + async def send_audio(): + for offset in range(0, len(audio), CHUNK_SAMPLES): + chunk = audio[offset:offset + CHUNK_SAMPLES] + await websocket.send((chunk * 32768.0).astype(np.int16).tobytes()) + # Pace at real time so tail latency reflects live streaming. Sleep by + # the chunk's ACTUAL duration so the final (shorter) chunk doesn't + # push audio_end late and understate tail latency. + await asyncio.sleep(len(chunk) / SAMPLE_RATE) + timing["audio_end"] = time.perf_counter() + await websocket.send(json.dumps({"type": "close_stream"})) + + async def receive_transcripts(): + nonlocal received_is_last + while True: + try: + message = await asyncio.wait_for(websocket.recv(), timeout=5.0) + data: dict[str, object] = json.loads(message) + except asyncio.TimeoutError: + break + + timing["last_message"] = time.perf_counter() + + if data.get("error"): + raise RuntimeError(f"Pulse error: {data.get('code')} {data.get('error')}") + + transcript = data.get("transcript") + if data.get("is_final"): + if not isinstance(transcript, str): + raise RuntimeError(f"is_final without string transcript: {data!r}") + segments.append(transcript) + + if data.get("is_last"): + received_is_last = True + timing["final"] = time.perf_counter() + break + + sender = asyncio.create_task(send_audio()) + try: + await receive_transcripts() + finally: + # If the receiver returned/raised before the sender finished, don't + # leave it pending (orphaned task warnings / unretrieved exceptions). + if not sender.done(): + sender.cancel() + try: + await sender + except BaseException: + pass + + if not received_is_last and not segments: + raise RuntimeError("session ended without is_last and no is_final segments (server stall?)") + + # If we stopped on the 5 s quiet timeout rather than is_last, use the time of + # the last message received as the final-transcript timestamp. + if timing["final"] is None and timing["last_message"] is not None: + timing["final"] = timing["last_message"] + + hypothesis = "".join(segments).strip() + tail_latency = (timing["final"] - timing["audio_end"]) if timing["final"] and timing["audio_end"] else float("nan") + + return {"hypothesis": hypothesis, "audio_seconds": audio_seconds, "tail_latency": tail_latency} + + +# ── Aggregation helpers ─────────────────────────────────────────────────────── + + +def percentile(values: list[float], fraction: float) -> float: + """Nearest-rank percentile of a list (fraction in [0, 1]).""" + + finite = sorted(value for value in values if value == value) # drop NaN + if not finite: + return float("nan") + index = min(len(finite) - 1, max(0, round(fraction * (len(finite) - 1)))) + return finite[index] + + +def read_metadata(data_dir: Path) -> list[tuple[str, str]]: + """Read metadata.csv -> list of (audio_filename, transcript).""" + + rows: list[tuple[str, str]] = [] + with (data_dir / "metadata.csv").open(newline="", encoding="utf-8") as handle: + reader = csv.DictReader(handle) + if reader.fieldnames is None or "audio_filename" not in reader.fieldnames or "transcript" not in reader.fieldnames: + raise SystemExit(f"metadata.csv must have headers 'audio_filename,transcript' (got {reader.fieldnames})") + for row in reader: + rows.append((row["audio_filename"].strip(), row["transcript"])) + return rows + + +# ── Main ────────────────────────────────────────────────────────────────────── + + +async def run(data_dir: Path, language: str, api_key: str) -> dict[str, object]: + + normalize = make_normalizer(language) + rows = read_metadata(data_dir) + + per_clip: list[dict[str, object]] = [] + total_subs = total_dels = total_ins = total_ref_words = 0 + + for audio_filename, reference in tqdm(rows, desc="streaming", unit="clip"): + audio_path = data_dir / "audio" / audio_filename + if not audio_path.exists(): + tqdm.write(f" [skip] missing audio: {audio_path}") + continue + + result = await stream_clip(audio_path, language, api_key) + hypothesis = str(result["hypothesis"]) + + norm_ref = normalize(reference) + norm_hyp = normalize(hypothesis) + measured = process_words(norm_ref, norm_hyp) if norm_ref.strip() else None + + if measured is not None: + subs, dels, ins = measured.substitutions, measured.deletions, measured.insertions + ref_words = len(measured.references[0]) + clip_wer = measured.wer * 100.0 + total_subs += subs + total_dels += dels + total_ins += ins + total_ref_words += ref_words + else: + subs = dels = ins = ref_words = 0 + clip_wer = float("nan") + + clip = { + "audio_filename": audio_filename, + "reference": reference, + "hypothesis": hypothesis, + "wer_pct": round(clip_wer, 2), + "subs": subs, + "dels": dels, + "ins": ins, + "ref_words": ref_words, + "audio_seconds": round(float(result["audio_seconds"]), 2), + "tail_latency_s": round(float(result["tail_latency"]), 3), + "norm_reference": norm_ref, + "norm_hypothesis": norm_hyp, + } + per_clip.append(clip) + + # ref/hyp shown here are the NORMALIZED strings actually scored. + tqdm.write( + f" {audio_filename}: WER={clip['wer_pct']}% " + f"tail={clip['tail_latency_s']}s\n" + f" ref: {norm_ref}\n" + f" hyp: {norm_hyp}" + ) + + corpus_wer = (100.0 * (total_subs + total_dels + total_ins) / total_ref_words) if total_ref_words else float("nan") + tails = [float(clip["tail_latency_s"]) for clip in per_clip] + + aggregate = { + "mode": "streaming", + "language": language, + "endpoint": PULSE_WS_URL, + "num_clips": len(per_clip), + "corpus_wer_pct": round(corpus_wer, 2), + "tail_p50_s": round(percentile(tails, 0.50), 3), + "tail_p90_s": round(percentile(tails, 0.90), 3), + "tail_p95_s": round(percentile(tails, 0.95), 3), + } + return {"aggregate": aggregate, "clips": per_clip} + + +def main(): + + parser = argparse.ArgumentParser(description="Ping prod Pulse streaming; report WER + tail latency.") + parser.add_argument("data_dir", type=Path, help="Directory with audio/ and metadata.csv") + parser.add_argument("--language", required=True, choices=["en", "hi"], help="Pulse language code") + args = parser.parse_args() + + api_key = os.environ.get("SMALLEST_API_KEY") + if not api_key: + raise SystemExit("SMALLEST_API_KEY is not set (run via `dotenv run`).") + if not (args.data_dir / "metadata.csv").exists(): + raise SystemExit(f"metadata.csv not found in {args.data_dir}") + + print(f"Pulse STREAMING lang={args.language} endpoint={PULSE_WS_URL}\n") + output = asyncio.run(run(args.data_dir, args.language, api_key)) + + aggregate = output["aggregate"] + print("\n── Aggregate ──") + print(f"clips\t\t\t{aggregate['num_clips']}") + print(f"corpus WER\t\t{aggregate['corpus_wer_pct']}%") + print(f"tail p50/p90/p95 (s)\t{aggregate['tail_p50_s']}\t{aggregate['tail_p90_s']}\t{aggregate['tail_p95_s']}") + + results_path = args.data_dir / f"results_streaming_{args.language}.json" + results_path.write_text(json.dumps(output, ensure_ascii=False, indent=2), encoding="utf-8") + print(f"\nWrote {results_path}") + + +if __name__ == "__main__": + main() From 4cb21c3c0e7565b4bf9e90d76d3a111cb7c9de76 Mon Sep 17 00:00:00 2001 From: Abhishek Mishra Date: Wed, 17 Jun 2026 15:27:12 +0530 Subject: [PATCH 2/3] fix(speech-to-text): README metadata.csv schema + output location MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Live end-to-end test on a fresh venv with a 5-clip eval dataset caught two README errors: 1. metadata.csv columns: header was documented as 'filename,reference' but both scripts validate for 'audio_filename,transcript' (and exit with a hard error on mismatch). Fixed README to match the script's actual contract. 2. JSON output location: README didn't mention the scripts write results to DATA_DIR/results__.json. Documented the filenames, shape, and a corrected jq example. Verified both scripts now run end-to-end against the corrected README: ping_pulse_offline.py --language en --model pulse-pro -> 4/5 clips at 0% WER, 1 at 66% (TTS clip too short — model truncated 'this is a test') -> corpus WER 10.26%, latency p50/p95 = 0.69/0.78s, RTFx ~32-49 ping_pulse_streaming.py --language en -> 5/5 clips at 0% WER -> tail latency p50/p95 = 250/310ms --- speech-to-text/benchmarks/README.md | 38 ++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 12 deletions(-) diff --git a/speech-to-text/benchmarks/README.md b/speech-to-text/benchmarks/README.md index 90ffa5f..7b3ef67 100644 --- a/speech-to-text/benchmarks/README.md +++ b/speech-to-text/benchmarks/README.md @@ -44,15 +44,18 @@ DATA_DIR/ └── metadata.csv # ground-truth transcripts ``` -`metadata.csv` is comma-separated with one row per clip: +`metadata.csv` is comma-separated with exactly two columns, header row required: ```csv -filename,reference +audio_filename,transcript clip_001.wav,Hello world this is a test. clip_002.mp3,The quick brown fox jumps over the lazy dog. ``` -The `filename` column matches a file in `audio/`. The `reference` column is the ground-truth transcript used for WER scoring. +- `audio_filename` matches a file in `audio/` (just the basename, not a full path). +- `transcript` is the ground-truth text used for WER scoring. + +The headers must be spelled exactly `audio_filename,transcript` — the scripts validate this and exit early on a mismatch. ## Language support @@ -62,7 +65,12 @@ Both scripts currently restrict `--language` to `en` (English) and `hi` (Hindi) ## Output -Both scripts print a JSON aggregate at the end: +Both scripts print per-clip results live + an aggregate summary table at the end. They also write a JSON results file to `DATA_DIR/`: + +- `ping_pulse_offline.py` → `DATA_DIR/results_batch__.json` +- `ping_pulse_streaming.py` → `DATA_DIR/results_streaming_.json` + +The JSON has an `aggregate` block and a `clips` array with per-clip detail: ```json { @@ -70,21 +78,27 @@ Both scripts print a JSON aggregate at the end: "mode": "batch", "language": "en", "model": "pulse-pro", - "num_clips": 250, - "corpus_wer_pct": 4.55, - "latency_p50_s": 0.42, - "latency_p95_s": 0.87, - "rtf_p50": 0.012, - "rtf_p95": 0.024 + "endpoint": "https://api.smallest.ai/waves/v1/stt/", + "num_clips": 5, + "corpus_wer_pct": 10.26, + "latency_p50_s": 0.693, + "latency_p90_s": 0.781, + "latency_p95_s": 0.781, + "rtf_p50": 0.2051, + "rtf_p90": 0.3753, + "rtf_p95": 0.3753 }, - "clips": [ ... ] + "clips": [ + {"audio_filename": "clip_001.wav", "reference": "...", "hypothesis": "...", + "wer_pct": 0.0, "latency_s": 0.69, "rtf": 0.27, "server_rtfx": 32.5} + ] } ``` Pipe to `jq` to extract just the headline numbers: ```bash -python ping_pulse_offline.py /path/to/data --language en | jq '.aggregate' +jq '.aggregate' /path/to/data/results_batch_pulse-pro_en.json ``` ## What this measures vs. what it doesn't From 18d706fec3019ee60699624637c428a666e34664 Mon Sep 17 00:00:00 2001 From: Abhishek Mishra Date: Wed, 17 Jun 2026 16:09:48 +0530 Subject: [PATCH 3/3] feat(speech-to-text/benchmarks): bundle ready-to-run sample datasets MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 20 English read-speech clips + 10 Hindi clips, each with a metadata.csv in the script's expected schema. Lets users run the benchmark scripts against real audio without first building their own eval dataset. Verified end-to-end on a fresh venv: ping_pulse_offline.py samples/english_samples --language en --model pulse-pro -> 20 clips, corpus WER 4.1%, latency p50/p95 = 0.76/0.92s, RTF p50 0.09 ping_pulse_streaming.py samples/hindi_samples --language hi -> 10 clips, corpus WER 7.5%, tail latency p50/p95 = 236/304 ms README updated with a 'Quick start — using the bundled samples' section showing exact commands. .gitignore in samples/ keeps runtime results_*.json files out of the repo. --- speech-to-text/benchmarks/README.md | 24 +++++++++++++++---- speech-to-text/benchmarks/samples/.gitignore | 1 + .../samples/english_samples/metadata.csv | 21 ++++++++++++++++ .../samples/hindi_samples/metadata.csv | 11 +++++++++ 4 files changed, 52 insertions(+), 5 deletions(-) create mode 100644 speech-to-text/benchmarks/samples/.gitignore create mode 100644 speech-to-text/benchmarks/samples/english_samples/metadata.csv create mode 100644 speech-to-text/benchmarks/samples/hindi_samples/metadata.csv diff --git a/speech-to-text/benchmarks/README.md b/speech-to-text/benchmarks/README.md index 7b3ef67..8d17baf 100644 --- a/speech-to-text/benchmarks/README.md +++ b/speech-to-text/benchmarks/README.md @@ -13,7 +13,14 @@ Both scripts are single-file, no imports from this repo. Copy either one anywher For the methodology behind each metric — what tail latency means, how RTF is computed, which transcripts to sample from — see the [Measuring Latency docs page](https://docs.smallest.ai/waves/documentation/speech-to-text-pulse/benchmarks/measuring-latency). -## Quick start +## Quick start — using the bundled samples + +The repo ships with two small ready-to-run sample datasets under `samples/`: + +| Dataset | Clips | Source | +|---|---|---| +| `samples/english_samples/` | 20 clips | English read speech, FLEURS-style | +| `samples/hindi_samples/` | 10 clips | Hindi read speech, Devanagari transcripts | ```bash # 1. Install dependencies (one time) @@ -24,11 +31,18 @@ pip install whisper-normalizer # 2. Set your API key export SMALLEST_API_KEY=sk_your_key_here -# 3. Prepare your eval folder (see "Dataset format" below) +# 3. Run against the bundled samples +python ping_pulse_offline.py samples/english_samples --language en --model pulse-pro +python ping_pulse_streaming.py samples/hindi_samples --language hi +``` -# 4. Run -python ping_pulse_offline.py /path/to/data_dir --language en --model pulse-pro -python ping_pulse_streaming.py /path/to/data_dir --language en +For a quick sanity check, the offline run on the bundled English samples typically reports corpus WER ~4-5% with Pulse Pro; streaming on the Hindi samples reports tail latency p50 around 240-300 ms. + +## Run against your own dataset + +```bash +python ping_pulse_offline.py /path/to/your/data_dir --language en --model pulse-pro +python ping_pulse_streaming.py /path/to/your/data_dir --language en ``` ## Dataset format diff --git a/speech-to-text/benchmarks/samples/.gitignore b/speech-to-text/benchmarks/samples/.gitignore new file mode 100644 index 0000000..b30f58b --- /dev/null +++ b/speech-to-text/benchmarks/samples/.gitignore @@ -0,0 +1 @@ +results_*.json diff --git a/speech-to-text/benchmarks/samples/english_samples/metadata.csv b/speech-to-text/benchmarks/samples/english_samples/metadata.csv new file mode 100644 index 0000000..ac05872 --- /dev/null +++ b/speech-to-text/benchmarks/samples/english_samples/metadata.csv @@ -0,0 +1,21 @@ +audio_filename,transcript +1036771894648013441.wav,as light pollution in their heyday was not the kind of problem it is today they are usually located in cities or at campuses easier to reach than those built in modern times +1038214857203833067.wav,through the night between 150 and 200 copies were made now known as dunlap broadsides +10492135991603561867.wav,in some areas boiling water for a minute is enough in others several minutes are needed +12018418318980824126.wav,this is called a chemical's ph you can make an indicator using red cabbage juice +11426556553252626005.wav,the harbor was the site of an infamous naval standoff in 1889 when seven ships from germany the us and britain refused to leave the harbor +11974497390699501411.wav,the find also grants insight into the evolution of feathers in birds +12382404601461539160.wav,on some routes the larger companies have their own planes but for other routes and smaller firms there was a problem +10622204403793540766.wav,for example each year students from bennet school in north carolina design a website about their trip to the state capital each year the website gets remodeled but old versions are kept online to serve as a scrapbook +12164368102080603077.wav,the governor's office said nineteen of the injured were police officers +1116401173651214130.wav,during this period of european history the catholic church which had become rich and powerful came under scrutiny +11060294228176938598.wav,as a result two fish species have become extinct and two others have become endangered including the humpback chub +10849792007407044602.wav,virtual teams are held to the same standards of excellence as conventional teams but there are subtle differences +11559094331412401332.wav,hydrogen ions are protons that had their electrons stripped off them since hydrogen atoms consist of one proton and one electron +125194026787628519.wav,cuomo 53 began his governorship earlier this year and signed a bill last month legalizing same-sex marriage +10471564175308895403.wav,but there are a lot of things about birds that still look like a dinosaur +11318597463051480163.wav,mrs kirchner announced her intention to run for president at the argentine theatre the same location she used to start her 2005 campaign for the senate as member of the buenos aires province delegation +1054805305879816018.wav,it was ruled by the vichy french these were french people who had made peace with the germans in 1940 and worked with the invaders instead of fighting them +10383288285905292158.wav,for example each year students from bennet school in north carolina design a website about their trip to the state capital each year the website gets remodeled but old versions are kept online to serve as a scrapbook +11668734600035674277.wav,the ruling party south west africa people's organisation swapo also retained a majority in the parliamentary elections +1146643745158029931.wav,a civilization is a singular culture shared by a significant large group of people who live and work co-operatively a society diff --git a/speech-to-text/benchmarks/samples/hindi_samples/metadata.csv b/speech-to-text/benchmarks/samples/hindi_samples/metadata.csv new file mode 100644 index 0000000..9c3832d --- /dev/null +++ b/speech-to-text/benchmarks/samples/hindi_samples/metadata.csv @@ -0,0 +1,11 @@ +audio_filename,transcript +hi_051.wav,प्राचीन संस्कृतियों और जनजातियों ने दूध बाल मांस और चमड़े को अपनी आसान पहुंच में बनाए रखना शुरू कर दिया +hi_096.wav,अमेरिकी जिमनास्टिक्स संयुक्त राज्य ओलंपिक समिति के पत्र का समर्थन करता है साथ ही सभी एथलीटों को एक सुरक्षित माहौल देने के लिए ओलंपिक परिवार के होने की अहमियत स्वीकार करता है +hi_004.wav,इंटरनेट पर यह खोज शत्रुतापूर्ण पर्यावरण पाठ्यक्रम के लिए अक्सर आपको एक स्थानीय कंपनी का पता प्रदान करेगी +hi_018.wav,कैल्शियम और पोटैशियम जैसे तत्वों को धातु माना जाता है बेशक चांदी और सोने जैसी धातुएं भी हैं +hi_000.wav,कुछ अणुओं में अस्थिर केंद्रक होता है जिसका मतलब यह है कि उनमें थोड़े या बिना किसी झटके से टूटने की प्रवृत्ति होती है +hi_028.wav,तकनीकी निर्धारण की ज़्यादातर व्याख्याएं दो सामान्य विचारों के बारे में बताती हैं तकनीकी विकास पर सांस्कृतिक या राजनीतिक असर का कोई भी प्रभाव नहीं पड़ता है और परिणामस्वरूप सामाजिक रूप से अनुकूलित होने के बजाय वह तकनीक निहित समाजों को प्रभावित करती है +hi_050.wav,यह शहर देश के बाकी शहरों से अलग है क्योंकि यह किसी अफ्रीकी शहर की बजाय अरब शहर लगता है +hi_037.wav,पुलिस अधीक्षक चंद्र शेखर सोलंकी ने कहा कि आरोपी ढके हुए चेहरे के साथ अदालत में पेश हुए थे +hi_042.wav,महिलाएं यह अनुशंसा की जाती है कि कोई भी महिला यात्री वास्तविक वैवाहिक स्थिति के बावजूद कहती है कि वह विवाहित है +hi_070.wav,बाघ की दहाड़ किसी शेर की तेज़ आवाज़ वाली दहाड़ की तरह नहीं होती है लेकिन उसकी बंधी हुई चिल्लाने जैसी होती है