feat: add end-to-end voice-agent evaluation skill and regression CLI

.github/workflows/test-sarvam-voice.yml

@@ -0,0 +1,51 @@
+name: Test Sarvam voice skill
+
+on:
+  pull_request:
+    paths:
+      - "test-sarvam-voice/**"
+      - "README.md"
+      - ".github/workflows/test-sarvam-voice.yml"
+  push:
+    branches: [main]
+    paths:
+      - "test-sarvam-voice/**"
+      - "README.md"
+      - ".github/workflows/test-sarvam-voice.yml"
+
+permissions:
+  contents: read
+
+jobs:
+  offline-test:
+    name: Python ${{ matrix.python-version }}
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.12"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install offline test tools
+        run: python -m pip install "PyYAML>=6,<7" "jiwer>=4,<5" "ruff>=0.12,<1" "uv>=0.8,<1"
+
+      - name: Validate skill metadata and run unit tests
+        run: python -m unittest discover -s test-sarvam-voice/tests -p "test_*.py" -v
+
+      - name: Check CLI entry points and PEP 723 environment
+        run: |
+          uv run --python "${{ matrix.python-version }}" test-sarvam-voice/scripts/voice_lab.py --help
+          uv run --python "${{ matrix.python-version }}" test-sarvam-voice/scripts/voice_lab.py plan --help
+          uv run --python "${{ matrix.python-version }}" test-sarvam-voice/scripts/voice_lab.py run --help
+          uv run --python "${{ matrix.python-version }}" test-sarvam-voice/scripts/voice_lab.py compare --help
+
+      - name: Lint without modifying files
+        run: |
+          ruff check test-sarvam-voice/scripts/voice_lab.py test-sarvam-voice/tests
+          ruff format --check test-sarvam-voice/scripts/voice_lab.py test-sarvam-voice/tests

.gitignore

@@ -3,3 +3,6 @@ skills-lock.json
 demo/
 .specstory/
 .cursorindexingignore
+__pycache__/
+*.pyc
+.venv/

README.md

@@ -27,6 +27,7 @@ These skills bridge that gap. Each one gives AI coding assistants the exact SDK
 | [text-to-speech](./text-to-speech) | Speech synthesis with Bulbul v3. Python & JS/TS quick starts, HTTP stream, WebSocket, pronunciation dictionaries, and unsupported parameter warnings. |
 | [translate](./translate) | Text translation with Sarvam-Translate v1 and Mayura v1. Python & JS/TS quick starts, model feature differences, and silent parameter failures. |
 | [voice-agents](./voice-agents) | Real-time voice agents with LiveKit and Pipecat (Python). JS/TS SDK reference for custom pipelines. |
+| [test-sarvam-voice](./test-sarvam-voice) | End-to-end voice-agent regression tests across Saaras v3, Sarvam-30B, and Bulbul v3, with quality, latency, cost, reliability, CI gates, and standalone HTML reports. |
 
 ## Installation
 
@@ -69,6 +70,8 @@ Each skill is a lean **correction layer** with both **Python** and **JavaScript/
 - **Parameters that error** (e.g., `pitch`/`loudness` returns 400 on Bulbul v3)
 - **Non-trivial SDK patterns** (e.g., Batch API job chain, WebSocket async connect)
 
+Workflow skills may also bundle deterministic scripts when reproducibility, offline validation, or machine-readable CI gates are part of the task.
+
 For everything else — full parameter tables, voice catalogs, language codes, rate limits, cookbook examples — the skill points to [llms.txt](https://docs.sarvam.ai/llms.txt), which is always up to date.
 
 ## Links

test-sarvam-voice/SKILL.md

@@ -0,0 +1,92 @@
+---
+name: test-sarvam-voice
+description: Benchmark Sarvam AI voice-agent pipelines end to end across Saaras v3, Sarvam-30B, and Bulbul v3. Use for prerecorded voice regression tests, multilingual ASR quality checks, VAD tuning, latency and cost analysis, reliability testing, baseline comparisons, or CI quality gates. Produces reproducible JSON, standalone HTML, and semantic-evaluation exports.
+license: Apache-2.0
+metadata:
+  author: sarvam-ai
+  version: "1.0"
+---
+
+# Test Sarvam Voice
+
+Measure the pipeline instead of testing STT, LLM, and TTS in isolation. Use the bundled CLI for deterministic validation, execution, reporting, and regression gates. Use `voice-agents` when building a LiveKit or Pipecat agent; use this skill when proving that an existing voice pipeline remains fast, accurate, reliable, and affordable.
+
+> [!IMPORTANT]
+> Keep `SARVAM_API_KEY` in the environment. Never put credentials, raw headers, private audio, or unredacted transcripts in a suite or report.
+
+## Run the workflow
+
+Run commands from the `sarvamai/skills` repository root.
+
+1. Copy `references/example-suite.yaml` and replace each audio path and reference transcript.
+2. Keep v1 input audio as mono 16-bit PCM WAV at 8 or 16 kHz. Convert other audio first:
+
+   ```bash
+   ffmpeg -i input.mp3 -ac 1 -ar 16000 -c:a pcm_s16le output.wav
+   ```
+
+3. Validate the suite and review the conservative cost projection before any network call:
+
+   ```bash
+   uv run test-sarvam-voice/scripts/voice_lab.py plan suite.yaml
+   ```
+
+4. Run the suite sequentially with no retry by default:
+
+   ```bash
+   export SARVAM_API_KEY="your-api-key"
+   uv run test-sarvam-voice/scripts/voice_lab.py run suite.yaml --out results/
+   ```
+
+5. Compare a candidate with a committed or downloaded baseline:
+
+   ```bash
+   uv run test-sarvam-voice/scripts/voice_lab.py compare \
+     baseline/report.json candidate/report.json --out comparison/
+   ```
+
+Treat exit `0` as success, `1` as a quality/regression gate failure, `2` as invalid input or a rejected budget, `3` as an incomplete live run, and `130` as interruption.
+
+## Design the suite
+
+- Cover the languages, acoustic conditions, intents, numbers, names, and code-mixing patterns that matter in production.
+- Use a stable human-verified reference transcript for each case.
+- Add `entities` only for critical values that must survive ASR, such as phone numbers, amounts, dates, people, and locations. Interpret `configured_entity_recall` as deterministic substring preservation, not Sarvam's LLM-based Entity Score.
+- Tag cases by language, channel, domain, and risk so the report exposes cohort regressions.
+- Set absolute gates for release requirements and regression thresholds for candidate-versus-baseline changes.
+- Keep `reasoning_effort: null` for latency-sensitive voice responses. Increase `max_tokens` if Sarvam-30B returns no speakable content.
+- Keep the budget ceiling conservative. The CLI refuses an over-budget run before creating a connection.
+
+Read [references/report-schema.md](references/report-schema.md) when adding CI gates, interpreting timings, or consuming JSON/JSONL artifacts.
+
+## Protect evaluation integrity
+
+- Default to one worker, zero retries, redacted text, and discarded TTS audio.
+- Add `--concurrency` only after checking account rate limits. Add bounded `--retries` only to measure recovery behavior; authentication, quota, and validation failures are never retried.
+- Add `--include-transcripts` only in a private output directory when semantic evaluation is required.
+- Add `--retain-audio` only for deliberate listening tests. Do not commit generated speech.
+- Use `--mock-events` for offline protocol regression tests. Never present mock latency as live service performance.
+- Compare distributions and P95 latency, not only averages. Inspect per-case failures before accepting a passing aggregate.
+
+## Interpret the outputs
+
+The run directory contains:
+
+- `report.json`: schema-versioned measurements, configuration, errors, gates, and suite/language/tag aggregates.
+- `report.html`: escaped, standalone comparison surface with no external assets.
+- `llm-wer.jsonl`: rows compatible with Sarvam's `llm_wer` dataset columns when transcript inclusion is enabled.
+- `llm-intent-entity.jsonl`: rows compatible with Sarvam's `llm_intent_entity` dataset columns when transcript inclusion is enabled.
+- `audio/`: synthesized responses only when explicitly retained.
+
+Use raw and normalized WER/CER together. Normalization removes punctuation, symbols, casing, and Unicode compatibility differences; it does not make semantic judgments. Treat all INR values as estimates tied to the dated pricing snapshot in the suite.
+
+## Refresh changing interfaces
+
+Before changing model IDs, supported languages, VAD parameters, limits, or pricing, fetch the current index and primary pages:
+
+- **https://docs.sarvam.ai/llms.txt** — current documentation index
+- [Pricing](https://docs.sarvam.ai/api/getting-started/pricing)
+- [Saaras v3](https://docs.sarvam.ai/api/getting-started/models/saaras)
+- [Sarvam-30B](https://docs.sarvam.ai/api/getting-started/models/sarvam-30b)
+- [Bulbul v3](https://docs.sarvam.ai/api/getting-started/models/bulbul)
+- [Credits and rate limits](https://docs.sarvam.ai/api/getting-started/ratelimits)

test-sarvam-voice/evals/evals.json

@@ -0,0 +1,23 @@
+{
+  "skill_name": "test-sarvam-voice",
+  "evals": [
+    {
+      "id": 1,
+      "prompt": "Create a Hindi-English customer-support voice regression suite covering phone numbers, amounts, noisy audio, and code-mixing. Keep the projected spend under INR 15 and explain how I should validate it before the first API call.",
+      "expected_output": "A version 1 YAML suite with valid Saaras v3, Sarvam-30B, and Bulbul v3 configuration; tagged cases; critical entities; gates; a dated pricing snapshot; and the plan command.",
+      "files": []
+    },
+    {
+      "id": 2,
+      "prompt": "Compare these two Sarvam voice reports, identify which gate regressed, and give me the exact command and exit-code behavior I can use in CI.",
+      "expected_output": "Use the bundled compare command, interpret candidate-minus-baseline quality/latency/cost/reliability deltas, and explain exit 0 versus exit 1.",
+      "files": []
+    },
+    {
+      "id": 3,
+      "prompt": "My Hindi telephony cohort has stable WER but EOS-to-first-audio P95 increased after changing VAD settings. Diagnose the report without exposing transcripts and propose a controlled follow-up test.",
+      "expected_output": "Inspect language/tag aggregates and VAD/finalization timing, keep transcripts redacted, vary one VAD parameter at a time, and compare repeated candidate reports against the same baseline.",
+      "files": []
+    }
+  ]
+}

test-sarvam-voice/references/example-suite.yaml

@@ -0,0 +1,93 @@
+version: 1
+name: support-agent-hi-en
+
+pipeline:
+  stt:
+    model: saaras:v3
+    language: unknown
+    mode: transcribe
+    sample_rate: 16000
+    chunk_ms: 250
+    realtime_pacing: true
+    vad:
+      high_sensitivity: false
+      signals: true
+      positive_speech_threshold: null
+      negative_speech_threshold: null
+      min_speech_frames: null
+      first_turn_min_speech_frames: null
+      negative_frames_count: null
+      negative_frames_window: null
+      start_speech_volume_threshold: null
+      interrupt_min_speech_frames: null
+      pre_speech_pad_frames: null
+      num_initial_ignored_frames: null
+  llm:
+    model: sarvam-30b
+    system_prompt: >-
+      You are a concise customer-support voice assistant. Answer in the user's
+      language, preserve critical numbers and names, and keep the response easy
+      to speak aloud.
+    temperature: 0.2
+    reasoning_effort: null
+    max_tokens: 160
+  tts:
+    model: bulbul:v3
+    language: hi-IN
+    speaker: shubh
+    pace: 1.0
+    temperature: 0.6
+    codec: wav
+    sample_rate: 24000
+
+execution:
+  repetitions: 3
+  timeout_seconds: 60
+  retries: 0
+  concurrency: 1
+  budget_inr: 25.0
+
+pricing:
+  currency: INR
+  retrieved_at: "2026-07-19"
+  source: https://docs.sarvam.ai/api/getting-started/pricing
+  stt_inr_per_hour: 30.0
+  llm_input_inr_per_million_tokens: 2.5
+  llm_cached_input_inr_per_million_tokens: 1.5
+  llm_output_inr_per_million_tokens: 10.0
+  tts_inr_per_10k_characters: 30.0
+
+gates:
+  max_normalized_wer: 0.20
+  max_normalized_cer: 0.12
+  min_configured_entity_recall: 1.0
+  max_eos_to_first_audio_p95_ms: 2200
+  min_success_rate: 0.99
+  max_estimated_cost_inr: 20.0
+
+regression:
+  max_normalized_wer_increase: 0.02
+  max_eos_to_first_audio_p95_increase_ms: 250
+  max_cost_increase_percent: 15
+  max_success_rate_drop: 0.01
+
+cases:
+  - id: change-phone-hi
+    audio: audio/change-phone-hi.wav
+    reference: मेरा फोन नंबर 9840950950 पर बदल दीजिए
+    language: hi-IN
+    tts_language: hi-IN
+    context: The caller is updating the phone number on an existing account.
+    tags: [hindi, support, numbers, clean]
+    entities:
+      - value: "9840950950"
+        aliases: ["9840 950 950"]
+
+  - id: delivery-status-en
+    audio: audio/delivery-status-en.wav
+    reference: Where is my order number 47291
+    language: en-IN
+    tts_language: en-IN
+    context: The caller wants a delivery update.
+    tags: [english, support, numbers, clean]
+    entities: ["47291"]

test-sarvam-voice/references/report-schema.md

@@ -0,0 +1,72 @@
+# Report contract and measurement semantics
+
+## Contents
+
+- [Artifacts](#artifacts)
+- [Timing boundaries](#timing-boundaries)
+- [Quality and reliability](#quality-and-reliability)
+- [Cost model](#cost-model)
+- [Regression gates](#regression-gates)
+- [Semantic exports](#semantic-exports)
+- [CI pattern](#ci-pattern)
+
+## Artifacts
+
+Treat `schema_version` as the compatibility boundary. Run reports use `sarvam.voice-eval/v1`; comparisons use `sarvam.voice-compare/v1`. Consumers must reject unknown major versions.
+
+Run reports contain the resolved suite configuration, pricing snapshot, absolute gate results, deterministic per-case records, and aggregates for the full suite, each language, and each tag. Report ordering is stable by case ID and repetition. HTML is a standalone escaped view of the same JSON data and loads no scripts, fonts, or stylesheets.
+
+Transcript, reference, context, configured entity, system-prompt, and LLM response text are absent by default. Their records contain only a SHA-256 digest, character count, and `redacted: true`. Semantic JSONL files are intentionally empty unless `--include-transcripts` is set.
+
+## Timing boundaries
+
+All live timings use the local monotonic clock:
+
+| Metric | Start | End |
+|---|---|---|
+| `stt_connection` | before entering the Saaras WebSocket context | connected socket available |
+| `vad_start`, `vad_end` | STT stage start | local receipt of the corresponding VAD event |
+| `stt_finalization_after_speech_end` | received `END_SPEECH`, or last audio chunk when absent | final non-empty transcript receipt |
+| `llm_time_to_first_token` | Sarvam-30B request start | first non-empty content delta |
+| `tts_time_to_first_byte` | Bulbul HTTP-stream request start | first non-empty binary chunk |
+| `eos_to_first_audio` | received `END_SPEECH`, or last audio chunk when absent | first synthesized audio byte |
+| `total_turn` | before STT connection | complete TTS stream |
+
+Mock fixtures use the same field names but are deterministic protocol replays. Do not mix mock and live reports in a performance baseline.
+
+## Quality and reliability
+
+`raw_wer` and `raw_cer` preserve the input strings. Normalized metrics apply Unicode NFKC, case folding, punctuation/symbol removal, and whitespace collapse before deterministic edit distance. Empty references score `0` only against an empty hypothesis and `1` otherwise.
+
+`configured_entity_recall` checks only suite-provided strings and aliases after deterministic text, compact-whitespace, and digit normalization. It is intentionally not Sarvam's LLM-based Entity Score and makes no semantic-equivalence claim.
+
+Success rate includes every requested repetition. A failed attempt records a sanitized exception type, message, and status code. The runner retries only transport/time-out and server-side failures when explicitly configured; it never retries authentication, quota/rate-limit, or invalid-request responses.
+
+P50 and P95 use linearly interpolated percentiles over successful samples. Each metric summary carries its own `sample_count`; always inspect it and the aggregate `failure_count` beside percentiles.
+
+## Cost model
+
+STT duration rounds up to a minimum of one billed second per case attempt. LLM input/output uses returned streaming usage when available. If usage is absent, a four-characters-per-token heuristic is used and `estimated: true` remains visible. TTS uses the response character count.
+
+`plan` projects a conservative maximum from audio duration, configured `max_tokens`, every allowed retry attempt, and the dated suite pricing snapshot. `run` rejects a projection above `execution.budget_inr` before importing the live client or opening a connection. The measured cost aggregate covers completed runs; use the conservative projection or provider billing when failed attempts may have consumed partial stages. Refresh pricing from the official source before using reports for financial planning.
+
+## Regression gates
+
+Run gates apply absolute requirements to the candidate suite. Comparison gates apply candidate-minus-baseline deltas:
+
+- WER increase must remain at or below its ratio limit.
+- EOS-to-audio P95 increase must remain at or below its millisecond limit.
+- estimated cost percentage increase must remain at or below its percentage limit.
+- success-rate drop must remain at or below its ratio limit.
+
+A skipped gate has a null threshold. Gate failure returns exit code `1`; incomplete live execution returns `3`, even if aggregate gates also fail.
+
+## Semantic exports
+
+With transcript inclusion enabled, `llm-wer.jsonl` exposes `transcription`, `prediction`, `audio_filepath`, and `language`, matching the configurable dataset columns consumed by Sarvam's `llm_wer` project.
+
+`llm-intent-entity.jsonl` exposes `ground_truth`, `asr_output`, `audio_file`, `language`, and `context`, matching the columns consumed by Sarvam's `llm_intent_entity` project. Run those projects separately when semantic judgments are needed; this skill does not bundle an LLM judge or its credentials.
+
+## CI pattern
+
+Keep live credentials out of pull-request CI. Replay a checked-in protocol fixture against generated temporary WAV input, compare the report to a reviewed baseline, and fail on exit `1`, `2`, or `3`. Run live suites only in a protected scheduled workflow with a strict INR ceiling and private artifact retention.