feat: add end-to-end voice-agent evaluation skill and regression CLI

[ayushhagarwal]

Problem

The repository explains how to build a Sarvam voice agent, but it does not provide a reproducible way to answer the release question: did an STT, prompt, VAD, model, or TTS change improve the whole turn without regressing quality, latency, reliability, or cost?

This PR adds test-sarvam-voice, an Agent Skill and self-contained PEP 723 CLI for end-to-end regression testing across Saaras v3, Sarvam-30B, and Bulbul v3.

What this adds

• A versioned YAML suite for pipeline settings, cases, repetitions, budgets, absolute gates, and baseline-regression thresholds.
• plan, run, and compare commands in one uv run-compatible Python script.
• Real-time-paced prerecorded WAV streaming into Saaras v3, streamed Sarvam-30B completions, and Bulbul v3 HTTP-stream synthesis.
• Raw and normalized WER/CER, deterministic configured-entity recall, VAD/STT/LLM/TTS timing boundaries, P50/P95 cohorts, success rate, attempts, billable units, and estimated INR.
• Schema-versioned JSON, escaped standalone HTML, and optional JSONL exports compatible with the dataset columns used by Sarvam's llm_wer and llm_intent_entity projects.
• Offline protocol-event fixtures, 24 deterministic tests, three Agent Skill evaluation prompts, and a path-filtered Python 3.10/3.12 workflow with no live API calls.

Difference from voice-agents

voice-agents is the construction guide for LiveKit and Pipecat applications. This skill is the measurement layer for an existing pipeline. It deliberately uses the individual Sarvam SDK stages so it can expose stage boundaries and compare a candidate against a reviewed baseline.

mono PCM WAV
  -> Saaras v3 WebSocket + VAD
  -> Sarvam-30B streamed response
  -> Bulbul v3 HTTP audio stream
  -> JSON / HTML / semantic JSONL / exit code

CLI

uv run test-sarvam-voice/scripts/voice_lab.py plan suite.yaml
uv run test-sarvam-voice/scripts/voice_lab.py run suite.yaml --out results/
uv run test-sarvam-voice/scripts/voice_lab.py compare \
  baseline/report.json candidate/report.json --out comparison/

The runner defaults to one worker, zero retries, redacted sensitive text, and discarded synthesized audio. Exit codes distinguish success (0), gate failure (1), invalid input or rejected budget (2), incomplete live execution (3), and interruption (130).

Privacy, cost, and reliability controls

• Accepts only mono 16-bit PCM WAV at 8 or 16 kHz and fails with an actionable conversion command.
• Redacts transcript, reference, context, configured entity, system-prompt, and response text by default; HTML escapes all dynamic values and loads no external assets.
• Keeps semantic exports empty unless transcript inclusion is explicitly enabled.
• Uses a dated, source-linked pricing snapshot and rejects a conservative projection above the suite budget before opening a connection. The projection includes every allowed retry attempt.
• Retries only transport, timeout, and server failures when configured; authentication, quota/rate-limit, and invalid-request failures are never retried.
• Rejects mock-versus-live comparisons because their latency clocks are not comparable.

Validation

• python -m unittest discover: 24/24 passed
• Agent Skill metadata validation: passed
• ruff check: passed
• ruff format --check: passed
• PEP 723 uv run help for all three commands: passed
• Fresh-context skill evaluations: suite authoring, report comparison, and VAD-regression diagnosis all completed with the intended safe workflow
• Diff whitespace, credential-pattern, and prohibited-token audits: passed

The live smoke test is not included yet because no SARVAM_API_KEY was available in the authoring environment. The PR is intentionally a Draft; the offline workflow never requires a key, and a sanitized sub-₹1 live result can be added before marking it ready.

Intentional v1 boundaries

• No microphone UI, LiveKit/Pipecat adapter, or bundled speech recording.
• No overlapping incremental LLM-to-TTS synthesis or persistent TTS WebSocket.
• No embedded semantic judge or claim that deterministic entity recall is Sarvam's LLM-based Entity Score.
• No retry by default and no retained transcript/audio by default.

Maintainer questions

1. Does a deterministic workflow skill of this size fit the repository's lean correction-layer direction, or would you prefer the CLI in a companion repository with only the skill entry here?
2. Is the proposed sarvam.voice-eval/v1 report boundary useful for future cookbook or CI examples?
3. Are the v1 timing boundaries—especially local VAD receipt and EOS-to-first-audio—aligned with the voice team's internal evaluation terminology?
4. Should the semantic JSONL adapters remain optional exports, or should a follow-up integrate the evaluation projects more directly?

[ayushhagarwal]

Hi @vinayak-sarvam ,
Gentle follow-up on this draft. It adds an end-to-end regression workflow for Sarvam voice agents covering quality, latency, reliability, and estimated cost.
I noticed main has moved and the PR now has conflicts. I’m happy to rebase and resolve them. Before I invest further, could you give me a quick scope signal: does this workflow belong in the skills repository, or would you prefer the CLI in a companion repository with only the Agent Skill here?