diff --git a/docs.json b/docs.json index 09399ee2a..5bf67cc38 100644 --- a/docs.json +++ b/docs.json @@ -56,6 +56,13 @@ "guides/fundamentals/custom-frame-processor", "guides/fundamentals/detecting-user-idle", "guides/fundamentals/ivr", + { + "group": "Evaluations", + "pages": [ + "guides/fundamentals/evaluations/overview", + "guides/fundamentals/evaluations/bluejay" + ] + }, "guides/fundamentals/metrics", "guides/fundamentals/user-input-muting", "guides/fundamentals/recording-audio", diff --git a/guides/fundamentals/evaluations/bluejay.mdx b/guides/fundamentals/evaluations/bluejay.mdx new file mode 100644 index 000000000..9a23da85c --- /dev/null +++ b/guides/fundamentals/evaluations/bluejay.mdx @@ -0,0 +1,159 @@ +--- +title: "Bluejay" +description: "Simulation, observability, and evaluation platform for voice AI agents with native Pipecat integration." +--- + +## Overview + +[Bluejay](https://getbluejay.ai) is a simulation, observability, and evaluation platform purpose-built for voice AI agents. It provides no-code simulation testing and production call monitoring that integrate directly with Pipecat, whether you're running on Pipecat Cloud or self-hosting. + + + + + +With Bluejay, you can: + +- Run automated simulations that call your agent and evaluate its responses +- Define test scenarios covering edge cases like interruptions, unexpected input, and multi-turn flows +- Monitor every production call with automated quality scoring +- Track evaluation metrics over time to catch regressions early + +## Pipecat Cloud integration + +If your agent is deployed on [Pipecat Cloud](/deployment/pipecat-cloud/introduction), Bluejay offers two zero-configuration integration paths: + + + + Enter your Pipecat Cloud API key and agent name in Bluejay's dashboard. + Bluejay connects directly to your agent's API to spin up simulation sessions + with no code changes required. + + + + Enter your agent's phone number into Bluejay and start running simulations + immediately. Bluejay calls your agent just like a real user would, testing + end-to-end behavior over telephony. + + + + + The telephony integration tests the full call stack, from phone network to + your agent and back, making it ideal for catching issues that only surface in + real call conditions. + + +## Self-hosted integration + +If you're running Pipecat on your own infrastructure, Bluejay integrates via a WebSocket connection. Point Bluejay at your agent's WebSocket endpoint and it will establish a session to run simulations against your agent directly. + +See the [Bluejay WebSocket integration guide](https://docs.getbluejay.ai/simulation-integrations/websockets) for setup instructions. + +## Observability + +Simulations cover pre-deployment testing, but observability ensures your agent maintains quality with real users. Bluejay's [Evaluate API](https://docs.getbluejay.ai/api-reference/endpoint/evaluate) lets you submit any production call for automated evaluation. + +```python +import requests + +url = "https://api.getbluejay.ai/v1/evaluate" +headers = {"X-API-Key": ""} +payload = { + "agent_id": "", + "start_time_utc": "2025-03-31T18:30:00Z", + "participants": [ + {"role": "AGENT", "name": "Healthcare Agent Harry"}, + {"role": "USER", "name": "John Doe"}, + ], + "recording_url": "https://s3.amazonaws.com/my-recordings/call-123.wav", +} + +response = requests.post(url, json=payload, headers=headers) +``` + + + Integrate the evaluate endpoint into your agent's session cleanup logic to + automatically evaluate every production call without manual intervention. + + +## Traces + +Bluejay supports [tracing](https://docs.getbluejay.ai/core-concepts/traces) to monitor and observe your agent's execution flow, latency, and performance in real-time. Traces conform to the [OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/) standard, so you can use any compatible instrumentation library, including OpenInference, Langfuse, and OpenLLMetry. + +To send traces to Bluejay: + +1. Instrument your application to export traces to Bluejay's OTLP endpoint +2. Link traces to call evaluations by including the `trace_id` in your [Evaluate API](https://docs.getbluejay.ai/api-reference/endpoint/evaluate) requests +3. View traces alongside your call evaluations in the Bluejay dashboard + +### Example: OpenTelemetry setup + +Configure the OpenTelemetry SDK to export traces to Bluejay: + +```python +from opentelemetry.sdk import trace as trace_sdk +from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter +from opentelemetry.sdk.trace.export import SimpleSpanProcessor +from opentelemetry.sdk.resources import SERVICE_NAME, Resource + +endpoint = "https://otlp.getbluejay.ai/v1/traces" +resource = Resource.create({SERVICE_NAME: "my-pipecat-agent"}) + +tracer_provider = trace_sdk.TracerProvider(resource=resource) +headers = { + "X-API-KEY": "", +} + +tracer_provider.add_span_processor( + SimpleSpanProcessor(OTLPSpanExporter(endpoint, headers=headers)) +) +``` + +Once the tracer provider is configured, use it with any OpenTelemetry-compatible instrumentation. For example, to automatically trace LLM calls: + +```python +from openinference.instrumentation.openai import OpenAIInstrumentor + +OpenAIInstrumentor().instrument(tracer_provider=tracer_provider) +``` + +## Next steps + + + + Full setup guides, API reference, and configuration options. + + + + Step-by-step guide for connecting Bluejay to your Pipecat agent. + + + + Learn about evaluation strategies for Pipecat agents. + + + + Capture conversation transcripts to use with evaluation tools. + + diff --git a/guides/fundamentals/evaluations/overview.mdx b/guides/fundamentals/evaluations/overview.mdx new file mode 100644 index 000000000..066a281ee --- /dev/null +++ b/guides/fundamentals/evaluations/overview.mdx @@ -0,0 +1,153 @@ +--- +title: "Evaluations" +sidebarTitle: "Overview" +description: "Test and improve your voice AI agents from local prompt iteration to production monitoring." +--- + +## Overview + +Building a voice AI agent is only half the challenge. You also need to know it handles real conversations reliably. A good evaluation strategy progresses through two phases: + +1. **Local testing**: Iterate on your LLM prompts quickly without needing live audio services, reducing cost and tightening the feedback loop during development. +2. **Production evaluation**: Automated simulations and observability for deployed agents, catching regressions and tracking quality over time with real user traffic. + +Starting locally and layering in production tooling as your agent matures gives you the fastest path to a reliable, well-tested agent. + +## Local prompt testing + +Before investing in full end-to-end simulations, focus on getting your LLM prompts right. Pipecat's architecture makes it straightforward to test your agent's conversational logic without running STT or TTS services, saving both time and cost during development. + +The most efficient way to iterate on prompts is to bypass audio entirely and send text directly to your LLM pipeline. This lets you validate conversational logic, function calling, and response quality in seconds rather than minutes. + +You can configure your pipeline to accept text input instead of audio by replacing STT with a transcript-based input: + +```python +from pipecat.frames.frames import TranscriptionFrame + +# Send a simulated user utterance directly into the pipeline +frame = TranscriptionFrame( + text="I'd like to schedule an appointment for tomorrow at 3pm", + user_id="test-user", + timestamp=0, +) +``` + +This approach lets you: + +- Test prompt variations rapidly without waiting for audio processing +- Validate function calling behavior with specific user inputs +- Build repeatable test cases for edge cases and failure modes +- Run tests in CI without audio infrastructure + +## Production evaluation + +Once your prompts are solid and you've validated the local experience, production evaluation tools help you scale testing and monitor quality across real deployments. This is where evaluation platforms come in. + +### Simulations + +Automated test conversations exercise your agent's behavior across scenarios, edge cases, and failure modes before they reach users. Simulation platforms can connect to your agent via API, WebSocket, or telephony to run scripted or AI-driven test calls. + +Key things to test with simulations: + +- **Multi-turn flows**: Verify your agent handles complete conversation paths correctly +- **Edge cases**: Test interruptions, unexpected input, silence, and barge-in +- **Telephony behavior**: End-to-end testing over real phone networks catches issues that only surface in production call conditions +- **Regressions**: Run simulation suites before each deployment to catch breaking changes + +### Observability + +Continuous evaluation of live calls lets you catch regressions, track quality over time, and close the loop between what you test and what users experience. Common approaches include: + +- Submitting call recordings and transcripts for automated quality scoring +- Tracking evaluation metrics over time to detect quality drift +- Using [OpenTelemetry traces](/server/utilities/opentelemetry) to monitor latency and execution flow + +Together, simulations and observability form a feedback loop: simulations validate changes before deployment, and observability surfaces issues that inform your next round of tests. + +### Evaluation platforms + +Several platforms offer simulation testing and production monitoring for voice AI agents: + + + + Simulation, observability, and evaluation platform with native Pipecat Cloud integration. Supports no-code API, WebSocket, and telephony testing. + + + + Evaluation and testing platform for voice AI agents with simulation and scoring capabilities. + + + + Automated testing and quality assurance platform for voice AI agents. + + + + + Building an evaluation integration for Pipecat? We welcome contributions to + this page. Open a PR on the [docs + repository](https://github.com/pipecat-ai/docs). + + +## Pipecat's built-in tools + +Pipecat provides several building blocks that feed into any evaluation workflow: + +- **[Metrics](/guides/fundamentals/metrics)**: Built-in TTFB, processing time, and usage tracking for LLM and TTS services +- **[Saving transcripts](/guides/fundamentals/saving-transcripts)**: Capture conversation transcripts for offline analysis and evaluation +- **[OpenTelemetry](/server/utilities/opentelemetry)**: Export traces to any OTel-compatible backend for latency and performance monitoring +- **[Observers](/server/utilities/observers/observer-pattern)**: Monitor frame flow without modifying the pipeline, useful for custom instrumentation + +## Next steps + + + + Monitor performance and LLM/TTS usage with Pipecat's built-in metrics. + + + + Capture conversation transcripts to use with evaluation tools. + + + + Export traces for performance monitoring and debugging. + + + + Build custom processors for evaluation-specific instrumentation. + + diff --git a/videos/pipecat_dashboard_demo.mp4 b/videos/pipecat_dashboard_demo.mp4 new file mode 100644 index 000000000..d614890c0 Binary files /dev/null and b/videos/pipecat_dashboard_demo.mp4 differ