diff --git a/README.md b/README.md index 58ca586..15e05c1 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,7 @@ End-to-end LLM evaluation toolkit for Dynatrace AI Observability. - Python `>=3.10` (dt-ai-ingest) - A Dynatrace environment with GenAI spans (`gen_ai.*` OTEL attributes) - A Dynatrace platform token (generated at [myaccount.dynatrace.com/platformTokens](https://myaccount.dynatrace.com/platformTokens) — `dt-evals doctor` walks you through it) -- Credentials for your judge provider (OpenAI, Anthropic, Google, AWS Bedrock, or Azure OpenAI) +- Credentials for your judge provider (OpenAI, Anthropic, Google Gemini Enterprise Platform (Vertex AI), AWS Bedrock, or Azure OpenAI) ## Install @@ -169,6 +169,7 @@ dt-evals run --since 2h --sample 20 --concurrency 8 --debug | `--concurrency ` | Number of parallel evaluation workers | | `--debug` | Per-step timing logs | | `--config ` | Path to a specific config file | +| `--store-evaluated-prompt` | Include the evaluated prompt/response in bizevents written to Dynatrace (overrides `storeEvaluatedPrompt` in config; default: false) | **GitHub Actions example:** @@ -314,7 +315,7 @@ DT_API_TOKEN=dt0c01.xxxxx ## Built-in Evaluators -13 built-in LLM judge evaluators plus statistical drift detection. +14 built-in LLM judge evaluators plus statistical drift detection. | Evaluator | Measures | |-----------|----------| @@ -340,14 +341,13 @@ DT_API_TOKEN=dt0c01.xxxxx | Provider | Default model | Notes | |----------|--------------|-------| -| `openai` | `gpt-5.4` | `OPENAI_API_KEY` | -| `anthropic` | `claude-sonnet-4-7` | `ANTHROPIC_API_KEY` | -| `vertex` | `gemini-3-pro` | Workload Identity / ADC by default — set `judge.project` + `judge.location` (or `GOOGLE_CLOUD_PROJECT` + `GOOGLE_CLOUD_LOCATION`). No `GOOGLE_API_KEY` required. | -| `gemini` | `gemini-3.1-flash-live` | `GOOGLE_API_KEY` | -| `bedrock` | `anthropic.claude-opus-4-7` | `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` | +| `openai` | `gpt-5-mini` | `OPENAI_API_KEY` | +| `anthropic` | `claude-sonnet-4-6` | `ANTHROPIC_API_KEY` | +| `vertex`, `gemini` | `gemini-3.1-flash-lite` | Google Gemini Enterprise Platform (Vertex AI) — use ADC by default with `vertex` and set `judge.project` + `judge.location` (or `GOOGLE_CLOUD_PROJECT` + `GOOGLE_CLOUD_LOCATION`); use `GOOGLE_API_KEY` with `gemini`. | +| `bedrock` | `us.anthropic.claude-haiku-4-5-20251001-v1:0` | `AWS_ACCESS_KEY_ID` + `AWS_SECRET_ACCESS_KEY` | | `azure-openai` | user-provided deployment name | `AZURE_OPENAI_API_KEY` + `AZURE_OPENAI_ENDPOINT` + `AZURE_OPENAI_API_VERSION` | -Override the model with `--model ` or set `judge.model` in config. +Override the model with `--model ` or set `judge.model` in config. See `dt-eval-cli/src/config/defaults.ts` for current defaults. --- @@ -361,7 +361,6 @@ name: travel-assistant-prod dynatrace: environmentUrl: https://your-env.live.dynatrace.com - apiToken: dt0c01.xxxxx judge: provider: openai @@ -397,9 +396,7 @@ judge: provider: bedrock model: us.anthropic.claude-3-5-haiku-20241022-v1:0 region: us-east-1 - # or use AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY env vars - apiKey: - secretKey: + # credentials via AWS_ACCESS_KEY_ID + AWS_SECRET_ACCESS_KEY env vars, IAM role, or SSO ``` **Azure OpenAI example:** @@ -426,9 +423,9 @@ JUDGE_MODEL=gpt-4.1 OPENAI_API_KEY=sk-... # Anthropic ANTHROPIC_API_KEY=sk-ant-... -# Google Gemini Developer API +# Google Gemini Enterprise Platform (Vertex AI) GOOGLE_API_KEY=... -# Google Vertex AI — uses Workload Identity / Application Default Credentials (no API key) +# Or use Workload Identity / Application Default Credentials instead of an API key GOOGLE_CLOUD_PROJECT=my-gcp-project GOOGLE_CLOUD_LOCATION=global # AWS Bedrock diff --git a/dt-eval-cli/README.md b/dt-eval-cli/README.md index 19a0602..f387d0b 100644 --- a/dt-eval-cli/README.md +++ b/dt-eval-cli/README.md @@ -24,7 +24,6 @@ The repository also includes `dt-eval-lib`, a reusable TypeScript evaluation eng npx @dynatrace-oss/dt-evals configure npx @dynatrace-oss/dt-evals validate npx @dynatrace-oss/dt-evals run --since 1h --sample 10 --concurrency 10 -npx @dynatrace-oss/dt-evals deploy --provider aws ``` Tune `--concurrency` (or `judge.concurrency` in your yaml) to control how many judge calls run in parallel — bump it for faster runs, drop it if the provider rate-limits you. Default is 5. @@ -40,7 +39,6 @@ Tune `--concurrency` (or `judge.concurrency` in your yaml) to control how many j - Go from low score to root cause with end-to-end AI observability - Correlate failures with prompts, retrieval, tool calls, latency, and dependencies - Reuse the same evaluator catalog in CI, app code, and local workflows with dt-eval-lib -- Deploy the runner to AWS Lambda, Google Cloud Run, Azure Functions or Docker for continuous evals - Trigger alerts and optional remediation from the same evaluation pipeline ## Packages @@ -53,12 +51,12 @@ Tune `--concurrency` (or `judge.concurrency` in your yaml) to control how many j ## What It Does -- **13 built-in judge metrics** for safety, grounding, relevance, quality, and retrieval quality +- **14 built-in judge metrics** for safety, grounding, relevance, quality, and retrieval quality - **Statistical drift detection** against a 7 day baseline of prior evaluation scores - **Flexible sampling** with random percentage, latest `N`, or `errors-only` - **PII masking before judge calls** for emails, phone numbers, credit cards, and SSNs - **Evaluated prompt/response excluded from bizevents by default** — opt in with `storeEvaluatedPrompt` (see [Configuration](#configuration)) -- **OpenAI and Anthropic support** with optional custom base URLs for gateways and proxies +- **OpenAI, Anthropic, Azure OpenAI, Google Gemini Enterprise Platform (Vertex AI), and Bedrock support** with optional custom base URLs for gateways and proxies - **CI friendly runs** with JSON output and non-zero exit codes on threshold breaches - **Local run history** with list, inspect, and export flows - **Scheduled runs** stored locally and triggerable on demand @@ -83,7 +81,7 @@ The current CLI path is Dynatrace specific. The product positioning is broader: - Node.js `>=20` - A Dynatrace environment with GenAI spans or OpenTelemetry-style `gen_ai.*` fields -- An OpenAI or Anthropic API key for the judge model +- Credentials for your judge provider (OpenAI, Anthropic, Azure OpenAI, Google Gemini Enterprise Platform (Vertex AI), or Bedrock) ## Install @@ -290,7 +288,6 @@ name: travel-assistant-prod dynatrace: environmentUrl: https://your-env.live.dynatrace.com - apiToken: dt0c01.xxxxx dtctlContext: my-prod-context judge: @@ -298,7 +295,7 @@ judge: model: gpt-4.1 timeout: 30000 maxRetries: 2 - concurrency: 5 # parallel judge calls; --concurrency overrides this + concurrency: 5 scope: service: travel-assistant @@ -321,9 +318,6 @@ alerts: relevance: 0.7 answer-completeness: 0.8 -# Off by default — the evaluated prompt/response are not written back to Dynatrace. -# Set to true to include gen_ai.evaluation.input.question/.answer/.system_prompt -# in the result bizevents. Overridable per-invocation with --store-evaluated-prompt. storeEvaluatedPrompt: false ``` @@ -346,10 +340,8 @@ and `destination` (write): dynatrace: origin: environmentUrl: https://prod.live.dynatrace.com - apiToken: dt0s16.xxxxx # needs storage:spans:read, storage:buckets:read destination: environmentUrl: https://eval-results.dev.apps.dynatracelabs.com - apiToken: dt0s16.yyyyy # needs storage:bizevents:read for drift, plus storage:events:write and storage:metrics:write ``` Top-level `environmentUrl` / `apiToken` (if present) act as fallbacks — if @@ -382,41 +374,50 @@ For `dt-evals alerts apply` (Dynatrace Workflows), grant these additionally on t ### Custom span field mapping By default, the CLI reads OTel GenAI semconv (`gen_ai.input.messages`, -`gen_ai.output.messages`, `gen_ai.output.message`, …) and OpenLLMetry conventions (`gen_ai.prompt.N.*`, -`gen_ai.completion.0.content`). If your spans expose the LLM I/O under -different attribute names, point the CLI at them via `scope.spanFields`. +`gen_ai.output.messages`, `gen_ai.output.message`, …). It also probes a +small set of legacy fallback attributes (`gen_ai.prompt.N.*`, +`gen_ai.completion.0.content`) for compatibility with older emitters. If +your spans expose the LLM I/O under different attribute names — for +example OpenInference (`llm.input_messages`, `llm.output_messages`, +`llm.model_name`) — point the CLI at them via `scope.spanFields`. See the +[`dynatrace-ai-agent-instrumentation-examples`](https://github.com/dynatrace-oss/dynatrace-ai-agent-instrumentation-examples) +for end-to-end instrumentation examples. Each entry accepts a single attribute or a list of candidates; the first non-null value wins, with the built-in defaults appended as fallback. -**Example 1 — non-semconv spans:** +**Example 1 — OTel GenAI variant** (some Bedrock / Vertex SDK emitters +serialize the full message array under the plural attribute +`gen_ai.output.messages` instead of the singular +`gen_ai.output.message`): ```yaml scope: - service: my-llm-service - since: 30m + service: pydantic-ai-music-agent spanFields: - input: llm.user_input # or [llm.user_input, my.custom.input] - output: llm.response - context: span.context # can point to any span field the user wants as evaluator context, e.g. RAG context or system prompt - systemInstruction: llm.system - model: llm.model + output: [gen_ai.output.message, gen_ai.output.messages] ``` -**Example 2 — OTel GenAI plural form** (some Bedrock / Vertex SDK -emitters serialize the full message array under the plural attribute -name `gen_ai.output.messages` instead of the singular `gen_ai.output.message`): +The runner stringifies whatever it finds, so a JSON array under +`gen_ai.output.messages` reaches the judge as the assistant turn(s). +This is still OTel GenAI-compatible — `spanFields` just lets you handle +emitter-specific variants explicitly. + +**Example 2 — OTel spans not following the GenAI SemConv (OpenInference):** ```yaml scope: - service: pydantic-ai-music-agent + service: my-openinference-service + since: 30m spanFields: - output: gen_ai.output.messages # JSON-encoded array of {role, parts} + input: [llm.input_messages, input.value] + output: [llm.output_messages, output.value] + model: llm.model_name ``` -The runner stringifies whatever it finds, so a JSON array under -`gen_ai.output.messages` reaches the judge as the assistant turn(s). -This is convention-friendly: list every variant you've seen and the -parser walks them in order. +OpenInference usually stores chat turns in `llm.input_messages` / +`llm.output_messages` and plain-text payloads in `input.value` / +`output.value`, so listing both keeps the mapping resilient across +instrumentations. ### Per-metric input routing @@ -429,22 +430,20 @@ turn in isolation, not the joined transcript: ```yaml metrics: enabled: - - faithfulness # legacy string form + - faithfulness # plain string form - id: user-frustration inputs: - input: userPrompt # latest user-role prompt slot + input: userPrompt - id: hallucination inputs: - context: context # use the canonical context field for this metric + context: context ``` Available canonical fields: `input`, `output`, `context`, `systemInstruction`, `model`, `userPrompt` (latest user-role prompt slot, extracted from -`gen_ai.prompt.N.role == "user"`). When the requested field is empty on a -given span, the slot falls back to `span.input` / `span.output` / -`span.context` so a single misconfiguration doesn't drop spans. `context` has -no built-in default span attribute; if a span does not provide it, evaluator -context is omitted unless you explicitly route `inputs.context` elsewhere. +`gen_ai.prompt.N.role == "user"`). `context` has no built-in default span +attribute — if a span does not provide it, evaluator context is omitted unless +you explicitly route `inputs.context` elsewhere. ### Full example @@ -470,23 +469,17 @@ scope: sampling: strategy: latest count: 50 - # Map custom span attributes to canonical fields. Defaults handle OTel - # GenAI semconv + OpenLLMetry; override only what you need. spanFields: - context: span.context # can point to any span field the user wants as evaluator context, e.g. RAG context or system prompt + context: span.context output: [gen_ai.output.message, gen_ai.output.messages] # input, systemInstruction, model use built-in defaults metrics: enabled: - # Plain string form — uses span.input / span.output / span.context - faithfulness - relevance - hallucination - answer-completeness - # Object form — overrides which canonical span field feeds the - # evaluator's `input` slot. user-frustration scores the user's turn - # alone instead of the full joined transcript. - id: user-frustration inputs: input: userPrompt @@ -521,10 +514,9 @@ OPENAI_BASE_URL=https://your-gateway.example.com/v1 ANTHROPIC_API_KEY=sk-ant-... ANTHROPIC_BASE_URL=https://your-proxy.example.com -# Gemini Developer API +# Google Gemini Enterprise Platform (Vertex AI) GOOGLE_API_KEY=... - -# Vertex AI — authenticates via Workload Identity / Application Default Credentials (no API key). +# Or authenticate via Workload Identity / Application Default Credentials instead of an API key. # Set the target project and region (or judge.project / judge.location in config): GOOGLE_CLOUD_PROJECT=my-gcp-project GOOGLE_CLOUD_LOCATION=global @@ -559,23 +551,24 @@ Drift results are written back as the same event type with `gen_ai.evaluation.ty ## Built-in Evaluators -`dt-evals` ships with 13 built-in LLM judge evaluators, plus drift detection as a separate statistical metric. +`dt-evals` ships with 14 built-in LLM judge evaluators, plus drift detection as a separate statistical metric. -| Evaluator | Measures | Required fields | -|-----------|----------|-----------------| -| `toxicity` | Harmful, offensive, or unsafe output | `input`, `output` | -| `faithfulness` | Whether the answer is grounded in provided context | `input`, `output`, `context` | -| `hallucination` | Unsupported or fabricated claims | `input`, `output`, `context` | -| `pii-leakage` | Personally identifiable information in the answer | `input`, `output` | -| `relevance` | Whether the answer addresses the user request | `input`, `output` | -| `factual-accuracy` | Accuracy against a reference answer | `input`, `output`, `expectedOutput` | -| `user-frustration` | Whether the user was left frustrated at the end of the conversation | `input` | -| `context-relevance` | Retrieval quality for supplied context | `input`, `context` | +| Evaluator | Measures | Fields used | +|-----------|----------|-------------| | `answer-completeness` | Whether all parts of the request were answered | `input`, `output` | -| `prompt-injection` | Prompt injection attempts in the input | `input`, `output` | | `bias` | Harmful bias or unfair framing | `input`, `output` | -| `summarization-quality` | Summary faithfulness, coverage, and conciseness | `input`, `output` | | `conciseness` | Whether the answer avoids filler and unnecessary padding | `input`, `output` | +| `context-relevance` | Retrieval quality for supplied context | `input` | +| `factual-accuracy` | Accuracy using world knowledge | `input`, `output` | +| `faithfulness` | Whether the answer is grounded in provided context | `input`, `output` | +| `fluency` | Grammar, clarity, and natural language quality | `input`, `output` | +| `hallucination` | Unsupported or fabricated claims | `input`, `output` | +| `pii-leakage` | Personally identifiable information in the answer | `input`, `output` | +| `prompt-injection` | Prompt injection attempts in the input | `input` | +| `relevance` | Whether the answer addresses the user request | `input`, `output` | +| `summarization-quality` | Summary faithfulness, coverage, and conciseness | `input`, `output` | +| `toxicity` | Harmful, offensive, or unsafe output | `output` | +| `user-frustration` | Frustration signals in the user's message | `input` | ### Custom evaluators @@ -627,104 +620,7 @@ Use it when you care about regressions that show up gradually, not just single-r ## TypeScript Library -The repository includes `dt-eval-lib` for programmatic usage. - -### Basic example - -```ts -import { evaluate, BuiltInMetric } from "@dynatrace-oss/dt-eval-lib"; - -const result = await evaluate( - BuiltInMetric.Faithfulness, - { - input: "Can I cancel my booking after check-in?", - output: "Yes, you can cancel any time and get a full refund.", - context: "Bookings can be canceled up to 24 hours before check-in. Refunds are not available after check-in.", - }, - { - provider: { - provider: "openai", - apiKey: process.env.OPENAI_API_KEY, - model: "gpt-4.1", - timeout: 30000, - maxRetries: 2, - }, - }, -); - -console.log(result.score); -console.log(result.explanation.summary); -console.log(result.explanation.reasoning); -``` - -### Inspect the evaluator catalog - -```ts -import { listPrompts, getPrompt, BuiltInMetric } from "@dynatrace-oss/dt-eval-lib"; - -const prompts = listPrompts(); -const relevance = getPrompt(BuiltInMetric.Relevance); - -console.log(prompts.map((prompt) => prompt.id)); -console.log(relevance.requiredFields); -``` - -### Override thresholds in code - -```ts -import { evaluate, BuiltInMetric } from "@dynatrace-oss/dt-eval-lib"; - -const result = await evaluate( - BuiltInMetric.Relevance, - { - input: "What are your support hours?", - output: "Our support team is available on weekdays.", - }, - { - provider: { - provider: "anthropic", - apiKey: process.env.ANTHROPIC_API_KEY, - }, - scoring: { - thresholdOverride: 0.8, - }, - }, -); - -console.log(result.score.label); -``` - -### Use a custom gateway or proxy - -```ts -import { evaluate, BuiltInMetric } from "@dynatrace-oss/dt-eval-lib"; - -await evaluate( - BuiltInMetric.Toxicity, - { - input: "Write a release note", - output: "The release fixes several issues.", - }, - { - provider: { - provider: "openai", - apiKey: process.env.OPENAI_API_KEY, - baseUrl: process.env.OPENAI_BASE_URL, - }, - }, -); -``` - -Library features: - -- OpenAI, Anthropic, Google Vertex AI, and Gemini providers -- Structured judge responses with score, summary, and reasoning -- Built-in prompt catalog via `BuiltInMetric`, `listPrompts()`, and `getPrompt()` -- Custom evaluator support by passing your own `PromptDefinition` -- Binary, continuous, and Likert scoring scales -- Threshold overrides per evaluation call -- Retry handling for transient provider failures -- Composable API that can be embedded in external eval and observability workflows +The repository also includes [`dt-eval-lib`](../dt-eval-lib/README.md) — a standalone TypeScript package for running the same judge-based metrics directly in code, tests, and CI pipelines without the CLI. It supports all six providers and the full built-in evaluator catalog. ## PII Handling @@ -739,16 +635,25 @@ The original values are not sent to the judge by the CLI path. ## Data Expectations -The CLI currently expects chat-style GenAI spans that include fields such as: - -- `gen_ai.provider.name` -- `gen_ai.request.model` -- `gen_ai.prompt..content` -- `gen_ai.prompt..role` -- `gen_ai.completion.0.content` -- `service.name` or `dt.entity.service` for service scoping - -This makes it a good fit for apps instrumented with OpenTelemetry GenAI conventions or OpenLLMetry-style span attributes. +The CLI reads OTel GenAI semconv fields by default: + +| Canonical field | Default attributes read | +|----------------|------------------------| +| `input` | `gen_ai.input.messages` | +| `output` | `gen_ai.output.message`, `gen_ai.output.messages` | +| `model` | `gen_ai.request.model` | +| `systemInstruction` | `gen_ai.system_instruction` | + +This makes the CLI a good fit for apps instrumented with OpenTelemetry +GenAI conventions. For other tracing schemas — including OpenInference — +use `scope.spanFields` to map their attributes to the canonical fields +above. OpenInference commonly uses attributes such as +`llm.input_messages`, `llm.output_messages`, `input.value`, +`output.value`, and `llm.model_name`. A small set of legacy fallback attributes +(`gen_ai.prompt..content/role`, `gen_ai.completion.0.content`) is also +probed for compatibility with older emitters. See +[`dynatrace-ai-agent-instrumentation-examples`](https://github.com/dynatrace-oss/dynatrace-ai-agent-instrumentation-examples) +for example instrumentations. ## Local Development @@ -756,10 +661,7 @@ This makes it a good fit for apps instrumented with OpenTelemetry GenAI conventi # from the repo root — install all workspace dependencies npm install -# build the library npm run build --workspace=dt-eval-lib - -# build the CLI npm run build --workspace=dt-eval-cli ``` diff --git a/dt-eval-cli/src/config/defaults.ts b/dt-eval-cli/src/config/defaults.ts index 03a6cc6..86a722d 100644 --- a/dt-eval-cli/src/config/defaults.ts +++ b/dt-eval-cli/src/config/defaults.ts @@ -39,7 +39,7 @@ export const DEFAULT_CONFIG: Omit & { since: '1h', sampling: { strategy: 'random', - percent: 100, + percent: 5, }, }, metrics: { diff --git a/dt-eval-cli/tests/config.test.ts b/dt-eval-cli/tests/config.test.ts index a4ab1a0..b03cd51 100644 --- a/dt-eval-cli/tests/config.test.ts +++ b/dt-eval-cli/tests/config.test.ts @@ -127,7 +127,7 @@ describe('config', () => { expect(config.scope.since).toBe('1h'); expect(config.scope.sampling.strategy).toBe('random'); - expect(config.scope.sampling.percent).toBe(100); + expect(config.scope.sampling.percent).toBe(5); expect(config.metrics.enabled).toEqual(['toxicity', 'relevance', 'faithfulness']); expect(config.judge.timeout).toBe(30000); expect(config.judge.maxRetries).toBe(2); diff --git a/dt-eval-lib/README.md b/dt-eval-lib/README.md index f081ca5..1f9f4b9 100644 --- a/dt-eval-lib/README.md +++ b/dt-eval-lib/README.md @@ -13,6 +13,15 @@ Minimal TypeScript library for running LLM-as-a-judge evaluations. npm install @dynatrace-oss/dt-eval-lib ``` +Then install the peer dependency for your provider: + +| Provider | Peer dependency | +|----------|-----------------| +| `openai`, `azure-openai` | `npm install openai` | +| `anthropic` | `npm install @anthropic-ai/sdk` | +| `vertex`, `gemini` | `npm install @google/genai` | +| `bedrock` | `npm install @aws-sdk/client-bedrock-runtime` | + ## Build ```bash @@ -50,103 +59,90 @@ console.log(result.explanation); // { summary: "...", reasoning: "..." } ## Available Metrics -| Metric | Enum | Type | Required Fields | -|--------|------|------|-----------------| -| `toxicity` | `BuiltInMetric.Toxicity` | binary | input, output | -| `faithfulness` | `BuiltInMetric.Faithfulness` | continuous | input, output, context | -| `hallucination` | `BuiltInMetric.Hallucination` | binary | input, output, context | -| `pii-leakage` | `BuiltInMetric.PiiLeakage` | binary | input, output | -| `relevance` | `BuiltInMetric.Relevance` | continuous | input, output | -| `factual-accuracy` | `BuiltInMetric.FactualAccuracy` | continuous | input, output, expectedOutput | -| `user-frustration` | `BuiltInMetric.UserFrustration` | binary | input | -| `context-relevance` | `BuiltInMetric.ContextRelevance` | continuous | input, context | -| `answer-completeness` | `BuiltInMetric.AnswerCompleteness` | continuous | input, output | -| `prompt-injection` | `BuiltInMetric.PromptInjection` | binary | input, output | -| `bias` | `BuiltInMetric.Bias` | binary | input, output | -| `summarization-quality` | `BuiltInMetric.SummarizationQuality` | continuous | input, output | -| `conciseness` | `BuiltInMetric.Conciseness` | continuous | input, output | - -> **Note:** The "Required Fields" column lists the `EvalInput` property names you pass to `evaluate()`. +All built-in metrics return a **continuous** score in `[0.0, 1.0]`. The score is labeled `"pass"` when `value >= threshold` and `"fail"` otherwise. Every built-in metric defaults to `threshold: 0.5` — override it per-call via `EvalConfig.scoring.thresholdOverride`. Source of truth: [`src/prompts/catalog-data.ts`](src/prompts/catalog-data.ts). + +| Metric | Enum | Score range | Fields used from `EvalInput` | +|--------|------|-------------|-------------------------------| +| `answer-completeness` | `BuiltInMetric.AnswerCompleteness` | 0.0 – 1.0 | `input`, `output` | +| `bias` | `BuiltInMetric.Bias` | 0.0 – 1.0 | `input`, `output` | +| `conciseness` | `BuiltInMetric.Conciseness` | 0.0 – 1.0 | `input`, `output` | +| `context-relevance` | `BuiltInMetric.ContextRelevance` | 0.0 – 1.0 | `input` | +| `factual-accuracy` | `BuiltInMetric.FactualAccuracy` | 0.0 – 1.0 | `input`, `output` | +| `faithfulness` | `BuiltInMetric.Faithfulness` | 0.0 – 1.0 | `input`, `output` | +| `fluency` | `BuiltInMetric.Fluency` | 0.0 – 1.0 | `input`, `output` | +| `hallucination` | `BuiltInMetric.Hallucination` | 0.0 – 1.0 | `input`, `output` | +| `pii-leakage` | `BuiltInMetric.PiiLeakage` | 0.0 – 1.0 | `input`, `output` | +| `prompt-injection` | `BuiltInMetric.PromptInjection` | 0.0 – 1.0 | `input` | +| `relevance` | `BuiltInMetric.Relevance` | 0.0 – 1.0 | `input`, `output` | +| `summarization-quality` | `BuiltInMetric.SummarizationQuality` | 0.0 – 1.0 | `input`, `output` | +| `toxicity` | `BuiltInMetric.Toxicity` | 0.0 – 1.0 | `output` | +| `user-frustration` | `BuiltInMetric.UserFrustration` | 0.0 – 1.0 | `input` | + +> **Note:** The "Fields used" column lists which fields the metric prompt actually reads. `EvalInput.output` is required at the TypeScript type level for all calls — pass `output: ""` for metrics that only use `input`. ## Providers -Supports **OpenAI**, **Anthropic**, **Vertex AI**, and **Gemini Developer API**. Configure via API key in code or environment variables. +Supports **OpenAI**, **Anthropic**, **Azure OpenAI**, **Google Gemini Enterprise Platform (Vertex AI)**, and **Amazon Bedrock**. Configure via explicit options in code or environment variables. ### Environment Variables ```bash # OpenAI -export OPENAI_API_KEY="sk-..." -export OPENAI_BASE_URL="https://your-proxy.example.com/v1" # optional +OPENAI_API_KEY=sk-... +OPENAI_BASE_URL=https://your-proxy.example.com/v1 # optional # Anthropic -export ANTHROPIC_API_KEY="sk-ant-..." -export ANTHROPIC_BASE_URL="https://your-proxy.example.com" # optional +ANTHROPIC_API_KEY=sk-ant-... +ANTHROPIC_BASE_URL=https://your-proxy.example.com # optional -# Google AI (Vertex AI & Gemini) — API key -export GOOGLE_API_KEY="AIza..." -``` +# Azure OpenAI (all three required) +AZURE_OPENAI_API_KEY=... +AZURE_OPENAI_ENDPOINT=https://.openai.azure.com +AZURE_OPENAI_API_VERSION=2024-02-01 -Or use a `.env` file (not committed to git): +# Google Gemini Enterprise Platform (Vertex AI) — ADC path (no API key needed) +GOOGLE_CLOUD_PROJECT=my-gcp-project +GOOGLE_CLOUD_LOCATION=us-central1 # optional, defaults to "global" -```bash -OPENAI_API_KEY=sk-... -ANTHROPIC_API_KEY=sk-ant-... -OPENAI_BASE_URL=https://your-proxy.example.com/v1 -ANTHROPIC_BASE_URL=https://your-proxy.example.com +# Google Gemini Enterprise Platform (Vertex AI) — API key path GOOGLE_API_KEY=AIza... + +# Amazon Bedrock — static credentials (optional if using IAM roles / SSO) +AWS_ACCESS_KEY_ID=... +AWS_SECRET_ACCESS_KEY=... +AWS_DEFAULT_REGION=us-east-1 # optional, defaults to "us-east-1" ``` -### Vertex AI Setup +Config is resolved in this order for each option: -1. Get an API key from [Google Cloud Console](https://console.cloud.google.com/apis/credentials) (Vertex AI Express Mode) -2. Set `GOOGLE_API_KEY` env var (or pass `apiKey` in provider config) +1. Explicit value in `provider` options +2. Environment variable -```ts -await evaluate(BuiltInMetric.Toxicity, input, { - provider: { - provider: "vertex", - apiKey: "AQ...", - }, -}); -``` +### Google Gemini Enterprise Platform (Vertex AI) Setup + +Use Application Default Credentials (ADC) with provider ID `vertex` when running on GKE or Cloud Run with Workload Identity, or pass `apiKey` with provider ID `gemini` for key-based auth. -### Gemini Developer API Setup +### Azure OpenAI Setup -1. Get an API key from [Google AI Studio](https://aistudio.google.com/apikey) -2. Set `GOOGLE_API_KEY` env var (or pass `apiKey` in provider config) +All four fields are required — `model` is your Azure deployment name, which is user-defined and has no default. ```ts await evaluate(BuiltInMetric.Toxicity, input, { provider: { - provider: "gemini", - apiKey: "AIza...", + provider: "azure-openai", + apiKey: "...", // or AZURE_OPENAI_API_KEY + baseUrl: "https://.openai.azure.com", // or AZURE_OPENAI_ENDPOINT + apiVersion: "2024-02-01", // or AZURE_OPENAI_API_VERSION + model: "my-gpt4-deployment", // your deployment name }, }); ``` -> **Note:** Both `vertex` and `gemini` use the `@google/genai` SDK and require Node.js ≥ 20. +### Amazon Bedrock Setup -When calling `evaluate()`, the library resolves config in this order: +Uses the AWS SDK credential chain — IAM roles, SSO, and `AWS_PROFILE` are all resolved automatically. Static credentials are only needed if you can't use role-based auth. -1. Explicit value in `provider` options (e.g., `provider.apiKey`, `provider.baseUrl`) -2. Environment variable (`OPENAI_API_KEY`, `OPENAI_BASE_URL`, etc.) - -```ts -// Option 1: explicit config -await evaluate(BuiltInMetric.Toxicity, input, { - provider: { - provider: "openai", - apiKey: "sk-...", - baseUrl: "https://your-proxy.example.com/v1", - }, -}); - -// Option 2: env vars (no apiKey/baseUrl needed) -await evaluate(BuiltInMetric.Toxicity, input, { - provider: { provider: "openai" }, -}); -``` +> **Note:** Google provider IDs `vertex` and `gemini` both use the `@google/genai` SDK; `azure-openai` and `openai` both use the `openai` SDK. ## Metric Identification @@ -164,35 +160,35 @@ Use `listPrompts()` and `getPrompt()` to discover available metrics: ```ts import { listPrompts, getPrompt, BuiltInMetric } from "@dynatrace-oss/dt-eval-lib"; -const all = listPrompts(); // all 13 built-in metrics +const all = listPrompts(); // all 14 built-in metrics const tox = getPrompt(BuiltInMetric.Toxicity); // single metric by ID ``` ## Configuration ```ts -import type { EvalConfig } from "@dynatrace-oss/dt-eval-lib"; +import type { EvalConfig, EvalInput } from "@dynatrace-oss/dt-eval-lib"; + +// What you pass to evaluate() +const input: EvalInput = { + input: "What is the capital of France?", // required — the user query + output: "The capital of France is Paris.", // required — the LLM response to evaluate + context: "...", // optional — retrieved documents (RAG) + expectedOutput: "...", // optional — reference answer +}; +// How to run the evaluation const config: EvalConfig = { provider: { - provider: "openai", // "openai" | "anthropic" | "vertex" | "gemini" - apiKey: "sk-...", // optional if env var is set - baseUrl: "https://...", // optional (openai/anthropic only) - model: "gpt-4.1", // optional — defaults to gpt-4.1 / claude-sonnet-4-6 / gemini-2.5-pro (vertex) / gemini-2.5-flash (gemini) - timeout: 30000, // optional — request timeout in ms (default 30000) - maxRetries: 2, // optional — retries on transient errors (default 2) + provider: "openai", // "openai" | "anthropic" | "azure-openai" | "vertex" | "gemini" | "bedrock" + apiKey: "sk-...", // optional if env var is set + baseUrl: "https://", // optional — openai and anthropic only + model: "...", // optional — see src/engine/providers/index.ts for per-provider defaults + timeout: 30000, // optional — request timeout in ms (default: 30000) + maxRetries: 2, // optional — retries on transient errors (default: 2) }, scoring: { - thresholdOverride: 0.8, // optional — override the metric's default threshold + thresholdOverride: 0.8, // optional — override the metric's pass threshold (default: 0.5) }, }; ``` - -## Threshold Override - -```ts -const result = await evaluate(BuiltInMetric.Relevance, input, { - provider: { provider: "openai", apiKey: "sk-..." }, - scoring: { thresholdOverride: 0.8 }, // stricter than default 0.5 -}); -``` diff --git a/dt-eval-lib/src/engine/types.ts b/dt-eval-lib/src/engine/types.ts index 9114d7f..84e1faa 100644 --- a/dt-eval-lib/src/engine/types.ts +++ b/dt-eval-lib/src/engine/types.ts @@ -21,7 +21,7 @@ export interface ProviderOptions { project?: string; /** GCP region for Vertex AI (e.g. "global" or "us-central1"). Falls back to GOOGLE_CLOUD_LOCATION env var. Default: global. */ location?: string; - /** Model override — defaults to gpt-4.1 / claude-sonnet-4-6 / gemini-2.5-pro (vertex) / gemini-2.5-flash (gemini) */ + /** Model override — see DEFAULT_MODELS in src/engine/providers/index.ts for per-provider defaults */ model?: string; /** Request timeout in ms — default 30000 */ timeout?: number;