Merge remote-tracking branch 'origin/main' into fix/legacy-genai-attributes

Author: mesutoezdil

DEVELOPMENT.md

@@ -50,7 +50,7 @@ Once running, submit a run with:
 ```bash
 curl -X POST http://localhost:8001/api/runs \
     -H 'content-type: application/json' \
-    -d '{"spec": {"approach": "trace_replay", "target": {"kind": "inline", "inline": {...}}, "evalConfig": {"metrics": ["tool_trajectory_avg_score"]}}}'
+    -d '{"spec": {"approach": "trace_replay", "target": {"kind": "inline", "inline": {...}}, "evalConfig": {"evaluators": [{"name": "tool_trajectory_avg_score", "type": "builtin"}]}}}'
 ```
 
 Then poll `GET /api/runs/{runId}` and `GET /api/runs/{runId}/results`. Without `storage.backend=postgres`, the `/api/runs` endpoints return 503 with a hint pointing at the env var.

README.md

@@ -250,7 +250,7 @@ See the [Custom Evaluators guide](docs/custom-evaluators.md) for the full protoc
 agentevals serve            # bundled UI on http://localhost:8001
 ```
 
-Upload traces and eval sets, select metrics, and view results with interactive span trees. Live-streamed traces appear in the "Local Dev" tab, grouped by session ID. For running from source, see [DEVELOPMENT.md](DEVELOPMENT.md).
+Upload traces and eval sets, select evaluators, and view results with interactive span trees. Live-streamed traces appear in the "Local Dev" tab, grouped by session ID. For running from source, see [DEVELOPMENT.md](DEVELOPMENT.md).
 
 Interactive API docs are available at `/docs` (Swagger) and `/redoc` while the server is running. The OTLP receiver on port 4318 serves its own docs at `http://localhost:4318/docs`.
 
@@ -397,6 +397,18 @@ Yes. A custom evaluator is any program that reads JSON from stdin and writes a s
 
 Yes. The OTLP receiver on port 4318 accepts standard `http/protobuf` and `http/json` trace exports, so it slots into any OpenTelemetry pipeline as just another exporter destination. If your pipeline uses gRPC (port 4317), place an [OTel Collector](https://opentelemetry.io/docs/collector/) in front to bridge gRPC to HTTP. The [Kubernetes example](examples/kubernetes/README.md) shows this pattern.
 
+**Can I use agentevals to evaluate Claude Code, Codex, or OpenCode?**
+
+Not today. agentevals scores agent behavior from OpenTelemetry GenAI traces (spans for model calls, tool calls, agent invocations following the [OTel GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)). The major coding agents do not currently emit telemetry in that shape:
+
+- **Claude Code** ships OTel telemetry as logs, not GenAI spans. A prior proof of concept on a feature branch made it work by stitching hook events into synthetic traces. Reviving that path is on the backlog, not a near-term commitment.
+- **Codex** exposes OTel, but in a different shape we have not yet validated against the GenAI semconv.
+- **OpenCode** did not have OTel support merged the last time we checked.
+
+Retrofitting agentevals to ingest each harness's bespoke telemetry is multiple thousands of lines of glue code per agent, for a use case where the dominant signal is "did the final output feel right," not "did the agent call the right tool with the right arguments in the right order." That kind of vibes evaluation is interesting work for harness and coding-agent vendors themselves, but it is not what agentevals is optimized for.
+
+agentevals is built for the opposite end of the spectrum: smaller, purpose-built, properly instrumented agents (kagent, agentregistry, custom Strands / ADK / LangChain / OpenAI Agents SDK flows) running in cloud native environments, where success is measurable through tool trajectories, response matching, and deterministic pass/fail gates. If that is your use case, we are a good fit. If you are evaluating long-running coding sessions end to end, you probably want a tool built specifically for that shape.
+
 **How does this compare to ADK's evaluations?**
 
 Unlike ADK's eval method, which couples agent execution with evaluation, agentevals only handles scoring: it takes pre-recorded traces and compares them against expected behavior using metrics like tool trajectory matching, response quality, and LLM-based judgments.
@@ -420,3 +432,7 @@ Langfuse is a full observability platform (requires Postgres, ClickHouse, Redis,
 **How does this compare to Opik?**
 
 Opik's primary evaluation path re-runs your application code against a dataset, incurring additional LLM costs per eval run. It also supports online evaluation rules that auto-score production traces. While Opik supports OpenTelemetry ingestion alongside its own SDK, its evaluation workflow still centers on re-execution against datasets. agentevals evaluates pre-recorded OTel traces from any framework without re-execution, and runs entirely locally with no cloud dependency.
+
+## Acknowledgements
+
+agentevals is built on top of [Google's Agent Development Kit](https://github.com/google/adk-python). ADK provides the evaluator protocol and the canonical eval data model (`Invocation`, `EvalSet`, `Evaluator`, prebuilt metrics) that this project extends. `google-adk` is licensed under [Apache 2.0](https://github.com/google/adk-python/blob/main/LICENSE), the same license as agentevals. Thanks to the ADK team and contributors.

charts/agentevals/templates/deployment.yaml

@@ -29,8 +29,9 @@ spec:
       securityContext:
         {{- toYaml .Values.podSecurityContext | nindent 8 }}
       serviceAccountName: {{ include "agentevals.serviceAccountName" . }}
-      {{- if .Values.ephemeralVolume.enabled }}
+      {{- if or .Values.ephemeralVolume.enabled .Values.extraVolumes }}
       volumes:
+        {{- if .Values.ephemeralVolume.enabled }}
         - name: agentevals-tmp
           {{- if or .Values.ephemeralVolume.sizeLimit (eq .Values.ephemeralVolume.medium "Memory") }}
           emptyDir:
@@ -43,6 +44,10 @@ spec:
           {{- else }}
           emptyDir: {}
           {{- end }}
+        {{- end }}
+        {{- with .Values.extraVolumes }}
+        {{- toYaml . | nindent 8 }}
+        {{- end }}
       {{- end }}
       containers:
         - name: agentevals
@@ -70,6 +75,10 @@ spec:
               value: "postgres"
             - name: AGENTEVALS_DATABASE_SCHEMA
               value: {{ .Values.database.postgres.schema | quote }}
+            - name: AGENTEVALS_AUTO_MIGRATE
+              value: {{ .Values.database.postgres.autoMigrate | quote }}
+            - name: AGENTEVALS_DB_CONNECT_TIMEOUT_S
+              value: {{ .Values.database.postgres.connectTimeoutSeconds | quote }}
             {{- if .Values.database.postgres.urlFile }}
             - name: AGENTEVALS_DATABASE_URL_FILE
               value: {{ .Values.database.postgres.urlFile | quote }}
@@ -135,10 +144,15 @@ spec:
               port: http
             initialDelaySeconds: 15
             periodSeconds: 20
-          {{- if .Values.ephemeralVolume.enabled }}
+          {{- if or .Values.ephemeralVolume.enabled .Values.extraVolumeMounts }}
           volumeMounts:
+            {{- if .Values.ephemeralVolume.enabled }}
             - name: agentevals-tmp
               mountPath: /tmp
+            {{- end }}
+            {{- with .Values.extraVolumeMounts }}
+            {{- toYaml . | nindent 12 }}
+            {{- end }}
           {{- end }}
       {{- with .Values.nodeSelector }}
       nodeSelector:

charts/agentevals/values.yaml

@@ -159,6 +159,16 @@ env: []
 # -- Extra envFrom sources (ConfigMapRef, SecretRef)
 envFrom: []
 
+# -- Extra volumes appended to the pod spec. Use this to mount additional
+# config files or secrets (e.g. result-sink credentials) into the pod.
+extraVolumes: []
+
+# -- Extra volumeMounts appended to the main container. Pair with
+# extraVolumes by name. securityContext.readOnlyRootFilesystem is true by
+# default; that only makes the root filesystem read-only, mounted paths
+# themselves are unaffected, so a writable extraVolumes entry works fine.
+extraVolumeMounts: []
+
 # ==============================================================================
 # STORAGE  (preview feature)
 #
@@ -195,6 +205,18 @@ database:
     urlFile: ""
     # -- Postgres schema to use for agentevals tables.
     schema: agentevals
+    # -- Apply pending database migrations during server startup before the
+    # HTTP listener opens. The Postgres advisory lock serialises concurrent
+    # replica starts so this is safe with replicaCount > 1. When set to
+    # false the server refuses to start if the schema is behind or dirty;
+    # run "agentevals migrate up" manually in that case.
+    autoMigrate: true
+    # -- Seconds the startup will spend retrying the initial Postgres
+    # connection before the pod aborts. Default 600s matches the chart's
+    # hard-coded startupProbe budget (failureThreshold 60 x periodSeconds
+    # 10). Going above 600s requires overriding the probe in your own
+    # downstream template.
+    connectTimeoutSeconds: 600
     # -- Bundled Postgres instance for development and evaluation only.
     # Not suitable for production. Deployed when enabled is true and url /
     # urlFile are not set.

docs/custom-evaluators.md

@@ -317,6 +317,26 @@ The `grader.evaluation_metric` field selects the similarity algorithm:
 | `rouge_1` through `rouge_5` | Unigram through 5-gram overlap (F-measure) |
 | `rouge_l` | Longest common subsequence overlap (F-measure) |
 
+### Label Model Grader
+
+Scores responses without a golden set. The model reads each response and assigns a label from a fixed list. Passing labels are defined in the config.
+
+```yaml
+evaluators:
+  - name: quality_check
+    type: openai_eval
+    grader:
+      type: label_model
+      model: gpt-4o-mini
+      input:
+        - role: user
+          content: "Rate this response: {{ item.actual_response }}"
+      labels: [good, bad]
+      passing_labels: [good]
+```
+
+The `threshold` field is not used for `label_model`. A response passes if its assigned label is in `passing_labels`.
+
 ### How it works
 
 Under the hood, agentevals creates an ephemeral eval on OpenAI, submits the actual and expected responses as JSONL items, polls for results, and cleans up. The agent's response and the golden reference are both placed in the `item` namespace (with `include_sample_schema: false`), so OpenAI only grades the provided text without generating any model outputs.

docs/eval-set-format.md

@@ -1,6 +1,6 @@
 # Eval Set Format
 
-An eval set is a JSON file containing golden reference data that metrics compare agent traces against. It follows the [Google ADK `EvalSet`](https://github.com/google/adk-python/blob/main/src/google/adk/evaluation/eval_set.py) schema, which means eval sets are portable between agentevals and ADK tooling.
+An eval set is a JSON file containing golden reference data that evaluators compare agent traces against. It follows the [Google ADK `EvalSet`](https://github.com/google/adk-python/blob/main/src/google/adk/evaluation/eval_set.py) schema, which means eval sets are portable between agentevals and ADK tooling.
 
 Most users will not need to author eval sets by hand. The web UI can generate them from live sessions (mark a session as golden, and the server builds the eval set automatically). This document is for users who want to create or edit eval sets directly, whether for CLI usage, CI pipelines, or version-controlled test suites.
 
@@ -203,9 +203,9 @@ The `parts` array can contain text, function calls, or function responses. Most
 
 Each `FunctionCall` has `name`, `args`, and `id`. Each `FunctionResponse` has `name`, `response`, and `id`. Match `id` values between calls and responses to pair them.
 
-## Which Metrics Use Eval Sets
+## Which Evaluators Use Eval Sets
 
-Not all metrics require an eval set. Use `agentevals list-metrics` to see which do:
+Not all evaluators require an eval set. Use `agentevals evaluator list --source builtin` to see which built-in evaluators do:
 
 | Metric | Needs Eval Set | What It Reads |
 |---|---|---|

examples/custom_evaluators/eval_config.yaml

@@ -32,3 +32,4 @@ evaluators:
     ref: evaluators/random_evaluator/random_evaluator.py
     threshold: 0.110
     executor: local
+

examples/custom_evaluators/eval_config_openai_eval.yaml

@@ -0,0 +1,18 @@
+# Eval config using OpenAI Evals API graders.
+# Requires OPENAI_API_KEY to be set.
+#
+# Run with:
+#   agentevals run samples/helm.json \
+#     --config examples/custom_evaluators/eval_config_openai_eval.yaml
+
+evaluators:
+  - name: quality_check
+    type: openai_eval
+    grader:
+      type: label_model
+      model: gpt-4o-mini
+      input:
+        - role: user
+          content: "Rate this response: {{ item.actual_response }}"
+      labels: [good, bad]
+      passing_labels: [good]

examples/dice_agent/README.md

@@ -149,7 +149,7 @@ Update `main.py` to test the new functionality.
 **After agent completes:**
 - Status changes to "EVALUATED"
 - Evaluation results appear as colored badges
-- Each metric shows: name and score (e.g., "tool_trajectory_avg_score: 1.00")
+- Each evaluator result shows: name and score (e.g., "tool_trajectory_avg_score: 1.00")
 
 **Multiple runs:**
 - Each run creates a new session with model name in ID

examples/kubernetes/README.md

@@ -221,7 +221,7 @@ This captures the GPT-5 session's tool trajectory and final responses as the gol
 2. Select both sessions (the `gpt-4.1-mini` session and the `gpt-5` session)
 3. Click **Evaluate**
 4. Select the `helm-agent-comparison` eval set
-5. Choose the metrics:
+5. Choose the evaluators:
    - **tool_trajectory_avg_score**: Did the agent call the correct tools in the correct order?
    - **response_match_score**: Did the agent produce responses consistent with the golden reference?
 6. Run the evaluation
@@ -241,7 +241,7 @@ Compare the two sessions in the results table:
 
 <img width="1914" height="1154" alt="image" src="https://github.com/user-attachments/assets/5939a8d4-3775-4cf1-9cf2-d3b6b4afd582" />
 
-You can also click an individual conversation and see a breakdown of each evaluators.
+You can also click an individual conversation and see a breakdown of each evaluator.
 
 <img width="1916" height="1348" alt="image" src="https://github.com/user-attachments/assets/984b3d29-8018-4fcb-9036-bb7c6e97d9ff" />