Skip to content

Commit ffc6558

Browse files
Merge pull request #420 from salesforcecli/js/edit-messages
W-22457867: quick edit of --help for "agent test run-eval"
2 parents 33e25a3 + a6f0e80 commit ffc6558

1 file changed

Lines changed: 14 additions & 11 deletions

File tree

messages/agent.test.run-eval.md

Lines changed: 14 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,27 @@
11
# summary
22

3-
Run evaluation tests against an Agentforce agent.
3+
Run rich evaluation tests against an Agentforce agent.
44

55
# description
66

7-
Execute rich evaluation tests against an Agentforce agent using the Einstein Evaluation API. Supports both YAML test specs (same format as `sf agent generate test-spec`) and JSON payloads.
7+
Specify the tests you want to run with one of these inputs to the --spec flag:
88

9-
When you provide a YAML test spec, the command automatically translates test cases into Evaluation API calls and infers the agent name from the spec's `subjectName` field. This means you can use the same test spec with both `sf agent test run` and `sf agent test run-eval`. YAML test specs also support contextVariables, which allow you to inject contextual data (such as CaseId or RoutableId) into agent sessions for testing with different contexts.
9+
- YAML test spec generated by the `agent generate test-spec` CLI command
10+
- JSON payload
1011

11-
When you provide a JSON payload, it's sent directly to the API with optional normalization. The normalizer auto-corrects common field name mistakes, converts shorthand references to JSONPath, and injects defaults. Use `--no-normalize` to disable this auto-normalization. JSON payloads can also include context_variables on agent.create_session steps for the same contextual testing capabilities.
12+
When you provide a YAML test spec, this command automatically translates test cases into internal state-based evaluation framework calls and infers the agent name from the test spec's `subjectName` field. As a result, you can use the same test spec with both the `agent test run` and `agent test run-eval` commands. YAML test specs also support context variables, which allow you to inject contextual data (such as CaseId or RoutableId) into agent sessions for testing with different contexts.
1213

13-
Supports 8+ evaluator types, including topic routing assertions, action invocation checks, string/numeric assertions, semantic similarity scoring, and LLM-based quality ratings.
14+
When you provide a JSON payload, it's sent directly to the evaluation framework with optional normalization. The normalizer auto-corrects common field name mistakes, converts shorthand references to JSONPath, and injects defaults. Use `--no-normalize` to disable this auto-normalization. JSON payloads can also include context_variables on agent.create_session steps for the same contextual testing capabilities as when you use a YAML test spec.
15+
16+
This command supports more than 8 evaluator types, including subagent routing assertions, action invocation checks, string/numeric assertions, semantic similarity scoring, and LLM-based quality ratings.
1417

1518
# flags.spec.summary
1619

1720
Path to test spec file (YAML or JSON). Supports reading from stdin when piping content.
1821

1922
# flags.api-name.summary
2023

21-
Agent DeveloperName (also called API name) to resolve agent_id and agent_version_id. Auto-inferred from the YAML spec's subjectName.
24+
Agent API name (also called DeveloperName) used to resolve agent_id and agent_version_id. Auto-inferred from the YAML spec's subjectName.
2225

2326
# flags.result-format.summary
2427

@@ -36,23 +39,23 @@ Disable auto-normalization of field names and shorthand references.
3639

3740
- Run tests using a YAML test spec on the org with alias "my-org":
3841

39-
<%= config.bin %> <%= command.id %> --spec tests/my-agent-testSpec.yaml --target-org my-org
42+
<%= config.bin %> <%= command.id %> --spec specs/my-agent-testSpec.yaml --target-org my-org
4043

4144
- Run tests using a YAML spec with explicit agent name override; use your default org:
4245

43-
<%= config.bin %> <%= command.id %> --spec tests/my-agent-testSpec.yaml --api-name My_Agent --target-org my-org
46+
<%= config.bin %> <%= command.id %> --spec specs/my-agent-testSpec.yaml --api-name My_Agent
4447

4548
- Run tests using a JSON payload:
4649

47-
<%= config.bin %> <%= command.id %> --spec tests/eval-payload.json --target-org my-org
50+
<%= config.bin %> <%= command.id %> --spec specs/eval-payload.json --target-org my-org
4851

4952
- Run tests and output results in JUnit format; useful for continuous integration and deployment (CI/CD):
5053

51-
<%= config.bin %> <%= command.id %> --spec tests/my-agent-testSpec.yaml --target-org my-org --result-format junit
54+
<%= config.bin %> <%= command.id %> --spec specs/my-agent-testSpec.yaml --target-org my-org --result-format junit
5255

5356
- Run tests with contextVariables to inject contextual data into agent sessions (add contextVariables to test cases in your YAML spec):
5457

55-
<%= config.bin %> <%= command.id %> --spec tests/agent-with-context.yaml --target-org my-org
58+
<%= config.bin %> <%= command.id %> --spec specs/agent-with-context.yaml --target-org my-org
5659

5760
- Pipe JSON payload from stdin (--spec flag is automatically populated from stdin):
5861

0 commit comments

Comments
 (0)