|
| 1 | +# @kbn/evals-suite-agent-builder-dashboards |
| 2 | + |
| 3 | +Evaluation test suite for Agent Builder Dashboards behavior, built on top of [`@kbn/evals`](../../kbn-evals/README.md). |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +This package contains in-code evaluation datasets for Agent Builder Dashboards behavior. The initial coverage focuses on skill selection and intent routing: |
| 8 | + |
| 9 | +- Dashboard requests should load dashboard management. |
| 10 | +- Standalone visualization requests should load visualization creation without creating a dashboard. |
| 11 | +- ES|QL query-writing requests should not use dashboard management. |
| 12 | + |
| 13 | +For general information about writing evaluation tests, configuration, reporting, and comparison, see the main [`@kbn/evals` documentation](../../kbn-evals/README.md). |
| 14 | + |
| 15 | +## Prerequisites |
| 16 | + |
| 17 | +### Configure EIS Connectors |
| 18 | + |
| 19 | +For local EIS-backed model runs, run the eval setup wizard: |
| 20 | + |
| 21 | +```bash |
| 22 | +node scripts/evals init |
| 23 | +``` |
| 24 | + |
| 25 | +When `node scripts/evals init` finishes, copy the printed connector export into the same shell where you will run evals: |
| 26 | + |
| 27 | +```bash |
| 28 | +export KIBANA_TESTING_AI_CONNECTORS="..." |
| 29 | +``` |
| 30 | + |
| 31 | +This makes EIS connector IDs available as Playwright projects, for example `eis-anthropic-claude-4-5-sonnet`. |
| 32 | + |
| 33 | +### Optional: Configure Phoenix and Tracing |
| 34 | + |
| 35 | +`node scripts/evals start` starts EDOT and Scout for you. If you want to export traces to Phoenix or a shared tracing cluster, configure the eval profiles with: |
| 36 | + |
| 37 | +```bash |
| 38 | +node scripts/evals init config |
| 39 | +``` |
| 40 | + |
| 41 | +See [`@kbn/evals` documentation](../../kbn-evals/README.md) for `TRACING_EXPORTERS`, `TRACING_ES_URL`, and Phoenix executor details. |
| 42 | + |
| 43 | +## Running Evaluations |
| 44 | + |
| 45 | +### Managed Stack |
| 46 | + |
| 47 | +Use `node scripts/evals start` when you want the CLI to start or reuse EDOT and Scout, enable EIS Cloud Connected Mode, and then run the suite: |
| 48 | + |
| 49 | +```bash |
| 50 | +node scripts/evals start \ |
| 51 | + --suite agent-builder-dashboards \ |
| 52 | + --project eis-anthropic-claude-4-5-sonnet \ |
| 53 | + --evaluation-connector-id eis-anthropic-claude-4-5-sonnet |
| 54 | +``` |
| 55 | + |
| 56 | +The Scout Kibana instance is usually available at <http://localhost:5620>, and Elasticsearch at <http://localhost:9220>. |
| 57 | + |
| 58 | +### Run a Single Eval |
| 59 | + |
| 60 | +Filter by Playwright test title with `--grep`: |
| 61 | + |
| 62 | +```bash |
| 63 | +node scripts/evals start \ |
| 64 | + --suite agent-builder-dashboards \ |
| 65 | + --grep "dashboards in chat smokescreen" \ |
| 66 | + --project eis-anthropic-claude-4-5-sonnet \ |
| 67 | + --evaluation-connector-id eis-anthropic-claude-4-5-sonnet |
| 68 | +``` |
| 69 | + |
| 70 | +Available skill-selection test titles: |
| 71 | + |
| 72 | +- `dashboards in chat smokescreen` |
| 73 | +- `visualization request does not create dashboard` |
| 74 | +- `esql query help does not create dashboard` |
| 75 | + |
| 76 | +After the eval stack is already running, use `run` for faster iteration: |
| 77 | + |
| 78 | +```bash |
| 79 | +node scripts/evals run \ |
| 80 | + --suite agent-builder-dashboards \ |
| 81 | + --grep "visualization request does not create dashboard" \ |
| 82 | + --project eis-anthropic-claude-4-5-sonnet \ |
| 83 | + --evaluation-connector-id eis-anthropic-claude-4-5-sonnet |
| 84 | +``` |
| 85 | + |
| 86 | +### Repetitions |
| 87 | + |
| 88 | +By default, each dataset example runs once. To run each example multiple times, pass `--repetitions`: |
| 89 | + |
| 90 | +```bash |
| 91 | +node scripts/evals start \ |
| 92 | + --suite agent-builder-dashboards \ |
| 93 | + --grep "dashboards in chat smokescreen" \ |
| 94 | + --project eis-anthropic-claude-4-5-sonnet \ |
| 95 | + --evaluation-connector-id eis-anthropic-claude-4-5-sonnet \ |
| 96 | + --repetitions 3 |
| 97 | +``` |
| 98 | + |
| 99 | +Equivalent environment variable: |
| 100 | + |
| 101 | +```bash |
| 102 | +EVALUATION_REPETITIONS=3 node scripts/evals run \ |
| 103 | + --suite agent-builder-dashboards \ |
| 104 | + --grep "dashboards in chat smokescreen" \ |
| 105 | + --project eis-anthropic-claude-4-5-sonnet \ |
| 106 | + --evaluation-connector-id eis-anthropic-claude-4-5-sonnet |
| 107 | +``` |
| 108 | + |
| 109 | +### Direct Playwright |
| 110 | + |
| 111 | +For lower-level debugging, run Playwright directly: |
| 112 | + |
| 113 | +```bash |
| 114 | +EVALUATION_CONNECTOR_ID=eis-anthropic-claude-4-5-sonnet \ |
| 115 | +node scripts/playwright test \ |
| 116 | + --config x-pack/platform/packages/shared/agent-builder-dashboards/kbn-evals-suite-agent-builder-dashboards/playwright.config.ts \ |
| 117 | + evals/skill_selection/skill_selection.spec.ts \ |
| 118 | + --project eis-anthropic-claude-4-5-sonnet \ |
| 119 | + --grep "esql query help does not create dashboard" |
| 120 | +``` |
| 121 | + |
| 122 | +Use `--list` to check what Playwright can discover: |
| 123 | + |
| 124 | +```bash |
| 125 | +EVALUATION_CONNECTOR_ID=eis-anthropic-claude-4-5-sonnet \ |
| 126 | +node scripts/playwright test \ |
| 127 | + --config x-pack/platform/packages/shared/agent-builder-dashboards/kbn-evals-suite-agent-builder-dashboards/playwright.config.ts \ |
| 128 | + --project eis-anthropic-claude-4-5-sonnet \ |
| 129 | + --list |
| 130 | +``` |
| 131 | + |
| 132 | +## Sample Data |
| 133 | + |
| 134 | +The skill-selection spec loads Kibana logs sample data before running: |
| 135 | + |
| 136 | +```ts |
| 137 | +await fetch('/api/sample_data/logs', { |
| 138 | + method: 'POST', |
| 139 | + version: '2023-10-31', |
| 140 | +}); |
| 141 | +``` |
| 142 | + |
| 143 | +To verify the index exists in the Scout Elasticsearch cluster: |
| 144 | + |
| 145 | +```bash |
| 146 | +curl -u elastic:changeme "http://localhost:9220/_cat/indices/kibana_sample_data_logs?v" |
| 147 | +curl -u elastic:changeme "http://localhost:9220/kibana_sample_data_logs/_count?pretty" |
| 148 | +``` |
| 149 | + |
| 150 | +## Stopping the Stack |
| 151 | + |
| 152 | +When you are done: |
| 153 | + |
| 154 | +```bash |
| 155 | +node scripts/evals stop |
| 156 | +``` |
0 commit comments