|
| 1 | +# Evaluation Guide |
| 2 | + |
| 3 | +How to validate the migration skill after making changes to prompt files. |
| 4 | + |
| 5 | +--- |
| 6 | + |
| 7 | +## Quick Reference |
| 8 | + |
| 9 | +| What | Command | |
| 10 | +| ------------------------------------- | ------------------------------------------------------------------- | |
| 11 | +| Structural check (instant, no Claude) | `mise run eval:check` | |
| 12 | +| Full evaluation (after migration run) | `python tools/eval_check.py --migration-dir <DIR> --fixture <NAME>` | |
| 13 | +| Guided evaluation (Claude assists) | Tell Claude: `Read tools/EVAL_SKILL.md and follow the instructions` | |
| 14 | + |
| 15 | +--- |
| 16 | + |
| 17 | +## When to Evaluate |
| 18 | + |
| 19 | +Run the evaluation harness whenever you: |
| 20 | + |
| 21 | +- Edit any prompt file under `features/migration-to-aws/skills/gcp-to-aws/` |
| 22 | +- Modify schema definitions, classification rules, or mapping tables |
| 23 | +- Change the fast-path table, rubric, or pricing cache |
| 24 | +- Add or remove questions from the Clarify phase |
| 25 | + |
| 26 | +--- |
| 27 | + |
| 28 | +## Step-by-Step Workflow |
| 29 | + |
| 30 | +### 1. Make your prompt changes |
| 31 | + |
| 32 | +Edit the relevant files under `features/migration-to-aws/skills/gcp-to-aws/`. |
| 33 | + |
| 34 | +### 2. Run the structural check |
| 35 | + |
| 36 | +```bash |
| 37 | +mise run eval:check |
| 38 | +``` |
| 39 | + |
| 40 | +This runs instantly with no Claude API calls. It verifies that critical directives |
| 41 | +still exist in the prompt files: |
| 42 | + |
| 43 | +- FORBIDDEN blocks (scope boundaries for each phase) |
| 44 | +- BigQuery specialist gate directive |
| 45 | +- Security rules (no hardcoded credentials, no wildcard IAM) |
| 46 | +- Dry-run default for generated scripts |
| 47 | + |
| 48 | +If this fails, a directive was accidentally deleted. The output tells you |
| 49 | +exactly which file and line to check. |
| 50 | + |
| 51 | +### 3. Pick a fixture |
| 52 | + |
| 53 | +Choose the fixture that best covers your change: |
| 54 | + |
| 55 | +| Fixture | Use when you changed... | |
| 56 | +| -------------------------- | ------------------------------------------------------------------------- | |
| 57 | +| `minimal-cloud-run-sql` | General prompt changes, state machine, phase ordering, generate phase | |
| 58 | +| `bigquery-specialist-gate` | BigQuery handling, specialist gate, Clarify advisory, analytics exclusion | |
| 59 | +| `ai-workload-openai` | AI detection, model mapping, lifecycle rules, Category F questions | |
| 60 | +| `user-preferences` | Clarify question flow, preference schema, Design preference consumption | |
| 61 | +| `negative-services` | Classification rules, auth exclusion, forbidden service mappings | |
| 62 | + |
| 63 | +For broad changes, run `minimal-cloud-run-sql` first (it has the most invariants), |
| 64 | +then any fixture specific to your change area. |
| 65 | + |
| 66 | +### 4. Run the migration skill against the fixture |
| 67 | + |
| 68 | +```bash |
| 69 | +cd tests/fixtures/minimal-cloud-run-sql |
| 70 | +``` |
| 71 | + |
| 72 | +Open Claude Code and trigger the migration: |
| 73 | + |
| 74 | +``` |
| 75 | +migrate from GCP to AWS |
| 76 | +``` |
| 77 | + |
| 78 | +The skill runs all 6 phases using the fixture's Terraform files and pre-seeded |
| 79 | +preferences. Wait for it to complete (you'll see "Migration planning complete" |
| 80 | +or similar). |
| 81 | + |
| 82 | +### 5. Run the evaluation |
| 83 | + |
| 84 | +**Option A — Direct (recommended for CI and scripting):** |
| 85 | + |
| 86 | +```bash |
| 87 | +python tools/eval_check.py \ |
| 88 | + --migration-dir tests/fixtures/minimal-cloud-run-sql/.migration/<RUN_ID> \ |
| 89 | + --fixture minimal-cloud-run-sql |
| 90 | +``` |
| 91 | + |
| 92 | +Replace `<RUN_ID>` with the directory name (e.g., `0419-1200`). The checker |
| 93 | +outputs JSON with pass/fail for every invariant. |
| 94 | + |
| 95 | +**Option B — Claude-assisted (recommended for understanding failures):** |
| 96 | + |
| 97 | +In Claude Code, type: |
| 98 | + |
| 99 | +``` |
| 100 | +Read tools/EVAL_SKILL.md and follow the instructions to evaluate the migration results |
| 101 | +``` |
| 102 | + |
| 103 | +Claude runs the checker, explains any failures in plain language, computes |
| 104 | +hashes, and writes `.eval-results.json`. |
| 105 | + |
| 106 | +### 6. Review results |
| 107 | + |
| 108 | +The checker output looks like: |
| 109 | + |
| 110 | +```json |
| 111 | +{ |
| 112 | + "summary": { |
| 113 | + "hard_total": 26, |
| 114 | + "hard_passed": 26, |
| 115 | + "hard_failed": 0, |
| 116 | + "hard_skipped": 0, |
| 117 | + "status": "pass" |
| 118 | + } |
| 119 | +} |
| 120 | +``` |
| 121 | + |
| 122 | +- **All pass** — your change is safe. Proceed to commit. |
| 123 | +- **Failures** — read the `details` field for each failed invariant. It tells |
| 124 | + you what was found, what was expected, and which prompt file to check. |
| 125 | +- **Skips** — a custom handler isn't implemented yet. Not a blocker. |
| 126 | + |
| 127 | +### 7. Commit the results |
| 128 | + |
| 129 | +```bash |
| 130 | +git add .eval-results.json |
| 131 | +git commit -m "eval: results for minimal-cloud-run-sql" |
| 132 | +``` |
| 133 | + |
| 134 | +The `.eval-results.json` file is SHA-bound to your commit and prompt file |
| 135 | +hashes. CI can verify it wasn't forged or stale. |
| 136 | + |
| 137 | +--- |
| 138 | + |
| 139 | +## Test Fixtures |
| 140 | + |
| 141 | +### minimal-cloud-run-sql |
| 142 | + |
| 143 | +The default fixture. 2 PRIMARY resources (Cloud Run + Cloud SQL), 3 SECONDARY |
| 144 | +resources. Tests the full 6-phase pipeline with all defaults. |
| 145 | + |
| 146 | +**26 hard invariants** covering: phase ordering, inventory schema, cluster |
| 147 | +integrity, preferences schema, design mappings, confidence values, cost |
| 148 | +estimation, security rules, dry-run defaults. |
| 149 | + |
| 150 | +**Location:** `tests/fixtures/minimal-cloud-run-sql/` |
| 151 | + |
| 152 | +### bigquery-specialist-gate |
| 153 | + |
| 154 | +Tests the BigQuery specialist gate — the rule that `google_bigquery_*` resources |
| 155 | +must map to "Deferred — specialist engagement" and be excluded from cost totals. |
| 156 | + |
| 157 | +**9 hard invariants** focused on: deferred mapping, no Redshift/Athena/Glue/EMR, |
| 158 | +non-BigQuery resources still get normal mappings. |
| 159 | + |
| 160 | +**Location:** `tests/fixtures/bigquery-specialist-gate/` |
| 161 | + |
| 162 | +### ai-workload-openai |
| 163 | + |
| 164 | +Tests AI workload detection for apps using the OpenAI SDK on GCP. Includes |
| 165 | +both Terraform and Python source code (`src/chatbot.py` with `openai` imports). |
| 166 | + |
| 167 | +**11 hard invariants** focused on: AI profile creation, model detection, |
| 168 | +confidence threshold, excluded/legacy model rules, Category F questions. |
| 169 | + |
| 170 | +**Location:** `tests/fixtures/ai-workload-openai/` |
| 171 | + |
| 172 | +### user-preferences |
| 173 | + |
| 174 | +Same infrastructure as minimal, but with pre-seeded user answers |
| 175 | +(target_region=us-west-2, availability=single-az, cutover_strategy=blue-green). |
| 176 | +Tests that Design output respects user preferences. |
| 177 | + |
| 178 | +**10 hard invariants** focused on: chosen_by="user" preserved, Design region |
| 179 | +matches preference, single-AZ honored. |
| 180 | + |
| 181 | +**Location:** `tests/fixtures/user-preferences/` |
| 182 | + |
| 183 | +### negative-services |
| 184 | + |
| 185 | +Tests that forbidden AWS services never appear in Design output. Contains |
| 186 | +auth resources (should be excluded from inventory), BigQuery (should be |
| 187 | +deferred), and Cloud Run (should map to Fargate, not Lightsail/Beanstalk). |
| 188 | + |
| 189 | +**8 hard invariants** focused on: auth exclusion, no Cognito, no Lightsail, |
| 190 | +no Elastic Beanstalk, no analytics services for BigQuery. |
| 191 | + |
| 192 | +**Location:** `tests/fixtures/negative-services/` |
| 193 | + |
| 194 | +--- |
| 195 | + |
| 196 | +## Understanding Invariants |
| 197 | + |
| 198 | +Invariants are defined in YAML files: |
| 199 | + |
| 200 | +- **Shared:** `tests/invariants.yml` — used by `minimal-cloud-run-sql` |
| 201 | +- **Per-fixture:** `tests/fixtures/<name>/invariants.yml` — used by all other fixtures |
| 202 | + |
| 203 | +Each invariant has: |
| 204 | + |
| 205 | +| Field | Purpose | |
| 206 | +| ------------- | ------------------------------------------------------------ | |
| 207 | +| `id` | Unique identifier (e.g., H25, BQ3, NEG1) | |
| 208 | +| `description` | What's being checked | |
| 209 | +| `source` | Which prompt file contains the rule (file:line) | |
| 210 | +| `check.type` | How it's checked (file_exists, content_absent, custom, etc.) | |
| 211 | + |
| 212 | +### Check types |
| 213 | + |
| 214 | +| Type | What it does | Needs Python handler? | |
| 215 | +| ----------------- | ----------------------------------------------------- | --------------------- | |
| 216 | +| `file_exists` | Assert a file exists in migration output | No | |
| 217 | +| `file_absent` | Assert a file does NOT exist | No | |
| 218 | +| `content_present` | Assert patterns exist in file(s) | No | |
| 219 | +| `content_absent` | Assert patterns do NOT exist in file(s) | No | |
| 220 | +| `json_path_value` | Assert a JSON field has a specific value | No | |
| 221 | +| `json_every` | Assert every item in a JSON array has required fields | No | |
| 222 | +| `uniqueness` | Assert all values at a path are unique | No | |
| 223 | +| `cross_file_join` | Assert values in one file match values in another | No | |
| 224 | +| `custom` | Delegate to a Python script in `tools/invariants/` | Yes | |
| 225 | + |
| 226 | +### Custom handlers |
| 227 | + |
| 228 | +Python scripts in `tools/invariants/` handle logic too complex for YAML. |
| 229 | +Each script: |
| 230 | + |
| 231 | +1. Takes the migration directory as `sys.argv[1]` |
| 232 | +2. Reads the relevant JSON file(s) |
| 233 | +3. Prints `{"status": "pass"}` or `{"status": "fail", "details": "..."}` |
| 234 | +4. Includes documentation with: invariant description, skill file reference, |
| 235 | + and pass/fail examples |
| 236 | + |
| 237 | +--- |
| 238 | + |
| 239 | +## Adding a New Invariant |
| 240 | + |
| 241 | +### For declarative checks (no Python): |
| 242 | + |
| 243 | +Add an entry to the fixture's `invariants.yml`: |
| 244 | + |
| 245 | +```yaml |
| 246 | +- id: H50 |
| 247 | + description: "New check description" |
| 248 | + source: "prompt-file.md:line" |
| 249 | + check: |
| 250 | + type: content_absent |
| 251 | + file: gcp-resource-inventory.json |
| 252 | + patterns: |
| 253 | + - "forbidden pattern" |
| 254 | +``` |
| 255 | +
|
| 256 | +### For complex checks (Python handler): |
| 257 | +
|
| 258 | +1. Add the YAML entry pointing to a handler: |
| 259 | +
|
| 260 | + ```yaml |
| 261 | + - id: H50 |
| 262 | + description: "New check description" |
| 263 | + source: "prompt-file.md:line" |
| 264 | + check: |
| 265 | + type: custom |
| 266 | + handler: "tools/invariants/h50_new_check.py" |
| 267 | + ``` |
| 268 | +
|
| 269 | +1. Create `tools/invariants/h50_new_check.py` following the pattern of |
| 270 | + existing handlers. Include the docstring with invariant description, |
| 271 | + skill file reference, and examples. |
| 272 | + |
| 273 | +1. Run the evaluation to verify your new invariant passes on existing output. |
| 274 | + |
| 275 | +--- |
| 276 | + |
| 277 | +## Troubleshooting |
| 278 | + |
| 279 | +### `pip install pyyaml` error |
| 280 | + |
| 281 | +The checker requires PyYAML. Install it: |
| 282 | + |
| 283 | +```bash |
| 284 | +pip install pyyaml |
| 285 | +``` |
| 286 | + |
| 287 | +### "No invariants found for fixture" |
| 288 | + |
| 289 | +The fixture name doesn't match any `invariants.yml`. Check: |
| 290 | + |
| 291 | +- `tests/fixtures/<name>/invariants.yml` exists, OR |
| 292 | +- `tests/invariants.yml` has `fixture: <name>` at the top |
| 293 | + |
| 294 | +### Handler returns "skip" |
| 295 | + |
| 296 | +The custom Python handler file doesn't exist yet. Either implement it or |
| 297 | +accept the skip — skipped invariants don't affect pass/fail. |
| 298 | + |
| 299 | +### False positive on H41 (hardcoded credentials) |
| 300 | + |
| 301 | +The pattern `password = "` catches Terraform attribute assignments with string |
| 302 | +values. If your generated Terraform has a legitimate use (e.g., |
| 303 | +`manage_master_user_password = true`), the check should already pass since we |
| 304 | +only match `password = "` (with a quote). If you encounter a new false |
| 305 | +positive, tighten the pattern in `tests/invariants.yml`. |
0 commit comments