Skip to content

Commit 1dc5a62

Browse files
baonguyenNavabaonguyenNava
authored
Merge pull request #2 from navapbc/baonguyen/build-using-strata-sdk-skill
Baonguyen/build using strata sdk skill
2 parents 9793e8a + 540d4f7 commit 1dc5a62

12 files changed

Lines changed: 1557 additions & 7 deletions

File tree

‎docs/superpowers/specs/2026-04-28-skill-eval-harness-design.md‎

Lines changed: 264 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,264 @@
1+
# Skill Evaluation Harness — Design
2+
3+
**Date:** 2026-04-28
4+
**Status:** Draft, pending implementation plan
5+
**Author:** Bao Nguyen
6+
**Subsystem:** 1 of 3 (Skill Eval Harness → Managed Agent → Non-Technical Builder UX)
7+
8+
## Goal
9+
10+
Build a GitHub Actions + AWS pipeline that evaluates each agent skill in this repo for **output quality** and **cost/latency** whenever a pull request modifies it. Results post back to the PR as a non-blocking comment so authors can iterate on skill design with measurable feedback.
11+
12+
This subsystem also lays foundational AWS infrastructure (IAM/OIDC, Bedrock AgentCore, Lambda, Step Functions) that subsequent subsystems — a managed agent and a non-technical builder UX — will reuse.
13+
14+
## Scope
15+
16+
### In scope
17+
18+
- GitHub Actions workflow triggered on pull requests touching `skills/**`
19+
- AWS-hosted execution of each affected skill against a curated prompt set
20+
- LLM-judge scoring against a rubric per skill
21+
- Cost and latency capture per prompt
22+
- PR comment with summary table (sticky, updated on re-run)
23+
- Repo-level `evals/` directory housing prompts and rubrics
24+
- IaC (CDK or Terraform; choice deferred to plan) for all AWS resources
25+
- Unit tests for Lambda handlers, schema validation, comment rendering
26+
- One-time manual validation against the smallest skill before merge
27+
28+
### Out of scope
29+
30+
- Hard merge gates / pass-fail thresholds (results are advisory only)
31+
- Cross-account or multi-region deployment (single region: `us-east-1`)
32+
- Historical-trend dashboards (CloudWatch metrics emitted, but dashboard is later work)
33+
- Auto-generation of test prompts (curated only)
34+
- The managed agent itself (subsystem 2)
35+
- The non-technical builder UX (subsystem 3)
36+
37+
## Non-Goals
38+
39+
- Replace human review of skill quality
40+
- Score every skill on every PR (only changed skills run)
41+
- Block merges or enforce minimum scores (intentionally a soft signal)
42+
43+
## Success Criteria
44+
45+
1. A skill author opens a PR modifying `skills/build-strata-rails-app/SKILL.md`. Within ~10 minutes a PR comment shows quality scores per prompt, total cost, and latency.
46+
2. Adding a new skill with a corresponding `evals/<skill>/` directory automatically gets evaluated on its first PR — no infra changes needed.
47+
3. AWS cost per PR run stays under $1 for a typical 3-prompt skill (target; observed via CloudWatch).
48+
4. Lambda + Step Function patterns are reusable for subsystem 2.
49+
50+
## Architecture
51+
52+
```
53+
GitHub PR (skills/** changed)
54+
|
55+
v
56+
+-----------------------------+
57+
| GH Action: skill-eval.yml |
58+
| - detect changed skills |
59+
| - OIDC -> AWS |
60+
| - upload input zip to S3 |
61+
| - start Step Function |
62+
| - poll until done |
63+
| - post PR comment |
64+
+-----------------------------+
65+
| AssumeRole (OIDC)
66+
v
67+
+----------------------------------------------+
68+
| AWS account |
69+
| |
70+
| Step Function: SkillEvalSM (Express) |
71+
| |- Map state (per changed skill) |
72+
| | |- Lambda: load-evals |
73+
| | |- Map state (per prompt, max 5) |
74+
| | | |- Lambda: run-skill |
75+
| | | | `- Bedrock AgentCore |
76+
| | | | (skill loaded as ctx) |
77+
| | | `- Lambda: judge |
78+
| | | `- Bedrock InvokeModel |
79+
| | | (Claude Sonnet judge) |
80+
| | `- Lambda: aggregate |
81+
| `- Final: build report.json -> S3 |
82+
| |
83+
| S3: skill-eval-artifacts-<account> |
84+
| /<run-id>/ |
85+
| input.zip, output.json, judge.json, |
86+
| report.json |
87+
| |
88+
| CloudWatch: metrics namespace `SkillEval` |
89+
+----------------------------------------------+
90+
```
91+
92+
### Auth and trust boundaries
93+
94+
- **GitHub -> AWS:** OIDC federation. A repo-scoped IAM role (`gh-oidc-skill-eval-role`) is assumable only by this repository's workflows. Permissions: `s3:PutObject` to a per-run prefix, `states:StartExecution` and `states:DescribeExecution` for `SkillEvalSM`, `s3:GetObject` for the report path.
95+
- **Lambdas:** Each Lambda has a least-privilege role. `run-skill` may invoke AgentCore. `judge` may invoke Bedrock models. All can write CloudWatch logs and metrics; only `aggregate` can write the final report.
96+
- No long-lived AWS keys anywhere.
97+
98+
### Region
99+
100+
Single region: `us-east-1`. Chosen for AgentCore + Bedrock model availability and to keep IaC simple.
101+
102+
## Components
103+
104+
### Repository additions
105+
106+
| Path | Purpose |
107+
| --- | --- |
108+
| `.github/workflows/skill-eval.yml` | Trigger, OIDC, SFN start, poll, comment |
109+
| `evals/<skill>/prompts.json` | Array of `{id, prompt, expected_skill_invoked, rubric_dims}` per skill |
110+
| `evals/<skill>/rubric.md` | Human-readable judge criteria |
111+
| `infra/` | IaC for all AWS resources (CDK or Terraform — chosen in plan) |
112+
| `lambdas/load-evals/` | Python 3.12. Reads `evals/<skill>/prompts.json` from `input.zip` in S3, validates schema, returns prompt list to SFN |
113+
| `lambdas/run-skill/` | Python 3.12. Loads SKILL.md as context, calls AgentCore, captures output + metrics |
114+
| `lambdas/judge/` | Python 3.12. Calls Bedrock with rubric + skill + prompt + output, returns score JSON |
115+
| `lambdas/aggregate/` | Python 3.12. Reads judge artifacts, builds report, emits CloudWatch metrics |
116+
| `scripts/post-comment.py` | Renders markdown table from `report.json` and posts a sticky PR comment |
117+
| `scripts/eval_local.py` | Runs full pipeline against AWS, skipping the GH layer (for skill authors) |
118+
119+
### AWS resources
120+
121+
- **IAM:** `gh-oidc-skill-eval-role` (assumed by GH), `lambda-exec-role-load-evals`, `lambda-exec-role-run-skill`, `lambda-exec-role-judge`, `lambda-exec-role-aggregate`
122+
- **Step Function:** `SkillEvalSM` (Express workflow, 5 min execution cap, max 5 concurrent prompts in inner Map)
123+
- **Lambdas:** `load-evals`, `run-skill`, `judge`, `aggregate` (Python 3.12, 1024 MB; provisioned concurrency = 1 for `run-skill` and `judge`)
124+
- **S3:** `skill-eval-artifacts-<account-id>` with 30-day lifecycle expiry on all objects
125+
- **Bedrock:** AgentCore agent `skill-eval-agent`, model access for `claude-sonnet-4-6`
126+
- **CloudWatch:** Log groups per Lambda; custom metrics under namespace `SkillEval` (`QualityScore`, `TokensIn`, `TokensOut`, `LatencyMs`, `CostUsd`)
127+
- **Budget alarm:** $50 daily threshold on the `SkillEval` cost-allocation tag, SNS to a human distro
128+
129+
### Models
130+
131+
- **Skill runtime:** Claude Sonnet 4.6 — matches what real users see in Claude Code, good cost/quality balance.
132+
- **Judge:** Claude Sonnet 4.6, `temperature=0` — same family, deterministic-as-possible scoring. Document expected ±1 point variance.
133+
134+
### Eval prompt schema
135+
136+
`evals/<skill>/prompts.json`:
137+
138+
```json
139+
[
140+
{
141+
"id": "happy-path-1",
142+
"prompt": "Scaffold a rails app",
143+
"expected_skill_invoked": true,
144+
"rubric_dims": ["correctness", "completeness", "follows_skill_steps"]
145+
}
146+
]
147+
```
148+
149+
`rubric_dims` keys must exist in the corresponding `rubric.md` so the judge can score each.
150+
151+
## Data Flow
152+
153+
1. PR opens or updates with changes under `skills/**`.
154+
2. `skill-eval.yml` runs.
155+
- `git diff --name-only origin/main...HEAD | grep ^skills/` -> list of changed skill directories. Empty -> exit 0.
156+
3. GH Action assumes the AWS role via OIDC (`aws-actions/configure-aws-credentials`).
157+
4. GH Action zips changed `skills/<x>/` and `evals/<x>/` directories and uploads to `s3://skill-eval-artifacts-<account>/<run_id>/input.zip` (avoids passing large payloads through Step Functions state).
158+
5. GH Action calls `StartExecution` on `SkillEvalSM` with input:
159+
```json
160+
{
161+
"run_id": "<pr-number>-<commit-sha>",
162+
"skills": ["build-strata-rails-app"],
163+
"pr_number": 42,
164+
"commit_sha": "abc123"
165+
}
166+
```
167+
6. Step Function:
168+
- Outer `Map` over `skills[]`:
169+
- `load-evals` Lambda reads `evals/<skill>/prompts.json` from `input.zip` in S3.
170+
- Inner `Map` over `prompts[]` (max concurrency 5):
171+
- `run-skill` Lambda:
172+
- Loads `skills/<skill>/SKILL.md` as system context
173+
- Invokes an AgentCore session with the user prompt
174+
- Captures response text, tool calls, token counts (input + output), wall time
175+
- Writes `s3://.../<run_id>/<skill>/<prompt_id>/output.json`
176+
- `judge` Lambda:
177+
- Reads output + rubric
178+
- Builds judge prompt: rubric dimensions + skill description + user prompt + skill output
179+
- Calls `bedrock:InvokeModel` (Claude Sonnet 4.6)
180+
- Parses `{quality: 0-10, dims: {...}, rationale: "..."}`. Retries up to 3 times on parse failure with stricter system prompt; if still failing, writes `{score: null, error: "parse-failed"}`.
181+
- Writes `s3://.../<run_id>/<skill>/<prompt_id>/judge.json`
182+
- `aggregate` Lambda:
183+
- Reads all judge + output artifacts
184+
- Computes: average quality, total tokens, total cost (price x tokens), p50 / p95 latency
185+
- Writes `s3://.../<run_id>/report.json`
186+
- Emits CloudWatch metrics under namespace `SkillEval`
187+
7. GH Action polls `DescribeExecution` until `SUCCEEDED` or `FAILED`. Timeout: 10 minutes.
188+
8. GH Action downloads `report.json`, runs `post-comment.py`:
189+
- Renders a markdown table: `skill | prompt | quality | tokens | cost ($) | latency (s)`
190+
- Posts or updates a sticky PR comment, identified by the marker `<!-- skill-eval-bot -->`
191+
192+
## Error Handling
193+
194+
| Failure | Handling |
195+
| --- | --- |
196+
| GH OIDC assume-role fails | GH job fails, no PR comment. Author retries via `Re-run`. |
197+
| Changed skill has no `evals/<skill>/` directory | `load-evals` returns empty list. Report row says `no evals defined`. PR comment soft-warns. |
198+
| Malformed `prompts.json` | `load-evals` fails fast with a schema error. SFN catches; aggregate marks the skill as `eval-config-error`. Comment reflects this. |
199+
| AgentCore session timeout (>2 min) | `run-skill` kills the session, writes output with `status: timeout` and partial response. Judge skips and marks `not-judged`. |
200+
| Bedrock throttling (`ThrottlingException`) | Lambda retries with exponential backoff (3 attempts, base 2s). SFN-level retry on `Lambda.ServiceException`. |
201+
| Judge returns non-JSON | Parser retries 3 times with stricter system prompt. On final failure, judge.json marks `score: null, error: parse-failed`. |
202+
| Lambda cold start blows latency budget | Provisioned concurrency = 1 on `run-skill` and `judge`. |
203+
| S3 write fails | SFN retry with backoff. Final failure -> run marked `infra-error`. PR comment shows error rather than score. |
204+
| SFN execution exceeds 5 minutes | Express workflows cap at 5 min. If real runs trend longer, switch to Standard workflow (decided during implementation). |
205+
| Cost runaway | Per-prompt token cap: 8K input / 4K output. SFN inner Map `maxConcurrency = 5`. Daily budget alarm at $50 -> SNS notification to a human. |
206+
| GH poll timeout (10 min) | Action posts an "eval still running, see CloudWatch link" comment with a non-blocking warning. |
207+
208+
**Sticky-comment policy:** every failure mode produces a comment. There are no silent failures.
209+
210+
**Idempotency:** `run_id = <pr-number>-<commit-sha>`. Re-runs on the same SHA reuse existing S3 artifacts (each Lambda checks S3 before re-computing). This keeps re-run cost near zero.
211+
212+
## Testing
213+
214+
### Unit tests (pytest, in repo)
215+
216+
- `tests/lambdas/test_run_skill.py` — mock AgentCore SDK; assert request shape, output schema, timeout handling
217+
- `tests/lambdas/test_judge.py` — mock Bedrock; assert rubric-aware prompt construction, JSON parse with retries, error path
218+
- `tests/lambdas/test_aggregate.py` — fixture S3 directory; assert `report.json` numbers (avg, p95, totals)
219+
- `tests/scripts/test_post_comment.py` — golden-file markdown render; mock GitHub API
220+
221+
### Schema tests
222+
223+
- `tests/test_eval_schema.py` — every `evals/<skill>/prompts.json` validates against a JSON schema
224+
- Extension to `scripts/lint_skills.py`: warn (not fail) if `skills/<x>/` exists without a matching `evals/<x>/`
225+
226+
### Integration
227+
228+
- `scripts/eval_local.py` — runs the full pipeline against AWS, bypassing the GH layer. Used by skill authors to sanity-check before pushing.
229+
- Nightly cron full-suite eval against `main` is **out of scope here** but planned as follow-up work feeding subsystem 2.
230+
231+
### CI smoke tests
232+
233+
- `tests/test_workflow_yaml.py` — parses `.github/workflows/skill-eval.yml`; asserts OIDC permissions, required steps
234+
- `tests/infra/test_cdk_synth.py` (or Terraform-plan equivalent) — synthesizes infra, snapshot-tests resources
235+
236+
### Manual validation (one-time, before merging the harness PR)
237+
238+
- End-to-end run on the `hello-world` skill (smallest blast radius)
239+
- Confirm: PR comment renders, S3 artifacts present, CloudWatch metrics emitted, cost matches estimate
240+
241+
### Knowingly untested
242+
243+
- Real Bedrock model output quality (judge variance) — accepted noise; mitigated by `temperature=0` and documented ±1 score variance
244+
- AgentCore SDK behavior changes — pin SDK version, monitor release notes
245+
246+
## Open Questions for Implementation Plan
247+
248+
These are deliberately deferred to the writing-plans step:
249+
250+
- IaC tool: CDK (TypeScript) vs Terraform. Tradeoffs: CDK aligns with TS-heavy AWS examples; Terraform aligns with broader Nava infra conventions.
251+
- Step Function flavor: Express (5 min cap) vs Standard. Default to Express; switch if observed runs exceed 5 min.
252+
- Concrete cost model and budget number for nightly + per-PR usage.
253+
- Where `eval_local.py` reads AWS credentials from (`AWS_PROFILE` vs SSO).
254+
255+
## How This Feeds Subsystem 2 (Managed Agent)
256+
257+
The infrastructure built here is deliberately reusable:
258+
259+
- **OIDC + IAM patterns** — same role-assumption pattern will gate an external chat UI calling AWS.
260+
- **Step Functions + Lambda** — the same fan-out / coordinate / aggregate pattern fits a multi-step agent workflow.
261+
- **AgentCore agent** — `skill-eval-agent` is a stripped-down version of what subsystem 2 will operate at scale.
262+
- **Skill-as-context loader** — the `run-skill` Lambda's mechanism for loading a skill into Claude's context is the exact mechanism the managed agent will use.
263+
264+
Subsystem 2 will likely promote `skill-eval-agent` to a richer agent with persistent state and tool access; the eval harness keeps it stateless and constrained.

‎references/ruby-version-check.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# Ruby Version Check (shared reference)
2+
3+
**Purpose:** verify the active Ruby matches what a Rails project requires before installing gems or running generators. Mismatched Ruby is a common cause of confusing `bundle install` and `bin/rails` failures.
4+
5+
**Used by:** any skill that touches a Rails project's gems, runs Rails generators, or runs `make test` / `make lint`. Reference this file from `<RAILS_DIR>` (the located Rails app directory) — all paths below are relative to it.
6+
7+
## Step A: Determine the required version
8+
9+
Check, in order:
10+
11+
1. `<RAILS_DIR>/.ruby-version` (one line, e.g. `3.3.4`)
12+
2. `<RAILS_DIR>/Gemfile` for a `ruby "X.Y.Z"` line
13+
3. `<RAILS_DIR>/.tool-versions` for a `ruby X.Y.Z` line (asdf)
14+
15+
Save the required version as `<REQUIRED_RUBY>`. If none of those files specify a Ruby version, skip the rest — there is no version to enforce.
16+
17+
## Step B: Compare against the active Ruby
18+
19+
```sh
20+
ruby -v
21+
```
22+
23+
If the major.minor.patch matches `<REQUIRED_RUBY>` → done. Continue with the calling skill.
24+
25+
If it does not match → continue to Step C.
26+
27+
## Step C: Ask which version manager the user uses
28+
29+
> **Your active Ruby is `<active>` but the project requires `<REQUIRED_RUBY>`. Which Ruby version manager do you use? (rbenv / asdf / rvm / chruby / other)**
30+
31+
## Step D: Switch to the required version
32+
33+
First check whether the version is installed; if not, install it. Then activate it.
34+
35+
| Manager | Check installed | Install if missing | Activate |
36+
|---------|-----------------|--------------------|----------|
37+
| **rbenv** | `rbenv versions \| grep -q <REQUIRED_RUBY>` | `rbenv install <REQUIRED_RUBY>` | `rbenv local <REQUIRED_RUBY>` (run inside `<RAILS_DIR>`) |
38+
| **asdf** | `asdf list ruby \| grep -q <REQUIRED_RUBY>` | `asdf install ruby <REQUIRED_RUBY>` | `asdf local ruby <REQUIRED_RUBY>` (run inside `<RAILS_DIR>`) |
39+
| **rvm** | `rvm list strings \| grep -q <REQUIRED_RUBY>` | `rvm install <REQUIRED_RUBY>` | `rvm use <REQUIRED_RUBY>` |
40+
| **chruby** | `chruby \| grep -q <REQUIRED_RUBY>` | install via `ruby-install <REQUIRED_RUBY>` (tell user if `ruby-install` is missing) | `chruby <REQUIRED_RUBY>` |
41+
| **other / unsure** | — | — | Stop. Ask the user to switch manually, then confirm before continuing. |
42+
43+
Notes:
44+
45+
- `rbenv install` / `asdf install ruby` may take several minutes (compiling Ruby). Tell the user before running.
46+
- After activating, **re-run `ruby -v`** to confirm. If still mismatched (shell hasn't picked up the new version), stop and ask the user to open a new shell or `source` their rc file.
47+
- Do not run `sudo` for any of these — version managers are per-user.
48+
49+
## Step E: Verify Bundler is available
50+
51+
```sh
52+
bundle -v
53+
```
54+
55+
If `bundle` is missing → `gem install bundler`. Then continue with the calling skill.
56+
57+
## Common pitfalls
58+
59+
| Problem | Fix |
60+
|---------|-----|
61+
| `ruby -v` still old after `rbenv local` | Open a new shell, or run `eval "$(rbenv init -)"` in current shell |
62+
| `asdf install ruby <ver>` fails with build errors | User missing build deps (openssl, readline). Direct them to asdf-ruby README. |
63+
| Multiple version managers installed (e.g. rbenv + asdf) | Ask which one is authoritative; mixing causes silent shadowing |
64+
| `.ruby-version` and `Gemfile` disagree | `.ruby-version` wins for the version manager; Gemfile `ruby` line is enforced by Bundler. Ask user to reconcile. |

0 commit comments

Comments
 (0)