Skip to content

Commit 89252ac

Browse files
mikepenaMSmikepenaMS
authored
Add generate-mcp-app skill for MCP Apps widget generation (#55)
* Add generate-mcp-app skill for MCP Apps widget generation New plugin that generates self-contained MCP App widgets (HTML) from tool output. Includes skill template, API reference, design guidelines, and a sample widget. * Fix README sample filename * Add contrast rules to design guidelines Text on brand backgrounds must use colorNeutralForegroundOnBrand (white), not colorNeutralForeground1 (dark). Prevents unreadable dark-on-blue text. * Add evals * Feedback * Update widget template and reference docs to align with Copilot UX guidelines Body padding changed from 16px to 24px per official Copilot UX guidelines. Generalized CDN library references in mcp-apps-reference.md and eval assertions. The UMD global collision pattern is preserved with generic placeholder names. Eval results confirm no regression (9.92/10 avg, 53/53 pass all layers). * Address PR feedback: rename skill, add error fallback, add Installation/License Renamed skill from generate-mcp-app to generate-mcp-app-ui to avoid confusion with create-mcp-app in the ext-apps repo. Added else branch in ontoolresult template to call renderError when structuredContent is missing (prevents infinite loading spinner). Added Installation and License sections to README. Reordered examples in SKILL.md. Added weather sample to eval runbook related files. Eval results: 9.75/10 avg (no regression from 9.77 baseline). * Name change
1 parent 4930073 commit 89252ac

10 files changed

Lines changed: 4667 additions & 0 deletions

File tree

‎.claude-plugin/marketplace.json‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,19 @@
3333
"microsoft"
3434
]
3535
},
36+
{
37+
"name": "mcp-apps",
38+
"source": "./plugins/mcp-apps",
39+
"description": "Generate MCP App widgets for MCP tools",
40+
"category": "development",
41+
"tags": [
42+
"mcp",
43+
"mcp-apps",
44+
"widget",
45+
"power platform",
46+
"microsoft"
47+
]
48+
},
3649
{
3750
"name": "canvas-apps",
3851
"source": "./plugins/canvas-apps",

‎evals/mcp-apps/generate-mcp-app-ui/eval-runbook.md‎

Lines changed: 168 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,168 @@
1+
# Eval Runbook for generate-mcp-app-ui
2+
3+
How to evaluate the `generate-mcp-app-ui` skill. Three layers, run in order.
4+
5+
## Related files
6+
7+
- **Skill definition:** `plugins/mcp-apps/skills/generate-mcp-app-ui/SKILL.md`
8+
- **Reference docs:** `plugins/mcp-apps/references/mcp-apps-reference.md`, `plugins/mcp-apps/references/design-guidelines.md`
9+
- **Sample widgets:** `plugins/mcp-apps/samples/flight-status-widget.html`, `plugins/mcp-apps/samples/weather-refresh-widget.html`
10+
11+
## Eval data
12+
13+
All eval definitions live in `evals.json` alongside this file. The file contains:
14+
15+
- `common_assertions`: 13 assertions every generated widget must pass
16+
- `evals`: 53 test cases, each with a `prompt`, inline `data`, per-widget `assertions`, and a `tier` field
17+
18+
The `data` field on each eval is the JSON object that an MCP tool would return at runtime. During widget generation, you paste it into the conversation as "the tool's test output." During rendering tests, it becomes the value of `result.structuredContent` in the JSON-RPC `toolResult` message that the host sends to the widget's iframe.
19+
20+
### Tiers
21+
22+
Each eval has a `tier` to support selective running:
23+
24+
| Tier | Count | Purpose |
25+
| -------- | ----- | -------------------------------------------------------------------------------------------------------- |
26+
| `smoke` | 7 | Diverse representatives (map, chart, table, cards, complex layout, structured, stress). Run on every PR. |
27+
| `full` | 44 | All remaining core widget types. Run nightly or pre-release. |
28+
| `stress` | 2 | Type-mismatch edge cases (string booleans, empty-string coordinates). Run with full suite. |
29+
30+
Eval id 51 is tagged `smoke` (not `stress`) so quick runs still exercise type coercion. Its assertions are stress-style (string-to-number parsing), but it runs with the smoke set rather than requiring the full suite.
31+
32+
## Quick start: running one eval
33+
34+
Here is a complete example using eval id 2 (weather widget).
35+
36+
1. Open Claude Code (or any Claude-powered tool with the skill installed).
37+
2. Send a message like:
38+
39+
> /generate-mcp-app-ui Create a weather widget showing current conditions and 5-day forecast.
40+
>
41+
> Here is the tool's test output:
42+
>
43+
> ```json
44+
> {
45+
> "city": "Seattle",
46+
> "temperature": 58,
47+
> "humidity": 72,
48+
> "conditions": "Partly Cloudy",
49+
> "forecast": [
50+
> { "day": "Mon", "high": 62, "low": 48, "conditions": "Sunny" },
51+
> { "day": "Tue", "high": 59, "low": 47, "conditions": "Cloudy" },
52+
> { "day": "Wed", "high": 55, "low": 44, "conditions": "Rain" },
53+
> { "day": "Thu", "high": 57, "low": 45, "conditions": "Partly Cloudy" },
54+
> { "day": "Fri", "high": 61, "low": 49, "conditions": "Sunny" }
55+
> ]
56+
> }
57+
> ```
58+
59+
3. Save the generated HTML file.
60+
4. **Layer 1 check:** Open the HTML in a text editor and verify: starts with `<!DOCTYPE html>`, has one `<script type="module">`, uses `result.structuredContent`, etc. Then check the per-widget assertions: "Shows current temperature prominently", "Shows 5-day forecast strip", "Uses weather icons/emoji for conditions."
61+
5. **Layer 2 check:** Open the HTML in a browser (it needs a JSON-RPC host to send it the tool data, see Step 3 below).
62+
6. **Layer 3 check:** Score the visual result against the rubric.
63+
64+
## How to run evals
65+
66+
### Step 1: Generate widgets
67+
68+
Invoke the `generate-mcp-app-ui` skill via Claude Code by running `/generate-mcp-app-ui` followed by the eval's `prompt`. Paste the eval's `data` JSON into the conversation as the tool's test output.
69+
70+
For each eval in `evals.json`:
71+
72+
- Use the `prompt` as the user message
73+
- Paste the `data` object as the tool's test output JSON
74+
75+
Save each generated HTML file for testing.
76+
77+
To run only a subset, filter by `tier` (e.g., smoke-only for quick validation).
78+
79+
### Step 2: Layer 1 - Static assertions
80+
81+
Check each generated HTML file against the `common_assertions` (13 checks) plus the eval's per-widget `assertions`.
82+
83+
**Common assertions verify:**
84+
85+
1. Complete HTML file starting with `<!DOCTYPE html>`
86+
2. Exactly one `<script type="module">` block
87+
3. Named import: `import { App } from ...`
88+
4. `app.ontoolresult` set before `app.connect()`
89+
5. `app.onhostcontextchanged` set before `app.connect()`
90+
6. `app.onteardown` set before `app.connect()`
91+
7. Uses `result.structuredContent` to access data
92+
8. Defines and uses `escapeHtml` for user data in innerHTML
93+
9. No `window.openai`
94+
10. No `max-width` on the main container (responsive `@media (max-width:...)` is fine)
95+
11. Uses `var(--color` Fluent design tokens
96+
12. Includes `<fluent-spinner>` loading state
97+
13. Includes an error state
98+
99+
These can be checked with text search / regex against the HTML source. No browser needed.
100+
101+
**Pass criteria:** Every widget passes all common assertions plus its own specific assertions.
102+
103+
### Step 3: Layer 2 - Rendering tests
104+
105+
Load each widget in a browser to verify it actually runs.
106+
107+
**What to check:**
108+
109+
- Widget loads without JavaScript errors in the console
110+
- Content renders (not stuck on "Loading..." or showing the error state)
111+
- Layout is not broken (no overlapping elements, no blank page)
112+
113+
**Important:** Widgets import ES modules from CDN, so they must be served over HTTP (e.g., `npx serve .` or `python -m http.server`). Opening via `file://` will fail due to CORS restrictions on module imports.
114+
115+
This can be done manually or automated with Playwright / Puppeteer. The widget needs a host page that simulates the MCP Apps JSON-RPC protocol, sending the eval's `data` as the `structuredContent` in a `toolResult` message.
116+
117+
To build a minimal test host: create an HTML page that loads the widget in an iframe and posts JSON-RPC messages via `postMessage`. The key message to send after the widget connects:
118+
119+
```json
120+
{
121+
"jsonrpc": "2.0",
122+
"method": "toolResult",
123+
"params": {
124+
"structuredContent": { "...eval data object here..." }
125+
}
126+
}
127+
```
128+
129+
The widget also expects an initial `hostContext` message for theming:
130+
131+
```json
132+
{
133+
"jsonrpc": "2.0",
134+
"method": "hostContext",
135+
"params": {
136+
"theme": "light",
137+
"fontFamily": "Segoe UI, sans-serif",
138+
"containerWidth": 600
139+
}
140+
}
141+
```
142+
143+
See the [MCP Apps protocol spec](https://modelcontextprotocol.io/specification/2025-03-26/extensions/apps) for the full message format.
144+
145+
**Pass criteria:** All widgets render with visible content, zero JS errors.
146+
147+
### Step 4: Layer 3 - UX scoring
148+
149+
Review each widget visually (screenshot or live) against this rubric:
150+
151+
| Category | 2 (Full) | 1 (Partial) | 0 (Fail) |
152+
| -------- | --------------------------------------------------- | -------------------------------- | -------------------------------------- |
153+
| Protocol | Correct MCP setup, structuredContent, escapeHtml | Minor issue | Wrong data access or missing connect() |
154+
| Code | Clean code, parseFloat for numerics, error handling | Minor issues | JS errors, broken logic |
155+
| Visual | Polished layout, good spacing, Fluent tokens | Decent but cramped or misaligned | Broken layout |
156+
| Renders | All data fields populated correctly | Some fields missing | Blank or wrong data |
157+
| Design | Right visual for the data, compact, no clutter | Reasonable but not optimal | Wrong visual type |
158+
159+
Max score: 10 per widget (5 categories x 2 points).
160+
161+
**Pass criteria:** Average score >= 8.5/10 across all widgets, no individual widget below 7/10.
162+
163+
## When to run evals
164+
165+
- After any change to the skill definition (SKILL.md)
166+
- After changes to reference docs (mcp-apps-reference.md, design-guidelines.md)
167+
- Before submitting a PR that modifies the skill
168+
- Smoke tier is sufficient for small changes; run full + stress for significant updates

0 commit comments

Comments
 (0)