Checklist YAML (feature x CRUD operations with point values)
→ Research Agent Phase 1: explore site, test each operation, pass/fail
→ Research Agent Phase 2: generate .spec.ts, debug without video (fast, no demowright delays)
→ Research Agent Phase 3: record with video (demowright delays + TTS + cursor overlay)
→ Final video: demo of all operations (successes + failures) + QA scorecard
- Improve Gemini evaluation score: average 3.7/10 → target 7/10
- Fix mux-reporter subtitles filter on macOS (path quoting)
- Reduce title card idle time (video starts recording before render begins)
- QA Research Agent (
src/agent/qa-research.ts) — 3-phase pipeline - Phase 1: headless LLM exploration, tests each CRUD operation, scores pass/fail
- Phase 2: generates demowright .spec.ts, debug loop with LLM auto-fix (3 retries)
- Phase 3: runs spec with video recording (demowright delays + TTS)
- Anthropic SDK with 60s timeout (fixes Bun fetch hanging)
- Phase 2 pass detection: regex "N passed"/"N failed" (not string match)
- Visual actions in system prompt (safeMove, hover, scroll required)
- QA scorecard: fullscreen HTML card with color-coded results + long narration
- Checklist YAMLs: registry-web, comfy-docs, comfy-website, download-data, embedded-editor
- CRUD operations: added create (search input), update (filter switch) operations
- cloud-comfy headless WebGL via Chromium --headless=new
- Gemini 3.1 Pro video evaluator (demo/evaluate-demos.ts)
- demowright setup API: segment({ setup, action }) for pre-narration navigation
- Deleted old demo-research.ts and demo-editor.ts
- Timeout increased to 20min for slow specs
- preserveOutput: "always" for webm retention
- mux-reporter webm fallback + subtitles fallback
- docs/research-agent.md with QA principles
| Product | Score |
|---|---|
| comfy-docs | 80% (4/5) |
| comfy-website | 80% (4/5) |
| comfy-registry | 77% (10/13) |
| embedded-workflow-editor | 75% (3/4) |
| download-data | 100% (5/5) |
| Video | Score |
|---|---|
| comfy-registry-qa | 5/10 |
| comfy-docs-qa | 4/10 |
| download-data-qa | 2/10 |
| Average | 3.7/10 |