Skip to content

Commit ac2628a

Browse files
committed
docs: add AGENTS.md to document the S2 evaluation system and update latest.json with a valid baseline result.
1 parent e38f555 commit ac2628a

3 files changed

Lines changed: 113 additions & 65 deletions

File tree

evals/AGENTS.md

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# S2 Evaluation System - AGENTS.md
2+
3+
This directory contains the automated evaluation pipeline for AntV S2. It is designed to benchmark and validate code generation capabilities of AI models (specifically Claude Code) against expert S2 patterns.
4+
5+
---
6+
7+
## 🤖 Agent Role: S2 Evaluation Orchestrator
8+
The goal of the agents in this directory is to execute, monitor, and visualize the performance of S2-specialized code generation.
9+
10+
- **Primary Goal**: Ensure that "Skill Injected" code (using S2 expert knowledge) outperforms "Baseline" code in correctness and aesthetic quality.
11+
- **Verification Method**: Every line of generated code MUST be rendered in a headless browser (Playwright) to confirm zero-runtime-error status.
12+
13+
## 🛠 Tech Stack
14+
- **Engine**: Node.js + TypeScript (`ts-node`)
15+
- **Agent Link**: Claude Code CLI (`claude` command)
16+
- **Validation**: Playwright (Headless Chromium)
17+
- **Visualization**: React (Vite) + Ant Design 5 + VisionSnap SDK
18+
- **Data**: JSON-based expert test cases (`datasets/s2-expert-cases.json`)
19+
20+
## ⚡ Key Commands
21+
Agents should use the following commands to interact with the system:
22+
23+
| Command | Action | Description |
24+
|:---|:---|:---|
25+
| `npm run eval` | **Execution** | Starts the concurrent evaluation process. Spawns isolated Claude sessions. |
26+
| `npm run eval -- --limit 5` | **Dry Run** | Runs the evaluation for only the first 5 cases (for debugging). |
27+
| `npm run dashboard` | **Visualization** | Starts the local dev server to view result reports at `localhost:5173`. |
28+
29+
## 🛡 Boundaries & Principles
30+
31+
### Always
32+
- Use **Isolated Sessions**: When spawning `claude`, always include `--session-id` with a unique UUID and `--no-session-persistence` to prevent context pollution between test cases.
33+
- **Extract Markdown**: Use robust regex to extract code blocks between ` ```tsx ` markers from the model's stdout.
34+
- **Concurrent Execution**: Utilize the built-in `pMap` concurrency (dynamically calculated based on system memory) to avoid running 100+ cases serially.
35+
36+
### Never
37+
- **Never rely on local file writes** inside the model evaluation prompt. Models must be instructed via `STRICT_SYSTEM_PROMPT` to only output code to `stdout`.
38+
- **Never use hardcoded sleep** for iframe rendering. Always use `waitForSelector` for reliable headless validation.
39+
40+
### Ask First
41+
- If `os.totalmem()` detection fails or the machine has less than 8GB RAM, ask before running high-concurrency tasks.
42+
43+
## 📂 Directory Structure Context
44+
- `run_eval.ts`: The main brain/scheduler. Orchestrates Claude, handles concurrency, and saves reports.
45+
- `validate.ts`: The headless judge. Boots the dashboard in `?mode=validate` to check if code rendering succeeds.
46+
- `datasets/`: The source of truth for benchmark prompts.
47+
- `results/`: The output warehouse of evaluation history.
48+
- `dashboard/`: The UI interface for human-in-the-loop review.

0 commit comments

Comments
 (0)