Skip to content

Commit f89b5ea

Browse files
committed
chore: 去掉 S2 的描述
1 parent ac2628a commit f89b5ea

1 file changed

Lines changed: 19 additions & 18 deletions

File tree

evals/AGENTS.md

Lines changed: 19 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,48 +1,49 @@
1-
# S2 Evaluation System - AGENTS.md
1+
# AntV Skills Evaluation System - AGENTS.md
22

3-
This directory contains the automated evaluation pipeline for AntV S2. It is designed to benchmark and validate code generation capabilities of AI models (specifically Claude Code) against expert S2 patterns.
3+
This directory contains the automated evaluation pipeline for all AntV chart visualization skills (G2, G6, S2, L7, F2, etc.). It is designed to benchmark and validate code generation capabilities of AI models (specifically Claude Code) against expert domain knowledge.
44

55
---
66

7-
## 🤖 Agent Role: S2 Evaluation Orchestrator
8-
The goal of the agents in this directory is to execute, monitor, and visualize the performance of S2-specialized code generation.
7+
## 🤖 Agent Role: AntV Skills Evaluation Orchestrator
8+
The goal of the agents in this directory is to execute, monitor, and visualize the performance of AntV-specialized code generation across different technical stacks.
99

10-
- **Primary Goal**: Ensure that "Skill Injected" code (using S2 expert knowledge) outperforms "Baseline" code in correctness and aesthetic quality.
11-
- **Verification Method**: Every line of generated code MUST be rendered in a headless browser (Playwright) to confirm zero-runtime-error status.
10+
- **Primary Goal**: Ensure that "Skill Injected" code (using AntV expert skills) outperforms "Baseline" code in correctness, configuration accuracy, and aesthetic quality.
11+
- **Verification Method**: Every line of generated code MUST be rendered in a headless browser (Playwright) via the dashboard's validation mode to confirm functional correctness.
1212

1313
## 🛠 Tech Stack
1414
- **Engine**: Node.js + TypeScript (`ts-node`)
1515
- **Agent Link**: Claude Code CLI (`claude` command)
1616
- **Validation**: Playwright (Headless Chromium)
1717
- **Visualization**: React (Vite) + Ant Design 5 + VisionSnap SDK
18-
- **Data**: JSON-based expert test cases (`datasets/s2-expert-cases.json`)
18+
- **Data**: JSON-based expert test cases located in `datasets/`.
1919

2020
## ⚡ Key Commands
2121
Agents should use the following commands to interact with the system:
2222

2323
| Command | Action | Description |
2424
|:---|:---|:---|
2525
| `npm run eval` | **Execution** | Starts the concurrent evaluation process. Spawns isolated Claude sessions. |
26-
| `npm run eval -- --limit 5` | **Dry Run** | Runs the evaluation for only the first 5 cases (for debugging). |
26+
| `npm run eval -- --limit 5` | **Dry Run** | Runs the evaluation for a limited number of cases. |
27+
| `npm run eval -- --skill g2` | **Switch Skill**| (Planned) Run evaluation for a specific AntV skill stack. |
2728
| `npm run dashboard` | **Visualization** | Starts the local dev server to view result reports at `localhost:5173`. |
2829

2930
## 🛡 Boundaries & Principles
3031

3132
### Always
3233
- Use **Isolated Sessions**: When spawning `claude`, always include `--session-id` with a unique UUID and `--no-session-persistence` to prevent context pollution between test cases.
33-
- **Extract Markdown**: Use robust regex to extract code blocks between ` ```tsx ` markers from the model's stdout.
34-
- **Concurrent Execution**: Utilize the built-in `pMap` concurrency (dynamically calculated based on system memory) to avoid running 100+ cases serially.
34+
- **Extract Markdown**: Use robust regex to extract code blocks from the model's stdout. Ensure the extraction logic handles various markdown flavors.
35+
- **Concurrent Execution**: Utilize system-memory-aware concurrency to run batch evaluations efficiently without overloading the local machine or hitting API rate limits.
3536

3637
### Never
37-
- **Never rely on local file writes** inside the model evaluation prompt. Models must be instructed via `STRICT_SYSTEM_PROMPT` to only output code to `stdout`.
38-
- **Never use hardcoded sleep** for iframe rendering. Always use `waitForSelector` for reliable headless validation.
38+
- **Never rely on local file writes** inside the evaluation prompt. Models must be strictly instructed to output code to `stdout` only.
39+
- **Never use hardcoded sleep** for iframe rendering. Always use dynamic `waitForSelector` or event-based triggers for reliable validation.
3940

4041
### Ask First
41-
- If `os.totalmem()` detection fails or the machine has less than 8GB RAM, ask before running high-concurrency tasks.
42+
- If the target dataset for a specific skill is missing in `datasets/`, ask before proceeding with a default fall-back.
4243

4344
## 📂 Directory Structure Context
44-
- `run_eval.ts`: The main brain/scheduler. Orchestrates Claude, handles concurrency, and saves reports.
45-
- `validate.ts`: The headless judge. Boots the dashboard in `?mode=validate` to check if code rendering succeeds.
46-
- `datasets/`: The source of truth for benchmark prompts.
47-
- `results/`: The output warehouse of evaluation history.
48-
- `dashboard/`: The UI interface for human-in-the-loop review.
45+
- `run_eval.ts`: The main orchestrator. Handles Claude spawning, concurrency limits, and result aggregation.
46+
- `validate.ts`: the rendering judge. Connects to the dashboard to perform browser-level checks.
47+
- `datasets/`: Storage for benchmark prompts (e.g., `s2-expert-cases.json`, `g2-expert-cases.json`).
48+
- `results/`: Historical evaluation reports.
49+
- `dashboard/`: The UI dashboard for comparing results and performing manual audits.

0 commit comments

Comments
 (0)