|
| 1 | +# S2 Evaluation System - AGENTS.md |
| 2 | + |
| 3 | +This directory contains the automated evaluation pipeline for AntV S2. It is designed to benchmark and validate code generation capabilities of AI models (specifically Claude Code) against expert S2 patterns. |
| 4 | + |
| 5 | +--- |
| 6 | + |
| 7 | +## 🤖 Agent Role: S2 Evaluation Orchestrator |
| 8 | +The goal of the agents in this directory is to execute, monitor, and visualize the performance of S2-specialized code generation. |
| 9 | + |
| 10 | +- **Primary Goal**: Ensure that "Skill Injected" code (using S2 expert knowledge) outperforms "Baseline" code in correctness and aesthetic quality. |
| 11 | +- **Verification Method**: Every line of generated code MUST be rendered in a headless browser (Playwright) to confirm zero-runtime-error status. |
| 12 | + |
| 13 | +## 🛠 Tech Stack |
| 14 | +- **Engine**: Node.js + TypeScript (`ts-node`) |
| 15 | +- **Agent Link**: Claude Code CLI (`claude` command) |
| 16 | +- **Validation**: Playwright (Headless Chromium) |
| 17 | +- **Visualization**: React (Vite) + Ant Design 5 + VisionSnap SDK |
| 18 | +- **Data**: JSON-based expert test cases (`datasets/s2-expert-cases.json`) |
| 19 | + |
| 20 | +## ⚡ Key Commands |
| 21 | +Agents should use the following commands to interact with the system: |
| 22 | + |
| 23 | +| Command | Action | Description | |
| 24 | +|:---|:---|:---| |
| 25 | +| `npm run eval` | **Execution** | Starts the concurrent evaluation process. Spawns isolated Claude sessions. | |
| 26 | +| `npm run eval -- --limit 5` | **Dry Run** | Runs the evaluation for only the first 5 cases (for debugging). | |
| 27 | +| `npm run dashboard` | **Visualization** | Starts the local dev server to view result reports at `localhost:5173`. | |
| 28 | + |
| 29 | +## 🛡 Boundaries & Principles |
| 30 | + |
| 31 | +### Always |
| 32 | +- Use **Isolated Sessions**: When spawning `claude`, always include `--session-id` with a unique UUID and `--no-session-persistence` to prevent context pollution between test cases. |
| 33 | +- **Extract Markdown**: Use robust regex to extract code blocks between ` ```tsx ` markers from the model's stdout. |
| 34 | +- **Concurrent Execution**: Utilize the built-in `pMap` concurrency (dynamically calculated based on system memory) to avoid running 100+ cases serially. |
| 35 | + |
| 36 | +### Never |
| 37 | +- **Never rely on local file writes** inside the model evaluation prompt. Models must be instructed via `STRICT_SYSTEM_PROMPT` to only output code to `stdout`. |
| 38 | +- **Never use hardcoded sleep** for iframe rendering. Always use `waitForSelector` for reliable headless validation. |
| 39 | + |
| 40 | +### Ask First |
| 41 | +- If `os.totalmem()` detection fails or the machine has less than 8GB RAM, ask before running high-concurrency tasks. |
| 42 | + |
| 43 | +## 📂 Directory Structure Context |
| 44 | +- `run_eval.ts`: The main brain/scheduler. Orchestrates Claude, handles concurrency, and saves reports. |
| 45 | +- `validate.ts`: The headless judge. Boots the dashboard in `?mode=validate` to check if code rendering succeeds. |
| 46 | +- `datasets/`: The source of truth for benchmark prompts. |
| 47 | +- `results/`: The output warehouse of evaluation history. |
| 48 | +- `dashboard/`: The UI interface for human-in-the-loop review. |
0 commit comments