Skip to content

README.en.md

Latest commit

 

History

History
347 lines (262 loc) · 10.8 KB

README.en.md

File metadata and controls

347 lines (262 loc) · 10.8 KB
Raw

AI Agent Security Benchmarks

中文 | English

An extensible benchmark platform for evaluating the security capabilities of AI agents. The primary track is a set of reproducible Web CTF challenges, with support for integrating external ranges such as Vulhub and VulnHub.

Goals

This project does not measure one-off payload hit rate. It evaluates whether an AI agent can perform the full security workflow end to end:

  • Discover the attack surface
  • Enumerate endpoints and assets
  • Understand authentication, authorization, and business logic
  • Validate vulnerabilities and build exploit chains
  • Capture flags and supporting evidence
  • Complete tasks reliably under limited steps and context

Core Principles

  • Design the platform and challenges from first principles
  • Prioritize maintainability, reproducibility, and observability before scale
  • Decouple platform from challenges, and challenges from runtimes
  • Use Docker-based web challenges as the primary path, with VM ranges as an extension
  • Keep Web Console, CLI, and Agent Runner on the same control-plane API

Positioning

This repository is not just a collection of 20 Docker targets. It is an extensible AI Agent Security Benchmark Platform:

  • Primary source: native Docker web challenges
  • Extended sources: Vulhub, VulnHub, Custom Compose, and future K8s / remote runtimes
  • Access modes: Web Console + CLI + API
  • Target deployment: local or private deployment on Windows / macOS / Linux

V1 Scope

V1 focuses on the most stable path:

  • 20 native web CTF challenges
  • A unified challenge.yml specification
  • Docker runtime
  • Control plane backend
  • Web administration console
  • CLI
  • Checker / grader / reset toolchain

V1 explicitly does not target:

  • Fully automated imports for arbitrary external ranges
  • Multi-tenant SaaS deployment
  • Large-scale distributed scheduling
  • Complete VM compatibility

Current Status

The repository already contains the V1 backbone:

  • 20 native Docker web challenges across Tier 1 to Tier 5
  • Node.js / TypeScript control-plane backend
  • React web console
  • API-driven CLI
  • Full checker coverage
  • Batch start / stop / reset operations
  • Single-challenge and batch checker execution
  • A semi-automated Vulhub import skeleton

Validated results:

  • 20/20 challenge checkers passing
  • Every challenge supports docker compose up/down
  • The Web Console can display catalog, details, runtime status, health, and logs
  • The backend exposes single-challenge and batch lifecycle APIs

Default Credentials

On first startup, if the backend has no existing users, it bootstraps a default administrator:

  • Username: admin
  • Password: AdminPass123

You can override these with environment variables:

  • BENCH_DEFAULT_ADMIN_USERNAME
  • BENCH_DEFAULT_ADMIN_PASSWORD

The first login must change the password. If you plan to expose the platform to a shared network or the public internet, set explicit admin credentials instead of relying on the fallback defaults.

Quick Start

1. Start the Backend

cd platform/backend
npm install
npm run dev

2. Start the Frontend

cd platform/frontend
npm install
npm run dev

Default endpoints:

  • Backend: http://127.0.0.1:8000
  • Frontend: http://127.0.0.1:5173

If the backend starts with no existing users, it bootstraps a default admin account. The username and password come from BENCH_DEFAULT_ADMIN_USERNAME and BENCH_DEFAULT_ADMIN_PASSWORD; if unset, they fall back to admin and AdminPass123. The first login must change the password before accessing protected resources. The web console uses an HttpOnly browser session, while agents and the CLI use separately issued Agent-Token credentials.

If you plan to expose the platform to a shared network or the public internet, set explicit admin credentials instead of relying on the fallback defaults.

3. Use the CLI

cd platform/cli
npm install
npm run build
node dist/benchctl.js login --username admin --password AdminPass123
node dist/benchctl.js whoami
node dist/benchctl.js tokens
node dist/benchctl.js overview
node dist/benchctl.js list --tier 2
node dist/benchctl.js start --tier 1
node dist/benchctl.js stop
node dist/benchctl.js reset web-01-robots-leak
node dist/benchctl.js status web-11-ssrf-internal
node dist/benchctl.js targets web-11-ssrf-internal
node dist/benchctl.js logs web-11-ssrf-internal --tail 50
node dist/benchctl.js check --tier 1
node dist/benchctl.js import-compose /path/to/compose/project
node dist/benchctl.js import-vulhub /path/to/vulhub/project --id demo-vulhub
node dist/benchctl.js import-vulnhub /path/to/image.ova --id demo-vm
node dist/benchctl.js list-imports
node dist/benchctl.js show-import custom-compose my-import
node dist/benchctl.js preflight-import custom-compose my-import
node dist/benchctl.js update-import custom-compose my-import --benchmark-file /tmp/benchmark.json
node dist/benchctl.js validate-imports
node dist/benchctl.js register-import custom-compose my-import
node dist/benchctl.js unregister-import custom-compose my-import

3.1 Build Release Artifacts

node tools/platform/publish.mjs --version v0.2.0
node tools/platform/publish.mjs --version v0.2.0 --only backend,cli

By default this generates the following under artifacts/releases/<version>/:

  • backend/
  • cli/
  • frontend/
  • manifest.json

4. Run the Full Checker Suite Locally

python3 tools/checker/run_all.py
python3 tools/checker/run_all.py --tier 3
python3 tools/checker/run_all.py web-11-ssrf-internal web-20-frankenstack
python3 tools/checker/run_all.py --json --output artifacts/checks/latest.json

4.1 Generate Grading Reports

python3 tools/grader/grade.py
python3 tools/grader/grade.py --tier 1 --output-dir artifacts/grades/tier-1
python3 tools/grader/grade.py --tier 1 --trace-file artifacts/traces/sample.json --output-dir artifacts/grades/tier-1
python3 tools/platform/manage.py grade --tier 1 --trace-file artifacts/traces/sample.json --output-dir artifacts/grades/tier-1

You can also submit runner traces to the backend first and export them again in a grader-compatible bundle:

node platform/cli/dist/benchctl.js record-trace artifacts/traces/sample.json
node platform/cli/dist/benchctl.js list-runs --challenge-id web-01-robots-leak --limit 20
node platform/cli/dist/benchctl.js export-traces --output artifacts/traces/exported.json
python3 tools/grader/grade.py --trace-file artifacts/traces/exported.json

4.2 Hand the Platform Directly to an Agent

Recommended flow:

  1. Log in to the backend
  2. Generate a dedicated token
  3. Pass the token, the OpenAPI document URL, and agent instructions to the agent
  4. Let the agent solve challenges directly through the control-plane API

Recommended documents for the agent:

Key APIs exposed by the control plane:

  • GET /openapi.json
  • GET /api/challenges
  • GET /api/challenges/{id}
  • POST /api/start_challenge
  • POST /api/hint
  • POST /api/submit
  • POST /api/stop_challenge

Authentication header:

Agent-Token: <issued_token>

4.3 Optional: Run the Agent Runner

If you still need a local orchestration layer, the repository also ships a minimal agent runner:

python3 tools/agent/runner.py \
  --api-base-url http://127.0.0.1:8000 \
  --username admin \
  --password AdminPass123 \
  --solver-cmd "python3 tools/agent/example_solver.py" \
  --tier 1 \
  --limit 2

The runner executes the following sequence:

  • Fetch challenges
  • Start the challenge
  • Invoke the external solver
  • Submit the flag
  • Stop the challenge
  • Upload the trace

See also:

5. Generate a Vulhub Import Draft

node platform/cli/dist/benchctl.js import-compose /path/to/compose/project
python3 adapters/vulhub/importer.py /path/to/vulhub/project imports/vulhub/my-import
node platform/cli/dist/benchctl.js import-vulhub /path/to/vulhub/project --id my-import
node platform/cli/dist/benchctl.js import-vulnhub /path/to/image.ova --id my-vm

Import drafts created through the control plane and CLI are stored in SQLite. They are not treated as fully normalized benchmarks with completed checker, flag, or healthcheck metadata. External adapter scripts may still emit their own intermediate files when needed.

You can inspect and validate these drafts with:

node platform/cli/dist/benchctl.js list-imports
node platform/cli/dist/benchctl.js show-import custom-compose my-import
node platform/cli/dist/benchctl.js preflight-import custom-compose my-import
node platform/cli/dist/benchctl.js update-import custom-compose my-import --benchmark-file /tmp/benchmark.json
node platform/cli/dist/benchctl.js validate-imports
node platform/cli/dist/benchctl.js register-import custom-compose my-import

Documentation Index

Repository Layout

aiagentsec-benchmarks/
  README.md
  README.en.md
  docs/
    architecture.md
    challenge-spec.md
    control-plane.md
    roadmap.md
  platform/
    backend/
    frontend/
    cli/
  runtimes/
    docker/
    vm/
  adapters/
    native/
    custom-compose/
    vulhub/
    vulnhub/
  tools/
    checker/
    grader/
    reset/
    validate/
  challenges/
    web/
      tier-01/
      tier-02/
      tier-03/
      tier-04/
      tier-05/

Challenge Progression

The 20 web challenges follow a human penetration-testing path instead of being grouped by vulnerability name alone:

  • Tier 1: obvious entry point, single-step exploitation
  • Tier 2: basic enumeration and two-step exploitation
  • Tier 3: understanding authentication, frontend/backend behavior, and business logic
  • Tier 4: state control, multi-service interaction, or lateral movement
  • Tier 5: exploit chains, deeper reasoning, and internal pivoting

External Content Strategy

Vulhub

Suitable as a Docker-based extension source via vulhub-adapter.

VulnHub

Suitable as a VM-based extension source via vm-runtime + vulnhub-adapter, but not as the V1 mainline.

Why:

  • It mostly ships OVA / OVF / VMDK virtual machine images
  • It lacks unified reset, checker, and flag interfaces
  • Entry points commonly depend on DHCP and dynamic IPs
  • Cross-hypervisor and cross-platform compatibility is unstable

Next Steps

Recommended implementation order:

  1. Batch checker execution and persistent batch run results
  2. Agent trace collection and finer-grained scoring
  3. VM non-HTTP service probing and log collection
  4. Leaderboards and comparative reports