Feature request: Add AgentThreatBench (OWASP Agentic Top 10) as a built-in security benchmark

[vgudur-dev]

Summary

AgentThreatBench is the first benchmark suite that operationalizes the OWASP Top 10 for Agentic Applications (2026) into executable evaluation tasks. It was merged into UKGovernmentBEIS/inspect_evals, the official evaluation suite maintained by the UK AI Safety Institute.

What it benchmarks

Three tasks targeting distinct agentic attack surfaces:

• Memory Poison (ASI06) — Tests whether agents correctly answer questions from a memory store that contains adversarial entries (direct injection, context poisoning, gradual poisoning, authority impersonation, delimiter escape, role hijack)
• Autonomy Hijack (ASI01) — Tests whether agents performing email triage resist indirect instruction injection embedded in email content returned by tools
• Data Exfiltration (ASI01) — Tests whether a customer-support agent can be redirected via indirect injection in lookup_customer output into leaking SSNs/account numbers via send_message

Scoring approach

Uses a dual-metric approach: utility (task completion) + security (attack resistance) scored independently. This maps well to deepeval's metric architecture.

Proposal

Add AgentThreatBench as a built-in security benchmark in deepeval, similar to how deepeval already supports custom red-team metrics. The dataset is open-source.

Benchmark docs: https://ukgovernmentbeis.github.io/inspect_evals/evals/safeguards/agent_threat_bench/
Source: https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/agent_threat_bench
OWASP reference: https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/

[vgudur-dev]

Hi @penguine-ip — tagging you as the active maintainer(s) here to ask if you'd be willing to take a look at this request.

Quick context: deepeval already supports custom metrics and red-team attack modules — AgentThreatBench's 10 task categories map cleanly to deepeval's AttackModule interface.

The ask: an AgentThreatBench attack module or a reference in the security evaluation docs

AgentThreatBench is already merged into the UK AI Safety Institute's official inspect_evals suite and is installable via pip install inspect_evals. The benchmark covers 10 executable tasks aligned to the OWASP Top 10 for Agentic Applications 2025.

Happy to provide a prototype integration, a code snippet, or a draft docs PR — whatever would be most useful. Would love to know if this is something the team would consider.