Awesome LLM Red Teaming

A comprehensive list of tools and resources for red-teaming large language models (LLMs), vulnerability research, and reverse engineering using various techniques like prompt manipulation, adversarial testing, and model interpretation.

LLM Red Teaming Resources by Technical Function

Playgrounds and practice targets

• Folly - Open-source LLM prompt injection and jailbreak playground for hands-on experimentation.
• DamnVulnerableMathLLM - Intentionally vulnerable maths LLM app for practising prompt injection and logic exploits.
• DamnVulnerableShoppingLLM - Intentionally vulnerable shopping assistant scenario for testing data leakage and unsafe tool use.
• RedTeam Arena - Competitive jailbreak challenges that train you to bypass model guardrails under time pressure.
• MyLLM Bank - WithSecure CTF where you exploit a multi-LLM banking app to learn real-world prompt injection chains.
• MyLLM Doctor - WithSecure medical chatbot challenge showing multi-stage prompt injection and policy evasion.
• Gandalf by Lakera - Public challenge where you try to extract a secret from an aligned model despite protections.

Red teaming frameworks and agent harnesses

• PyRIT - Microsoft framework for systematic, repeatable red teaming of generative AI systems with automation support.
• Rigging - Lightweight agent and tool-calling harness for orchestrating offensive LLM experiments in controlled environments.

Attack generation and jailbreak toolkits

• BrokenHill - Implementation of the Greedy Coordinate Gradient attack that automatically crafts adversarial prompts.
• llm-attacks - Research code for universal adversarial suffixes that transfer across aligned LLMs.
• Spikee - Simple prompt injection toolkit for probing and exploiting LLM applications.

Obfuscation and payload encoding utilities

• Invisible Unicode Tags Playground - Converts text to invisible Unicode tags to smuggle hidden instructions past filters.
• Emoji Variation Selector Playground - Hides data in emoji variation selectors to evade naïve content checks.
• ASCII to Unicode Character Reducer - Re-encodes ASCII as compact Unicode lookalikes to defeat brittle filters.

Traffic interception and support tooling

• LLM-itM - Man-in-the-middle proxy for OpenAI-compatible APIs that lets you inspect and modify prompts and responses.

Defences, standards and guardrails to test against

• Anthropic Constitutional Classifiers - Rule-based alignment classifiers that red teamers can evaluate and attempt to bypass.
• Prompt Guard 86M - Small model for detecting unsafe prompts that can be assessed and circumvented.
• OWASP Top 10 for LLMs - Community guidance on common LLM application risks and mitigations.

Research papers and technical articles

• Best-of-N Jailbreaks - Black-box strategy that samples many prompts and selects those most likely to bypass guardrails.
• Universal and Transferable Attacks - Demonstrates universal adversarial suffixes that elicit unsafe outputs across models.
• AutoDAN - Dual-LLM method that crafts gradient-inspired adversarial prompts to defeat filters.
• Faster-GCG - Efficiency improvements that speed up discrete optimisation for jailbreak prompts.
• GenBFA - Shows bit-flip weight corruption can induce malicious behaviour in generative models.
• Manipulating LLM Recommendations - Examines how adversarial prompts can game LLM-driven recommenders.
• GCG on OpenReview - Peer-review discussion of coordinate gradient jailbreak optimisation.
• GCG explainer by Nightfall - Plain-language overview of GCG and why it works.
• BrokenHill overview by Bishop Fox - Practitioner write-up on applying GCG in the wild.
• ACG by Haize Labs - Introduces an accelerated variant that dramatically reduces attack time.

Case studies and exploitation write-ups

• Microsoft on evolving guardrail attacks - Details jailbreak patterns observed in the wild and defensive responses.
• Lessons from red teaming 100 AI products - Summarises lessons from large-scale AI product red teaming.
• Multi-chain prompt injection attacks - Explains chaining attacks across stages in complex LLM systems.
• Context pollution and delayed tool use - Shows how polluting context can trigger dangerous automated tool use later.
• Terminal ANSI sequence abuse - Demonstrates how hidden escape codes in outputs can lead to local code execution.
• Air Canada chatbot case - Real-world legal impact when an AI system produces binding misinformation.

Bug bounties and programmes

• Microsoft AI Bounty - Official programme that accepts AI vulnerability reports with rewards.
• 0DIN AI - Platform focused on GenAI bug bounties and evaluations.

Contributing

Your contributions are always welcome! Please read the contribution guidelines first. We follow the Contributor Covenant Code of Conduct. Please make sure to review and adhere to this code of conduct when contributing.

License

This project is licensed under the MIT License - see the LICENSE.md file for details.