v3.5.0

What's New

🚀 Attacks

• Add Semantic Game attack (#178, thanks @rudiandradi)

We Need Your Feedback

If you have suggestions, encounter any issues, or want to share your experiences using LLAMATOR, please don't hesitate to reach out! You can find us in Telegram: @llamator

v3.4.0

What's New

🛠 Improvements

• Refactor test preset functions to improve clarity.
• Improve saving attacker's and client's answers, including empty tested client answer in case of error.
• Rename get_tested_client_prompts into get_attack_prompts.

🚀 Attacks

• Add Composition of Principles (CoP) attack.
• Add Repetition Token Attack (OWASP LLM10:2025 Unbounded Consumption).

We Need Your Feedback

If you have suggestions, encounter any issues, or want to share your experiences using LLAMATOR, please don't hesitate to reach out! You can find us in Telegram: @llamator

v3.3.0

What's New

🛠 Improvements

1. Redesigned the output of testing parameter presets. Added the following presets: all, owasp:llm01, owasp:llm07, owasp:llm09, llm, vlm, eng, rus.
2. Add new tag - model: llm / vlm
3. README update - Enterprise Version announce

🚀 Attacks

1. Added a new Linguistic Sandwich attack. An adversarial prompt in a low-resource language is sandwiched between benign prompts in other languages.
2. In the System Prompt Leakage attack, the heuristiс evaluation has been replaced with LLM-as-a-judge. This checks the similarity between the system's output and the intended prompt based on the system description.
3. The static Past Tense attack has become the dynamic Time Machine attack. The attacking model now alters the temporal context of the adversarial prompt.

We Need Your Feedback

If you have suggestions, encounter any issues, or want to share your experiences using LLAMATOR, please don't hesitate to reach out! You can find us in Telegram: @llamator

v3.2.0

What's New

🚀 New Attacks

• Added Deceptive Delight (thanks @EgorovM)
• Added Dialogue Injection Continuation (thanks @3ndetz)
• Added VLM Lowres PDFs Attack
• Added VLM M-Attack
• Added VLM Text Hallucination Attack

🧠 VLM Support

• Introduced support for Vision Language Model (VLM) attacks, expanding the framework’s multimodal testing capabilities. Thanks @ti3c2 and @svyatocheck for these cool attacks!

🛠 Improvements

• Added Dialogue Injection Developer Mode (formerly "Dialog Injection")
• Renamed Harmful Behavior Multistage to PAIR and add scoring with the Judge Model
• Revised and translated Harmbench dataset into Russian
• Added language column to datasets and enabled filtering attacks by language
• Updated start_testing to return a dictionary object with test results for using in CI/CD pipeline

🔥 Removed

• Removed Complimentary Transition
• Removed Typoglycemia Attack
• Removed legacy RU_* attacks (now handled via language-based dataset filtering)

We Need Your Feedback

If you have suggestions, encounter any issues, or want to share your experiences using LLAMATOR, please don't hesitate to reach out! You can find us in Telegram: @llamator

v3.1.0

What's New

• Add Autodan Turbo Attack (2410.05295v3) – thanks @wearetyomsmnv for initial code!
• Add Dialogue Injection Attack (2503.08195) – thanks @3ndetz!
• Enhance documentation and add judge model validation checks
• Switch parquet engine from fastparquet to pyarrow

We Need Your Feedback

If you have suggestions, encounter any issues, or want to share your experiences using LLAMATOR, please don't hesitate to reach out! You can find us in Telegram: @llamator

v3.0.0

What's New

Killer Features

• Add a new config for the judge model, allowing it to be specified as a separate model
• Add Shuffle Inconsistency attack (Original Paper: https://arxiv.org/html/2501.04931)
• Change the way of setting parameters for the test start function: attack class now includes dictionaries with descriptions of various aspects of an attack
• Add to attacks with datasets custom parameter for another dataset

Important Improvements

• Add a function for displaying templates with written attack presets;
• Add verification for attack parameters;
• Add handling for emergency attack stoppages;
• Refactor judge models interaction for Ethical Compliance, Logical Inconsistencies, Sycophancy tests;
• Improve console output and progress bars;
• Update the logging order of attack steps;
• Update LangChain versions;
• Update examples in Jupyter notebooks;

We Need Your Feedback

If you have suggestions, encounter any issues, or want to share your experiences using LLAMATOR, please don't hesitate to reach out! You can find us in Telegram: @llamator

v2.3.1

What's New

• Add video guides about Red Teaming and LLAMATOR (thanks @RomiconEZ)
• Update Documentation: copyright, guides section
• Fix null checking for multistage attacks (thanks @nizamovtimur)
• Enhance sycophancy

We Need Your Feedback

If you have suggestions, encounter any issues, or want to share your experiences using LLAMATOR, please don't hesitate to reach out! You can find us in Telegram: @llamator

v2.2.0

What's New

• Add Suffix Attack and New System Prompt Leakage Requests (we're happy to see in contributors @Shine-afk)
• Add HarmBench Prompts to Harmful Behavior Attack (thanks @NickoJo)
• Other minor improvements and bug fixes

We Need Your Feedback

If you have suggestions, encounter any issues, or want to share your experiences using LLAMATOR 2.1.0, please don't hesitate to reach out! You can find us in Telegram: @llamator

v2.1.0

What's New

• Add BON attack (@NickoJo)
• Add Crescendo attack (@nizamovtimur)
• Add Docker example with Jupyter Notebook and installed LLAMATOR (@RomiconEZ)
• Improve attack system prompt for Prompt Leakage (@nizamovtimur)
• Other minor improvements and bug fixes

We Need Your Feedback

If you have suggestions, encounter any issues, or want to share your experiences using LLAMATOR 2.1.0, please don't hesitate to reach out! You can find us in Telegram: @llamator

v2.0.1

What's New

• Add the strip_client_responses parameter for ChatSession
• Other small improvements in attacks