StructTransform Bench: Evaluating Structure Transformation Attacks on Safety-Aligned LLMs

—— A simple benchmark for evaluating safety-alignment of models against structure transformation attacks ——

Table of Contents

• StructTransform Bench: Evaluating Structure Transformation Attacks on Safety-Aligned LLMs
  • Table of Contents
  • Updates
  • About
    • What is StructTransform?
  • Benchmark Transformations
    • Notes
    • Content Transformations
    • Structure Transformations
  • Experimental results
  • ⚙️ Setup
    • Judge Models
  • 🚀 Usage
    • Configuration Setup
    • Environment Variables
    • Key Configuration Options
    • Example Workflow
    • Reproducing Results
  • 🖊️ Citing StructTransform

Updates

• (2025-01-26) Initial release

About

What is StructTransform?

See example outputs from our attacks here.

StructTransform is an advanced attack technique targeting safety-aligned Large Language Models (LLMs) by transforming malicious intents into alternative syntaxes and computational structures. Unlike traditional adversarial prompts that rely on natural language variations, StructTransform encodes harmful prompts into structured formats such as SQL, JSON, and even novel syntaxes generated by LLMs. This approach combined with existing jailbreaks (e.g. encoding and roleplay) significantly expands the attack surface and exposes vulnerabilities in current safety mechanisms. To aid in evaluating and improving model robustness, StructTransform Bench systematically assesses LLMs' defenses against these structural transformations, highlighting critical gaps in alignment techniques.

Benchmark Transformations

Structure / Transformation	No Transformation	Encoding	Roleplay	Encoding + Roleplay
JSON	✅	✅	✅	✅
SQL	✅	✅	✅	✅
Cypher	✅	✅	✅	✅
SymLogix*	✅	✅	✅	✅

Notes

1. SymLogix
   SymLogix is a custom syntax generated through GPT‑4o.

2. Encoding
   Encoding includes both utf-16 for complex models (corresponding to combination_1), and url-based for smaller models (corresponding to combination_2).

Content Transformations

• No Transformation
  Raw payload with no modifications.
• Encoding
  Obfuscates payload using UTF-16/url-based, making detection harder.
• Roleplay
  Wraps payload in a playful or misleading storyline to avoid scrutiny.
• Encoding + Roleplay
  Combines both obfuscation and storyline techniques.

Structure Transformations

• JSON
  Standard JSON-based payload.
• SQL
  SQL-based statements with different potential for injection.
• Cypher
  Used for graph databases (e.g., Neo4j).
• SymLogix
  A custom syntax generated with GPT‑4o.

Experimental results

Our StructTransform benchmark achieves 100% Attack Success Rate (ASR) on most state-of-the-art safety-aligned models, including Claude 3.5 when using adaptive attacks that combine structure and content transformations. We further get >75% ASR on o1. Simple transformations like SQL or JSON formats also yielded high ASRs, showcasing the benchmark's ability to expose vulnerabilities across diverse syntactic domains. The results, summarized in the figure below, highlight the critical gaps in existing alignment techniques.

⚙️ Setup

The project requires python>=3.9. Set up the project by following the below instructions.

1. Clone the repo

git clone https://github.com/StructTransform/Benchmark.git

2. Navigate to project directory and install requirements

cd Benchmark
pip install -r requirements.txt

3. The implementation is built on Easyjailbreak, and hence requires installation of easyjailbreak module through the below command:

pip install -e .

4. We assume the judge models are accessible via OpenAI API.

Judge Models

We rely on the following judge models:

• cais/HarmBench-Llama-2-13b-cls: Llama-2-13b model that is trained to detect harmful content.
• allenai/wildguard: Mistral-7B model that is trained to detect harmful request, harmful response and refusal. We only use the refusal detection component.

We used vllm to host the judge models and OpenAI API to make requests.

🚀 Usage

The benchmark relies on config.yaml, as this will be the main method of configuring the benchmark. More instruction are given below on running the benchmark.

Configuration Setup

Update the config.yaml file with your experiment settings. Here's the structure:

# Define the models and their configuration
models:
  target:
    model_name: "anthropic/claude-3.5-sonnet"
    base_url: "https://openrouter.ai/api/v1/"
    api_key_env: "OPENROUTER_API_KEY"
    generation_config:
      temperature: 0.0
      max_tokens: 1024

  # NOT needed if only running benchmark tests, replace if using attack generation with deepseek
  attack:
    model_name: "dummy model"
    base_url: "https://www.dummy.com/"
    api_key_env: "API_KEY"
    generation_config:
      temperature: 1
      top_p: 0.9
      max_tokens: 1024

  eval:
    model_name: "cais/HarmBench-Llama-2-13b-cls"
    base_url: "http://localhost:9200/v1"
    api_key_env: "EMPTY"
    generation_config:
      # Do NOT change
      temperature: 0.0
      max_tokens: 1

  refusal_eval:
    model_name: "allenai/wildguard"
    base_url: "http://localhost:9201/v1"
    api_key_env: "EMPTY"
    generation_config:
      # Do NOT change
      temperature: 0.0
      max_tokens: 20


attack_settings:
  # 1 = use hex encoding for more capable models
  # 2 = use url encoding for smaller models
  combination_mode: 2

structures:
  # Comment out the structures you don't want included in test
  JSON: "benchmark/json_dataset.pkl"
  SQL: "benchmark/sql_dataset.pkl"
  Cypher: "benchmark/cypher_dataset.pkl"
  SymLogix: "benchmark/symlogix_dataset.pkl"

paths:
  # path where the results should be saved
  save_path: "benchmark_results/claude-3.5-sonnet"

Environment Variables

Set required API keys in your environment:

# For target model (OpenRouter)
export OPENROUTER_API_KEY="your_openrouter_api_key"

# (optional) For attack model (DeepSeek)
export DEEPSEEK_PLATFORM_API_KEY="your_deepseek_api_key"

3. Execute the benchmark with:

python examples/run_StructuredAttack.py

Key Configuration Options

Section	Key	Description
models	api_key_env	Environment variable name containing API key
attack_settings	combination_mode	1: Test combination_1, 2: Test combination_2
structures	SQL, JSON, Cypher, SymLogix	Include keys for each structure that will part of the run, exclude otherwise. Paths to structure-specific dataset files
paths	save_path	Root directory for saving all experiment results

---

Example Workflow

# 1. Update config.yaml based on setup
nano config.yaml  

# 2. Set environment variables
export OPENROUTER_API_KEY="sk-your-key-here"
export DEEPSEEK_PLATFORM_API_KEY="sk-your-deepseek-key"

# 3. Run the benchmark
python examples/run_StructuredAttack.py

Results will be saved in nested directories under paths.save_path following this structure:

benchmark_results/
└── claude-3.5-sonnet/
    ├── JSON/
    ├── JSONCombined/
    ├── SQL/
    └── ...

Reproducing Results

• For reproducibility, we provide pickle files containing the Deepseek-v3 generated structure transformation. Structure transformations can be re-generated by uncommenting the following line in examples/run_StructuredAttack.py:

attacker.generate_attack_prompts(pkl_file_path)

• For convenience, all the generated artifacts will be uploaded to a cloud folder.
• Content warning: the pickle files and artifacts contain text that is offensive, harmful, or otherwise inappropriate in nature.

🖊️ Citing StructTransform

Under Review