This repository provides an implementation of WISA, an adversarial attack framework designed for text summarization models such as Pegasus, BART, and T5. The core idea is to identify and perturb important input regions—sentences and words—based on ROUGE-driven scoring and guided synonym substitution using large language models (LLMs) like Llama3.
- 🧠 Sentence- and word-level importance ranking based on ROUGE-1 F1 drop
- 🎯 Weighted Sampling to prioritize impactful perturbations
- 🔁 LLM-based synonym substitution using locally served models via Ollama
- 📊 Comprehensive evaluation metrics:
- ROUGE (1/2/L)
- BLEU
- Exact Match (EM)
- F1 Score
- Semantic Similarity (Sentence-BERT)
- Perplexity (GPT-2)
- Modification Rate
- 🧩 Model-agnostic summarization via HuggingFace Transformers (Pegasus, BART, T5)
Requires Python 3.8+
- Clone the repository and install dependencies:
pip install -r requirements.txt
- Download NLTK sentence tokenizer:
python -m nltk.downloader punkt
WISA leverages local LLMs (e.g., Llama3) via Ollama.
Visit: https://ollama.com
Follow platform-specific instructions to install the Ollama runtime.
ollama serve # or ./ollama serve
ollama pull llama3 # download Llama3 locally
ollama run llama3
Alternatively, run from Python:
python llms.py
To launch the attack:
python attack.py
- Make sure to edit attack.py to adjust hyperparameters such as:
- top_k sentences
- LLM model name (llama3)
- sampling size
- batch size, etc.
You may also test individual modules directly:
- Sentence importance scoring:
python sis.py
- Word importance analysis:
python wis.py
- Word substitution and synonym generation:
python syn_model.py
- Metric evaluation preview:
python metrics.py
- HuggingFace Transformers
- Sentence-Transformers
- Ollama for local LLM support
- GPT-2 for Perplexity calculation
- BERT for synonym replacement