STINER: Structured Threat Intelligence Named Entity Recognition

A comprehensive framework for Named Entity Recognition (NER) in Cyber Threat Intelligence (CTI), designed to extract structured information from Twitter/X social media posts.

Overview

This repository contains the complete pipeline for training and evaluating NER models on cyber threat intelligence data, including:

• Encoder-based models: BERT, RoBERTa, SecBERT, CySecBERT, DarkBERT (with/without CRF)
• Large Language Models: Llama-3.1, Gemma-2, Qwen2.5 (zero-shot & fine-tuned)
• GLiNER: Zero-shot evaluation

Repository Structure

STINER/
├── notebooks/          # Research pipeline (01-14)
├── data/
│   ├── processed/     # Train/val/test splits (shared)
│   └── tweet_ids.txt  # Tweet IDs for data rehydration
├── results/           # Benchmark results and figures
└── scripts/           # Utility scripts

Quick Start

Installation

# Clone repository
git clone https://github.com/ChammakhYasir/STINER.git
cd STINER

# Create environment
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Set Hugging Face token (for gated models)
export HF_TOKEN="your_token_here"

Data

The repository includes:

• ✅ data/processed/train.json, val.json, test.json - Annotated splits
• ✅ data/tweet_ids.txt - Tweet IDs for rehydration

To reproduce experiments, use the provided splits or rehydrate tweets using the IDs.

Running Experiments

# Core pipeline
01_data_preparation.ipynb      # Data preprocessing
02_data_splitting.ipynb         # Train/val/test split
03_dataset_analysis.ipynb       # Dataset statistics

# Model training & evaluation
04_encoder_training.ipynb       # Train encoder models
05_encoder_error_analysis.ipynb # Error analysis
06_encoder_crf_training.ipynb   # Encoder + CRF
07_gliner_evaluation.ipynb      # GLiNER zero-shot
08_llm_zero_shot_evaluation.ipynb  # LLM zero-shot
09_llm_finetuning.ipynb         # LLM fine-tuning (QLoRA)
10_performance_analysis.ipynb   # Per-entity metrics

# Optional: Twitter analysis
11_twitter_data_processing.ipynb      # Process raw tweets
12_threat_landscape_analysis.ipynb    # DarkBERT inference
13_twitter_data_cleaning.ipynb        # URL cleaning
14_europe_threat_analysis.ipynb       # Regional analysis

Entity Types

The STINER dataset includes 9 entity types:

• ACTOR - Threat actors/groups
• TARGET - Targeted organizations/individuals
• LOCATION - Geographic locations
• SECTOR - Industry sectors
• DATE - Temporal references
• CAMPAIGN - Operation/campaign names
• TOOL - Attack tools/malware
• DATA_TYPE - Types of data stolen/affected
• PRICE - Ransom amounts

Results

Benchmark results are available in results/tables/:

• encoder_benchmark_summary.csv - Encoder model results
• benchmark_results_encoders_crf.csv - Encoder+CRF results
• llm_benchmark_results.csv - LLM results

License

MIT License