A real-time network intrusion detection system that combines a Random Forest classifier trained on NSL-KDD with live packet capture, SHAP-based explainability, MITRE ATT&CK mapping, and a Streamlit dashboard.
- Live packet capture via Scapy, with NSL-KDD-style feature extraction computed in real time using rolling 2-second and 100-connection windows
- Binary anomaly detection (Normal vs Anomaly) using a Random Forest classifier
- Multi-class attack categorization (DoS, Probe, R2L, U2R) via a second Random Forest trained on attack subtypes
- Severity scoring (Low / Medium / High) based on model confidence
- SHAP explainability — every flagged anomaly includes the top 3 features that drove the prediction, with direction and magnitude
- MITRE ATT&CK mapping — attack categories are mapped to representative ATT&CK technique IDs (e.g. T1498 for DoS)
- Dual persistence — recent activity in
live_logs.csvfor the live dashboard view, full history innids_logs.db(SQLite) for historical analytics - Streamlit dashboard — live feed, attack type breakdown, top attacking IPs, and an all-time historical sidebar
- Adversarial robustness testing — a standalone script that measures how detection rate degrades under realistic evasion techniques (low-and-slow, service diversification, error-rate masking)
- Dockerized — sniffer and dashboard run as separate containers via docker-compose
Live traffic is captured by the sniffer container, which extracts NSL-KDD-style features, runs them through the binary and category models, computes SHAP explanations and MITRE mappings for anomalies, and writes results to both CSV and SQLite. The dashboard container reads from both stores to render the live feed and historical analytics.
Live traffic
|
v
[sniffer container]
Feature extraction
|
v
Binary RF model --------> Category RF model
| |
v v
SHAP explainer MITRE ATT&CK mapping
\____________ ____________/
\/
Log writer
/ \
v v
live_logs.csv nids_logs.db
(last 100) (full history)
\ /
v v
[dashboard container]
Live feed view | Sidebar analytics
|
v
Browser: localhost:8501
ai-nids/
├── data/
│ ├── KDDTrain+.txt
│ └── KDDTest+.txt
├── train.py # Trains binary + category RF models, saves artifacts
├── live_sniffer_advanced.py # Real-time packet capture, inference, logging
├── app_adv.py # Streamlit dashboard
├── adversarial_test.py # Standalone evasion/robustness evaluation
├── nids_model.pkl # Binary anomaly detector (generated)
├── category_model.pkl # Attack category classifier (generated)
├── label_encoder.pkl # Category label encoder (generated)
├── scaler.pkl # Feature scaler (generated)
├── expected_columns.pkl # Feature column order (generated)
├── feature_importance.png # Top-15 feature importance plot (generated)
├── live_logs.csv # Last 100 packets (generated, runtime)
├── nids_logs.db # Full history SQLite database (generated, runtime)
├── Dockerfile
├── docker-compose.yml
├── requirements.txt
└── .dockerignore
Place KDDTrain+.txt and KDDTest+.txt in a data/ directory at the project root.
python3 train.pyThis produces nids_model.pkl, category_model.pkl, label_encoder.pkl, scaler.pkl, expected_columns.pkl, and feature_importance.png. It also prints classification reports for both the binary and category models.
docker compose up --buildOpen http://localhost:8501 for the dashboard. The sniffer container requires host networking and NET_ADMIN/NET_RAW capabilities to capture packets, which only works on Linux.
sudo python3 live_sniffer_advanced.py
streamlit run app_adv.pypython3 adversarial_test.pyThis crafts perturbed versions of real DoS attack samples — simulating low-and-slow pacing, service diversification, and TCP-handshake completion (SF instead of S0/REJ flags) — and reports how the detection rate degrades under each evasion strategy. The output documents the model's blind spots, particularly around Slowloris-style low-and-slow attacks that fall below the sniffer's connection-count thresholds.
- Feature extraction: for each captured packet, the sniffer computes the same 41 features NSL-KDD uses — connection counts, error rates, service/flag one-hot encodings, and rolling-window statistics over the last 2 seconds and last 100 connections.
- Binary classification: the scaled feature vector is passed to the Random Forest binary model, which outputs Normal or Anomaly with a confidence score.
- Thresholding and false-positive suppression: predictions below a confidence threshold are ignored; high-volume but well-formed traffic (e.g. large downloads — completed SF connections with high same-service rates) is excluded from anomaly flags to reduce false positives.
- Category classification: if flagged as an anomaly, a second Random Forest (trained only on attack samples, with balanced class weights) predicts the attack category — DoS, Probe, R2L, or U2R.
- Severity scoring: confidence is bucketed into Low (<65%), Medium (65-85%), and High (>85%) severity.
- Explainability: SHAP TreeExplainer computes per-feature contributions for the binary model's prediction; the top 3 by absolute value are logged as the "reasons" for the flag.
- MITRE ATT&CK mapping: the attack category is mapped to a representative ATT&CK technique (DoS -> T1498, Probe -> T1046, R2L -> T1110, U2R -> T1068).
- Logging: results are appended to
live_logs.csv(rolling, last 100 rows shown live) andnids_logs.db(full history, indexed by status, source IP, and date).
- Live traffic features (
duration,dst_bytes,logged_in) are approximated since they're not directly observable from a single packet — this creates some distribution mismatch versus NSL-KDD's training data. - The model was trained on NSL-KDD, a 1998-derived dataset; it reflects attack patterns from that era and may not generalize to modern attack techniques without retraining on more recent data.
- As shown by
adversarial_test.py, low-and-slow attacks that stay under the 2-second/100-connection window thresholds can evade detection — a known weakness of flow-statistic-based detectors.
Python, scikit-learn (Random Forest), SHAP, Scapy, pandas, SQLite, Streamlit, Docker.