Log Analysis: Investigating Server Outages and Bot Traffic

This project investigates unusual traffic patterns and potential bot activity causing server outages for a small media startup. The analysis was carried out in multiple stages, each documented in its own numbered folder. Each folder represents the next step in narrowing down the sources of malicious, automated, or abnormal traffic.

---

Folder Overview

1) sample-log/

Purpose: Initial Exploration

• This was the starting point for analysis using the full, unfiltered logs.
• Scripts in this folder parsed raw logs to extract:
  • Total requests per hour
  • Top IPs and user agents
  • Frequently accessed endpoints
• Helped identify abnormal peaks in traffic volume.

---

2) No_Banned_UA/

Purpose: User-Agent Filtering

• Building on the initial analysis, this stage focused on identifying known malicious or suspicious user agents such as:
  • sqlmap, curl, Burp Suite, python-requests, etc.
• The logs were filtered to remove entries with these UAs.
• Discovered that these accounted for roughly 4.5 of total traffic.

---

3) No_Banned_User/

Purpose: Targeted Login Attempt Analysis

• With noisy user agents removed, patterns emerged around high-frequency login attempts.
• Certain usernames (e.g., frank.miller, admin, bob, etc.) were being targeted hundreds of times from multiple global IPs.
• Showed signs of brute-force or credential stuffing attacks.
• Suggested restricting login attempts by geo-IP or frequency.
• These inclusion constituted another 5% of the total traffic

---

4) No_SPAM/

Purpose: Clearing up the remaining spam present

• Investigated a massive traffic spike on 03/07/2025 at 12:00.
• Found that IP addresses in the 194.168.1.1 – 194.168.1.8 range generated a disproportionately high number of requests during peak hours.
• These requests used randomized or spoofed User Agents, allowing them to bypass UA-based filters, that may have been present.
• This final section also contributed around 4% of the total traffic

---

Summary

Each stage progressively removed noise and revealed deeper layers of suspicious activity. The result is a 3-stage filter:

1. User Agent filtering (static tool-based bots),
2. Login abuse filtering (credential stuffing attempts),
3. Local IP range filtering (randomized botnets or probes).

All scripts, logs, and output CSVs are preserved in their respective folders for traceability and reproducibility.

Also if your using the dockerfile use the command column -s, -t < file.csv | less -#2 -N -S to make the csv sppear in columns