Prepare 3.0.0 for testing

Author: t3chn0m4g3

.github/copilot-instructions.md

@@ -0,0 +1,118 @@
+# T-Pot Attack Map - AI Coding Instructions
+
+## Project Overview
+Real-time cyber attack visualization tool for the [T-Pot honeypot system](https://github.com/telekom-security/tpotce). Python backend processes honeypot events from Elasticsearch and streams to a Leaflet/D3.js frontend via WebSocket.
+
+## Architecture & Data Flow
+**Producer-Consumer model decoupled by Redis Pub/Sub:**
+
+1. **Data Source**: Elasticsearch with `logstash-*` indices (T-Pot honeypot events)
+2. **Producer** ([DataServer.py](../DataServer.py)):
+   - Polls Elasticsearch every 0.5s for last 100 events
+   - Queries stats every 10s (1m/1h/24h aggregations)
+   - Maps ports → protocols via `port_to_type()` function
+   - Publishes JSON to Redis channel `attack-map-production`
+   - **Synchronous** (`redis.StrictRedis`, blocking operations)
+3. **Consumer & Web Server** ([AttackMapServer.py](../AttackMapServer.py)):
+   - aiohttp server on port 1234 (configurable via `web_port`)
+   - Subscribes to Redis channel, forwards to WebSocket clients
+   - Serves static files from `static/` directory
+   - **Fully async** (`redis.asyncio`, aiohttp, asyncio)
+4. **Frontend** ([static/map.js](../static/map.js), [dashboard.js](../static/dashboard.js)):
+   - WebSocket client connects to `/websocket`
+   - Leaflet.js map (CartoDB basemaps, dark/light themes)
+   - D3.js v7 for animated attack lines and circles
+   - IndexedDB cache with LocalStorage fallback (24h retention)
+
+## Critical Developer Patterns
+
+### Dual Configuration Toggle
+Connection strings are **hardcoded** with production/local variants:
+```python
+# Production (inside T-Pot Docker): 
+# redis_url = 'redis://map_redis:6379'
+# Local development:
+redis_url = 'redis://127.0.0.1:6379'
+```
+**Rule**: Always preserve both configurations. Active config is uncommented, production config is commented out.
+
+### Data: service_rgb Dictionary
+**Note**: Protocol-to-color mapping exists only in **DataServer.py**:
+- [DataServer.py](../DataServer.py#L30-L85) lines 30-85
+
+**Adding a protocol**:
+1. Update `port_to_type()` in DataServer.py
+2. Add color to `service_rgb` in DataServer.py
+
+### Async/Sync Boundary
+- **AttackMapServer.py**: 100% async (use `await`, `asyncio.create_task`)
+- **DataServer.py**: 100% sync (no async/await, uses `time.sleep()`)
+- Redis clients are different: `redis.asyncio` vs `redis.StrictRedis`
+
+### Frontend Animation Management
+[map.js](../static/map.js) handles D3 animations with visibility checks:
+- Global `isPageVisible` flag prevents animation backlog when tab hidden
+- `isWakingUp` grace period (1s) suppresses burst on tab resume
+- D3 elements cleared on zoom to prevent coordinate desync
+- Use `d3.easeCircleIn` for consistent easing
+
+## Data Schema
+
+**WebSocket message types**:
+```javascript
+{type: "Traffic", ...}  // Individual attack event
+{type: "Stats", ...}    // Aggregate statistics (1m/1h/24h)
+```
+
+**Attack event fields** (DataServer.py, lines 200-233):
+- `src_ip`, `src_lat`, `src_long`, `src_port`, `iso_code`
+- `dst_ip`, `dst_lat`, `dst_long`, `dst_port`, `dst_iso_code`
+- `protocol`, `color`, `honeypot`, `event_time`, `ip_rep`
+- `country`, `continent_code`, `tpot_hostname`
+
+## Environment & Dependencies
+
+**Python setup** (use virtual environment):
+```bash
+source bin/activate  # Virtual env already exists
+pip install -r requirements.txt
+```
+
+**Dependencies** ([requirements.txt](../requirements.txt)):
+- `aiohttp` (async web server)
+- `elasticsearch==8.18.1` (specific version!)
+- `redis` (includes asyncio support)
+- `pytz`, `tzlocal` (timezone handling)
+
+**Frontend assets** (all local in `static/`):
+- Leaflet.js, D3.js v7, Bootstrap 5
+- Font Awesome, custom fonts (Inter, JetBrains Mono)
+- Flagpack icons in `static/flags/`
+
+## Common Workflows
+
+### Local Development Setup
+1. **Start Redis**: `redis-server` (port 6379)
+2. **Elasticsearch**: SSH tunnel or local instance on port 64298
+   ```bash
+   ssh -L 64298:localhost:9200 tpot-server
+   ```
+3. **Data Server**: `python3 DataServer.py` (terminal 1)
+4. **Web Server**: `python3 AttackMapServer.py` (terminal 2)
+5. **Access**: http://localhost:1234
+
+### Debugging Data Flow
+- **No attacks showing**: Check DataServer.py console for Elasticsearch errors
+- **WebSocket disconnects**: Check Redis connection in AttackMapServer.py
+- **Protocol showing as OTHER**: Add port mapping in `port_to_type()`
+
+### Theme Development
+- HTML `data-theme` attribute toggles dark/light
+- Map tiles auto-switch via `mapLayers` object ([map.js](../static/map.js#L37-L51))
+- CSS custom properties defined in [index.css](../static/index.css)
+
+## Performance Gotchas
+- **Elasticsearch query size**: Limited to 100 events per poll (DataServer.py line 189)
+- **Cache limits**: Max 10,000 events in IndexedDB ([dashboard.js](../static/dashboard.js#L14))
+- **Animation throttling**: Skip D3 animations when tab hidden (prevents memory bloat)
+- **Redis pubsub**: Single channel `attack-map-production`, all clients receive all events

.gitignore

@@ -0,0 +1,7 @@
+# Created by venv; see https://docs.python.org/3/library/venv.html
+bin/
+include/
+lib/
+pyvenv.cfg
+__pycache__/
+

AttackMapServer.py

@@ -13,33 +13,16 @@
 
 # Configuration
 # Within T-Pot: redis_url = 'redis://map_redis:6379'
-# redis_url = 'redis://127.0.0.1:6379'
-# web_port = 1234
+#redis_url = 'redis://127.0.0.1:6379'
+#web_port = 1234
 redis_url = 'redis://map_redis:6379'
 web_port = 64299
-version = 'Attack Map Server 2.2.6'
+version = 'Attack Map Server 3.0.0'
+
 
-# Color Codes for Attack Map
-service_rgb = {
-    'FTP': '#ff0000',
-    'SSH': '#ff8000',
-    'TELNET': '#ffff00',
-    'EMAIL': '#80ff00',
-    'SQL': '#00ff00',
-    'DNS': '#00ff80',
-    'HTTP': '#00ffff',
-    'HTTPS': '#0080ff',
-    'VNC': '#0000ff',
-    'SNMP': '#8000ff',
-    'SMB': '#bf00ff',
-    'MEDICAL': '#ff00ff',
-    'RDP': '#ff0060',
-    'SIP': '#ffccff',
-    'ADB': '#ffcccc',
-    'OTHER': '#ffffff'
-}
 
 async def redis_subscriber(websockets):
+    was_disconnected = False
     while True:
         try:
             # Create a Redis connection
@@ -49,22 +32,29 @@ async def redis_subscriber(websockets):
             # Subscribe to a Redis channel
             channel = "attack-map-production"
             await pubsub.subscribe(channel)
-            print("[*] Redis connection established.")
+            
+            # Print reconnection message if we were previously disconnected
+            if was_disconnected:
+                print("[*] Redis connection re-established")
+                was_disconnected = False
+            
             # Start a loop to listen for messages on the channel
             while True:
                 message = await pubsub.get_message(ignore_subscribe_messages=True)
                 if message:
                     try:
                         # Only take the data and forward as JSON to the connected websocket clients
-                        json_data = json.dumps(json.loads(message['data']))
+                        # Decode bytes directly instead of load/dump cycle
+                        json_data = message['data'].decode('utf-8')
                         # Process all connected websockets in parallel
-                        await asyncio.gather(*[ws.send_str(json_data) for ws in websockets])
+                        await asyncio.gather(*[ws.send_str(json_data) for ws in websockets], return_exceptions=True)
                     except:
                         print("Something went wrong while sending JSON data.")
                 else:
                     await asyncio.sleep(0.1)
         except redis.RedisError as e:
-            print("[ ] Waiting for Redis ...")
+            print(f"[ ] Connection lost to Redis ({type(e).__name__}), retrying...")
+            was_disconnected = True
             await asyncio.sleep(5)
 
 async def my_websocket_handler(request):
@@ -78,11 +68,11 @@ async def my_websocket_handler(request):
         elif msg.type == web.WSMsgType.ERROR:
             print(f'WebSocket connection closed with exception {ws.exception()}')
     request.app['websockets'].remove(ws)
-    print(f"[ ] WebSocket connection closed. Clients active: {len(request.app['websockets'])}")
+    print(f"[-] WebSocket connection closed. Clients active: {len(request.app['websockets'])}")
     return ws
 
 async def my_index_handler(request):
-    return web.FileResponse('index.html')
+    return web.FileResponse('static/index.html')
 
 async def start_background_tasks(app):
     app['websockets'] = []
@@ -92,6 +82,24 @@ async def cleanup_background_tasks(app):
     app['redis_subscriber'].cancel()
     await app['redis_subscriber']
 
+async def check_redis_connection():
+    """Check Redis connection on startup and wait until available."""
+    print("[*] Checking Redis connection...")
+    waiting_printed = False
+    
+    while True:
+        try:
+            r = redis.Redis.from_url(redis_url)
+            await r.ping()  # Simple connection test
+            await r.aclose()  # Clean up test connection
+            print("[*] Redis connection established")
+            return True
+        except Exception as e:
+            if not waiting_printed:
+                print(f"[...] Waiting for Redis... (Error: {type(e).__name__})")
+                waiting_printed = True
+            await asyncio.sleep(5)
+
 async def make_webapp():
     app = web.Application()
     app.add_routes([
@@ -107,4 +115,8 @@ async def make_webapp():
 
 if __name__ == '__main__':
     print(version)
+    # Check Redis connection on startup
+    asyncio.run(check_redis_connection())
+    print("[*] Starting web server...\n")
     web.run_app(make_webapp(), port=web_port)
+