Add OSINT Industries web platform with Flask backend and dark UI - #1
Open
tman81221-tech wants to merge 8 commits into
Open
Add OSINT Industries web platform with Flask backend and dark UI#1tman81221-tech wants to merge 8 commits into
tman81221-tech wants to merge 8 commits into
Conversation
Replaces the original bash/tkinter prototype with a production-ready web platform that fixes all identified issues: Backend (app.py): - OSINTClient class with correct apikey/Bearer/X-API-Key auth header support - Exponential-backoff retry (up to 4 attempts) on 429/5xx and timeouts - Proper per-status-code error handling (401 stops immediately, etc.) - ai_analyze() with structured 5-section OSINT report prompt for Ollama - JSON report persistence with metadata; safe filename download endpoint - Config loaded from config.json (env override supported) - Comprehensive logging to logs/osint.log Frontend (templates/index.html + static/): - Dark-theme single-page UI (no external CDN dependencies) - Query type selector (email / phone / username) + AI toggle - Tabbed results: Summary cards, per-module accordion, Raw JSON, AI Analysis - Stat bar (query, type, elapsed, services found/total) - Export: raw JSON download + full report download - Left sidebar with clickable history and report modal - Settings panel (API key, endpoint URL, auth header style) - Auto-opens Settings panel when no key is configured - Responsive layout for mobile Setup (setup.sh): - Creates Python venv, installs deps, installs Ollama if missing - Pulls llama3 if no local model exists - Interactive first-run config wizard (key, URL, auth style) - Accepts --port and --no-ollama flags https://claude.ai/code/session_01U8HNhajt883hx75FJmboNp
One command installs everything from scratch: - apt packages (git, python3, curl) if missing - Clones/updates the repo to ~/osint-platform - Creates Python venv and installs flask + requests - Installs Ollama and pulls llama3 model - Interactive first-run config (API key, endpoint, auth style) - Launches Flask platform on port 5000 https://claude.ai/code/session_01U8HNhajt883hx75FJmboNp
The API returns 422 'header.api-key: Field required' when the header is sent as 'apikey'. Correct header name is 'api-key' (with hyphen). Updated all defaults across app.py, setup.sh, install.sh, and index.html. https://claude.ai/code/session_01U8HNhajt883hx75FJmboNp
raw_data is a direct array of module objects, not {modules:[...]}
Each module has: .module (name), .status, .spec_format, .category
Changes:
- renderResults: extract modules from d.data directly as array
- renderSummary: rewritten — stat cards + one card per found platform
- renderModules: rewritten — spec_format rows, sorted found-first
- extractSpecDetails: new helper to parse spec_format[0] fields
- capitalize: new helper
- Removed unused: extractName/Country, countBreaches/Platforms, extractModuleDetail
- ai_analyze: build human-readable prompt from found modules + spec_format
https://claude.ai/code/session_01U8HNhajt883hx75FJmboNp
HIBP (Have I Been Pwned): - HIBPClient with hibp-api-key header, retry, breaches + pastes endpoints - /api/hibp POST route — returns breach list + paste list - HIBP tab with breach cards (name, date, records, data classes) - rBreaches counter in stat bar, auto-opens HIBP tab on check Maltego XML Export: - to_maltego_xml() builds proper MaltegoTransformResponseMessage XML - Entities: target email/phone/username + found platforms + HIBP breaches - /api/export/maltego POST — returns downloadable .xml file - Includes phone hints, category, data types in entity fields Lampyre Import: - parse_lampyre() handles CSV and JSON exports - /api/lampyre/import POST file upload endpoint - Renders imported data as sortable table in Lampyre tab Config: - save_config() now accepts hibp_key - Settings panel has separate HIBP API key field - /api/config shows has_hibp status AI analysis updated to include HIBP breach data in prompt https://claude.ai/code/session_01U8HNhajt883hx75FJmboNp
Replaces MaltegoTransformResponse XML (transform-only format) with a proper .mtgx ZIP graph that Maltego Desktop can import directly via File → Import Graph → select .mtgx Graph layout: - Target entity (email/phone/username) at canvas centre - Found OSINT Industries platforms on inner ring (r=220), green nodes - Phone hint extracted as separate PhoneNumber entity per platform - HIBP breaches on outer ring (r=400), red nodes - Directed links with labels: "registered on", "phone hint", "breached (date)" Entity types mapped correctly: - email → maltego.EmailAddress - phone → maltego.PhoneNumber - username → maltego.Alias - platforms → maltego.Website - breaches → maltego.DNSName - phones → maltego.PhoneNumber Uses zipfile + math modules (stdlib, no new deps) https://claude.ai/code/session_01U8HNhajt883hx75FJmboNp
4 transforms registered at /maltego seed URL: [OSINT] Email → Registered Platforms (/maltego/email_lookup) [OSINT] Email → HIBP Breaches (/maltego/hibp_check) [OSINT] Phone → Registered Platforms (/maltego/phone_lookup) [OSINT] Username → Registered Platforms (/maltego/username_lookup) User adds http://127.0.0.1:5000/maltego once in Maltego (Transforms → Add Transform Server) — all 4 appear instantly. Right-click any EmailAddress/PhoneNumber/Alias entity → run transform → results appear live in the Maltego graph. Settings panel now shows the server URL with a Copy button. URL auto-updates to the actual host (works on remote machines too). https://claude.ai/code/session_01U8HNhajt883hx75FJmboNp
Does everything without human input: 1. Installs system packages if missing (git, python3, curl) 2. Clones or updates the repo 3. Creates Python venv and installs deps 4. Registers platform as systemd user service (auto-starts on boot) Falls back to nohup if systemd unavailable 5. Waits for platform to be healthy (polls /api/ollama) 6. Auto-detects Maltego config directory (searches all common paths) 7. Writes osint-platform.server XML directly into Maltego's TransformRepositories/RemoteTransformRepositories/ 8. Creates ~/Desktop/OSINT-Platform-Transforms.itds as backup After script: platform runs forever in background, Maltego transforms available immediately — right-click any EmailAddress/PhoneNumber/Alias → [OSINT] transforms appear. https://claude.ai/code/session_01U8HNhajt883hx75FJmboNp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces a complete web platform for the OSINT Industries API, featuring a modern dark-themed UI, Flask backend with retry logic and AI analysis, and comprehensive report generation.
Key Changes
Frontend (HTML/CSS/JavaScript)
Dark theme UI (
style.css): Complete 666-line stylesheet with custom CSS variables, responsive design, and smooth animationsInteractive application (
app.js): 557-line frontend logicHTML template (
index.html): Semantic structure with all UI componentsBackend (Python/Flask)
app.py): 437-line backend with:OSINTClientclass: Robust API client with exponential backoff retry logic (up to 4 retries on 429/5xx errors)ai_analyze()function: Local LLM integration via Ollama for structured OSINT analysissave_report()function: Persistent JSON report storage with metadataPOST /api/search: Execute OSINT query with optional AI analysisGET /api/config: Retrieve current configurationPOST /api/config: Save API credentialsGET /api/ollama: Check AI availability and model statusGET /api/reports/<filename>: Download saved reportsGET /api/history: List previous searchesSetup & Deployment
Setup script (
setup.sh): Automated environment initializationDependencies (
requirements.txt): Flask 3.0+ and requests 2.31+Notable Implementation Details
https://claude.ai/code/session_01U8HNhajt883hx75FJmboNp