Bellingcat OSINT Pipeline — n8n Implementation

A low-code OSINT analysis pipeline built in n8n, implementing large-scale corpus ingestion, vector embeddings, entity relationship extraction, and a RAG chatbot over the Bellingcat article archive.

This project is one half of a deliberate architectural comparison: the same analysis pipeline is implemented in both n8n (this repo) and Python (bellingcat-py). The comparison is a practical guide to choosing between low-code/no-code and pro-code approaches for GenAI pipelines — covering trade-offs in accessibility, flexibility, performance, and operational overhead.

Pipeline Overview

Four interconnected n8n workflows providing a complete OSINT analysis pipeline:

1. Scraper — Automated content collection from Bellingcat (paginated, deduplicated)
2. Vector Index Creator — Semantic embedding generation and PGVector storage
3. RAG Chatbot — Conversational interface for querying the collected corpus
4. Relationship Entity Mapping (REM) — LLM-powered entity and relationship extraction

Comparison with Python Implementation

This n8n implementation can be compared with the Python-based bellingcat-py project.

n8n Implementation Advantages

Visual Development:

• No-code/low-code approach accessible to non-programmers
• Visual workflow representation makes logic easy to understand
• Drag-and-drop interface for rapid prototyping

Built-in Integrations:

• Native database connectors and API integrations
• Pre-built LangChain nodes for AI workflows
• Automatic error handling and retry mechanisms

Operational Benefits:

• Built-in scheduling and monitoring
• Web-based execution environment
• Easy deployment and scaling options
• Workflow version control and collaboration features

Maintenance:

• Reduced code maintenance overhead
• Visual debugging capabilities
• Community-supported node ecosystem

n8n Implementation Disadvantages

Flexibility Limitations:

• Less flexibility for complex custom logic
• Limited debugging capabilities compared to traditional code
• Constrained by available node types

Vendor Lock-in:

• Dependent on n8n platform and ecosystem
• Migration complexity if switching platforms
• Limited customization of core functionality

Performance Considerations:

• Visual workflows may have overhead compared to optimized code
• Limited control over execution optimization
• Potential bottlenecks in complex data processing

Python Implementation Advantages

Full Programming Control:

• Complete flexibility in implementation
• Advanced debugging and profiling capabilities
• Custom optimization for performance-critical operations

Ecosystem:

• Access to entire Python ecosystem
• Custom library integration
• Advanced data processing capabilities

Deployment Flexibility:

• Multiple deployment options (Docker, serverless, etc.)
• Infrastructure-as-code integration
• Custom monitoring and logging solutions

Python Implementation Disadvantages

Development Complexity:

• Requires programming expertise
• More time-intensive development process
• Infrastructure setup and maintenance overhead

Operational Overhead:

• Manual scaling and monitoring setup
• Error handling implementation required
• Deployment complexity

Workflows

1. Content Scraper

Purpose: Automated collection of articles from Bellingcat website

Key Features:

• Scheduled execution for continuous data collection
• Pagination handling (scrapes 100 pages by default)
• POST requests to Bellingcat's FacetWP API endpoint
• HTML to Markdown conversion for consistent text processing
• Deduplication to prevent collecting the same articles
• Automatic PostgreSQL storage
• Chain execution of vector index creation upon completion

Technical Details:

• Uses FacetWP API: https://www.bellingcat.com/wp-json/facetwp/v1/refresh
• Filters for news articles only (/news/ URL pattern)
• Stores articles in markdown table with URL and content fields

2. Vector Index Creator

Purpose: Generate semantic embeddings for collected content

Key Features:

• Reads articles from PostgreSQL database
• Text chunking with configurable size (2000 chars) and overlap (200 chars)
• Mistral Cloud embeddings for semantic search capabilities
• PGVector storage for efficient similarity searches
• Metadata preservation (URL tracking for source attribution)
• Deduplication to avoid reprocessing existing content

Technical Details:

• Uses Mistral Cloud embedding model
• Chunk size: 2000 characters with 200 character overlap
• Stores embeddings in PGVector database
• Triggered automatically after scraping or manually

3. RAG Chatbot

Purpose: Conversational interface for querying collected content

Key Features:

• Chat trigger for user interaction
• AI Agent with OpenRouter GPT model (gpt-oss-120b)
• Semantic search using vector embeddings
• PostgreSQL chat memory for conversation persistence
• Top-K retrieval (20 documents) for comprehensive context
• System prompt optimized for investigative support

Technical Details:

• Uses OpenRouter API for language model access
• Mistral Cloud embeddings for query vectorization
• Vector similarity search with k=20
• Persistent conversation memory
• Restricts responses to available source material

4. Relationship Entity Mapping (REM)

Purpose: Extract entities and relationships for investigative analysis

Key Features:

• Structured information extraction using LLM
• Comprehensive schema for investigation-relevant relationships
• Entity types: person, organization, location, event, document, weapon, etc.
• Relationship types: associated_with, operating_in, led_by, funded_by, etc.
• Batch processing for efficiency
• PostgreSQL storage in structured format

Technical Details:

• Uses OpenRouter GPT model with structured output
• JSON schema validation for consistent extraction
• Processes articles in batches of 5
• Stores relationships in rel table with source/target entities
• Handles extraction errors gracefully

Setup

Prerequisites

• n8n instance (self-hosted or cloud)
• PostgreSQL database with PGVector extension
• API credentials for (can be switched to ollama for offline/privacy/cost):
  • Mistral Cloud (embeddings)
  • OpenRouter (language models)

Database Setup

TODO: Workflow for database setup (tables)

Create the required PostgreSQL tables:

-- Articles storage
CREATE TABLE markdown (
    url TEXT PRIMARY KEY,
    markdown TEXT
);

-- Relationships storage  
CREATE TABLE rel (
    idx SERIAL PRIMARY KEY,
    relationship TEXT NOT NULL,
    source_name TEXT NOT NULL,
    source_type TEXT NOT NULL,
    target_name TEXT NOT NULL,
    target_type TEXT NOT NULL,
    properties JSONB,
    origin TEXT NOT NULL
);

-- Enable PGVector extension for embeddings
CREATE EXTENSION IF NOT EXISTS vector;

n8n Configuration

1. Import Workflows: Import the four JSON workflow files into your n8n instance
2. Configure Credentials:
   • PostgreSQL connection (n8n-bellingcat)
   • Mistral Cloud API key
   • OpenRouter API key
3. Set Workflow Connections: Link the Collection workflow to trigger Vector Index creation
4. Activate Workflows: Enable the Collection workflow for scheduled execution

Usage

Running the Pipeline

1. Start Collection: Execute the Collection workflow to begin scraping
2. Monitor Progress: Check execution logs for scraping status
3. Vector Index: Automatically triggered after collection, or run manually
4. Query Content: Use the RAG workflow chat interface
5. Extract Relationships: Run REM workflow for entity analysis

Chat Interface

Access the RAG chatbot through the webhook URL provided by n8n. Ask questions about the collected content:

• "What has Bellingcat reported about surveillance technology?"
• "Find articles related to social media investigations"
• "What techniques are mentioned for geolocation?"

Technical Architecture

Data Flow

1. Collection: Articles scraped from Bellingcat → PostgreSQL storage
2. Processing: Text chunking → Mistral embeddings → PGVector storage
3. Query: User question → Vector search → LLM response with sources
4. Analysis: Article content → Entity extraction → Relationship storage

Database Schema

-- Core article storage
markdown (url, markdown)

-- Vector embeddings (managed by PGVector)
-- Automatic table creation by n8n LangChain nodes

-- Extracted relationships
rel (idx, relationship, source_name, source_type, target_name, target_type, properties, origin)

-- Chat memory (managed by PostgreSQL Chat Memory node)
-- Automatic table creation by n8n LangChain nodes

Use Cases

This implementation is suitable for:

• Rapid Prototyping: Quick setup of OSINT analysis pipelines
• Non-Technical Users: Accessible to journalists and researchers without programming background
• Educational Purposes: Visual demonstration of AI/ML workflows
• Small to Medium Scale: Efficient for thousands of documents
• Integration Projects: Easy connection with existing business tools

Future Enhancements

• Data Visualization: Integration with visualization tools for relationship mapping
• Multi-source Scraping: Extend to additional OSINT sources
• Advanced Analytics: Time-series analysis and trend detection
• Export Capabilities: Integration with investigation tools and report generation (reference Bellingcat toolkit)
• Real-time Monitoring: Automated alerting for new content matching specific criteria

Contributing

This project serves as a reference implementation for comparison with the Python approach. Contributions for improvements, additional workflows, or enhanced documentation are welcome.

License

This project is provided for educational and research purposes. Please respect Bellingcat's robots.txt and terms of service when using the scraping workflows.