Skip to content

An open-source, modular FIX protocol ingestion and monitoring engine that supports high-performance message processing, multi-source integration, real-time dashboards, and fault-tolerant, cloud-ready deployment for financial market data.

Notifications You must be signed in to change notification settings

OnePunchMonk/FIX-Feeder

Repository files navigation

FIX Feeder

Project Status: Stable

Modular, Fault-Tolerant FIX Protocol Feed Ingestion & Monitoring Engine

Throughput Metrics
β€’ Peak Throughput: 139,534 messages/second
β€’ Data Volume: ~17.4 MB/second
β€’ Connection Handling: 20 concurrent connections sustained
β€’ Memory Usage: Stable throughout test duration
β€’ Error Rate: 0% (all messages processed successfully)

System Resource Utilization
β€’ CPU Usage: ~45% peak during burst
β€’ Memory: ~200MB steady state
β€’ Network I/O: Saturated during peak load
β€’ Disk I/O: Efficient batch writes to database

FixFeeder Dashboard

FixFeeder is an open-source, extensible, and production-ready ingestion engine for the Financial Information Exchange (FIX) protocol. It is designed for high-performance, reliability, and observability, making it ideal for trading firms, quantitative analysts, and infrastructure engineers.

πŸ“‹ Table of Contents

🎯 Problem Statement

Financial markets generate massive volumes of structured messages through protocols like FIX (Financial Information Exchange). Organizations face several challenges:

Core Challenges

  1. Real-time Processing: Markets operate at microsecond latencies, requiring systems that can ingest and process thousands of messages per second
  2. Data Reliability: Financial data must be captured with 100% accuracy - missing or corrupted messages can result in significant financial losses
  3. Multi-Source Integration: Data comes from various sources (TCP sockets, files, message queues) requiring unified processing
  4. Observability: Systems must provide real-time monitoring, alerting, and audit trails for regulatory compliance
  5. Scalability: Architecture must handle varying load patterns from market open/close surges

Current Market Gap

Existing solutions are either:

  • Proprietary & Expensive: Vendor-locked solutions costing hundreds of thousands annually
  • Limited Flexibility: Rigid architectures that don't adapt to changing requirements
  • Poor Observability: Black-box systems with limited monitoring capabilities
  • Complex Deployment: Monolithic architectures difficult to deploy and maintain

FixFeeder Solution

FixFeeder addresses these challenges by providing:

  • Open Source: No vendor lock-in, fully customizable
  • Modular Architecture: Plug-and-play components for different requirements
  • Production-Ready: Built-in monitoring, logging, and fault tolerance
  • High Performance: Capable of processing 100,000+ messages/second
  • Cloud-Native: Containerized with Docker, ready for Kubernetes deployment

πŸ”§ Tech Stack

Core Technologies

  • Python 3.9+: Primary runtime for rapid development and extensive ecosystem
  • Apache Kafka: Distributed streaming platform for real-time message processing
  • PostgreSQL: Primary database with JSONB support for flexible message storage
  • SQLite: Development/testing database for local environments
  • Flask: Web framework for dashboard and REST API
  • Prometheus: Metrics collection and monitoring
  • Grafana: Visualization and alerting dashboards

Infrastructure & DevOps

  • Docker: Containerization for consistent deployments
  • Docker Compose: Multi-service orchestration
  • GitHub Actions: CI/CD pipeline for automated testing
  • YAML: Configuration management

Python Libraries

  • kafka-python: Kafka client library
  • psycopg2: PostgreSQL adapter
  • prometheus_client: Metrics instrumentation
  • PyYAML: Configuration parsing
  • pytest: Testing framework

✨ Features

πŸ”Œ Multiple Ingestion Sources

  • TCP Socket Listener: Real-time FIX session connectivity
  • File Reader: Batch processing of historical log files
  • Kafka Consumer: Integration with existing streaming infrastructure
  • Replay Engine: Simulated live feeds from historical data with configurable speed

🎯 Flexible Output Sinks

  • Kafka Producer: Real-time streaming to downstream systems
  • Database Writer: Persistent storage with PostgreSQL/SQLite support
  • Dual-sink Architecture: Simultaneous streaming and archival

πŸ“Š Real-time Monitoring

  • Prometheus Metrics: Message counts, error rates, processing latency
  • Grafana Dashboards: Visual monitoring with pre-built panels
  • Custom Metrics: Extensible monitoring framework

πŸ–₯️ Live Dashboard

  • Web Interface: Real-time message viewer with filtering
  • REST API: Programmatic access to message data and statistics
  • Audit Trail: Complete message history for compliance

πŸ”§ Extensible Design

  • Protocol Abstraction: Easy addition of ITCH, OUCH, and other protocols
  • Pluggable Components: Swap implementations without code changes
  • Configuration-Driven: YAML-based configuration for all components

☁️ Cloud-Native

  • Containerized: Docker images for consistent deployment
  • Orchestration: Docker Compose for development, Kubernetes-ready
  • Scalable: Horizontal scaling support for high-volume environments

πŸ—οΈ Architecture

FixFeeder follows a modular, pipeline-based architecture designed for scalability and maintainability:

System Summary

graph TD
    A[External FIX Source] -->|TCP Socket| B[SocketListener]
    A2[Log File] -->|File Read| B2[FileReader]
    A3[Kafka Topic] -->|Consume| B3[KafkaConsumer]
    A4[Replay File] -->|Timed Replay| B4[ReplayEngine]
    
    B --> C[FixParser]
    B2 --> C
    B3 --> C
    B4 --> C
    
    C -->|Parsed Message| D[KafkaProducer]
    C -->|Parsed Message| E[DBWriter]
    
    D -->|JSON Messages| F[Kafka Topic: fix.messages]
    E -->|Structured Data| G[PostgreSQL/SQLite]
    
    H[Prometheus Client] -->|Metrics| I[Prometheus Server]
    I -->|Data| J[Grafana Dashboard]
    
    K[Flask Dashboard] -->|Query| G
    L[REST API] -->|Query| G
    
    M[Monitor Module] -->|Counters| H
    C -.->|Increment Counters| M
    
    N[main.py] -->|Orchestrates| B
    N -->|Manages| C
    N -->|Starts| K
    N -->|Starts| L
    N -->|Starts| H
    
    O[Signal Handler] -->|Graceful Shutdown| N
    P[Configuration] -->|YAML Config| N
    
    style A fill:#f9f9f9
    style F fill:#e1f5fe
    style G fill:#e8f5e8
    style I fill:#fff3e0
    style J fill:#fce4ec
Loading

For detailed architecture information, see ARCHITECTURE.md.

High-Level Components

  1. Source Layer: Message ingestion from multiple sources
  2. Parser Layer: Protocol-specific message parsing and enrichment
  3. Queue Layer: Asynchronous message distribution via Kafka
  4. Storage Layer: Persistent storage for audit and analysis
  5. Monitoring Layer: Metrics collection and observability
  6. API Layer: Web dashboard and REST API for access

🌊 Data Flow

Understanding how data flows through FixFeeder is crucial for optimization and troubleshooting:

1. Message Ingestion

External Source β†’ FixFeeder Source Component
  • TCP Socket: Live FIX sessions connect directly to SocketListener
  • File: FileReader processes log files line by line
  • Kafka: KafkaConsumer subscribes to raw message topics
  • Replay: ReplayEngine simulates live feeds from historical data

2. Protocol Parsing

Raw Message β†’ FixParser β†’ Structured Message
  • Input: Raw FIX string (e.g., 8=FIX.4.2\x0135=D\x01...)
  • Processing:
    • Split by SOH delimiter (\x01)
    • Parse tag=value pairs
    • Validate message structure
    • Apply filtering rules (if configured)
    • Enrich with metadata (timestamps, human-readable values)
  • Output: JSON structure with raw and enriched data

3. Message Distribution

Structured Message β†’ [KafkaProducer, DBWriter]
  • Kafka Stream: Real-time distribution to downstream consumers
  • Database: Persistent storage for audit and analysis
  • Monitoring: Metrics collection (success/error counters)

4. Data Access

Stored Data β†’ [Dashboard, REST API]
  • Dashboard: Web interface for real-time monitoring
  • API: Programmatic access for external systems
  • Grafana: Metrics visualization and alerting

Message Structure Example

{
  "raw": {
    "8": "FIX.4.2",
    "35": "D",
    "55": "AAPL",
    "54": "1",
    "38": "100",
    "ingestion_timestamp": "2025-01-07T12:30:00.123456Z"
  },
  "enriched": {
    "tags": {
      "BeginString": {
        "tag": "8",
        "value": "FIX.4.2",
        "hr_value": "FIX.4.2"
      },
      "MsgType": {
        "tag": "35",
        "value": "D",
        "hr_value": "D (NewOrderSingle)"
      },
      "Symbol": {
        "tag": "55",
        "value": "AAPL",
        "hr_value": "AAPL"
      }
    }
  }
}

Error Handling & Resilience

  • Parse Failures: Logged and counted, don't stop processing
  • Connection Issues: Automatic reconnection with exponential backoff
  • Graceful Shutdown: Signal handling ensures clean resource cleanup
  • Data Consistency: Transactional writes to prevent partial data

πŸš€ Getting Started

Prerequisites

  • Docker & Docker Compose: For containerized deployment
  • Git: For cloning the repository
  • Python 3.9+: For local development (optional)

Quick Start

  1. Clone the repository:

    git clone https://github.com/your-username/FixFeeder.git
    cd FixFeeder
  2. Configure FixFeeder: Review and modify config/config.yml for your environment:

    source:
      type: socket  # socket, file, kafka, replay
      host: 0.0.0.0
      port: 9876
    
    storage:
      type: postgresql  # postgresql, sqlite
    
    monitoring:
      enabled: true
      prometheus_port: 8000
  3. Launch the complete stack:

    docker-compose up --build
  4. Verify deployment:

Service Ports

Service Port Description
FixFeeder Socket 9876 FIX message ingestion
Dashboard 5001 Web interface
Prometheus 8000 Metrics endpoint
Grafana 3000 Monitoring dashboards
PostgreSQL 5432 Database access
Kafka 9092 Message streaming

Local Development

For development without Docker:

  1. Install dependencies:

    pip install -r requirements.txt
  2. Start external services:

    docker-compose up kafka postgres prometheus grafana
  3. Run FixFeeder locally:

    python main.py

πŸ“‘ Sending Mock Data

Method 1: Using netcat (Linux/Mac/WSL)

echo -e "8=FIX.4.2\x0135=D\x0155=BASH_TEST\x0111=CMD_123\x0154=1\x0138=100\x0152=20250807-00:30:00\x0110=042\x01" | nc localhost 9876

Method 2: Using PowerShell (Windows)

$message = "8=FIX.4.2`0135=D`0155=BASH_TEST`0111=CMD_123`0154=1`0138=100`0152=20250807-00:30:00`0110=042`01"
$bytes = [System.Text.Encoding]::UTF8.GetBytes($message)
$tcpClient = New-Object System.Net.Sockets.TcpClient("localhost", 9876)
$stream = $tcpClient.GetStream()
$stream.Write($bytes, 0, $bytes.Length)
$tcpClient.Close()

Method 3: Using Python Script

import socket

def send_fix_message(message, host='localhost', port=9876):
    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.connect((host, port))
        s.sendall(message.encode('utf-8'))
        print(f"Sent: {message}")

# Example usage
fix_msg = "8=FIX.4.2\x0135=D\x0155=AAPL\x0111=ORDER_123\x0154=1\x0138=100\x0152=20250807-10:30:00\x0110=123\x01"
send_fix_message(fix_msg)

Sample FIX Messages

New Order Single (35=D):

8=FIX.4.2\x0135=D\x0155=AAPL\x0154=1\x0138=100\x0140=2\x0144=150.50\x0111=ORD_001\x0152=20250807-10:30:00\x0110=123\x01

Order Cancel Request (35=F):

8=FIX.4.2\x0135=F\x0111=CANCEL_001\x0141=ORD_001\x0155=AAPL\x0154=1\x0138=100\x0152=20250807-10:31:00\x0110=456\x01

⚑ Stress Testing

FixFeeder includes a comprehensive stress testing tool to validate performance under high load conditions.

Running Stress Tests

The stress test simulates multiple concurrent clients sending bursts of messages:

# Test with default settings (10,000 messages, 10 clients)
python stress_test.py

# Custom test configuration
python stress_test.py --messages 100000 --clients 20

Stress Test Results

Test Configuration:

  • Messages: 100,000 total
  • Clients: 20 concurrent connections
  • Message Size: ~125 bytes per message
  • Test Environment: Local Docker setup

Performance Results:

Preparing to send 100000 messages using 20 concurrent clients...
All clients started. Triggering barrier to send messages now!
...
--- Stress Test Complete ---
Sent 100000 messages in 0.72 seconds.
Average throughput: 139,534.78 messages/sec.

Performance Analysis

Throughput Metrics:

  • Peak Throughput: 139,534 messages/second
  • Data Volume: ~17.4 MB/second
  • Connection Handling: 20 concurrent connections sustained
  • Memory Usage: Stable throughout test duration
  • Error Rate: 0% (all messages processed successfully)

System Resource Utilization:

  • CPU Usage: ~45% peak during burst
  • Memory: ~200MB steady state
  • Network I/O: Saturated during peak load
  • Disk I/O: Efficient batch writes to database

Stress Test Architecture

The stress testing tool (stress_test.py) implements:

  1. Barrier Synchronization: All clients start simultaneously for accurate throughput measurement
  2. Concurrent Connections: Simulates real-world multi-client scenarios
  3. Realistic Messages: Uses valid FIX message structures
  4. Performance Metrics: Detailed timing and throughput analysis

Interpreting Results

Key Performance Indicators:

  • Messages/Second: Primary throughput metric
  • Latency Distribution: P50, P95, P99 processing times
  • Error Rate: Parse failures or connection drops
  • Resource Efficiency: CPU/Memory usage per message

Scaling Considerations:

  • Linear Scaling: Performance scales with CPU cores
  • Memory Bounded: Large message volumes require memory tuning
  • Network Limited: Gigabit networks can become bottleneck
  • Database Performance: Bulk inserts optimize storage throughput

πŸ“Š Monitoring & Observability

Prometheus Metrics

FixFeeder exposes comprehensive metrics for monitoring:

# Message processing metrics
fix_messages_ingested_total{} 12450
fix_parse_errors_total{} 3

# System health metrics  
fixfeeder_uptime_seconds{} 3600
fixfeeder_active_connections{} 5

Grafana Dashboards

Pre-configured dashboards provide visual monitoring:

  • Message Throughput: Real-time ingestion rates
  • Error Tracking: Parse failures and system errors
  • Connection Monitoring: Active socket connections
  • Resource Utilization: CPU, memory, and network usage

Log Analysis

Structured logging provides detailed operational insights:

2025-01-07 10:30:00 INFO [SocketListener] Connected by ('192.168.1.100', 54321)
2025-01-07 10:30:00 DEBUG [FixParser] Parsed message: {"35": "D", "55": "AAPL"}
2025-01-07 10:30:00 INFO [DBWriter] Wrote message to database

βš™οΈ Configuration

FixFeeder uses YAML configuration for all components:

# config/config.yml

# Message source configuration
source:
  type: socket        # socket, file, kafka, replay
  host: 0.0.0.0      # Bind address
  port: 9876         # TCP port
  file_path: ./logs/fix_sample.log  # For file source
  kafka_brokers: ["kafka:9092"]     # For Kafka source
  kafka_topic: "fix-raw-in"         # Input topic

# Message filtering
filter:
  enabled: true
  include_only:
    "55": ["AAPL", "GOOG", "MSFT"]  # Only these symbols

# Output configuration
queue:
  enabled: true
  type: kafka
  brokers: ["kafka:9092"]
  topic: fix.messages

storage:
  enabled: true
  type: postgresql
  postgres_config:
    user: "user"
    password: "password"
    host: "postgres"
    port: 5432
    database: "fixfeeder"

# Monitoring
monitoring:
  enabled: true
  prometheus_port: 8000

dashboard:
  enabled: true
  host: 0.0.0.0
  port: 5001

# Logging
core:
  log_level: INFO     # DEBUG, INFO, WARNING, ERROR

Environment-Specific Configuration

Create environment-specific configurations:

# Development
cp config/config.yml config/config.dev.yml

# Production  
cp config/config.yml config/config.prod.yml

Specify configuration file:

export FIXFEEDER_CONFIG=config/config.prod.yml
python main.py

πŸ”Œ API Reference

REST API Endpoints

Get Recent Messages:

GET /api/messages?limit=50

Get Specific Message:

GET /api/message/12345

Response Format:

{
  "id": 12345,
  "msg_seq_num": 42,
  "sending_time": "2025-01-07T10:30:00Z",
  "message": {
    "raw": {"8": "FIX.4.2", "35": "D"},
    "enriched": {...}
  }
}

Metrics Endpoints

Prometheus Metrics:

GET :8000/metrics

Health Check:

GET :5001/health

πŸ›£οΈ Future Roadmap

Phase 1: Enhanced Protocol Support

  • ITCH Protocol: NASDAQ market data feeds
  • OUCH Protocol: Order entry system integration
  • FIX/FAST: Compressed FIX message support
  • WebSocket Sources: Modern real-time data feeds

Phase 2: Advanced Processing

  • Message Validation: Schema-based validation engine
  • Complex Event Processing: Multi-message pattern detection
  • Machine Learning: Anomaly detection and prediction
  • Real-time Analytics: Stream processing with Apache Flink

Phase 3: Enterprise Features

  • Multi-tenancy: Isolated environments per organization
  • Authentication: OAuth2/SAML integration
  • Encryption: End-to-end message encryption
  • Compliance: SOX/RegT audit trail capabilities

Phase 4: Cloud & Scale

  • Kubernetes: Native cloud deployment
  • Auto-scaling: Dynamic resource allocation
  • Global Distribution: Multi-region deployment
  • Edge Processing: Latency-optimized edge nodes

Phase 5: Advanced Analytics

  • Time Series Database: InfluxDB integration
  • Data Lake: S3/HDFS long-term storage
  • Business Intelligence: Tableau/PowerBI connectors
  • Algorithmic Trading: Strategy backtesting framework

🀝 Contributing

We welcome contributions from the community! Here's how to get started:

Development Setup

  1. Fork the repository
  2. Clone your fork:
    git clone https://github.com/your-username/FixFeeder.git
  3. Create a feature branch:
    git checkout -b feature/your-feature-name
  4. Install development dependencies:
    pip install -r requirements.txt
    pip install -r requirements-dev.txt

Running Tests

# Unit tests
pytest tests/

# Integration tests
pytest tests/test_integration.py

# Coverage report
pytest --cov=. tests/

Code Quality

We maintain high code quality standards:

# Code formatting
black .

# Linting
flake8 .

# Type checking
mypy .

Contribution Guidelines

  • Code Style: Follow PEP 8 with Black formatting
  • Testing: Maintain 80%+ test coverage
  • Documentation: Update docs for new features
  • Performance: Benchmark performance-critical changes
  • Security: Follow security best practices

Submitting Changes

  1. Ensure tests pass:
    pytest tests/
  2. Update documentation
  3. Submit pull request with:
    • Clear description of changes
    • Test coverage for new features
    • Performance impact analysis
    • Breaking change notifications

πŸ“„ License

This project is licensed under the MIT License - see the LICENSE file for details.

πŸ†˜ Support

  • GitHub Issues: Bug reports and feature requests
  • Discussions: Community Q&A and ideas
  • Documentation: Comprehensive guides and API docs
  • Examples: Sample configurations and use cases

FixFeeder is designed by financial engineers, for financial engineers. We understand the unique challenges of market data processing and are building a solution that scales with your business.

About

An open-source, modular FIX protocol ingestion and monitoring engine that supports high-performance message processing, multi-source integration, real-time dashboards, and fault-tolerant, cloud-ready deployment for financial market data.

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published