-
Notifications
You must be signed in to change notification settings - Fork 464
Architecture
Deep-dive into ExaBGP's internal architecture and design
This document provides a comprehensive technical overview of how ExaBGP works internally, its design principles, component architecture, and code organization.
- Overview
- Core Design Principles
- High-Level Architecture
- Component Architecture
- BGP Protocol Implementation
- Process and API Architecture
- Code Organization
- Extension Points
- Performance Considerations
- Version Differences
ExaBGP is a pure BGP protocol implementation designed for programmable network automation. Unlike traditional BGP implementations (BIRD, FRRouting, Cisco IOS), ExaBGP does NOT manipulate routing tables or forward packets.
- ✅ Complete BGP-4 protocol implementation (RFC 4271)
- ✅ Support for 55+ BGP RFCs and extensions
- ✅ API for dynamic route control
- ✅ Multiprotocol BGP (IPv4, IPv6, VPN, FlowSpec, EVPN, BGP-LS)
- ✅ Language-agnostic external process integration
- ❌ Does NOT maintain RIB (Routing Information Base)
- ❌ Does NOT manipulate FIB (Forwarding Information Base)
- ❌ Does NOT forward packets
- ❌ Does NOT run IGP protocols (OSPF, IS-IS)
- ❌ Does NOT modify kernel routing tables
Key Insight: ExaBGP speaks the BGP protocol. External processes decide what to announce. The OS kernel or other software performs forwarding.
┌────────────────────────────────────────────────────┐
│ Business Logic (Health Checks, DDoS Detection) │ ← Your Code
├────────────────────────────────────────────────────┤
│ BGP Protocol (Session Management, Messages) │ ← ExaBGP
├────────────────────────────────────────────────────┤
│ Route Installation (ip route add, FRR, BIRD) │ ← External Tools
├────────────────────────────────────────────────────┤
│ Packet Forwarding (Kernel, Hardware) │ ← OS/Hardware
└────────────────────────────────────────────────────┘
Rationale:
- Focus: ExaBGP focuses on BGP protocol correctness
- Flexibility: Business logic in any language
- Simplicity: No complex routing policy language
- Security: Reduced attack surface (no kernel modifications)
Everything is controlled via STDIN/STDOUT pipes between ExaBGP and external processes.
Command Flow:
External Process → STDOUT → ExaBGP STDIN → BGP UPDATE → Neighbor Router
Message Flow:
Neighbor Router → BGP UPDATE → ExaBGP → Process STDIN → External Process
Advantages:
- Language-agnostic (Python, Shell, Go, Rust, etc.)
- Simple integration (just read/write text)
- No complex SDKs or libraries required
- Easy testing (pipe commands manually)
Traditional BGP Implementation:
BGP Peer → Receive Route → Update RIB → Select Best → Install in FIB → Forward Packets
ExaBGP:
BGP Peer → Receive Route → Send to External Process → [External Process Decides]
Why?
- Flexibility: External process can use any routing logic
- Simplicity: No complex route selection algorithm
- Integration: Easy to integrate with existing routing systems
- Separation: BGP protocol separate from routing policy
Implication: If you want routes installed in the kernel, you must write code to call ip route add.
See also: RFC Support - No RIB/FIB Manipulation
┌──────────────────────────────────────────────────────────────────┐
│ External Processes │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Health Check │ │ DDoS Detect │ │ Custom Logic │ │
│ │ (Python) │ │ (Shell) │ │ (Go) │ │
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
│ │ STDOUT │ STDOUT │ STDOUT │
│ │ Commands │ Commands │ Commands │
└─────────┼─────────────────┼─────────────────┼────────────────────┘
▼ ▼ ▼
┌──────────────────────────────────────────────────────────────────┐
│ ExaBGP Core │
│ │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ Process Manager │ │
│ │ • Launch/monitor external processes │ │
│ │ • Manage STDIN/STDOUT pipes │ │
│ │ • Restart crashed processes │ │
│ └────────────────────────────────────────────────────────────┘ │
│ │
│ ┌────────────────┐ ┌──────────────┐ ┌────────────────────┐ │
│ │ Parser │ │ Encoder │ │ Configuration │ │
│ │ (Commands) │ │ (JSON/Text) │ │ Parser │ │
│ └────────────────┘ └──────────────┘ └────────────────────┘ │
│ │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ BGP Protocol Engine │ │
│ │ │ │
│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │ │
│ │ │ Neighbor │ │ Message │ │ Capability │ │ │
│ │ │ Management │ │ Builder │ │ Negotiation │ │ │
│ │ └──────────────┘ └──────────────┘ └──────────────────┘ │ │
│ │ │ │
│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │ │
│ │ │ FSM │ │ Timer │ │ Attribute │ │ │
│ │ │ (RFC 4271) │ │ Management │ │ Handling │ │ │
│ │ └──────────────┘ └──────────────┘ └──────────────────┘ │ │
│ └────────────────────────────────────────────────────────────┘ │
│ │
└─────────────┬────────────────────────────────────────────────────┘
│ TCP Port 179
│ BGP Protocol
▼
┌──────────────────────────────────────────────────────────────────┐
│ BGP Neighbor (Router) │
│ (Cisco, Juniper, Arista, FRRouting, BIRD, etc.) │
└──────────────────────────────────────────────────────────────────┘
Purpose: Manage external processes that control ExaBGP via API.
Responsibilities:
- Launch processes - Fork configured processes at startup
- Monitor health - Detect crashed processes
- Auto-restart - Restart failed processes (configurable)
- Pipe management - Establish STDIN/STDOUT pipes
- Clean shutdown - Terminate processes on ExaBGP shutdown
Configuration Example:
process healthcheck {
run python3 -m exabgp healthcheck --cmd "curl http://localhost";
encoder text;
}
process ddos-detect {
run /etc/exabgp/scripts/ddos-detect.py;
encoder json;
}Process Lifecycle:
ExaBGP Startup
↓
Read Configuration
↓
For Each Configured Process:
↓
Fork Process → Create Pipes → Monitor
↓
Process Running ←──┐
↓ │
Process Crash? │
↓ Yes │
Auto-restart ──────┘
↓ No
Continue
See also: Process Configuration
Purpose: Parse commands from external processes and execute them.
Input: STDOUT from external processes Output: Internal command queue
Supported Formats:
- Text API - Human-readable text commands
- JSON API - Structured JSON commands
Text Command Example:
announce route 10.0.0.0/24 next-hop 192.0.2.1 community [65001:100]
withdraw route 10.0.0.0/24
announce flow route { match { source 10.0.0.0/8; } then { discard; } }
shutdownJSON Command Example:
{
"exabgp": "4.0",
"type": "announce",
"route": {
"nlri": "10.0.0.0/24",
"next-hop": "192.0.2.1",
"community": [[65001, 100]]
}
}Parsing Flow:
External Process STDOUT
↓
Read Line
↓
Detect Format (text vs json)
↓
Parse Command
↓
Validate Syntax
↓
Queue for Execution
↓
Execute Command
↓
Send ACK (5.x only)
See also: Text API Reference, JSON API Reference
Purpose: Format received BGP messages for external processes.
Input: BGP messages from neighbors Output: Formatted messages to process STDIN
Text Format:
neighbor 192.0.2.1 192.0.2.2 received update route 10.0.0.0/24 \
next-hop 192.0.2.1 origin igp as-path [65001 65002]
JSON Format:
{
"exabgp": "4.0.0",
"time": 1699999999.0,
"type": "update",
"neighbor": {
"address": {"local": "192.0.2.2", "peer": "192.0.2.1"},
"asn": {"local": 65000, "peer": 65001}
},
"message": {
"update": {
"announce": {
"ipv4 unicast": {
"192.0.2.1": [{
"nlri": "10.0.0.0/24",
"attribute": {
"origin": "igp",
"as-path": [65001, 65002],
"next-hop": "192.0.2.1"
}
}]
}
}
}
}
}Message Types Encoded:
-
update- Route announcements/withdrawals -
open- BGP OPEN message received -
notification- BGP NOTIFICATION (errors) -
keepalive- BGP KEEPALIVE -
refresh- Route refresh request -
state- BGP session state changes -
connected/down- Session events
See also: API Overview
Purpose: Parse ExaBGP configuration file.
Syntax: INI-style with hierarchical blocks
Example:
[exabgp.api]
ack = true
neighbor 192.168.1.1 {
router-id 192.168.1.2;
local-as 65001;
peer-as 65000;
family {
ipv4 unicast;
ipv4 flow;
}
static {
route 10.0.0.0/24 next-hop 192.0.2.1;
}
api {
processes [ healthcheck ];
}
}
process healthcheck {
run python3 -m exabgp healthcheck;
encoder text;
}Parsing Features:
- Template inheritance
- Include files
- Environment variable expansion
- Syntax validation
See also: Configuration Syntax
RFC 4271 - ExaBGP implements the complete BGP-4 FSM.
States:
┌──────┐
│ Idle │ ← Initial state, session not established
└───┬──┘
│ Start event (configured neighbor)
▼
┌─────────┐
│ Connect │ ← Initiating TCP connection
└────┬────┘
│ TCP connection established
▼
┌──────────┐
│OpenSent │ ← OPEN message sent, waiting for peer OPEN
└────┬─────┘
│ Received valid OPEN message
▼
┌─────────────┐
│OpenConfirm │ ← OPEN confirmed, waiting for KEEPALIVE
└──────┬──────┘
│ Received KEEPALIVE
▼
┌─────────────┐
│Established │ ← Session established, exchanging routes
└─────────────┘
State Transitions:
- Idle → Connect - Configuration loaded
- Connect → OpenSent - TCP connection established
- OpenSent → OpenConfirm - Valid OPEN received
- OpenConfirm → Established - KEEPALIVE received
- Established → Idle - Error, NOTIFICATION, or manual shutdown
See also: BGP State Machine
ExaBGP implements all BGP-4 message types:
Purpose: Session establishment and capability negotiation
Fields:
- Version (always 4)
- AS Number (16-bit or 32-bit)
- Hold Time
- BGP Identifier (Router ID)
- Optional Parameters (capabilities)
Capabilities Negotiated:
- Multiprotocol Extensions (IPv6, VPN, FlowSpec, etc.)
- 4-byte ASN Support
- ADD-PATH
- Graceful Restart
- Route Refresh
- Extended Message Support
See also: Capabilities
Purpose: Route announcements and withdrawals
Structure:
┌────────────────────────────────────┐
│ Withdrawn Routes Length (2 bytes) │
├────────────────────────────────────┤
│ Withdrawn Routes (variable) │
├────────────────────────────────────┤
│ Path Attributes Length (2 bytes) │
├────────────────────────────────────┤
│ Path Attributes (variable) │
│ - ORIGIN │
│ - AS_PATH │
│ - NEXT_HOP │
│ - LOCAL_PREF │
│ - COMMUNITY │
│ - etc. │
├────────────────────────────────────┤
│ NLRI (Network Layer Reachability) │
│ - Prefix length + IP prefix │
└────────────────────────────────────┘
Multiprotocol (MP-BGP): For non-IPv4 unicast, routes are carried in MP_REACH_NLRI and MP_UNREACH_NLRI attributes.
See also: Attribute Reference
Purpose: Error reporting and session termination
Error Codes:
- Message Header Error
- OPEN Message Error
- UPDATE Message Error
- Hold Timer Expired
- Finite State Machine Error
- Cease
Subcodes: Specific error within each code
See also: Debugging Guide
Purpose: Session maintenance
Frequency: Sent every (hold-time / 3) seconds
Format: Empty message (header only)
Purpose: Request re-advertisement of routes (RFC 2918)
Use Case: Policy changes without session reset
Purpose: Construct BGP UPDATE messages from API commands.
Flow:
API Command
↓
Parse Attributes
↓
Validate Syntax
↓
Build BGP Attributes
↓
Construct UPDATE Message
↓
Send to Neighbor
Attribute Encoding:
- Type-Length-Value (TLV) format
- Flags: Optional/Well-Known, Transitive/Non-Transitive, Partial, Extended
- Correct byte ordering (network byte order)
Address Family Support:
- IPv4 Unicast (traditional NLRI)
- IPv6, VPN, FlowSpec, EVPN, BGP-LS (MP_REACH_NLRI)
See also: Address Families
Purpose: Manage BGP sessions with configured neighbors.
Per-Neighbor State:
- TCP connection
- BGP FSM state
- Hold timer / Keepalive timer
- Capabilities negotiated
- Session statistics
Capabilities Negotiation:
Local Capabilities ∩ Peer Capabilities = Negotiated Capabilities
Timer Management:
- Hold Timer: Countdown from hold-time, reset on UPDATE/KEEPALIVE
- Keepalive Timer: Trigger KEEPALIVE every (hold-time / 3)
Purpose: Agree on optional BGP features during OPEN exchange.
Common Capabilities:
- Multiprotocol Extensions (RFC 4760) - AFI/SAFI support
- 4-byte ASN (RFC 4893) - 32-bit AS numbers
- ADD-PATH (RFC 7911) - Multiple paths per prefix
- Graceful Restart (RFC 4724) - Forwarding preservation
- Route Refresh (RFC 2918) - Route re-advertisement
- Extended Message (RFC 8654) - Messages > 4096 bytes
Negotiation Process:
- Local ExaBGP advertises capabilities in OPEN
- Peer advertises capabilities in OPEN
- Intersection of capabilities is used
- Unsupported capabilities are ignored
See also: Capabilities, RFC Compliance
Bi-directional pipes:
┌────────────────┐
│ External │
│ Process │
│ │
│ stdin stdout │
└───▲────────┬───┘
│ │
│ │ Commands
│ ▼
┌───┴────────▼───┐
│ ExaBGP │
│ │
│ BGP Messages │
└────────────────┘
External Process Perspective:
- Read from STDIN - Receive BGP messages from ExaBGP
- Write to STDOUT - Send commands to ExaBGP
- Flush after write - Critical for real-time communication
ExaBGP Perspective:
- Read from Process STDOUT - Receive commands
- Write to Process STDIN - Send BGP messages
ExaBGP 4.x and 5.x feature (enabled by default) - Sends ACK after executing commands.
Responses:
-
done- Command executed successfully -
error <message>- Command failed with error -
shutdown- ExaBGP is shutting down
Example Flow:
import sys
import select
import time
def wait_for_ack(expected_count=1, timeout=30):
"""Wait for ACK with polling loop. Handles text and JSON formats."""
import json
received = 0
start_time = time.time()
while received < expected_count:
if time.time() - start_time >= timeout:
return False
ready, _, _ = select.select([sys.stdin], [], [], 0.1)
if ready:
line = sys.stdin.readline().strip()
# Parse response (could be text or JSON)
answer = None
if line.startswith('{'):
try:
data = json.loads(line)
answer = data.get('answer')
except:
pass
else:
answer = line
if answer == "done":
received += 1
elif answer == "error":
return False
elif answer == "shutdown":
raise SystemExit(0)
else:
time.sleep(0.1)
return True
# Send command
sys.stdout.write("announce route 10.0.0.0/24 next-hop 192.0.2.1\n")
sys.stdout.flush()
# Wait for ACK (with polling)
if not wait_for_ack():
sys.exit(1) # Command failedAvailable in both 4.x and 5.x (enabled by default).
See also: API Overview - ACK Feature
ExaBGP uses a polling-based event loop (reactor pattern):
┌────────────────────────────────┐
│ Event Loop │
│ │
│ ┌──────────────────────────┐ │
│ │ Loop with Small Sleep │ │
│ │ - Select() for I/O │ │
│ │ • TCP sockets │ │
│ │ • Process pipes │ │
│ │ - Check Timers │ │
│ │ • Hold timer │ │
│ │ • Keepalive timer │ │
│ └──────────────────────────┘ │
│ │
│ Loop Iteration: │
│ ├─ Select(timeout) for I/O │
│ ├─ TCP Data? → Process BGP │
│ ├─ Process Data? → Parse │
│ ├─ Check Timers → Action │
│ ├─ Sleep briefly │
│ └─ Repeat │
└────────────────────────────────┘
Implementation Details:
-
I/O: Uses
select()for non-blocking socket/pipe I/O -
Timers: Implemented as loop-based polling - not event-driven
- Each iteration checks if timers have expired
- Small sleep between iterations (reduces CPU usage)
- Not using timer file descriptors (timerfd) or true event-driven timers
- Single-threaded: Simplicity, no locking required
- Non-blocking I/O: Efficient resource usage
Why Polling for Timers?
- Simplicity: No platform-specific timer APIs
- Portability: Works consistently across OS platforms
- Sufficient: BGP timers are typically seconds/minutes (not milliseconds)
Main codebase location: /src/exabgp/
src/exabgp/
├── application/ # ExaBGP main application entry point
├── bgp/ # BGP protocol implementation
│ ├── message/ # BGP message types (OPEN, UPDATE, etc.)
│ │ ├── open/ # OPEN message and capabilities
│ │ ├── update/ # UPDATE message and attributes
│ │ ├── notification/ # NOTIFICATION messages
│ │ └── refresh.py # ROUTE-REFRESH
│ ├── neighbor.py # Neighbor management
│ └── fsm.py # Finite State Machine
├── configuration/ # Configuration parsing
│ ├── neighbor/ # Neighbor configuration
│ ├── family/ # Address family configuration
│ └── process/ # Process configuration
├── protocol/ # Protocol utilities
│ ├── ip/ # IP address handling
│ ├── family.py # AFI/SAFI definitions
│ └── attributes.py # BGP attribute handling
├── reactor/ # Event loop (reactor pattern)
│ ├── loop.py # Main event loop
│ ├── api/ # API (process communication)
│ └── network/ # Network I/O
├── rib/ # Adj-RIB-In/Out (limited use)
└── version.py # Version information
Key Modules:
| Module | Purpose |
|---|---|
application/ |
Main entry point, CLI |
bgp/message/ |
BGP message encoding/decoding |
bgp/neighbor.py |
Neighbor state management |
configuration/ |
Config file parsing |
reactor/ |
Event loop and I/O |
protocol/ |
Protocol utilities |
File: src/exabgp/bgp/neighbor.py
Responsibilities:
- Manage BGP session with one peer
- Track FSM state
- Store negotiated capabilities
- Maintain timers
Key Methods:
-
connect()- Initiate TCP connection -
open()- Send OPEN message -
update()- Send UPDATE message -
keepalive()- Send KEEPALIVE
File: src/exabgp/bgp/message/
Hierarchy:
Message (abstract)
├── Open
├── Update
│ ├── Announce
│ └── Withdraw
├── Notification
├── KeepAlive
└── RouteRefresh
Responsibilities:
- Encode BGP messages to wire format
- Decode received BGP messages
- Validate message correctness
File: src/exabgp/bgp/message/update/attribute/
Supported Attributes:
- ORIGIN, AS_PATH, NEXT_HOP
- LOCAL_PREF, MED
- COMMUNITY, EXTENDED_COMMUNITY, LARGE_COMMUNITY
- MP_REACH_NLRI, MP_UNREACH_NLRI
- ORIGINATOR_ID, CLUSTER_LIST
- And 20+ more
See also: Attribute Reference
File: src/exabgp/reactor/loop.py
Responsibilities:
- Event-driven I/O multiplexing
- Process BGP messages from neighbors
- Process commands from external processes
- Manage timers
- Handle signals (reload, shutdown)
ExaBGP is designed to be extended. Here are the main extension points:
Add support for new AFI/SAFI combinations:
- Define AFI/SAFI in
src/exabgp/protocol/family.py - Implement NLRI encoding/decoding
- Add configuration parser support
- Add API command support
Example: SRv6 Mobile User Plane (MUP) was added this way.
Add support for new BGP attributes:
- Implement attribute class in
src/exabgp/bgp/message/update/attribute/ - Add encoding/decoding logic
- Add to attribute registry
- Update configuration parser
See also: Generic Attribute Support
Create new API formats beyond text/json:
- Implement encoder in
src/exabgp/reactor/api/encoder/ - Register encoder in configuration
- Handle message formatting
ExaBGP supports plugins for extending functionality without modifying core code.
Plugin use cases:
- Custom health check logic
- Specialized message processing
- Integration with external systems
Design: ExaBGP uses a single-threaded event loop (reactor pattern).
Implications:
- ✅ Simple, no locking required
- ✅ Predictable performance
- ❌ CPU-bound operations block event loop
- ❌ Limited to single CPU core
Best Practice: Keep external process logic fast. Use separate processes for CPU-intensive tasks.
ExaBGP does NOT store full routing tables:
- No RIB/FIB → Very low memory footprint
- Only tracks: configured static routes, active sessions
- Typical memory: < 100 MB
Contrast: Traditional BGP implementations store millions of routes (GB of RAM).
Neighbor scaling:
- Tested with 100+ BGP neighbors
- Limited by event loop (single thread)
Route scaling:
- No limit on announced routes (external process decides)
- UPDATE message rate limited by neighbor capacity
Process scaling:
- Multiple external processes supported
- Each process runs independently
Characteristics:
- Python 2 only
- Limited address family support
- Text API only
- No JSON encoder
Status: Deprecated, no longer maintained
Characteristics:
- Python 3 support
- Full multiprotocol support (IPv6, VPN, FlowSpec, EVPN, BGP-LS)
- JSON API
- Wide adoption
API: No command acknowledgment
Configuration: Environment variables + config file
See also: Migration from 3.4 to 4.x
New Features:
-
Command acknowledgment (ACK) -
done,error,shutdownresponses - flush route command - Withdraw all routes
- Enhanced JSON API format
- Improved error messages
- SRv6, BGP-MUP support
Breaking changes from 4.x:
- ACK changes API behavior (must handle responses)
- Some configuration syntax updates
- Python 3.8+ required (Python 2 removed)
See also: Migration from 4.2 to 5.0, API Overview - ACK
New Features:
- Async Reactor - Modern async/await-based event loop (now default)
-
Shell Completion - Bash, Zsh, Fish support (
exabgp shell install) - Enhanced CLI - Tab completion, JSON pretty-printing, inline help
-
Health Monitoring API -
session pinganddaemon statuscommands -
Migration Tool -
exabgp migratefor config and API conversion - Type Annotations - Full codebase type hints
Breaking changes from 5.x:
- Python 3.12+ required (3.7-3.11 dropped)
- BGP-LS JSON API field changes (ip→prefix, sr-adj→sr-adjs, etc.)
- Async reactor now default (generators still work inside async)
See also: Migration from 5.x to 6.0.0
- Configuration Syntax - Config file format
- API Overview - API architecture
- Text API Reference - Text API commands
- JSON API Reference - JSON API format
- BGP State Machine - FSM implementation
- Attribute Reference - All BGP attributes
- Command Reference - All commands
- DDoS Mitigation - FlowSpec architecture
- Service High Availability - Health check integration
- Anycast Management - Anycast architecture
- RFC Compliance - Implemented RFCs
- Examples Index - 86 configuration examples
Getting Started
Configuration
- Configuration Syntax
- Neighbor Configuration
- Directives A-Z
- Templates
- Environment Variables
- Process Configuration
API
- API Overview
- Text API Reference
- JSON API Reference
- API Commands
- Writing API Programs
- Error Handling
- Production Best Practices
Address Families
- Overview
- IPv4 Unicast
- IPv6 Unicast
- FlowSpec
- EVPN
- L3VPN
- BGP-LS
- VPLS
- SRv6 / MUP
- Multicast
- RT Constraint
Features
Use Cases
Tools
Operations
Reference
- Architecture
- Design
- Attribute Reference
- Command Reference
- BGP State Machine
- Capabilities
- Communities
- Examples Index
- Glossary
- RFC Support
Integration
Migration
Community
External