🕷️ Lyzr Crawl

A high-performance web crawler API built with Go
Extract content, discover URLs, and crawl websites at scale with real-time progress updates

Features • Quick Start • API • Examples • Contributing

✨ Features

🚀 Performance Concurrent crawling with configurable workers MongoDB storage for persistent job tracking Smart rate limiting to respect server resources Automatic retries with exponential backoff	🎯 Capabilities Multi-format extraction: HTML, Markdown, Clean Text JavaScript rendering via headless Chrome/Firefox Robots.txt compliance with configurable override Pattern-based filtering for URLs
📊 Real-time Monitoring WebSocket live updates for crawl progress Detailed job tracking with statistics Progress tracking with completion percentages RESTful API with Swagger documentation	🔒 Security & Reliability API key authentication for access control Domain restrictions to prevent abuse Request timeouts and circuit breakers Graceful error handling and recovery

🚀 Quick Start

🐳 Using Docker (Recommended)

# 1. Clone the repository
git clone https://github.com/LyzrCore/lyzr-crawl.git
cd lyzr-crawl

# 2. Set up environment
cp .env.example .env
# Edit .env with your MongoDB connection string

# 3. Start the crawler
docker-compose up -d

# 4. Check health
curl http://localhost:8080/health

💻 Local Development

Click to expand local setup instructions

# Prerequisites: Go 1.21+, MongoDB, RabbitMQ (optional)

# Install dependencies
go mod download

# Run with custom configuration
go run . \
  -mongo-uri="mongodb://localhost:27017/crawler" \
  -rabbitmq-url="amqp://localhost:5672/" \
  -port=8080

# Build binary
go build -o crawler .
./crawler

📖 API Documentation

🔑 Authentication

All API endpoints require authentication:

# Using header
curl -H "X-API-Key: your-api-key-here" http://localhost:8080/api/...

# Using Bearer token
curl -H "Authorization: Bearer your-api-key-here" http://localhost:8080/api/...

📍 Endpoints

Method	Endpoint	Description
`GET`	`/health`	Health check with system status
`POST`	`/crawl`	Start a new crawl job
`POST`	`/content`	Extract content from URLs
`GET`	`/jobs`	List all crawl jobs
`GET`	`/jobs/{id}`	Get specific job details
`WS`	`/ws/{id}`	WebSocket for live updates
`GET`	`/notforhumans/`	Swagger UI documentation

📝 Examples

Start a Basic Crawl

curl -X POST http://localhost:8080/crawl \
  -H "X-API-Key: your-api-key-here" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "max_pages": 50,
    "max_depth": 2
  }'

📋 View full crawl options

{
  "url": "https://example.com",
  "max_pages": 100,
  "max_depth": 3,
  "allowed_domains": ["example.com", "blog.example.com"],
  "exclude_patterns": ["/admin", "/private"],
  "include_patterns": ["/blog", "/docs"],
  "respect_robots_txt": true,
  "crawl_delay": 1000,
  "timeout": 30,
  "max_concurrent": 5,
  "user_agent": "MyBot/1.0"
}

Extract Content from Multiple URLs

curl -X POST http://localhost:8080/content \
  -H "X-API-Key: your-api-key-here" \
  -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://example.com/page1",
      "https://example.com/page2"
    ],
    "concurrency": 50
  }'

Monitor Progress with WebSocket

const ws = new WebSocket('ws://localhost:8080/ws/job-id-here');

ws.onmessage = (event) => {
  const update = JSON.parse(event.data);
  console.log(`Progress: ${update.progress}%`);
  console.log(`URLs found: ${update.urls_found}`);
};

🏗️ Architecture

graph TB
    Client[Client Application] --> API[REST API]
    API --> MongoDB[(MongoDB)]
    API --> RabbitMQ[RabbitMQ]
    API --> Crawler[Crawler Engine]
    
    Crawler --> |Tier 1| Static[Static HTML]
    Crawler --> |Tier 2| Browser[Headless Browser]
    
    RabbitMQ --> WebSocket[WebSocket Handler]
    WebSocket --> Client
    
    style Client fill:#e1f5fe
    style API fill:#fff3e0
    style MongoDB fill:#c8e6c9
    style RabbitMQ fill:#ffccbc
    style Crawler fill:#f8bbd0

⚙️ Configuration

Environment Variables

# Core Configuration
MONGO_URI=mongodb://localhost:27017/crawler  # MongoDB connection
RABBITMQ_URL=amqp://localhost:5672/         # RabbitMQ (optional)
API_KEY=your-secure-api-key                  # API authentication
PORT=8080                                    # Server port
GIN_MODE=release                            # Framework mode

# Optional Services
SCRAPEOPS_API_KEY=your-key                  # Proxy rotation service

Crawl Parameters

Parameter	Type	Default	Description
`max_pages`	int	50	Maximum pages to crawl
`max_depth`	int	3	Maximum crawl depth
`crawl_delay`	int	1000	Delay between requests (ms)
`timeout`	int	30	Request timeout (seconds)
`max_concurrent`	int	5	Concurrent requests
`respect_robots_txt`	bool	true	Follow robots.txt rules

🛠️ Development

Project Structure

lyzr-crawl/
├── config/              # Configuration files
│   ├── database.go     # Database config
│   ├── rabbitmq.go     # RabbitMQ config
│   └── scrapeops.go    # ScrapeOps integration
├── handlers/            # HTTP request handlers
│   ├── content.go      # Content extraction endpoints
│   ├── crawl.go        # Crawl job management
│   ├── health.go       # Health check endpoints
│   ├── jobs.go         # Job listing and status
│   └── websocket.go    # WebSocket connections
├── services/            # Core business logic
│   ├── crawler.go      # Main crawling engine
│   ├── database.go     # MongoDB operations
│   ├── messaging.go    # RabbitMQ messaging
│   ├── robots.go       # Robots.txt parser
│   ├── sitemap.go      # Sitemap parser
│   └── stealth.go      # Anti-detection features
├── models/              # Data structures
│   ├── content.go      # Content response models
│   ├── crawl.go        # Crawl request/response
│   ├── events.go       # Event models
│   ├── job.go          # Job tracking models
│   └── sitemap.go      # Sitemap models
├── middleware/          # HTTP middleware
│   ├── auth.go         # API key authentication
│   └── logging.go      # Request logging
├── utils/               # Helper functions
│   └── url.go          # URL utilities
├── docs/                # API documentation
│   ├── docs.go         # Generated docs
│   ├── swagger.json    # Swagger spec
│   └── swagger.yaml    # Swagger spec
├── ui/                  # Web UI assets
│   └── index.html      # Simple web interface
├── main.go              # Application entry point
├── server.go            # HTTP server setup
├── docker-compose.yml   # Docker composition
├── Dockerfile           # Container definition
└── README.md            # This file

🤝 Contributing

We love contributions! Please see our Contributing Guide for details.

Quick Contribution Guide

Fork the repository
Create your feature branch (git checkout -b feature/AmazingFeature)
Make your changes
Test your changes locally
Commit your changes (git commit -m 'Add some AmazingFeature')
Push to the branch (git push origin feature/AmazingFeature)
Open a Pull Request

Development Setup

# Install dependencies
go mod download

# Build the project
go build -o lyzr-crawl .

# Run locally
./lyzr-crawl -mongo-uri="mongodb://localhost:27017/crawler"

# Generate Swagger docs
swag init -g server.go

📊 Performance Benchmarks

Lyzr Crawl Performance

Scenario	Requests/sec	Pages/min	Success Rate
Light Load (5 concurrent)	~8 req/sec	~480 pages/min	95%+
Medium Load (25 concurrent)	~25 req/sec	~1,500 pages/min	94%+
Heavy Load (50 concurrent)	~40 req/sec	~2,400 pages/min	92%+
Stress Test (100 concurrent)	~60 req/sec	~3,600 pages/min	89%+

Benchmark Notes

JavaScript rendering: Adds ~50-100ms overhead per page

🐛 Troubleshooting

Common issues and solutions

MongoDB Connection Failed

# Check if MongoDB is running
docker-compose ps

# Verify connection string
echo $MONGO_URI

# Test connection
mongosh "$MONGO_URI"

WebSocket Not Connecting

Ensure RabbitMQ is running
Check CORS settings if connecting from browser
Verify job ID is valid

High Memory Usage

Reduce max_concurrent setting
Lower max_pages limit
Enable swap if needed

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

Built with these amazing tools:

Go - The programming language
Rod - Browser automation
Gorilla Mux - HTTP router
MongoDB - Database
RabbitMQ - Message broker (optional)

Lyzr Crawl - Part of the Lyzr.ai ecosystem

Report Bug • Request Feature

Name		Name	Last commit message	Last commit date
Latest commit History 22 Commits
.idea		.idea
config		config
docs		docs
handlers		handlers
middleware		middleware
models		models
services		services
ui		ui
utils		utils
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
docker-compose.yml		docker-compose.yml
docs.go		docs.go
entrypoint.sh		entrypoint.sh
go.mod		go.mod
go.sum		go.sum
init-mongo.js		init-mongo.js
init.sh		init.sh
main.go		main.go
rabbitmq.conf		rabbitmq.conf
run.sh		run.sh
server.go		server.go

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

🕷️ Lyzr Crawl

✨ Features

🚀 Performance

🎯 Capabilities

📊 Real-time Monitoring

🔒 Security & Reliability