Soldexer

Soldexer is a high-throughput data service for accessing Solana's historical and real-time data. It processes 10MB+ per request with ~2-3 second latency, making it significantly faster than running your own node.

Table of Contents

• What is Soldexer
• Why Soldexer?
• Development Status
• Getting Started
  • Prerequisites
  • Installation
  • Configuration
  • Database Setup
  • Running the Project
• How It Works
• Related Repositories

Why Soldexer?

Unlike running your own decicated node (~$2k+/mo) or paying for proprietary APIs, Soldexer gives you scalable access, full filtering, and built-in transformations—at a fraction of the effort and cost.

Development Status

Soldexer is under active development. Open Beta is coming soon.

Getting Started

Prerequisites

• Yarn package manager
• Docker (if using local ClickHouse)

Installation

Install the project dependencies:

yarn install

Configuration

The project uses a soldexer.json configuration file. Here's what each field means:

Required Configuration

• portalUrl (string): The URL of the Soldexer portal
• pipes (object): Configuration for each data pipeline

Optional Configuration

• clickhouse (object): ClickHouse database configuration (uses defaults if not provided)

Example Configuration

{
    "portalUrl": "https://portal.sqd.dev",
    "clickhouse": {
        "url": "http://localhost:8123",
        "database": "soldexer",
        "username": "default",
        "password": ""
    },
    "pipes": {
        "swaps": {
            "fromBlock": 332557468
        },
        "metaplex": {
            "fromBlock": 332557468
        },
        "pumpfun": {
            "fromBlock": 332557468
        }
    }
}

Pipes Configuration

Each pipe requires:

• fromBlock (number): Starting block number for indexing
• toBlock (number, optional): Ending block number (if not provided, runs indefinitely)

Available pipes: swaps, metaplex, pumpfun

ClickHouse Configuration

If not provided, the following defaults are used:

• url: http://localhost:8123
• database: default
• username: default
• password: "" (empty string)

Database Setup

You have two options for the database:

Option 1: Local ClickHouse (Recommended for Development)

Start ClickHouse using Docker Compose:

docker compose up -d

This will start a local ClickHouse instance accessible at http://localhost:8123.

Option 2: External ClickHouse Database

If you have your own ClickHouse instance, update the clickhouse section in soldexer.json:

{
    "clickhouse": {
        "url": "https://your-clickhouse-url:8123",
        "database": "your_database_name",
        "username": "your_username",
        "password": "your_password"
    }
}

Running the Project

Once you have installed dependencies, configured soldexer.json, and set up your database, start the indexer:

yarn start

The indexer will:

1. Read your configuration from soldexer.json
2. Connect to the specified ClickHouse database
3. Start all configured pipes simultaneously
4. Begin indexing Solana data from the specified starting blocks

How It Works

1. Client sends a query to the Portal.
2. Portal:
   • Pulls recent data from local hotblocks storage.
   • Splits and routes historical data queries to decentralized workers.
3. Hotblocks Data Service streams live Solana data from RPCs into the Portal.
4. Workers fetch and return compressed historical data stored in S3.

Related Repositories

Name	Description
solana-ingest	Extracts Solana data and uploads compressed chunks to S3.
solana-data-service	Streams live Solana blocks to Portals via RPC.
sqd-portal	Handles incoming queries and routes to workers or hotblocks.
worker-rs	Decentralized worker that queries and serves data chunks from S3.

Example: Solana Swaps

# Install dependencies
yarn install

# Start ClickHouse
docker compose up -d

# Start the swaps indexer
yarn ts-node src/main.ts