Skip to content

Development

Jump to bottom
Artem Poltorzhitskiy edited this page Dec 11, 2025 · 2 revisions

Development Guide

Prerequisites

Before setting up the development environment, ensure you have:

  • Git - Version control system
  • Go 1.25+ - Latest Go runtime (download from https://go.dev/dl/)
  • Docker & Docker Compose - For containerized services
  • Timescale+ - Database (or use Docker version)
  • Make - Build automation tool (pre-installed on macOS/Linux)

Initial Setup

1. Clone the Repository

git clone https://github.com/celenium-io/celestia-indexer.git
cd celestia-indexer

2. Environment Configuration

Create a .env file in the project root with required variables:

cp .env.example .env

Edit .env and set the following required variables:

# Celestia Node Configuration
CELESTIA_NODE_URL=http://localhost:26657              # Consensus node RPC
CELESTIA_NODE_API_URL=http://localhost:1317           # LCD API endpoint
CELESTIA_DAL_API_URL=http://localhost:7654            # Data availability API
CELESTIA_NODE_AUTH_TOKEN=<token-from-node>           # Node auth token

# Database Configuration
POSTGRES_HOST=db                                       # or localhost
POSTGRES_PORT=5432
POSTGRES_USER=postgres
POSTGRES_PASSWORD=<secure-password>
POSTGRES_DB=celestia

# Cache Configuration (optional)
VALKEY_HOST=cache
VALKEY_PORT=6379

# API Configuration
API_PORT=9876
PRIVATE_API_PORT=9877

# Logging
LOG_LEVEL=info                                         # debug, info, warn, error

# Indexing Parameters
INDEXER_BLOCK_PERIOD=15                              # Seconds between blocks
INDEXER_REQUEST_BULK_SIZE=10                         # Blocks per request

# Optional: Monitoring
SENTRY_DSN=<sentry-dsn>
PYROSCOPE_ENABLED=false
PYROSCOPE_URL=http://localhost:4040

3. Install Dependencies

go mod download
go mod tidy

4. Initialize Development Environment

Run the initialization script:

make init

This will set up the development environment and prepare the database.

Running the Application

Using Docker Compose (Recommended for Local Development)

Start all services:

make compose

This starts:

  • PostgreSQL database (localhost:5432)
  • Valkey cache (localhost:6379)
  • Indexer service
  • API service (localhost:9876)
  • Private API service (localhost:9877)

View logs:

docker-compose logs -f [service-name]

Stop services:

docker-compose down

Running Individual Services Locally

Start the Indexer

make indexer

Or manually:

cd cmd/indexer && go run . -c ../../configs/dipdup.yml

Start the Public API

make api

Or manually:

cd cmd/api && go run . -c ../../configs/dipdup.yml

Start the Private API

make private_api

Start the Celestials Service

make celestials

Build Process

Building Binaries

Create optimized binaries:

make build

This generates executables in the bin/ directory:

  • bin/indexer - Main indexing service
  • bin/api - REST API service
  • bin/private_api - Internal API service
  • bin/celestials - Celestials service

Build individual binaries:

cd cmd/indexer && go build -a -o ../../bin/indexer .

Building Docker Images

Build all Docker images:

docker-compose build

Build a specific image:

docker-compose build indexer
docker-compose build api

Testing

Run All Tests

make test

With coverage:

go test -cover -coverprofile=coverage.out ./...
go tool cover -html=coverage.out

Run Specific Tests

# Test a single package
go test ./internal/storage/...

# Test a specific test
go test ./internal/storage/... -run TestBlockRepository

# Verbose output
go test -v ./...

# Parallel execution
go test -p 8 ./...

Test Fixtures

The codebase uses testfixtures for test data. Fixtures are located in test data directories and loaded before tests run.

Code Quality

Linting

Run the linter:

make lint

Or manually:

golangci-lint run

Formatting

Format code:

go fmt ./...

Code Generation

Generate code for mocks, interfaces, and SQL:

make generate

This regenerates:

  • Mock implementations
  • Storage interfaces
  • ORM generated code
  • Node type definitions

Database Management

Database Initialization

The database is automatically initialized on first run with:

  • Table schemas
  • Indexes
  • Partitions
  • Materialized views
  • Functions

Running Migrations

Migrations are applied automatically by the indexer on startup.

To manually apply scripts:

# Using psql
psql -h localhost -U postgres -d celestia < database/views/00_block_stats_by_minute.sql

Database Shell

Access the database:

docker exec -it celestia-indexer-db-1 psql -U postgres -d celestia

Or locally if PostgreSQL is installed:

psql -h localhost -U postgres -d celestia

API Development

Swagger/OpenAPI Documentation

Generate API documentation:

make api-docs

Access API docs at: http://localhost:9876/v1/docs

Adding New Endpoints

  1. Create handler in cmd/api/handler/
  2. Add request/response models
  3. Add enpoint in routes cmd/api/init.go
  4. Document with Swagger comments
  5. Add tests to route tests

Example handler structure:

// GetBlock returns block information
// @Summary Get block by height
// @Tags Block
// @Produce json
// @Param height path integer true "Block Height"
// @Success 200 {object} BlockResponse
// @Router /v1/blocks/{height} [get]
func (h *Handler) GetBlock(c echo.Context) error {
    // handler implementation
}

Debugging

Enabling Debug Logging

Set LOG_LEVEL=debug in .env:

LOG_LEVEL=debug

Using Delve Debugger

# Install delve
go install github.com/go-delve/delve/cmd/dlv@latest

# Debug indexer
dlv debug ./cmd/indexer

# At dlv prompt
(dlv) break main.main
(dlv) continue

Profiling

The application integrates Pyroscope for performance profiling:

  1. Enable in .env:
PYROSCOPE_ENABLED=true
PYROSCOPE_URL=http://localhost:4040
  1. View profiles at the Pyroscope URL

Checking Database Connections

# Connect to database
psql -h localhost -U postgres -d celestia

# Check active connections
SELECT count(*) FROM pg_stat_activity;

# View slow queries
SELECT query, calls, mean_exec_time 
FROM pg_stat_statements 
ORDER BY mean_exec_time DESC 
LIMIT 10;

Performance Tuning

Database Optimization

  1. Enable Query Stats (Docker Compose default):

    • -cpg_stat_statements.track=all flag enables query tracking

  2. Review Slow Queries:

    SELECT * FROM pg_stat_statements 
ORDER BY mean_exec_time DESC LIMIT 20;

  3. Analyze Table Plans:

    EXPLAIN ANALYZE SELECT * FROM blocks;

Indexer Performance

Adjust bulk size in .env:

INDEXER_REQUEST_BULK_SIZE=20  # Increase for faster processing

Adjust block period:

INDEXER_BLOCK_PERIOD=15  # Lower for more frequent checks

Development Best Practices

  1. Always use transactions for multi-step operations
  2. Handle context cancellation gracefully
  3. Use dependency injection for testability
  4. Write unit tests for new functions
  5. Document public APIs with godoc comments
  6. Use structured logging with zerolog
  7. Validate input in API handlers
  8. Use error wrapping with pkg/errors
  9. Follow Go conventions (gofmt, naming)

Contributing

When contributing to the project:

  1. Create a feature branch: git checkout -b feature/your-feature
  2. Make changes and write tests
  3. Run tests and linting
  4. Push and create a pull request
  5. Address review comments
  6. Merge when approved

Clone this wiki locally