Skip to content

Development

Jump to bottom
Artem Poltorzhitskiy edited this page Mar 24, 2026 · 2 revisions

Development Guide

Prerequisites

Tool Minimum Version
Go 1.26.x
Docker + Docker Compose latest stable
Make any
golangci-lint latest

For DB integration tests, Docker must be running (tests spin up a real TimescaleDB container via testcontainers).

Initial Setup

git clone https://github.com/celenium-io/celestia-indexer.git
cd celestia-indexer
make init      # chmod scripts + first-time dev setup

Configuration

Configuration is provided via configs/dipdup.yml using ${ENV_VAR:-default} substitution. Set environment variables in your shell or in a .env file loaded before running services.

Full Environment Variable Reference

Datasources

Variable Default Description
CELESTIA_NODE_URL — CometBFT RPC endpoint
CELESTIA_NODE_RPS — RPC requests per second limit
CELESTIA_NODE_TIMEOUT — RPC request timeout
CELESTIA_NODE_API_URL — Celestia node REST API endpoint
CELESTIA_NODE_API_RPS — REST API requests per second limit
CELESTIA_NODE_API_TIMEOUT — REST API timeout
CELESTIA_NODE_WS_URL — CometBFT WebSocket endpoint
CELESTIA_DAL_API_URL — DAL API endpoint (blob retrieval)
CELESTIA_DAL_API_RPS — DAL API requests per second limit
CELESTIA_DAL_API_TIMEOUT — DAL API timeout
CELESTIALS_API_URL — Celestials profile API endpoint
CELESTIALS_API_RPS — Celestials API requests per second limit
CELESTIALS_API_TIMEOUT — Celestials API timeout
CELENIUM_BLOBS_API_URL — Celenium blobs API endpoint
CELENIUM_BLOBS_API_RPS — Celenium blobs API requests per second limit
CELENIUM_BLOBS_API_TIMEOUT — Celenium blobs API timeout

Database

Variable Default Description
POSTGRES_HOST db PostgreSQL host
POSTGRES_PORT 5432 PostgreSQL port
POSTGRES_USER — PostgreSQL user
POSTGRES_PASSWORD — PostgreSQL password
POSTGRES_DB celestia Database name
POSTGRES_MAX_OPEN_CONNECTIONS 100 Max open connections
POSTGRES_MAX_IDLE_CONNECTIONS 100 Max idle connections
POSTGRES_MAX_LIFETIME_CONNECTIONS 0 Max connection lifetime (0 = unlimited)

Indexer

Variable Default Description
LOG_LEVEL info Log level (debug, info, warn, error)
INDEXER_NAME celestia_indexer Indexer instance name
INDEXER_BLOCK_PERIOD 15s Block polling period
INDEXER_SCRIPTS_DIR ./database Directory with SQL scripts and views
INDEXER_START_LEVEL — Block height to start indexing from
INDEXER_REQUEST_BULK_SIZE 10 Number of blocks fetched per bulk request
INDEXER_FETCH_CONCURRENCY 1 Parallel block fetch goroutines
INDEXER_DISABLE_GZIP false Disable gzip compression for node requests
NETWORK — Network name (celestia, mocha, arabica)

Public API

Variable Default Description
API_HOST 0.0.0.0 API bind address
API_PORT 9876 API bind port
API_RATE_LIMIT 0 Requests per second limit (0 = unlimited)
API_PROMETHEUS_ENABLED true Enable Prometheus /metrics endpoint
API_REQUEST_TIMEOUT 30 Request timeout in seconds
API_WEBSOCKET_ENABLED true Enable WebSocket endpoint
API_WEBSOCKET_CLIENTS_PER_IP 10 Max WebSocket connections per IP
BLOB_RECEIVER dal_api Blob retrieval backend (dal_api or celenium_blobs)
CACHE_URL — Valkey/Redis connection URL
CACHE_DEFAULT_TTL 5 Default cache TTL in seconds
HYPERLANE_NODE_URL — Hyperlane node URL for chain store
SENTRY_DSN — Sentry DSN for error tracking

Private API

Variable Default Description
PRIVATE_API_HOST 0.0.0.0 Private API bind address
PRIVATE_API_PORT 9877 Private API bind port
PRIVATE_API_RATE_LIMIT 0 Rate limit (0 = unlimited)
PRIVATE_API_REQUEST_TIMEOUT 30 Request timeout in seconds

Celestials

Variable Default Description
CELESTIALS_CHAIN_ID celestia-1 Chain ID for Celestials indexer

Other

Variable Default Description
CELENIUM_ENV production Environment name
PROFILER_SERVER — Pyroscope profiler server URL

Running Services

All services via Docker Compose (recommended)

make compose
# equivalent to: docker-compose up -d --build

This starts: TimescaleDB 15.8, Valkey 8.0.2, indexer, public API, private API.

Individual services

make indexer      # go run ./cmd/indexer -c ./configs/dipdup.yml
make api          # go run ./cmd/api -c ./configs/dipdup.yml
make private_api  # go run ./cmd/private_api -c ./configs/dipdup.yml
make celestials   # go run ./cmd/celestials -c ./configs/dipdup.yml

Build binaries

make build     # builds all binaries to ./bin/
make clean     # removes ./bin/

Testing

make test                                         # run all tests (parallel, 120s timeout)
go test ./internal/storage/postgres/... -v        # run DB integration tests
go test ./cmd/api/handler/... -timeout 30s        # run API handler tests
go test ./... -run TestBlockByHeight -v           # run a specific test
make test-api                                     # Newman API integration tests
make cover                                        # generate HTML coverage report

Note: DB integration tests require Docker running — they spin up a real TimescaleDB container via testcontainers.

Code Generation

make generate    # regenerate mocks (go.uber.org/mock) and enum types
make api-docs    # regenerate Swagger docs (swag init)
make ga          # generate + api-docs combined

Run make generate after:

  • Adding or changing a storage interface
  • Adding or changing enum types in internal/storage/types/

Never edit *_enum.go files manually — they are fully code-generated.

Linting

make lint    # golangci-lint run

Active linters to pay attention to: zerologlint, musttag, gosec, containedctx.

Database Migrations

Migrations live in internal/storage/postgres/migrations/, named by date (e.g., 20260320000001_description.go).

Each file registers one migration via init():

func init() {
    Migrations.MustRegister(upXxx, downXxx)
}

Both up and down functions are required. Never leave down as a stub.

PostgreSQL-specific rollback notes:

  • Removing an added enum value: DELETE FROM pg_enum WHERE enumlabel = '...' AND enumtypid = (SELECT oid FROM pg_type WHERE typname = '...') — only safe if no rows use that value
  • Dropping a column: ALTER TABLE ... DROP COLUMN IF EXISTS ...
  • Dropping an index: DROP INDEX IF EXISTS ...

Debugging

Enable debug logging

Set LOG_LEVEL=debug in your environment.

Delve debugger

dlv debug ./cmd/api -- -c ./configs/dipdup.yml

Pyroscope profiling

Set PROFILER_SERVER to your Pyroscope server URL. The profiler is initialized automatically when the variable is set.

Direct DB access

docker-compose exec db psql -U $POSTGRES_USER -d $POSTGRES_DB

License Headers

All new Go source files must include SPDX license headers:

// SPDX-FileCopyrightText: 2025 Bb Strategy Pte. Ltd. <celenium@baking-bad.org>
// SPDX-License-Identifier: MIT

Apply headers to all files automatically:

make license-header

Clone this wiki locally