Midnight Indexer

The Midnight Indexer (midnight-indexer) is a set of components designed to optimize the flow of blockchain data from a Midnight Node to end-user applications. It retrieves the history of blocks, processes them, stores indexed data efficiently, and provides a GraphQL API for queries and subscriptions. This Rust-based implementation is the next-generation iteration of the previous Scala-based indexer, offering improved performance, modularity, and ease of deployment.

                                ┌─────────────────┐
                                │                 │
                                │                 │
                                │      Node       │
                                │                 │
                                │                 │
                                └─────────────────┘
                                         │
┌────────────────────────────────────────┼────────────────────────────────────────┐
│                                        │                                        │
│                                        │ fetch                                  │
│                                        │ blocks                                 │
│                                        ▼                                        │
│                               ┌─────────────────┐                               │
│                               │                 │                               │
│                               │                 │                               │
│                               │      Chain      │                               │
│             ┌─────────────────│     Indexer     │                               │
│             │                 │                 │                               │
│             │                 │                 │                               │
│             │                 └─────────────────┘                               │
│             │                          │                                        │
│             │                          │ save blocks                            │
│             │                          │ and transactions                       │
│             │    save relevant         ▼                                        │
│             │     transactions   .───────────.                                  │
│      notify │        ┌─────────▶(     DB      )───────────────────┐             │
│ transaction │        │           `───────────'                    │             │
│     indexed │        │                                            │ read data   │
│             ▼        │                                            ▼             │
│    ┌─────────────────┐                                   ┌─────────────────┐    │
│    │                 │                                   │                 │    │
│    │                 │                                   │                 │    │
│    │     Wallet      │                                   │     Indexer     │    │
│    │     Indexer     │◀──────────────────────────────────│       API       │    │
│    │                 │  notify                           │                 │    │
│    │                 │  wallet                           │                 │    │
│    └─────────────────┘  connected                        └─────────────────┘    │
│                                                                   ▲             │
│                                                           connect │             │
│                                                                   │             │
└───────────────────────────────────────────────────────────────────┼─────────────┘
                                                                    │
                                 ┌─────────────────┐                │
                                 │                 │                │
                                 │                 │                │
                                 │     Wallet      │────────────────┘
                                 │                 │
                                 │                 │
                                 └─────────────────┘

Components

Chain Indexer: Connects to the Midnight Node, fetches blocks and transactions, and stores indexed data.
Wallet Indexer: Associates connected wallets with relevant transactions, enabling personalized queries and subscriptions.
Indexer API: Exposes a GraphQL API for queries, mutations, and subscriptions.

Features

Fetch and query blocks, transactions and contract actions at specific block hashes, heights, transaction identifiers or contract addresses.
Real-time subscriptions to new blocks, contract actions and wallet-related events through WebSocket connections.
Secure wallet sessions enabling clients to track only their relevant transactions.
Configurable for both cloud (microservices) and standalone (single binary) deployments.
Supports both PostgreSQL (cloud) and SQLite (standalone) storage backends.
Extensively tested with integration tests and end-to-end scenarios.

Running

To run the Midnight Indexer Docker images are provided under the midnightntwrk organization. It is supposed that users are familiar with running Docker images, e.g. via Docker Compose or Kubernetes.

Standalone Mode

The standalone Indexer combines the Chain Indexer, Indexer API and Wallet Indexer components in a single executable alongside an in-process SQLite database. Therefore the only Docker image to be run is indexer-standalone.

By default it connects to a local Midnight Node at ws://localhost:9944 and exposes its GraphQL API at 0.0.0.0:8088. All configuration has defaults except for the secret used to encrypt stored sensitive data which must be provided via the APP__INFRA__SECRET environment variable as valid hex-encoded 32 bytes.

indexer-standalone be configured by the following environment variables:

Env Var	Meaning	Default
APP__APPLICATION__NETWORK_ID	Network ID	`undeployed`
APP__INFRA__STORAGE__CNN_URL	SQlite connection URL	`/data/indexer.sqlite`
APP__INFRA__NODE__URL	WebSocket Endpoint of Midnight Node	`ws://localhost:9944`
APP__INFRA__API__PORT	Port of the GraphQL API	`8088`
APP__INFRA__SECRET	Hex-encoded 32-byte secret to encrypt stored sensitive data	-

For the full set of configuration options see config.yaml.

Cloud Mode

The Chain Indexer, Indexer API and Wallet Indexer can be run as separate executables, interacting with a PostgreSQL database and a NATS messaging system. Running PostgreSQL and NATS is out of scope of this document. The respective Docker images are:

`chain-indexer` Configuration

Env Var	Meaning	Default
APP__APPLICATION__NETWORK_ID	Network ID	`undeployed`
APP__INFRA__STORAGE__HOST	PostgreSQL host	`localhost`
APP__INFRA__STORAGE__PORT	PostgreSQL port	`5432`
APP__INFRA__STORAGE__DBNAME	PostgreSQL database name	`indexer`
APP__INFRA__STORAGE__USER	PostgreSQL database user	`indexer`
APP__INFRA__PUB_SUB__URL	NATS URL	`localhost:4222`
APP__INFRA__PUB_SUB__USERNAME	NATS username	`indexer`
APP__INFRA__LEDGER_STATE_STORAGE__URL	NATS URL	`localhost:4222`
APP__INFRA__LEDGER_STATE_STORAGE__USERNAME	NATS username	`indexer`
APP__INFRA__NODE__URL	WebSocket Endpoint of Midnight Node	`ws://localhost:9944`

For the full set of configuration options see config.yaml.

`indexer-api` Configuration

Env Var	Meaning	Default
APP__APPLICATION__NETWORK_ID	Network ID	`undeployed`
APP__INFRA__STORAGE__HOST	PostgreSQL host	`localhost`
APP__INFRA__STORAGE__PORT	PostgreSQL port	`5432`
APP__INFRA__STORAGE__DBNAME	PostgreSQL database name	`indexer`
APP__INFRA__STORAGE__USER	PostgreSQL database user	`indexer`
APP__INFRA__PUB_SUB__URL	NATS URL	`localhost:4222`
APP__INFRA__PUB_SUB__USERNAME	NATS username	`indexer`
APP__INFRA__LEDGER_STATE_STORAGE__URL	NATS URL	`localhost:4222`
APP__INFRA__LEDGER_STATE_STORAGE__USERNAME	NATS username	`indexer`
APP__INFRA__API__PORT	Port of the GraphQL API	`8088`
APP__INFRA__SECRET	Hex-encoded 32-byte secret to encrypt stored sensitive data	-

For the full set of configuration options see config.yaml.

`wallet-indexer` Configuration

Env Var	Meaning	Default
APP__APPLICATION__NETWORK_ID	Network ID	`undeployed`
APP__INFRA__STORAGE__HOST	PostgreSQL host	`localhost`
APP__INFRA__STORAGE__PORT	PostgreSQL port	`5432`
APP__INFRA__STORAGE__DBNAME	PostgreSQL database name	`indexer`
APP__INFRA__STORAGE__USER	PostgreSQL database user	`indexer`
APP__INFRA__PUB_SUB__URL	NATS URL	`localhost:4222`
APP__INFRA__PUB_SUB__USERNAME	NATS username	`indexer`
APP__INFRA__SECRET	Hex-encoded 32-byte secret to encrypt stored sensitive data	-

For the full set of configuration options see config.yaml.

Running Locally

For development, you can use Docker Compose or run components manually:

Using Docker Compose

A docker-compose.yaml file is provided that defines services for the Indexer components as well as for dependencies like postgres, nats, and node. The latter are particularly interesting when running Indexer components "manually".

Manual Startup

The justfile defines recipes for each Indexer component to start it alongside its dependencies. E.g. run the Chain Indexer like this:

just run-chain-indexer

Development Setup

Requirements

Rust: The latest stable version, see rust-toolchain.toml.
Cargo: Comes with Rust.
cargo-nextest: For running tests, see nextest.
Just: A command-runner used for tasks. Install from just.
direnv: For a reproducible development environment.
Docker: For integration tests and running services locally.
subxt-cli: For fetching metadata from the node, see subxt. Note: Version used must match version in Cargo.toml.
sql-formatter: For formatting SQL files. Install from sql-formatter and either integrate into your editor or run find . -type f -name "*.sql" -exec sh -c 'sql-formatter -o "$1" "$1"' _ {} \;.

Environment Variables

As we allow zero secrets in the git repository, you need to define a couple of environment variables for build (tests) and runtime (tests). Notice that the values are just used locally for testing and can be chosen arbitrarily; APP__INFRA__SECRET must be a hex-encoded 32-byte value.

It is recommended to provide these environment variables via a ~/.midnight-indexer.envrc or ./.envrc.local file which is sourced by the .envrc file:

export APP__INFRA__STORAGE__PASSWORD=postgres
export APP__INFRA__PUB_SUB__PASSWORD=nats
export APP__INFRA__LEDGER_STATE_STORAGE__PASSWORD=nats
export APP__INFRA__SECRET=303132333435363738393031323334353637383930313233343536373839303132

Required Configuration for private Repositories

You may need access to private Midnight repositories and container registries. To achieve this:

GitHub Personal Access Token (PAT)

Create a classic PAT with:

repo (all)
read:packages
read:org

~/.netrc Setup

machine github.com
login <YOUR_GITHUB_ID>
password <YOUR_GITHUB_PAT>

Docker Authentication

echo $GITHUB_TOKEN | docker login ghcr.io -u <YOUR_GITHUB_ID> --password-stdin

GPG Setup (Signed Git Commits)

All contributors are required to cryptographically sign their Git commits using GPG. This confirms the identity of each contributor and marks commits as verified.

Step 0: Prerequisites (once)

macOS

brew install gnupg pinentry-mac
# Optional: ensure macOS uses GUI pinentry for passphrase prompts
echo "pinentry-program $(which pinentry-mac)" > ~/.gnupg/gpg-agent.conf
killall gpg-agent || true

Linux (Ubuntu/Debian)

sudo apt-get update && sudo apt-get install -y gnupg pinentry-curses
# pinentry-curses gives a passphrase prompt in the terminal
# On desktops, you can install pinentry-gtk2 instead

Step 1: Generate a GPG key (ed25519)

gpg --quick-generate-key "Your Name <[email protected]>" ed25519 sign 2y

Replace Your name and [email protected] with the one you use for Git.
sign limits the key’s usage to signing (good hygiene).
2y sets the key to expire in 2 years (you can renew later).

Already have a key? List them with:

gpg --list-secret-keys --keyid-format=long

Copy the key ID (the long hex after sec) for the next step—looks like ABCDEF1234567890.

Step 2: Tell Git to always sign

Set this globally (applies to all repositories):

git config --global user.name "Your Name"
git config --global user.email "[email protected]"

# Use OpenPGP (GPG) for signing
git config --global gpg.format openpgp

# If your key ID is ABCDEF1234567890:
git config --global user.signingkey ABCDEF1234567890

# Always sign commits and tags
git config --global commit.gpgsign true
git config --global tag.gpgsign true

Step 3: Add the GPG_TTY line to your shell config (critical)

This ensures the pinentry prompt works in your terminal.

macOS (zsh default)

echo 'export GPG_TTY=$(tty)' >> ~/.zshrc
source ~/.zshrc

Linux (bash)

echo 'export GPG_TTY=$(tty)' >> ~/.bashrc
source ~/.bashrc

Step 4: Upload your public key to your Git host (GitHub/GitLab)

Export your public key:

gpg --armor --export ABCDEF1234567890

Copy the entire block including:

-----BEGIN PGP PUBLIC KEY BLOCK-----
...
-----END PGP PUBLIC KEY BLOCK-----

Then upload it to your Git host:

GitHub: Settings → SSH and GPG keys → “New GPG key” GitLab: Preferences → GPG Keys → “Add key”

Make sure your Git email matches the email on the key and on the remote host account settings.

Step 5: Test it

In any repo:

git commit --allow-empty -m "test: signed commit"
git log --show-signature -1

You should see a “Good signature” line. If it asks for a passphrase, that’s normal—the agent can cache it.

LICENSE

Apache 2.0.

SECURITY.md

Provides a brief description of the Midnight Foundation's security policy and how to properly disclose security issues.

CONTRIBUTING.md

Provides guidelines for how people can contribute to the Midnight project.

CODEOWNERS

Defines repository ownership rules.

CLA Assistant

The Midnight Foundation appreciates contributions, and like many other open source projects asks contributors to sign a contributor License Agreement before accepting contributions. We use CLA assistant (https://github.com/cla-assistant/cla-assistant) to streamline the CLA signing process, enabling contributors to sign our CLAs directly within a GitHub pull request.

Dependabot

The Midnight Foundation uses GitHub Dependabot feature to keep our projects dependencies up-to-date and address potential security vulnerabilities.

Checkmarx

The Midnight Foundation uses Checkmarx for application security (AppSec) to identify and fix security vulnerabilities. All repositories are scanned with Checkmarx's suite of tools including: Static Application Security Testing (SAST), Infrastructure as Code (IaC), Software Composition Analysis (SCA), API Security, Container Security and Supply Chain Scans (SCS).

Unito

Facilitates two-way data synchronization, automated workflows and streamline processes between: Jira, GitHub issues and Github project Kanban board.

Name		Name	Last commit message	Last commit date
Latest commit History 331 Commits
.cargo		.cargo
.config		.config
.github		.github
.node/0.18.0-rc.3		.node/0.18.0-rc.3
chain-indexer		chain-indexer
docs		docs
indexer-api		indexer-api
indexer-common		indexer-common
indexer-standalone		indexer-standalone
indexer-tests		indexer-tests
qa		qa
wallet-indexer		wallet-indexer
.dockerignore		.dockerignore
.envrc		.envrc
.gitignore		.gitignore
.midnight.txt		.midnight.txt
.sql-formatter.json		.sql-formatter.json
CHANGELOG.md		CHANGELOG.md
CODEOWNERS		CODEOWNERS
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
LICENSE		LICENSE
NODE_VERSION		NODE_VERSION
README.md		README.md
SECURITY.md		SECURITY.md
cliff.toml		cliff.toml
coverage.sh		coverage.sh
deny.toml		deny.toml
docker-compose.yaml		docker-compose.yaml
generate_node_data.sh		generate_node_data.sh
generate_txs.sh		generate_txs.sh
get_node_metadata.sh		get_node_metadata.sh
justfile		justfile
license_headers.sh		license_headers.sh
rust-toolchain.toml		rust-toolchain.toml
rustfmt.toml		rustfmt.toml
taplo.toml		taplo.toml

License

midnightntwrk/midnight-indexer

Folders and files

Latest commit

History

Repository files navigation

Midnight Indexer

Components

Features

Running

Standalone Mode

Cloud Mode

chain-indexer Configuration

indexer-api Configuration

wallet-indexer Configuration

Running Locally

Using Docker Compose

Manual Startup

Development Setup

Requirements

Environment Variables

Required Configuration for private Repositories

GitHub Personal Access Token (PAT)

~/.netrc Setup

Docker Authentication

GPG Setup (Signed Git Commits)

Step 0: Prerequisites (once)

Step 1: Generate a GPG key (ed25519)

Step 2: Tell Git to always sign

Step 3: Add the GPG_TTY line to your shell config (critical)

Step 4: Upload your public key to your Git host (GitHub/GitLab)

Step 5: Test it

LICENSE

SECURITY.md

CONTRIBUTING.md

CODEOWNERS

CLA Assistant

Dependabot

Checkmarx

Unito

About

Resources

License

Code of conduct

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases 12

Uh oh!

Contributors 13

Uh oh!

Languages

`chain-indexer` Configuration

`indexer-api` Configuration

`wallet-indexer` Configuration