Skip to content

Commit d66cb95

Browse files
0xneves0xneves
committed
refactor: concise readme per directory
1 parent 11b00be commit d66cb95

File tree

4 files changed

+375
-153
lines changed

4 files changed

+375
-153
lines changed

‎README.md‎

Lines changed: 17 additions & 153 deletions
Original file line numberDiff line numberDiff line change
@@ -1,176 +1,40 @@
1-
# Kadena Indexer
1+
# Kadindexer - Kadena Indexer
22

33
This project is a monorepo that contains the following packages:
44

5-
- `@kadena-indexer/indexer`: The indexer package, which is responsible for scanning and storing blocks for Kadena blockchain.
6-
- `@kadena-indexer/terraform`: The Terraform configuration for provisioning the infrastructure required to run the indexer and the node.
5+
- [`@kadena-indexer/indexer`](indexer/README.md): The indexer package, which is responsible for scanning and storing blocks for Kadena blockchain.
6+
- [`@kadena-indexer/terraform`](terraform/README.md): The Terraform configuration for provisioning the infrastructure required to run the indexer and the node.
7+
- [`@kadena-indexer/backfill`](backfill/README.md): The backfill package, which is responsible for backfilling the indexer data.
78

8-
## Prerequisites
9+
## Requirements
910

10-
- [Terraform](https://www.terraform.io/downloads.html)
11-
- [AWS CLI](https://aws.amazon.com/cli/)
12-
- [AWS Account](https://aws.amazon.com/)
13-
- [AWS Access Key](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html#access-keys-and-secret-access-keys)
14-
15-
### Dev Container
16-
17-
This project is configured to run in a dev container. You can use the `Dev Containers: Open Folder in Container` command in VSCode to open the project in a dev container. This will automatically install the required dependencies and set up the environment. To use the dev container, you need to have Docker installed on your machine.
18-
19-
If you don't have Dev Containers installed, you can install it from the [VSCode Marketplace](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers).
20-
21-
### Configure Environment Variables
22-
23-
Under the `/terraform` directory, create an `.env` file using the `.env.template` as a reference and set the environment variables accordingly.
24-
25-
```bash
26-
cp terraform/.env.template terraform/.env
27-
```
28-
29-
`TF_VAR_AWS_ACCESS_KEY_ID` is your AWS access key ID.
30-
`TF_VAR_AWS_SECRET_ACCESS_KEY` is your AWS secret access key.
31-
`TF_VAR_AWS_ACCOUNT_ID` is your AWS account ID.
32-
`TF_VAR_AWS_USER_NAME` is the name of the user you created in AWS.
33-
`TF_VAR_AWS_DB_USERNAME` is the username for the postgress database.
34-
`TF_VAR_AWS_DB_PASSWORD` is the password for the postgress database.
35-
36-
Under the `/indexer` directory, create an `.env` file using the `.env.template` as a reference and set the environment variables accordingly.
37-
38-
```bash
39-
cp indexer/.env.template indexer/.env
40-
```
41-
42-
`AWS_S3_REGION` is the region where the S3 bucket is located.
43-
`AWS_S3_BUCKET_NAME` is the name of the S3 bucket where the data will be stored.
44-
`AWS_ACCESS_KEY_ID` is the access key ID for the S3 bucket.
45-
`AWS_SECRET_ACCESS_KEY` is the secret access key for the S3 bucket.
46-
47-
`SYNC_BASE_URL` is the base URL for the Kadena node.
48-
`SYNC_MIN_HEIGHT` is the minimum height to start syncing from.
49-
`SYNC_FETCH_INTERVAL_IN_BLOCKS` is the interval in blocks to fetch.
50-
`SYNC_TIME_BETWEEN_REQUESTS_IN_MS` is the time between requests in milliseconds.
51-
`SYNC_ATTEMPTS_MAX_RETRY` is the maximum number of attempts to retry.
52-
`SYNC_ATTEMPTS_INTERVAL_IN_MS` is the interval in milliseconds between attempts.
53-
`SYNC_NETWORK` is the network to sync.
54-
55-
`DB_USERNAME` is the username for the postgress database.
56-
`DB_PASSWORD` is the password for the postgress database.
57-
`DB_NAME` is the name of the postgress database.
58-
`DB_HOST` is the host for the postgress database. You have the host after the resource creation, so you can check for this information in the AWS console or in terraform output (postgres_db_host).
59-
60-
### Initialize Terraform
61-
62-
Initialize your Terraform workspace, which will download the provider and initialize it with the values provided in the terraform.`tfvars`` file.
63-
64-
```bash
65-
terraform init
66-
```
67-
68-
### Deploy Infrastructure
69-
70-
Plan and apply the Terraform configuration to provision your AWS resources:
71-
72-
```bash
73-
yarn terraform plan
74-
yarn terraform apply
75-
```
76-
77-
### Destroy Infrastructure
78-
79-
If you want to destroy the infrastructure created, you can use the following command:
80-
81-
```bash
82-
yarn terraform destroy
83-
```
11+
- Install dependencies
12+
- See individual package READMEs for specific prerequisites
8413

8514
## Installation
8615

87-
Set up the indexer with the following commands:
16+
Install dependencies with the following command:
8817

8918
```bash
90-
yarn && yarn indexer build
19+
yarn install
9120
```
9221

93-
## Features
22+
## Quick Start
9423

95-
### Run processing
24+
This is the quickest way to get the indexer running.
9625

97-
Continuous process of streaming, headers, payloads and missing blocks from node to s3 bucket and from s3 bucket to database
26+
Install [Docker](https://www.docker.com/).
9827

99-
```bash
100-
yarn indexer dev:run
101-
```
102-
103-
## Additional Commands
104-
105-
### Running with Docker
28+
Fill the `.env` file in the `indexer` folder. See [Environment Variables Reference](../indexer/README.md#32-environment-variables-reference).
10629

10730
```bash
108-
sudo docker build -t kadena-indexer:latest .
109-
sudo docker run --env-file ./indexer/.env -p 3000:3000 kadena-indexer:latest
110-
```
111-
112-
### Backfilling Blocks
113-
114-
Scan for and store historical blocks.
115-
116-
```bash
117-
yarn indexer dev:backfill
118-
```
119-
120-
### Streaming Blocks
121-
122-
Listen for new blocks and store them in real-time.
123-
124-
```bash
125-
yarn indexer dev:streaming
126-
```
127-
128-
### Identifying Missing Blocks
129-
130-
Scan for and store any blocks that were missed.
131-
132-
```bash
133-
yarn indexer dev:missing
134-
```
135-
136-
### Processing Headers
137-
138-
Start the header processing from S3 to the database.
139-
140-
```bash
141-
yarn indexer dev:headers
142-
```
143-
144-
### Processing Payloads
145-
146-
Start the payload processing from S3 to the database.
147-
148-
```bash
149-
yarn indexer dev:payloads
150-
```
151-
152-
## Advanced Usage
153-
154-
### Local Workflow Testing
155-
156-
For testing workflows locally, act is required. Install it using Homebrew:
157-
158-
```bash
159-
brew install act
31+
cp indexer/.env.template indexer/.env
16032
```
16133

162-
### Run Terraform Workflow Manually
163-
164-
If you want to run the terraform workflow manually, you can use the following command:
165-
34+
To start all services:
16635
```bash
167-
yarn run-terraform-workflow
36+
yarn indexer dev
16837
```
16938

170-
### Run Indexer Workflow Manually
171-
172-
If you want to run the indexer workflow manually, you can use the following command:
39+
**NOTE:** Using the image on with the composer require the database `DB_USERNAME` to default to `postgres`.
17340

174-
```bash
175-
yarn run-indexer-workflow
176-
```

‎backfill/README.md‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
# Kadena Indexer Backfill
2+
3+
## 1. Introduction
4+
5+
The Kadindexer Backfill is a utility tool designed to synchronize historical blockchain data from the Kadena network into your local database. It allows you to fetch and index past blocks and transactions, ensuring your database has a complete history of the chain. The backfill process can be configured to sync data from any specified block height, making it useful for both initial data population and recovery scenarios where data needs to be resynced from a particular point.
6+
7+
## 2. Prerequisites
8+
9+
- [Docker](https://www.docker.com/)
10+
- Kadena Indexer PostgreSQL database running
11+
- Network access to the Kadena network
12+
- Running your own Kadena node
13+
14+
## 3. Setup
15+
16+
### 3.1. Starting Docker
17+
Start Docker Desktop from command line or via IOS application.
18+
19+
```bash
20+
# MacOS - Start Docker Desktop from command line
21+
open -a Docker
22+
23+
# Linux - Start Docker daemon
24+
sudo systemctl start docker
25+
```
26+
27+
### 3.2. Environment Variables
28+
29+
| Variable | Description | Example |
30+
|----------|-------------|---------|
31+
| `CERT_PATH` | Path to SSL certificate bundle | `./global-bundle.pem` |
32+
| `SYNC_BASE_URL` | Base URL for the Chainweb API | `https://api.chainweb.com/chainweb/0.0` |
33+
| `CHAIN_ID` | ID of the chain to backfill | `0` |
34+
| `NETWORK` | Kadena network to sync from | `mainnet01` |
35+
| `SYNC_MIN_HEIGHT` | Starting block height for backfill | `5370495` |
36+
| `SYNC_FETCH_INTERVAL_IN_BLOCKS` | Number of blocks to fetch in each interval | `100` |
37+
| `SYNC_ATTEMPTS_MAX_RETRY` | Maximum number of retry attempts | `5` |
38+
| `SYNC_ATTEMPTS_INTERVAL_IN_MS` | Interval between retry attempts in milliseconds | `500` |
39+
| `DB_USERNAME` | PostgreSQL database username | `postgres` |
40+
| `DB_PASSWORD` | PostgreSQL database password | `password` |
41+
| `DB_NAME` | Name of the database | `indexer` |
42+
| `DB_HOST` | Database host address | `localhost` |
43+
| `DB_PORT` | Database port number | `5432` |
44+
45+
**NOTE:** The example Kadena node API from chainweb will not work for the indexer purpose. You will need to run your own Kadena node and set the `NODE_API_URL` to your node's API URL.
46+
47+
## 4. Usage
48+
49+
### 4.1. Start the Kadindexer services
50+
51+
Please refer to the [Kadena Indexer README](../indexer/README.md) for instructions on how to start the Kadindexer services.
52+
53+
### 4.2. Build the backfill image
54+
55+
Build the image:
56+
```bash
57+
docker build -t chainbychain -f Dockerfile .
58+
```
59+
60+
### 4.3. Run the container
61+
62+
#### Dockerfile (Chain by Chain)
63+
This Dockerfile is designed to run the backfill process for a single chain at a time. It's useful when you need to:
64+
- Sync data for a specific chain ID
65+
- Have more granular control over the backfill process
66+
- Debug issues with a particular chain
67+
- Manage resources more efficiently
68+
69+
#### Dockerfile.indexes
70+
This Dockerfile is specifically for recreating database indexes. Use this when you need to:
71+
- Rebuild corrupted indexes
72+
- Optimize existing indexes
73+
- Add new indexes to improve query performance
74+
- Perform database maintenance
75+
76+
#### Dockerfile.middle-backfill
77+
This Dockerfile orchestrates the backfill process across all chains simultaneously. It's beneficial when you want to:
78+
- Perform a complete system backfill
79+
- Sync data for all chains in parallel
80+
- Save time by running multiple chain syncs concurrently
81+
- Ensure consistency across all chains
82+
83+
For single chain backfill:
84+
```bash
85+
docker build -t chainbychain -f Dockerfile .
86+
docker run --rm --name chainbychain --env-file .env chainbychain
87+
```
88+
89+
For rebuilding indexes:
90+
```bash
91+
docker build -t rebuild-indexes -f Dockerfile.indexes .
92+
docker run --rm --name rebuild-indexes --env-file .env rebuild-indexes
93+
```
94+
95+
For all chains backfill:
96+
```bash
97+
docker build -t all-chains -f Dockerfile.middle-backfill .
98+
docker run --rm --name all-chains --env-file .env all-chains
99+
```

0 commit comments

Comments
 (0)