Skip to content

Commit 18129be

Browse files
nman98nman98
committed
Added some documentation, added docker file and docker compose, removed some tests that were more of an experiment and some duplicates, added workflow that tests go code
1 parent c7f229a commit 18129be

14 files changed

Lines changed: 579 additions & 335 deletions

File tree

‎.github/workflows/gotest.yml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
name: Go Test
2+
on:
3+
push:
4+
branches: [dev]
5+
pull_request:
6+
branches: [main, dev]
7+
8+
jobs:
9+
lint:
10+
runs-on: ubuntu-latest
11+
steps:
12+
- name: Checkout code
13+
uses: actions/checkout@v4
14+
15+
- name: Setup Go
16+
uses: actions/setup-go@v4
17+
with:
18+
go-version: '1.25.1'
19+
20+
- name: Run tests
21+
run: go test ./...

‎Dockerfile‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
FROM golang:1.25.1-bullseye AS builder
2+
3+
WORKDIR /app
4+
5+
COPY . .
6+
7+
RUN go build -o indexer indexer/main.go
8+
9+
RUN chmod +x indexer
10+
11+
FROM debian:bullseye-slim
12+
13+
WORKDIR /app
14+
15+
COPY --from=builder /app/indexer /app/indexer
16+
17+
ENTRYPOINT ["/app/indexer"]

‎Makefile‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,21 @@
1-
.PHONY: build-indexer install-indexer clean experimental-indexer experimental-install-indexer
1+
.PHONY: build install clean build-experimental install-experimental
22

3-
build-indexer:
3+
build:
44
mkdir -p build
55
go build -o build/indexer indexer/main.go
66

7-
install-indexer:
7+
install:
88
go install indexer/main.go
99

1010
clean:
1111
rm -rf build
1212

1313
# experimental build with greentea garbage collection
1414
# use at your own risk
15-
experimental-indexer:
15+
build-experimental:
1616
GOEXPERIMENT=greenteagc go build -o build/indexer indexer/main.go
1717

1818
# experimental install with greentea garbage collection
1919
# use at your own risk
20-
experimental-install-indexer:
20+
install-experimental:
2121
GOEXPERIMENT=greenteagc go install indexer/main.go

‎README.md‎

Lines changed: 155 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,155 @@
1-
# spectra_gnoland_indexer
2-
A Gnoland blockchain indexer for storing data. It's main usecase is for Spectra explorer but it can be used for any kind of data/analytics program or app.
1+
# The Spectra 💫 Gnoland Indexer
2+
3+
The Spectra Gnoland Indexer(SGI) is a tool that records the data from the Gnoland blockchain and stores the data
4+
in the timeseries database, Timescale DB. This indexer will be a part of the Spectra explorer and will be used to
5+
store the data for the explorer. This program can be used for any kind of data/analytics program or app.
6+
7+
The biggest problem when dealing with the blockchain data is how to store the data in a way that is easy to query
8+
and analyze. Some projects rely on the direct access to the data from the blockchain nodes and the problem in this
9+
case is that the nodes are meant for multiple purposes. They usually provide at least one, sometimes even more,
10+
endpoints for querying the data, then this node need to be able to work with other nodes and to acquire the block
11+
data from other nodes via P2P network. Most nodes can be used as a miners/validators that need to sign the blocks,
12+
and maintain the blockchain state. While they are the first point of contact for data and contact with the chain
13+
they are not most efficient when it comes to making some queries easy and fast. Plus the nodes are made to be fast
14+
and efficient when it comes to preformance of the nodes, so they are not most efficient when it comes to storing
15+
the data.
16+
17+
## Content
18+
- [The solution](#the-solution)
19+
- [Why TimescaleDB? And can it work on other SQL databases?](#why-timescaledb-and-can-it-work-on-other-sql-databases)
20+
- [The indexer and how it works](#the-indexer-and-how-it-works)
21+
- [What is indexed and what is not](#what-is-indexed-and-what-is-not)
22+
- [Pros and cons of the SGI](#pros-and-cons-of-the-sgi)
23+
- [🦾 Pros:](#pros)
24+
- [🐞 Cons:](#cons)
25+
26+
## The solution
27+
28+
The data in the blockchain is mostly tied to blocks, however when any kind of analytics is needed we need some
29+
parameter that falls familiar to us from the daily life. So it comes more natural to compare the data with some
30+
sort of time parameter then to a block height. Then the data needs to be stored in a way that is easy to query and
31+
analyze. And since we are dealing with the time parameter, we need to have the ability to aggregate the data over
32+
some time period so we can timeseries and easily plot the data.
33+
34+
A lot of indexers use NoSQL databases for this use case. However they usually have their own problems and
35+
limitations. They are not as flexible as the SQL databases when it comes to the data types and the query language.
36+
that is the reason why the TimescaleDB is a perfect fit for this use case.
37+
38+
Anyone with some experience with the SQL, especially with the PostgreSQL, can easily understand the TimescaleDB.
39+
It is a extension of the PostgreSQL that adds the ability to store the time series data in a way that is easy to
40+
query and analyze.
41+
42+
## Why TimescaleDB? And can it work on other SQL databases?
43+
44+
TimescaleDB (Tiger Data is their commercial version) is a extension of the PostgreSQL that adds the ability to
45+
store the time series data in a way that is easy to query and analyze. It also has a lot of features that can
46+
extend the capabilities of the PostgreSQL and make it more powerful.
47+
48+
Technically speaking the indexer can work on Postgres database, however you would need to create the tables and
49+
types manually. This might be added later in the future if there is a demand for it but for now the focus is on the
50+
TimescaleDB.
51+
52+
There are some other SQL databases that could work in theory. For them to work they would need to have:
53+
- The postgres wire protocol
54+
- Equivalent of Numeric type ( some databases might have it as DECIMAL, but not all of them )
55+
- The ability to create custom types
56+
57+
If all of the above are met then the indexer can work on them. It might work on CockroachDB for example but this is
58+
out of the scope of this project. Maybe in the future it might be an interesting idea to support it.
59+
60+
There were some other contenders for the database choice. They were: InfluxDB and QuestDB.
61+
62+
At the time when the cosmos indexer was made the InfluxDB had a opensource version 2 but at the time it was lacking
63+
some of the features that are now present in the version 3. Maybe it could have been used for the indexer but at
64+
the time it seemed to be not the best choice.
65+
66+
QuestDB was a interesting choice and it is probably even better choice than TimescaleDB. But it does have some
67+
limitations but nothing that would make it impossible to use it. However with the TimescaleDB it is pretty much an
68+
extention of Postgres and it is more mature and has more features and it can be extended with any other extensions
69+
that are available for the Postgres.
70+
71+
The TimescaleDB has also feature of it's own that is not present in the Postgres. The user could extend the indexer
72+
by adding the data aggregation features, automatic jobs scheduling, hyperfunctions and more.
73+
74+
## The indexer and how it works
75+
76+
The indexer has 2 main modes of operation:
77+
- Live mode
78+
- Historic mode
79+
80+
Live mode is the mode that is used to index the data in the real time. It will sync up the database to the latest
81+
block height and will continue to index the data in the real time.
82+
83+
Historic mode is the mode that allows the user to index a certain range of blocks. For example you might not need
84+
the whole chain history or just need a part of it. This mode is useful for the testing, partial indexing of the
85+
chain or gradual indexing of the chain.
86+
87+
Durring both modes the indexer will use fan out method to send the requests to the RPC node. Then all of the data
88+
is collected and processed and decoded in parallel. All of the addresses are collected and stored in the database
89+
and the indexer stores them in it's own cache. Then any address that is found in the transaction is referanced by
90+
the integer value in the transaction tables. At the end of the processing the data is inserted in batches into the
91+
database.
92+
93+
## What is indexed and what is not
94+
95+
The indexer stores the esential data that is related to the blocks, transactions, messages, and accounts.
96+
It also stores the validator block signings and the validator addresses.
97+
This way it is able to provide a lot of data for the analytics and the explorer.
98+
The data is stored in the following tables:
99+
- blocks
100+
- transactions general data
101+
- messages (each message type has its own table)
102+
- regular and validator addresses (each address type has its own table)
103+
- validator block signings
104+
- validator addresses
105+
- address transactions (to tie the addresses to the transactions)
106+
107+
Some of the data is not indexed and it is not planned to be indexed in the future. Such as:
108+
109+
### Blocks:
110+
Stored data:
111+
- Basic block hash
112+
- Block height
113+
- Block timestamp
114+
- Block chain ID
115+
- Block proposer address
116+
- Block transactions hashes
117+
118+
Not stored data:
119+
- Last commit hash
120+
- App hash
121+
- Data hash
122+
- Validators hash
123+
- Next validators hash
124+
- Consensus hash
125+
126+
### Validator signings:
127+
Stored data:
128+
- Validator block signing height
129+
- Validator block signing timestamp
130+
- Validator block signing signed validators
131+
132+
Not stored data:
133+
- Missed validators
134+
- Precommits and all of the hashes and other data, so only a confirmation that the validator signed the block
135+
136+
### Transactions:
137+
Almost all of the data regarding to the transaction and the messages are stored with the exception for the
138+
VM message Add Package and Call where in theory one could extract even the body of the smart contract. So this is unnecessary to store for explorers and other analytics tools. This might be added in the future if there is a demand for it.
139+
140+
## Pros and cons of the SGI
141+
142+
### 🦾 Pros:
143+
144+
- The indexer process the data using goroutines and channels, which can provide a faster processing of the data.
145+
- The program has 2 modes that can be used for the indexing of the data. This can be useful for the testing, partial indexing of the chain or gradual indexing of the chain.
146+
- The data is stored ready to be used for any kind of analytics and visualization with any programming language.
147+
- No need to deal with Amino encoding and decoding for the messagess as the indexer decodes the messages and stores them in the database.
148+
- It comes with a REST API to get you started quickly.(To be added in the future)
149+
- It relies on a SQL database, which can provide a easier experience for any user that is familiar with the SQL.
150+
- It uses TimescaleDB, a PostgresSQL extenstion that can be extended with any other extensions, plus the TimescaleDB has a lot of features that are not present in the Postgres.
151+
152+
### 🐞 Cons:
153+
154+
- The address cache is very preformant but it intorduces a complexity for the user. The developer that plans to use the indexer needs to understand how the addresses are stored and how to use them. The REST API provies a easy way to interact with the data and to get the data in a readable format. But if you are planning on making something custom you will need to fully understand the data structure and how to use it.
155+
- The indexer relies on the RPC node for the data. If the RPC node is not available the indexer will not be able to index the data. ( although in the future the indexer might be able to use multiple RPC nodes )

‎docker-compose.yml‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
services:
2+
3+
timescaledb:
4+
image: timescale/timescaledb-ha:pg17
5+
command: -c 'max_connections=500'
6+
container_name: timescale-gnoland
7+
restart: always
8+
ports:
9+
- "127.0.0.1:5432:5432"
10+
environment:
11+
POSTGRES_PASSWORD: 12345678
12+
POSTGRES_DB: gnoland
13+
volumes:
14+
- timescale_data:/home/postgres/pgdata
15+
networks:
16+
- gnoland
17+
healthcheck:
18+
test: ["CMD-SHELL", "pg_isready -U postgres -d gnoland"]
19+
interval: 10s
20+
timeout: 5s
21+
retries: 5
22+
23+
# first build the docker image
24+
# all of the envs you will need to change to values
25+
# as for the indexer mode the default here is live however look at deployment.md
26+
# for more information
27+
#indexer:
28+
# image: cogwheel/spectra-gnoland-indexer:latest
29+
# container_name: indexer-gnoland
30+
# restart: always
31+
# environment:
32+
# DB_HOST: timescale-gnoland
33+
# DB_PORT: 5432
34+
# DB_USER: postgres
35+
# DB_PASSWORD: 12345678
36+
# DB_NAME: gnoland
37+
# networks:
38+
# - gnoland
39+
# depends_on:
40+
# timescaledb:
41+
# condition: service_healthy
42+
#
43+
# volumes:
44+
# - ./config.yml:/app/config.yml
45+
# command: ["app/indexer", "live", "--config", "app/config.yml"]
46+
47+
48+
networks:
49+
gnoland:
50+
driver: bridge
51+
52+
volumes:
53+
timescale_data:

‎docs/README.md‎

Whitespace-only changes.

‎docs/deployment.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Live mode with custom config
2+
docker run your-indexer live --config /path/to/config.yml --skip-db-check
3+
4+
# Historic mode
5+
docker run your-indexer historic --from-height 1000 --to-height 2000
6+
7+
# Show all available commands
8+
docker run your-indexer --help
9+
10+
# Show help for specific command
11+
docker run your-indexer live --help

‎docs/requriements.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Requirements
2+
3+
The indexer could can run on one machine while the database is located on another machine. You can also use the
4+
Tiger Data (Timescaledb cloud edition) which is a managed service by Tiger Data which is a company that builds the
5+
Timescaledb.
6+
7+
However for the ease of understanding through the documentation will refer that the indexer and the database are located on the same machine.
8+
9+
## Hardware requirements
10+
11+
To run the indexer you need to have the following system requirements:
12+
13+
Minimum system requirements:
14+
15+
- 2vCPUs
16+
- 8GB RAM
17+
18+
Recommended system requirements:
19+
20+
- 4vCPUs
21+
- 16GB RAM
22+
23+
The indexer could probably run on ARM64 architecture but it is not tested yet. So stick with the x86_64 architecture.
24+
25+
For the storage it can work on the HDD, but if the Gnoland chain grows in popularity and number of users increases,
26+
the HDD might not be a good choice in the long term. So maybe a regular SATA SSD could be a better choice.
27+
But if you are just running some testing or just a small deployment, the HDD will do the job most of the time.
28+
As for the size it depends on the amount of the data that you are indexing.
29+
For example the integration test did about 400K blocks and 1 million
30+
transactions. So it took around 2.2 GB of disk space. There are a lot of things that can affect the size of the
31+
database but at least use this as some reference point to how much space you might need.
32+
33+
For the RAM and CPU it kinda depends but for now this is a good starting point. As the database size grows, you
34+
might need to increase the RAM and CPU.
35+
36+
## Software and OS requirements
37+
38+
The following software and OS requirements are required to run the indexer:
39+
40+
- Go 1.25.1
41+
- TimescaleDB 2.18 or higher but with PostgresSQL 16 or higher
42+
- OS: Linux, anything based on Debian(Ubuntu, Mint, etc.) or RHEL(CentOS stream, Rocky Linux, etc.) should work, openSUSE also ok
43+
- Docker (optional)

0 commit comments

Comments
 (0)