You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<palign="center"style="font-style: italic;">This project is still in development. Future releases might have some breaking changes.</p>
5
6
6
7
The Spectra Gnoland Indexer(SGI) is a tool that records the data from the Gnoland blockchain and stores the data
7
8
in the timeseries database, Timescale DB. This indexer will be a part of the Spectra explorer and will be used to
8
9
store the data for the explorer. This program can be used for any kind of data/analytics program or app.
9
10
10
11
The biggest problem when dealing with the blockchain data is how to store the data in a way that is easy to query
11
12
and analyze. Some projects rely on the direct access to the data from the blockchain nodes and the problem in this
12
-
case is that the nodes are meant for multiple purposes. They usually provide at least one, sometimes even more,
13
-
endpoints for querying the data, then this node need to be able to work with other nodes and to acquire the block
14
-
data from other nodes via P2P network. Most nodes can be used as a miners/validators that need to sign the blocks,
15
-
and maintain the blockchain state. While they are the first point of contact for data and contact with the chain
16
-
they are not most efficient when it comes to making some queries easy and fast. Plus the nodes are made to be fast
17
-
and efficient when it comes to preformance of the nodes, so they are not most efficient when it comes to storing
18
-
the data.
13
+
case is that the nodes are meant for multiple purposes.
14
+
15
+
They usually provide at least one, sometimes even more, endpoints for querying the data, then this node need to be
16
+
able to work with other nodes and to acquire the block data from other nodes via P2P network. Most nodes can be
17
+
used as a miners/validators that need to sign the blocks. Even if these nodes are only to be used as a RPC nodes
18
+
they can still provide the data but depending on the underlying SDK, tech stack and other factors it might be
19
+
not the best choice. And finally the nodes are not the best when it comes to storing the data and scalability.
19
20
20
21
## Table of content
21
22
-[Table of content](#table-of-content)
22
23
-[The solution](#the-solution)
23
24
-[Why TimescaleDB? And can it work on other SQL databases?](#why-timescaledb-and-can-it-work-on-other-sql-databases)
24
-
-[The indexer and how it works](#the-indexer-and-how-it-works)
25
+
-[How does the indexer work?](#how-does-the-indexer-work)
25
26
-[What is indexed and what is not](#what-is-indexed-and-what-is-not)
26
27
-[Blocks:](#blocks)
27
28
-[Validator signings:](#validator-signings)
@@ -53,6 +54,12 @@ TimescaleDB (Tiger Data is their commercial version) is a extension of the Postg
53
54
store the time series data in a way that is easy to query and analyze. It also has a lot of features that can
54
55
extend the capabilities of the PostgreSQL and make it more powerful.
55
56
57
+
The TimescaleDB has also feature of it's own that is not present in the Postgres. The user could extend the indexer
58
+
by adding the data aggregation features, automatic jobs scheduling, hyperfunctions and more.
59
+
60
+
This database extension also has a data compression feature that can be used to reduce the storage space and
61
+
segment the data into smaller chunks by using time based intervals making the queries faster.
62
+
56
63
Technically speaking the indexer can work on Postgres database, however you would need to create the tables and
57
64
types manually. This might be added later in the future if there is a demand for it but for now the focus is on the
58
65
TimescaleDB.
@@ -65,21 +72,7 @@ There are some other SQL databases that could work in theory. For them to work t
65
72
If all of the above are met then the indexer can work on them. It might work on CockroachDB for example but this is
66
73
out of the scope of this project. Maybe in the future it might be an interesting idea to support it.
67
74
68
-
There were some other contenders for the database choice. They were: InfluxDB and QuestDB.
69
-
70
-
At the time when the cosmos indexer was made the InfluxDB had a opensource version 2 but at the time it was lacking
71
-
some of the features that are now present in the version 3. Maybe it could have been used for the indexer but at
72
-
the time it seemed to be not the best choice.
73
-
74
-
QuestDB was a interesting choice and it is probably even better choice than TimescaleDB. But it does have some
75
-
limitations but nothing that would make it impossible to use it. However with the TimescaleDB it is pretty much an
76
-
extention of Postgres and it is more mature and has more features and it can be extended with any other extensions
77
-
that are available for the Postgres.
78
-
79
-
The TimescaleDB has also feature of it's own that is not present in the Postgres. The user could extend the indexer
80
-
by adding the data aggregation features, automatic jobs scheduling, hyperfunctions and more.
81
-
82
-
## The indexer and how it works
75
+
## How does the indexer work?
83
76
84
77
The indexer has 2 main modes of operation:
85
78
- Live mode
@@ -136,6 +129,7 @@ Stored data:
136
129
- Validator block signing height
137
130
- Validator block signing timestamp
138
131
- Validator block signing signed validators
132
+
- Proposer address
139
133
140
134
Not stored data:
141
135
- Missed validators
@@ -149,11 +143,12 @@ VM message Add Package and Call where in theory one could extract even the body
149
143
150
144
### 🦾 Pros:
151
145
152
-
- The indexer process the data using goroutines and channels, which can provide a faster processing of the data. Some early test on the real data showed about 2vCPU can index 10K blocks in about 2m30s while the 4vCPU can index 10K blocks in about 1m30s.
146
+
- The indexer process the data using goroutines and channels, which can provide a faster processing of the data.
147
+
- Fast data processing. [see benchmarks](./docs/benchmarks.md)
153
148
- The program has 2 modes that can be used for the indexing of the data. This can be useful for the testing, partial indexing of the chain or gradual indexing of the chain.
154
149
- The data is stored ready to be used for any kind of analytics and visualization with any programming language.
155
150
- No need to deal with Amino encoding and decoding for the messagess as the indexer decodes the messages and stores them in the database.
156
-
- It comes with a REST API to get you started quickly.(To be added in the future)
151
+
- It comes with a REST API to get you started quickly.
157
152
- It relies on a SQL database, which can provide a easier experience for any user that is familiar with the SQL.
158
153
- It uses TimescaleDB, a PostgresSQL extenstion that can be extended with any other extensions, plus the TimescaleDB has a lot of features that are not present in the Postgres.
159
154
@@ -165,4 +160,4 @@ VM message Add Package and Call where in theory one could extract even the body
165
160
166
161
## In depth documentation
167
162
168
-
For more detailed documentation, please refer to the [docs](https://github.com/Cogwheel-Validator/spectra-gnoland-indexer/blob/main/docs/README.md) directory.
163
+
For more detailed documentation, please refer to the [docs](./docs/README.md) directory.
With this indexer you can use the REST API from the api directory. It is built with chi framework and huma.
4
+
5
+
Out of the box you get a [Spotlight UI](https://stoplight.io/) to interact with the API on the /docs route.
6
+
You can also use some other API UIs but you will need to make the changes yourself. See docs about changing the UI [here](https://huma.rocks/features/api-docs/).
7
+
8
+
The huma is framework agnostic and you could modify the API to use some other framework, use another middleware
9
+
or maybe use the stdlib http package. This API provides the most basic and necessary features for querying the database.
10
+
11
+
## API routes
12
+
13
+
There are total of 5 routes available. This are the basic routes that are needed for the most part. This might be expanded in the future.
14
+
15
+
### Blocks
16
+
17
+
- /block/{height} - Get a specific block data by height
18
+
- /blocks/{from_height}/{to_height} - Get a range of blocks data by height range
19
+
- /blocks/{block_height}/signers - Get all of the validators that signed that block + the proposer
20
+
21
+
### Transactions
22
+
23
+
- /transaction/{tx_hash} - Get a specific basic transaction data by hash, this gives the basic data about the transaction like hash, timestamp, block height, gas used, gas wanted, fee and more.
24
+
- /transaction/{tx_hash}/message - Get a specific transaction message data by hash, this gives more detailed data about type of transaction, specific data for that message type and more.
25
+
26
+
### Addresses
27
+
28
+
- /address/{address}/txs?from_timestamp={from_timestamp}&to_timestamp={to_timestamp} - Get all of the transactions for a given address for a certain time period
29
+
30
+
## Setup API
31
+
32
+
To setup the API you can use the config file. The example config file is in the root under config-api.yml.example.
33
+
```yaml
34
+
# Example config file for the API
35
+
host: 127.0.0.1
36
+
port: 8080
37
+
cors_allowed_origins:
38
+
- "*"
39
+
cors_allowed_methods:
40
+
- "GET"
41
+
cors_allowed_headers:
42
+
- "Origin"
43
+
- "Content-Type"
44
+
- "Accept"
45
+
cors_max_age: 600
46
+
chain_name: gnoland
47
+
```
48
+
49
+
Some of the environment variables are located under the .env file. The example .env file is in the root under .env.example.
50
+
51
+
```env
52
+
# Example .env file for the API
53
+
# do not use password default unless for development or testing!!!
54
+
API_DB_HOST=127.0.0.1
55
+
API_DB_PORT=5432
56
+
API_DB_USER=reader
57
+
API_DB_SSLMODE=disable
58
+
API_DB_PASSWORD=12345678
59
+
API_DB_NAME=gnoland
60
+
61
+
# these are the default values for the database connection pool
62
+
# if they are not filled the API will load the default values
63
+
API_DB_POOL_MAX_CONNS=50
64
+
API_DB_POOL_MIN_CONNS=10
65
+
API_DB_POOL_MAX_CONN_LIFETIME=10s
66
+
API_DB_POOL_MAX_CONN_IDLE_TIME=5m
67
+
API_DB_POOL_HEALTH_CHECK_PERIOD=1m
68
+
API_DB_POOL_MAX_CONN_LIFETIME_JITTER=1m
69
+
```
70
+
71
+
You can make the API by running the following command from the project root:
72
+
73
+
```bash
74
+
make build-api
75
+
```
76
+
This command will build the API and it will be located in the build directory.
77
+
78
+
To run the API you can use the following command:
79
+
80
+
```bash
81
+
./build/api -c config-api.yml
82
+
```
83
+
84
+
You can also use the following command to run the API with HTTPS if you have the cert and key files:
* The 16 core/32 threads benchmark was ran 2/4 times the resault was at 6.5s however at this point the RPC node might be a bottleneck. One time it got 10 seconds and another time it was 5.4 seconds.
15
+
16
+
This blocks are mostly empty of transactions so it might not be the best benchmark. But it is a good indication of the performance and can be used as a reference.
17
+
18
+
However for the safety of the indexer I would recommend to use at the 50 - 100 chunks for now. Having 200 is a bit risky at this point untill more testing is done.
The indexer could can run on one machine while the database is located on another machine. You can also use the
4
+
Tiger Data (Timescaledb cloud edition) which is a managed service by Tiger Data which is a company that builds the
5
+
Timescaledb.
6
+
7
+
However for the ease of understanding through the documentation will refer that the indexer and the database are located on the same machine.
8
+
9
+
## Hardware requirements
10
+
11
+
To run the indexer you need to have the following system requirements:
12
+
13
+
Minimum system requirements:
14
+
15
+
- 2vCPUs
16
+
- 8GB RAM
17
+
18
+
Recommended system requirements:
19
+
20
+
- 4vCPUs
21
+
- 16GB RAM
22
+
23
+
The indexer could probably run on ARM64 architecture but it is not tested yet. So stick with the x86_64
24
+
architecture. There shouldn't be any major difference but if you really want to run it on ARM64 you might need to compile
25
+
the indexer from the source code.
26
+
27
+
For storage HDD should be enough. Durring the whole time of the development it was tested on old smal laptop HDD
28
+
with 5400 RPM i think. So it should work for the most part. But for any bigger projects where write and read
29
+
speeds are important you might needs SSD depenging on what you are doing. The SATA SSD is a good choice for the
30
+
most part but you can also use the NVMe SSD for better performance.
31
+
32
+
As for the size it depends on the amount of the data that you are indexing.
33
+
For example the integration test did about 400K blocks and 1 million
34
+
transactions. So it took around 2.2 GB of disk space. There are a lot of things that can affect the size of the
35
+
database but at least use this as some reference point to how much space you might need.
36
+
37
+
For the RAM and CPU it kinda depends but for now this is a good starting point. As the database size grows, you
38
+
might need to increase the RAM and CPU.
39
+
40
+
Aditional info for RAM. It is not that simple for more info look [here](https://docs.tigerdata.com/use-timescale/latest/hypertables/improve-query-performance/).
41
+
42
+
To make queries and compression efficient you need to have the necessary amount of RAM. This is kinda hard to calculate but according to the Tiger Data documentation:
43
+
44
+
```text
45
+
The default chunk interval is 7 days. However, best practice is to set chunk_interval so that prior to processing,
46
+
the indexes for chunks currently being ingested into fit within 25% of main memory. For example, on a system with
47
+
64 GB of memory, if index growth is approximately 2 GB per day, a 1-week chunk interval is appropriate. If index
48
+
growth is around 10 GB per day, use a 1-day interval.
49
+
```
50
+
51
+
So this is a bit hard to calculate. For example the Spectra cosmos indexer works in a similar way. At the testing
52
+
phase on Osmosis the 100K of block data took about 400MB-700MB of storage space. Now not all of the data is indexed
53
+
but for the sake of example let's say it is 700MB. The Osmosis has around 1s block production rate so this is 2
54
+
days and maybe around 2/3 of one day. Let's assume that for the week it will take arround 2GB. If Gnoland ever
55
+
reached these amount of data you might need to increase the RAM size or modify the chunk interval. But this is if
56
+
the indexed data was 2GB. Realistically this is not the case and the indexer data would need a lot of indexed
57
+
data.
58
+
59
+
Good rule of thumb is if the queries are slow increase the RAM size or modify the chunk interval or look at the
60
+
queries and see if they are efficient.
61
+
62
+
8 GB of RAM is a good starting point. 16 GB should be enough for most cases. Depending on how popular the chain is
63
+
and how much data is indexed you might need to increase the RAM size.
64
+
65
+
## Software and OS requirements
66
+
67
+
The following software and OS requirements are required to run the indexer:
68
+
69
+
- Go 1.25.1
70
+
- TimescaleDB 2.18 or higher but with PostgresSQL 16 or higher
71
+
- OS: Linux, anything based on Debian(Ubuntu, Mint, etc.) or RHEL(CentOS stream, Rocky Linux, etc.) should work, openSUSE also ok
0 commit comments