You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The data in the blockchain is mostly tied to blocks, however when any kind of analytics is needed we need some
38
+
The data in the blockchain is mostly tied to blocks, however when any kind of analytics is needed we need some
38
39
parameter that falls familiar to us from the daily life. So it comes more natural to compare the data with some
39
-
sort of time parameter then to a block height. Then the data needs to be stored in a way that is easy to query and
40
-
analyze. And since we are dealing with the time parameter, we need to have the ability to aggregate the data over
40
+
sort of time parameter then to a block height. Then the data needs to be stored in a way that is easy to query and
41
+
analyze. And since we are dealing with the time parameter, we need to have the ability to aggregate the data over
41
42
some time period so we can timeseries and easily plot the data.
42
43
43
-
A lot of indexers use NoSQL databases for this use case. However they usually have their own problems and
44
+
A lot of indexers use NoSQL databases for this use case. However they usually have their own problems and
44
45
limitations. They are not as flexible as the SQL databases when it comes to the data types and the query language.
45
46
that is the reason why the TimescaleDB is a perfect fit for this use case.
46
47
47
48
Anyone with some experience with the SQL, especially with the PostgreSQL, can easily understand the TimescaleDB.
48
-
It is a extension of the PostgreSQL that adds the ability to store the time series data in a way that is easy to
49
+
It is a extension of the PostgreSQL that adds the ability to store the time series data in a way that is easy to
49
50
query and analyze.
50
51
51
52
## Why TimescaleDB? And can it work on other SQL databases?
52
53
53
-
TimescaleDB (Tiger Data is their commercial version) is a extension of the PostgreSQL that adds the ability to
54
-
store the time series data in a way that is easy to query and analyze. It also has a lot of features that can
54
+
TimescaleDB (Tiger Data is their commercial version) is a extension of the PostgreSQL that adds the ability to
55
+
store the time series data in a way that is easy to query and analyze. It also has a lot of features that can
55
56
extend the capabilities of the PostgreSQL and make it more powerful.
56
57
58
+
The TimescaleDB sits between the OLTP and the OLAP databases. It is a hybrid database that can be used for both.
59
+
It is a good fit for the time series data and it is a good fit for the analytics and the exploration of the data.
60
+
57
61
The TimescaleDB has also feature of it's own that is not present in the Postgres. The user could extend the indexer
58
62
by adding the data aggregation features, automatic jobs scheduling, hyperfunctions and more.
59
63
60
-
This database extension also has a data compression feature that can be used to reduce the storage space and
64
+
This database extension also has a data compression feature that can be used to reduce the storage space and
61
65
segment the data into smaller chunks by using time based intervals making the queries faster.
62
66
63
-
Technically speaking the indexer can work on Postgres database, however you would need to create the tables and
64
-
types manually. This might be added later in the future if there is a demand for it but for now the focus is on the
67
+
Technically speaking the indexer can work on Postgres database, however you would need to create the tables and
68
+
types manually. This might be added later in the future if there is a demand for it but for now the focus is on the
65
69
TimescaleDB.
66
70
67
71
There are some other SQL databases that could work in theory. For them to work they would need to have:
72
+
68
73
- The postgres wire protocol
69
74
- Equivalent of Numeric type ( some databases might have it as DECIMAL, but not all of them )
70
75
- The ability to create custom types
71
76
72
-
If all of the above are met then the indexer can work on them. It might work on CockroachDB for example but this is
77
+
If all of the above are met then the indexer can work on them. It might work on CockroachDB for example but this is
73
78
out of the scope of this project. Maybe in the future it might be an interesting idea to support it.
74
79
75
80
## How does the indexer work?
76
81
77
82
The indexer has 2 main modes of operation:
83
+
78
84
- Live mode
79
85
- Historic mode
80
86
81
-
Live mode is the mode that is used to index the data in the real time. It will sync up the database to the latest
82
-
block height and will continue to index the data in the real time.
87
+
Live mode is the mode that is used to index the data in the real time. It will sync up the database to the latest
88
+
block height and will continue to index the data in the real time. It works by using near-real-time batch processing method. It will collect all of the data up to the latest block height and then it will process it in
89
+
batches. So the process is not instant but it is very fast and it is able to keep up with the latest block height.
83
90
84
-
Historic mode is the mode that allows the user to index a certain range of blocks. For example you might not need
85
-
the whole chain history or just need a part of it. This mode is useful for the testing, partial indexing of the
91
+
Historic mode is the mode that allows the user to index a certain range of blocks. For example you might not need
92
+
the whole chain history or just need a part of it. This mode is useful for the testing, partial indexing of the
86
93
chain or gradual indexing of the chain.
87
94
88
-
Durring both modes the indexer will use fan out method to send the requests to the RPC node. Then all of the data
89
-
is collected and processed and decoded in parallel. All of the addresses are collected and stored in the database
90
-
and the indexer stores them in it's own cache. Then any address that is found in the transaction is referanced by
91
-
the integer value in the transaction tables. At the end of the processing the data is inserted in batches into the
95
+
During both modes the indexer will use fan out method to send the requests to the RPC node. Then all of the data
96
+
is collected and processed and decoded in parallel. All of the addresses are collected and stored in the database
97
+
and the indexer stores them in it's own cache. Then any address that is found in the transaction is referenced by
98
+
the integer value in the transaction tables. At the end of the processing the data is inserted in batches into the
92
99
database.
93
100
94
101
## What data is stored in the database?
95
102
96
-
The indexer stores the esential data that is related to the blocks, transactions, messages, and accounts.
103
+
The indexer stores the essential data that is related to the blocks, transactions, messages, and accounts.
97
104
It also stores the validator block signings and the validator addresses.
98
-
This way it is able to provide a lot of data for the analytics and the explorer.
105
+
The data schema is denormalized to decrease the space needed and to provide the easy and fast access for the
106
+
explorer and any other visualizations and analytics tools.
107
+
99
108
The data is stored in the following tables:
109
+
100
110
- blocks
101
111
- transactions general data
102
112
- transaction messages (each message type has its own table)
@@ -107,8 +117,10 @@ The data is stored in the following tables:
107
117
108
118
Some of the data is not indexed and it is not planned to be indexed in the future. Such as:
109
119
110
-
### Blocks:
120
+
### Blocks
121
+
111
122
Stored data:
123
+
112
124
- Basic block hash
113
125
- Block height
114
126
- Block timestamp
@@ -117,47 +129,52 @@ Stored data:
117
129
- Block transactions hashes
118
130
119
131
Not stored:
132
+
120
133
- Last commit hash
121
134
- App hash
122
135
- Data hash
123
136
- Validators hash
124
137
- Next validators hash
125
138
- Consensus hash
126
139
127
-
### Validator signings:
140
+
### Validator signings
141
+
128
142
Stored data:
143
+
129
144
- Validator block signing height
130
145
- Validator block signing timestamp
131
146
- Validator block signing signed validators
132
147
- Proposer address
133
148
134
149
Not stored:
150
+
135
151
- Missed validators
136
152
- Precommits and all of the hashes and other data, so only a confirmation that the validator signed the block
137
153
138
-
### Transactions:
139
-
Almost all of the data regarding to the transaction and the messages are stored with the exception for the
154
+
### Transactions
155
+
156
+
Almost all of the data regarding to the transaction and the messages are stored with the exception for the
140
157
VM message Add Package and Call where in theory one could extract even the body of the smart contract. So this is unnecessary to store for explorers and other analytics tools. This might be added in the future if there is a demand for it.
141
158
142
159
## Pros and cons of the SGI
143
160
144
-
### 🦾 Pros:
161
+
### 🦾 Pros
145
162
146
163
- The indexer process the data using goroutines and channels, which can provide a faster processing of the data.
147
164
- Fast data processing. [see benchmarks](./docs/benchmarks.md)
148
165
- The program has 2 modes that can be used for the indexing of the data. This can be useful for the testing, partial indexing of the chain or gradual indexing of the chain.
149
166
- The data is stored ready to be used for any kind of analytics and visualization with any programming language.
150
-
- No need to deal with Amino encoding and decoding for the messagess as the indexer decodes the messages and stores them in the database.
167
+
- No need to deal with Amino encoding and decoding for the messages as the indexer decodes the messages and stores them in the database.
151
168
- It comes with a REST API to get you started quickly.
152
169
- It relies on a SQL database, which can provide a easier experience for any user that is familiar with the SQL.
153
-
- It uses TimescaleDB, a PostgresSQL extenstion that can be extended with any other extensions, plus the TimescaleDB has a lot of features that are not present in the Postgres.
170
+
- It uses TimescaleDB, a PostgresSQL extension that can be extended with any other extensions, plus the TimescaleDB has a lot of features that are not present in the Postgres.
154
171
155
-
### 🐞 Cons:
172
+
### 🐞 Cons
156
173
157
-
- The indexer has a address cache of all of the addresses that were ever used in the transactions. This gives the indexer ability to swap the addresses with the their integer index in the database. However this intorduces complexity. Anyone who plans to use the indexer and plans to make some custom solution on working with the data will need to fully understand the data structure and how to use it. The REST API provides a easy way to interact with the data and to get the data in a readable format.
174
+
- The indexer has a address cache of all of the addresses that were ever used in the transactions. This gives the indexer ability to swap the addresses with the their integer index in the database. However this introduces complexity. Anyone who plans to use the indexer and plans to make some custom solution on working with the data will need to fully understand the data structure and how to use it. The REST API provides a easy way to interact with the data and to get the data in a readable format.
158
175
- The indexer relies on the RPC node for the data. If the RPC node is not available the indexer will not be able to index the data. ( although in the future the indexer might be able to use multiple RPC nodes )
159
176
- Technically the indexer has a limit of 2 billion addresses. If at any point the Gnoland grows to that size the indexer would need to be updated to support it. It is not a problem for now but it is something to keep in mind.
160
177
161
178
## In depth documentation
162
179
163
-
For more detailed documentation, please refer to the [docs](./docs/README.md) directory.
180
+
For more detailed documentation, please refer to the [docs](./docs/README.md) directory.
0 commit comments