You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+30-64Lines changed: 30 additions & 64 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,7 +16,7 @@
16
16
<ahref="CHANGELOG.md">Changelog</a>
17
17
</div>
18
18
19
-
# Flare FTSO indexer
19
+
# Flare C-Chain Indexer
20
20
21
21
This code implements a fast and parallelized indexer of C-chain that fetches data needed for
22
22
various Flare protocols. It saves the data in a MySQL database.
@@ -30,59 +30,12 @@ docker-compose.yaml file for automatic deployment of a database).
30
30
31
31
The configuration is read from a `toml` file. Config file can be specified using the command line parameter `--config`, e.g., `./flare-ftso-indexer --config config.toml`.
32
32
The default config file name is `config.toml`.
33
-
Below is the list of configuration parameters, most are self-explanatory. Note that the chain node, to which the indexer connects
34
-
(parameter `node_url`), needs to allow many simultaneous request if the indexer is about to index big amount of data.
35
-
36
-
```toml
37
-
[indexer]
38
-
start_index = 0# the number of the block that the indexer will start with; will default to the appropriate block number according to the history drop configuration. Does not usually need to be set unless disabling history drop.
39
-
stop_index = 0# the number of the block that the indexer will stop with; set 0 or skip to index indefinitely
40
-
num_parallel_req = 100# the number of threads doing requests to the chain in parallel
41
-
batch_size = 1000# the number of blocks that will be pushed to a database in a batch (should be divisible by num_parallel_req)
42
-
log_range = 10# the size of the interval of blocks used to request logs in each request; suggested value is log_range = batch_size / num_parallel_req; note that a blockchain node might have an upper bound on this
43
-
new_block_check_millis = 1000# interval for checking for new blocks
44
-
45
-
[[indexer.collect_transactions]]
46
-
contract_address = "22474d350ec2da53d717e30b96e9a2b7628ede5b"# address of the contract (can be "undefined")
47
-
func_sig = "f14fcbc8"# signature of the function on the contract (can be "undefined")
48
-
status=true# boolean indicating if it should be checked if the transaction succeeded
49
-
collect_events=true# boolean indicating if the logs of the emitted events should be saved to the database
50
-
signature = true# boolean indicating if the transaction signature should be saved to the database
contract_address = "b682deef4f8e298d86bfc3e21f50c675151fb974"# address of the contract calling the log (can be "undefined")
60
-
topic = "undefined"# topic0 of the log (can be "undefined")
61
-
62
-
[db]
63
-
host = "localhost"
64
-
port = 3306
65
-
database = "flare_ftso_indexer"
66
-
username = "root"
67
-
password = "root"
68
-
log_queries = false
69
-
drop_table_at_start = false# set to true to drop existing tables at the start of the indexing - useful to force re-indexing
70
-
history_drop = 604800# Enable deleting the transactions and logs in DB that are older (timestamp of the block) than history_drop (in seconds); set 0 to turn off; defaults to 7 days for Flare/Songbird and 2 days for Coston*
71
-
72
-
[logger]
73
-
level = "INFO"
74
-
file = "./logger/logs/flare-ftso-indexer.log"
75
-
console = true
76
-
77
-
[chain]
78
-
node_url = "http://127.0.0.1:8545/"# or NODE_URL environment variable
79
-
# api_key = ... or NODE_API_KEY environment variable
80
-
# chain_type = 1 # default Avalanche based chain=1, Ethereum based chain=2
81
-
82
-
[timeout]
83
-
backoff_max_elapsed_time_seconds = 300# optional, defaults to 300s = 5 minutes. Affects how long the indexer will keep retrying in case of a complete outage of the node provider. Set to 0 to retry indefinitely.
84
-
timeout_milis = 1000# optional, defaults to 1000ms = 1s. Try increasing if you see timeout errors often.
85
-
```
33
+
Use [`config.example.toml`](config.example.toml) as the single config template. It includes full mode/indexer/db/chain/logger/timeout examples and comments.
34
+
35
+
#### Mode selection
36
+
37
+
-`indexer.mode = "fsp"`: use this when running as part of the FSP provider stack. Required FSP transactions/logs are hardcoded and auto-applied, so you do not need to specify `[[indexer.collect_transactions]]` or `[[indexer.collect_logs]]` in config (you can still add extra entries; they are merged and deduplicated). FSP startup also does selective indexing for required reward-epoch metadata windows instead of blindly indexing full historical ranges, which makes startup significantly faster. In this mode, `indexer.start_index` and `db.history_drop` are ignored; retention is derived from `indexer.history_epochs`.
38
+
-`indexer.mode = "full"`: use this for a generic C-chain indexer. In this mode you should define what to index via `[[indexer.collect_transactions]]` and `[[indexer.collect_logs]]`.
86
39
87
40
If the C chain indexer has been previously run and there is existing data in the database,
88
41
subsequent runs will resume indexing from after the last indexed block. Only when starting with an
@@ -101,41 +54,54 @@ set it back to false afterwards to avoid losing data on subsequent runs.
101
54
102
55
### Database
103
56
104
-
In `database/docker` we provide a simple database. Navigate to the folder and run
57
+
In `internal/database/docker` we provide a simple database. Navigate to the folder and run
105
58
106
59
```bash
107
60
docker-compose up
108
61
```
109
62
110
-
### Running FTSO indexer
63
+
### Running indexer
111
64
112
65
Simply run
113
66
114
67
```bash
115
-
go run main.go --config config.toml
68
+
go run ./cmd/indexer --config config.toml
116
69
```
117
70
118
71
or build and run the binaries with
119
72
120
73
```bash
121
-
go build
74
+
go build -o flare-ftso-indexer ./cmd/indexer
122
75
./flare-ftso-indexer --config config.toml
123
76
```
124
77
78
+
### Health endpoint
79
+
80
+
The indexer exposes `GET /health` on port `8080`.
81
+
82
+
- Returns `503` while startup catchup/backfill is still running.
83
+
- Returns `200` after startup is complete and the indexer has entered continuous indexing mode.
84
+
85
+
Example:
86
+
87
+
```bash
88
+
curl -i http://localhost:8080/health
89
+
```
90
+
125
91
### Tests
126
92
127
93
There is an integration test which checks the historical indexing against known transactions and
128
94
logs on Coston2. To run this test you will need a MySQL server and a Coston2 node, preferably one that is not rate-limited.
129
-
The integration test is configured via `testing/config_test.toml`. You can execute it with:
95
+
The integration test is configured via `test/config_test.toml`. You can execute it with:
130
96
131
97
```bash
132
-
$ go test ./main_test.go
98
+
$ go test ./cmd/indexer
133
99
```
134
100
135
-
Additionally, a unit test with a mockedchain node is available in `indexer/indexer_test.go`. Like the integration test, it uses `testing/config_test.toml` for configuration. You can run it using:
101
+
Additionally, a mocked-chain integration test is available in `test/indexer_test.go`. It uses `test/config_test.toml` for configuration. You can run it using:
136
102
137
103
```bash
138
-
go test ./indexer
104
+
go test ./test
139
105
```
140
106
141
107
To run tests with coverage analysis across all packages, save the results to `coverage.out`, and convert the report into an interactive HTML file run:
@@ -147,8 +113,8 @@ go tool cover -html=coverage.out
147
113
148
114
### Benchmarks
149
115
150
-
File `benchmarks/songbird_test.go` contains a benchmark test for indexing the FTSO protocol on the Songbird network. It processes 10,000 blocks and analyzes them. The test configuration is specified in `benchmarks/config_benchmark.toml`. To run the benchmark (replacing 10x with any desired number of repetitions), use:
116
+
File `benchmarks/songbird_test.go` contains a benchmark test for indexing the FTSO protocol on the Songbird network. It processes 10,000 blocks and analyzes them. The test configuration is specified in `benchmarks/config_banchmark.toml`. To run the benchmark (replacing 10x with any desired number of repetitions), use:
151
117
152
118
```bash
153
-
go test -benchmem -run=^$ -benchtime 10x -bench ^BenchmarkBlockRequests$ flare-ftso-indexer/benchmarks
119
+
go test -benchmem -run=^$ -benchtime 10x -bench ^BenchmarkBlockRequests$ ./benchmarks
0 commit comments