You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Jul 30, 2025. It is now read-only.
Copy file name to clipboardExpand all lines: apps/nextra/pages/en/build/indexer/txn-stream/self-hosted.mdx
+92-1Lines changed: 92 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,4 +8,95 @@ import { IndexerBetaNotice } from '@components/index';
8
8
9
9
<IndexerBetaNotice />
10
10
11
-
Coming soon!
11
+
In order to run Self-Hosted Transaction Stream Service, you will need to run the following components.
12
+
13
+
Indexer FN [https://github.com/aptos-labs/aptos-core/tree/main/ecosystem/indexer-grpc/indexer-grpc-fullnode]: A FN with indexer grpc functionality enabled. Typically your data service will need to access all historical data, therefore your FN need to sync from genesis in order to bootstrap the whole stack. The pruner can be deleted (through pruner) later on once the data is persisted into file store.
14
+
15
+
GrpcManager [https://github.com/aptos-labs/aptos-core/tree/main/ecosystem/indexer-grpc/indexer-grpc-manager]: A centrilized component that manages all the components in the stack. It can run in two mode (master and non-master), only 1 master is allowed. When it is running as master mode, it will also pull data from the upstream FN, and persistent the data into file store (which can be a local file system or Gooogle Cloud Storage).
16
+
17
+
DataService [https://github.com/aptos-labs/aptos-core/tree/main/ecosystem/indexer-grpc/indexer-grpc-data-service-v2]: Provides the client facing streaming Grpc service. It can run in 2 modes. The live mode serves data from its local cache, the historical mode serves data from the file store.
18
+
19
+
## Example Configs
20
+
21
+
- FN
22
+
```
23
+
indexer_grpc:
24
+
enabled: true
25
+
address: 0.0.0.0:50051 # The address to service Grpc request.
26
+
processor_task_count: 32
27
+
processor_batch_size: 100
28
+
output_batch_size: 100
29
+
30
+
indexer_table_info:
31
+
table_info_service_mode: IndexingOnly
32
+
parser_task_count: 10
33
+
parser_batch_size: 100
34
+
```
35
+
- GrpcManager
36
+
```
37
+
health_check_port: 8081 # The port for monitoring purpose.
38
+
server_config:
39
+
is_master: true # Whether running in master mode.
40
+
service_config:
41
+
listen_address: 0.0.0.0:50052 # The port that serves Grpc requests.
42
+
self_advertised_address: 0.0.0.0:50052
43
+
grpc_manager_addresses: # All GrpcManager addresses in the stack, need to point to the server_config.self_advertised_address in GrpcManager config.
44
+
- >-
45
+
http://0.0.0.0:50052
46
+
fullnode_addresses: # All upstream FN addresses in the stack, need to point to the indexer_grpc.address in FN config.
Call GrpcManager.GetDataServiceForRequest first, it will return the address of a data service instance.
85
+
86
+
Then Call DataService.GetTransactions.
87
+
- Use DataService directly
88
+
89
+
Call DataService.GetTransactions directly. In this case you might want to run both live data service and historical data service together.
90
+
91
+
## Advanced Usage
92
+
93
+
- Do not keep full history
94
+
If your stream never needs to serve old data and you don't want to keep the full history, for example you want to start a stream now and only care about data in the future, you can choose to not sync from genesis.
95
+
96
+
In order to do that, first you can start your FN and do a fast sync. Then download the most recent table info database from https://console.cloud.google.com/storage/browser/aptos-indexer-grpc-mainnet-table-info-backup (for testnet, replace `mainnet` with `testnet`), unzip to the db folder in your FN.
97
+
98
+
Then start your GrpcManager, it will generate the `metadata.json` in your file store (it could be your local file stream or GCS based on your config). Manually update the version to the next version you want to start processing. (the version must be a multiple of 100000 plus 1, e.g. 1000000001, and your FN must have data at this version).
99
+
100
+
Then restart all your binaries, it should start working.
0 commit comments