Skip to content
This repository was archived by the owner on Jul 30, 2025. It is now read-only.

Commit 1d4b314

Browse files
Update self-hosted.mdx (#908)
Co-authored-by: Greg Nazario <greg@gnazar.io>
1 parent 8f37dca commit 1d4b314

File tree

1 file changed

+92
-1
lines changed

1 file changed

+92
-1
lines changed

apps/nextra/pages/en/build/indexer/txn-stream/self-hosted.mdx

Lines changed: 92 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,4 +8,95 @@ import { IndexerBetaNotice } from '@components/index';
88

99
<IndexerBetaNotice />
1010

11-
Coming soon!
11+
In order to run Self-Hosted Transaction Stream Service, you will need to run the following components.
12+
13+
Indexer FN [https://github.com/aptos-labs/aptos-core/tree/main/ecosystem/indexer-grpc/indexer-grpc-fullnode]: A FN with indexer grpc functionality enabled. Typically your data service will need to access all historical data, therefore your FN need to sync from genesis in order to bootstrap the whole stack. The pruner can be deleted (through pruner) later on once the data is persisted into file store.
14+
15+
GrpcManager [https://github.com/aptos-labs/aptos-core/tree/main/ecosystem/indexer-grpc/indexer-grpc-manager]: A centrilized component that manages all the components in the stack. It can run in two mode (master and non-master), only 1 master is allowed. When it is running as master mode, it will also pull data from the upstream FN, and persistent the data into file store (which can be a local file system or Gooogle Cloud Storage).
16+
17+
DataService [https://github.com/aptos-labs/aptos-core/tree/main/ecosystem/indexer-grpc/indexer-grpc-data-service-v2]: Provides the client facing streaming Grpc service. It can run in 2 modes. The live mode serves data from its local cache, the historical mode serves data from the file store.
18+
19+
## Example Configs
20+
21+
- FN
22+
```
23+
indexer_grpc:
24+
enabled: true
25+
address: 0.0.0.0:50051 # The address to service Grpc request.
26+
processor_task_count: 32
27+
processor_batch_size: 100
28+
output_batch_size: 100
29+
30+
indexer_table_info:
31+
table_info_service_mode: IndexingOnly
32+
parser_task_count: 10
33+
parser_batch_size: 100
34+
```
35+
- GrpcManager
36+
```
37+
health_check_port: 8081 # The port for monitoring purpose.
38+
server_config:
39+
is_master: true # Whether running in master mode.
40+
service_config:
41+
listen_address: 0.0.0.0:50052 # The port that serves Grpc requests.
42+
self_advertised_address: 0.0.0.0:50052
43+
grpc_manager_addresses: # All GrpcManager addresses in the stack, need to point to the server_config.self_advertised_address in GrpcManager config.
44+
- >-
45+
http://0.0.0.0:50052
46+
fullnode_addresses: # All upstream FN addresses in the stack, need to point to the indexer_grpc.address in FN config.
47+
- >-
48+
http://0.0.0.0:50051
49+
- >-
50+
http://other-fullnode.xyz:50051
51+
file_store_config:
52+
file_store_type: GcsFileStore
53+
gcs_file_store_bucket_name: indexer
54+
gcs_file_store_service_account_key_path: /secrets/indexer-sa-key
55+
chain_id: 1
56+
```
57+
- DataService
58+
```
59+
health_check_port: 8081 # The port for monitoring purpose.
60+
server_config:
61+
chain_id: 1
62+
self_advertised_address: 0.0.0.0:50053
63+
grpc_manager_addresses: # All GrpcManager addresses in the stack, need to point to the server_config.self_advertised_address in GrpcManager config.
64+
- >-
65+
http://0.0.0.0:50052
66+
service_config:
67+
listen_address: 0.0.0.0:50053
68+
live_data_service_config: # For live data service.
69+
enabled: true
70+
num_slots: 5000000 # Max number of transactions to cache.
71+
size_limit_bytes: 10000000000 # Cache size in bytes.
72+
historical_data_service_config: # For historical data service.
73+
enabled: true
74+
file_store_config:
75+
file_store_type: GcsFileStore
76+
gcs_file_store_bucket_name: indexer
77+
gcs_file_store_service_account_key_path: /secrets/indexer-sa-key
78+
```
79+
80+
## Usage
81+
82+
- Use GrpcManager for routing / load balancing
83+
84+
Call GrpcManager.GetDataServiceForRequest first, it will return the address of a data service instance.
85+
86+
Then Call DataService.GetTransactions.
87+
- Use DataService directly
88+
89+
Call DataService.GetTransactions directly. In this case you might want to run both live data service and historical data service together.
90+
91+
## Advanced Usage
92+
93+
- Do not keep full history
94+
If your stream never needs to serve old data and you don't want to keep the full history, for example you want to start a stream now and only care about data in the future, you can choose to not sync from genesis.
95+
96+
In order to do that, first you can start your FN and do a fast sync. Then download the most recent table info database from https://console.cloud.google.com/storage/browser/aptos-indexer-grpc-mainnet-table-info-backup (for testnet, replace `mainnet` with `testnet`), unzip to the db folder in your FN.
97+
98+
Then start your GrpcManager, it will generate the `metadata.json` in your file store (it could be your local file stream or GCS based on your config). Manually update the version to the next version you want to start processing. (the version must be a multiple of 100000 plus 1, e.g. 1000000001, and your FN must have data at this version).
99+
100+
Then restart all your binaries, it should start working.
101+
102+

0 commit comments

Comments
 (0)