Skip to content

Commit 9d71821

Browse files
Document RPC BACKFILL flag and refresh sample config (#2789)
* Document RPC BACKFILL flag and refresh sample config The sample config in configuring.mdx predates two flags. Add both, with the usage text from the RPC source. - BACKFILL = false - SERVE_LEDGERS_FROM_DATASTORE = false Add a section for BACKFILL. It gives the version, the default, the datastore prerequisite, and the retention window that sets the size. It links to Data Lake Integration for datastore setup. Partially addresses #2602 * Match sample config to gen-config-file output Verified the sample block against the real generator. Built the gen-config-file body from stellar-rpc v28.0.0 and diffed it. - comment out BACKFILL and SERVE_LEDGERS_FROM_DATASTORE, because the generator comments out zero-value defaults - add NETWORK - add STELLAR_CAPTIVE_CORE_HTTP_QUERY_PORT and STELLAR_CAPTIVE_CORE_HTTP_QUERY_SNAPSHOT_LEDGERS - correct MAX_GET_LEDGERS_EXECUTION_DURATION from 5s to 10s Every environment-independent key now matches the generator exactly. * Do not promise exact backfill coverage Copilot flagged the retention sentence as an over-promise. It is right. Verified in internal/ingest/backfill.go at v27.1.1 and v28.0.0: - nBackfill = min(retentionWindow, currentTipLedger) - start clamped by dsInfo.sequences.First - forward fill ends at PrevCheckpoint(currentTipLedger) - ledgers already in the local database are not refetched Upstream's own comment says the window is "at least approximately" met. So call the value a target and name the three limits. Also label the error code fence as text. * add hints about datastore configurations --------- Co-authored-by: Elliot <elliot@stellar.org>
1 parent af0051c commit 9d71821

1 file changed

Lines changed: 40 additions & 1 deletion

File tree

docs/data/apis/rpc/admin-guide/configuring.mdx

Lines changed: 40 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,10 @@ The resulting configuration should look like this:
2424
# from the Internet and does not use TLS. "" (default) disables the admin server
2525
# ADMIN_ENDPOINT = ""
2626

27+
# Populates database with `history-retention-window` ledgers synchronously on
28+
# startup. This defaults to a week of ledgers if unspecified
29+
# BACKFILL = false
30+
2731
# path to additional configuration for the Stellar Core configuration file used
2832
# by captive core. It must, at least, include enough details to define a quorum
2933
# set
@@ -102,7 +106,7 @@ MAX_GET_LATEST_LEDGER_EXECUTION_DURATION = "5s"
102106
# The maximum duration of time allowed for processing a getLedgers request. When
103107
# that time elapses, the rpc server would return -32001 and abort the request's
104108
# execution
105-
MAX_GET_LEDGERS_EXECUTION_DURATION = "5s"
109+
MAX_GET_LEDGERS_EXECUTION_DURATION = "10s"
106110

107111
# The maximum duration of time allowed for processing a getLedgerEntries
108112
# request. When that time elapses, the rpc server would return -32001 and abort
@@ -154,6 +158,9 @@ MAX_SIMULATE_TRANSACTION_EXECUTION_DURATION = "15s"
154158
# Maximum amount of transactions allowed in a single getTransactions response
155159
MAX_TRANSACTIONS_LIMIT = 200
156160

161+
# Specifies the desired Stellar network, 'pubnet', 'testnet', or 'futurenet'.
162+
# NETWORK = ""
163+
157164
# Network passphrase of the Stellar network transactions should be signed for.
158165
# Commonly used values are "Test SDF Future Network ; October 2022", "Test SDF
159166
# Network ; September 2015" and "Public Global Stellar Network ; September 2015"
@@ -222,6 +229,14 @@ SOROBAN_FEE_STATS_RETENTION_WINDOW = 50
222229
# HTTP port for Captive Core to listen on (0 disables the HTTP server)
223230
STELLAR_CAPTIVE_CORE_HTTP_PORT = 11626
224231

232+
# HTTP port for Captive Core to listen on for high-performance queries like
233+
# /getledgerentry (must not conflict with CAPTIVE_CORE_HTTP_PORT)
234+
STELLAR_CAPTIVE_CORE_HTTP_QUERY_PORT = 11628
235+
236+
# Size of ledger history in Captive Core's high-performance query server (don't
237+
# touch unless you know what you are doing)
238+
STELLAR_CAPTIVE_CORE_HTTP_QUERY_SNAPSHOT_LEDGERS = 4
239+
225240
# path to stellar core binary
226241
STELLAR_CORE_BINARY_PATH = "/usr/bin/stellar-core"
227242

@@ -235,8 +250,16 @@ STELLAR_CORE_TIMEOUT = "2s"
235250
# fields in the config toml from being parsed.
236251
# STRICT = false
237252

253+
# Fetch historical ledgers from the datastore if they're not available locally. This entry should be followed by a datastore configuration.
254+
# SERVE_LEDGERS_FROM_DATASTORE = false
238255
```
239256

257+
:::info
258+
259+
Example datastore configurations can be found on the [data lake integration page](./data-lake-integration.mdx).
260+
261+
:::
262+
240263
Note that the above generated configuration contains the default values and you need to substitute them with proper values to run the image. For instance, when using a container, it is recommended to create a volume for the Captive Core and RPC persistent storage and point `CAPTIVE_CORE_STORAGE_PATH` and `DB_PATH` to it accordingly.
241264

242265
Then, you should create a configuration file for [Stellar Core](https://github.com/stellar/stellar-core). You can find sample configuration files for [Testnet](https://github.com/stellar/go-stellar-sdk/blob/main/ingest/ledgerbackend/configs/captive-core-testnet.cfg) and [Pubnet](https://github.com/stellar/go-stellar-sdk/blob/main/ingest/ledgerbackend/configs/captive-core-pubnet.cfg)
@@ -259,6 +282,22 @@ Then you would mount that volume using by adding the following parameter: `-v /U
259282

260283
Your running container would mount that volume at the path `/opt/stellar`
261284

285+
## Backfilling History on Startup
286+
287+
Set `BACKFILL = true` to populate the database with a trailing window of history synchronously on startup, before live ingestion begins. This option was added in RPC v25.1.0 and defaults to `false`.
288+
289+
`HISTORY_RETENTION_WINDOW` sets the target size of that window. The default is `120960` ledgers, which is about 7 days.
290+
291+
RPC treats this value as a target, not an exact count. Datastore coverage, ledgers already in the local database, and checkpoint boundaries can each reduce what the node fetches.
292+
293+
`BACKFILL` also requires datastore serving. Set `SERVE_LEDGERS_FROM_DATASTORE = true`. If you enable `BACKFILL` alone, RPC fails to start with this error:
294+
295+
```text
296+
backfill requires serving ledgers from datastore to be enabled. See the `--serve-ledgers-from-datastore` flag
297+
```
298+
299+
To set up the datastore, see [Data Lake Integration](./data-lake-integration.mdx). That page also explains which requests the datastore serves directly.
300+
262301
## Next Step
263302

264303
After installation is complete, you are now ready to proceed to [Running RPC](./running.mdx)!

0 commit comments

Comments
 (0)