You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Document RPC BACKFILL flag and refresh sample config (#2789)
* Document RPC BACKFILL flag and refresh sample config
The sample config in configuring.mdx predates two flags. Add both,
with the usage text from the RPC source.
- BACKFILL = false
- SERVE_LEDGERS_FROM_DATASTORE = false
Add a section for BACKFILL. It gives the version, the default, the
datastore prerequisite, and the retention window that sets the size.
It links to Data Lake Integration for datastore setup.
Partially addresses #2602
* Match sample config to gen-config-file output
Verified the sample block against the real generator. Built the
gen-config-file body from stellar-rpc v28.0.0 and diffed it.
- comment out BACKFILL and SERVE_LEDGERS_FROM_DATASTORE, because the
generator comments out zero-value defaults
- add NETWORK
- add STELLAR_CAPTIVE_CORE_HTTP_QUERY_PORT and
STELLAR_CAPTIVE_CORE_HTTP_QUERY_SNAPSHOT_LEDGERS
- correct MAX_GET_LEDGERS_EXECUTION_DURATION from 5s to 10s
Every environment-independent key now matches the generator exactly.
* Do not promise exact backfill coverage
Copilot flagged the retention sentence as an over-promise. It is right.
Verified in internal/ingest/backfill.go at v27.1.1 and v28.0.0:
- nBackfill = min(retentionWindow, currentTipLedger)
- start clamped by dsInfo.sequences.First
- forward fill ends at PrevCheckpoint(currentTipLedger)
- ledgers already in the local database are not refetched
Upstream's own comment says the window is "at least approximately" met.
So call the value a target and name the three limits.
Also label the error code fence as text.
* add hints about datastore configurations
---------
Co-authored-by: Elliot <elliot@stellar.org>
# Fetch historical ledgers from the datastore if they're not available locally. This entry should be followed by a datastore configuration.
254
+
# SERVE_LEDGERS_FROM_DATASTORE = false
238
255
```
239
256
257
+
:::info
258
+
259
+
Example datastore configurations can be found on the [data lake integration page](./data-lake-integration.mdx).
260
+
261
+
:::
262
+
240
263
Note that the above generated configuration contains the default values and you need to substitute them with proper values to run the image. For instance, when using a container, it is recommended to create a volume for the Captive Core and RPC persistent storage and point `CAPTIVE_CORE_STORAGE_PATH` and `DB_PATH` to it accordingly.
241
264
242
265
Then, you should create a configuration file for [Stellar Core](https://github.com/stellar/stellar-core). You can find sample configuration files for [Testnet](https://github.com/stellar/go-stellar-sdk/blob/main/ingest/ledgerbackend/configs/captive-core-testnet.cfg) and [Pubnet](https://github.com/stellar/go-stellar-sdk/blob/main/ingest/ledgerbackend/configs/captive-core-pubnet.cfg)
@@ -259,6 +282,22 @@ Then you would mount that volume using by adding the following parameter: `-v /U
259
282
260
283
Your running container would mount that volume at the path `/opt/stellar`
261
284
285
+
## Backfilling History on Startup
286
+
287
+
Set `BACKFILL = true` to populate the database with a trailing window of history synchronously on startup, before live ingestion begins. This option was added in RPC v25.1.0 and defaults to `false`.
288
+
289
+
`HISTORY_RETENTION_WINDOW` sets the target size of that window. The default is `120960` ledgers, which is about 7 days.
290
+
291
+
RPC treats this value as a target, not an exact count. Datastore coverage, ledgers already in the local database, and checkpoint boundaries can each reduce what the node fetches.
292
+
293
+
`BACKFILL` also requires datastore serving. Set `SERVE_LEDGERS_FROM_DATASTORE = true`. If you enable `BACKFILL` alone, RPC fails to start with this error:
294
+
295
+
```text
296
+
backfill requires serving ledgers from datastore to be enabled. See the `--serve-ledgers-from-datastore` flag
297
+
```
298
+
299
+
To set up the datastore, see [Data Lake Integration](./data-lake-integration.mdx). That page also explains which requests the datastore serves directly.
300
+
262
301
## Next Step
263
302
264
303
After installation is complete, you are now ready to proceed to [Running RPC](./running.mdx)!
0 commit comments