Skip to content

Commit 6cae4d0

Browse files
committed
Make the Complete History Archive section a practical operator guide
Rework the archivist docs around what an operator is actually trying to do — install the tool, decide where to run it, pick a command, then scan / repair / mirror: - Drop --verify from the default examples and give it its own section covering what it checks and what it costs. - Add a command-selection table, and document --low/--high. - Note that stellar-archivist is separate from stellar-core: run it off the validator host, and never stop the validator to work on its archive. - Remove the stray `systemctl start stellar-core` at the end of the section, which paired with nothing and implied Core had been stopped. - Fix an invalid ':::attention' admonition and copyedit throughout. Also establish that a Full Validator's archive is a complete one. Neither the Full Validator definition nor the Tier 1 requirements mentioned archive depth, so nothing said an archive should reach back to the first ledger. Tier 1 delegates its archive requirement to the Full Validator definition, so stating completeness there covers both. The archives page now frames publishing as two stages, so the Backfilling section no longer implies you are finished when the archive holds only recent checkpoints.
1 parent ff58a1c commit 6cae4d0

2 files changed

Lines changed: 62 additions & 24 deletions

File tree

docs/validators/README.mdx

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ The basic flow, which you can navigate through using the "Admin Guide" on the le
4949

5050
## Types of validator nodes {/* #types-of-nodes */}
5151

52-
There are two types of validator nodes, and they perform the same basic functions: they run Stellar Core, connect to peers, submit transactions, and store the state of the ledger. The difference is this: a **Basic Validator** does not publish a history archive; a **Full Validator** does.
52+
There are two types of validator nodes, and they perform the same basic functions: they run Stellar Core, connect to peers, submit transactions, and store the state of the ledger. The difference is this: a **Basic Validator** does not publish a history archive; a **Full Validator** publishes one covering the network's complete history.
5353

5454
:::info
5555

@@ -67,9 +67,9 @@ The advantage: signatures can serve as official endorsements of specific ledgers
6767

6868
### Full Validator
6969

70-
#### Validating, offers public archive
70+
#### Validating, offers a complete public archive
7171

72-
A Full Validator is the same as a Basic Validator except that it also publishes a [History Archive](./admin-guide/environment-preparation.mdx) containing snapshots of the ledger, including all transactions and their results. A Full Validator writes to an internet-facing blob store — such as AWS or Azure — so it's a bit more expensive and complex to run, but it also does the most to support the network’s resilience and decentralization.
72+
A Full Validator is the same as a Basic Validator except that it also publishes a [History Archive](./admin-guide/environment-preparation.mdx) covering the network's **complete history** snapshots of the ledger, including all transactions and their results. A Full Validator writes to an internet-facing blob store — such as AWS or Azure — so it's a bit more expensive and complex to run, but it also does the most to support the network’s resilience and decentralization.
7373

7474
When other nodes join the network — or experience difficulty and temporarily fall out of sync — they can consult archives offered by Full Validators to catch up on the history of the network. Redundant archives prevent a single point of failure, and allow network participants to verify the veracity of a given history.
7575

docs/validators/admin-guide/publishing-history-archives.mdx

Lines changed: 59 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,12 @@ title: Publishing History Archives
33
sidebar_position: 50
44
---
55

6-
If you want to run a [Full Validator](../README.mdx#full-validator), you need to set up your node to publish a history archive. You can host an archive using a blob store such as Amazon's S3 or Digital Ocean's spaces, or you can simply serve a local archive directly via an HTTP server such as Nginx or Apache. If you're setting up a [Basic Validator](../README.mdx#basic-validator), you can skip this section. No matter what kind of node you're planning to run, make sure to set it up to `get` history, which is covered in [Environment Preparation](./environment-preparation.mdx).
6+
If you want to run a [Full Validator](../README.mdx#full-validator), you need to set up your node to publish a history archive covering the network's complete history. You can host an archive using a blob store such as Amazon's S3 or Digital Ocean's spaces, or you can simply serve a local archive directly via an HTTP server such as Nginx or Apache. If you're setting up a [Basic Validator](../README.mdx#basic-validator), you can skip this section. No matter what kind of node you're planning to run, make sure to set it up to `get` history, which is covered in [Environment Preparation](./environment-preparation.mdx).
7+
8+
Getting to a complete archive takes two stages:
9+
10+
- Configuring your node to publish, described in the sections below. This fills the archive forward from the moment your node starts.
11+
- [Completing the archive](#complete-history-archive), which backfills everything from before that point.
712

813
:::caution[One archive per node]
914

@@ -160,7 +165,7 @@ server {
160165

161166
Given the choice, it's best to configure your history archive _prior to_ your node's initial sync with an existing network. That way your validator's history publishes as you join, and subsequently sync with, the network.
162167

163-
However, if you have not published an archive during the node's initial sync, the steps required to create a history archive for an existing validator — in other words, to upgrade a Basic Validator to a Full Validator — are quite straightforward. First, you'll need to stop your `stellar-core` instance:
168+
However, if you have not published an archive during the node's initial sync, the steps required to start publishing from an existing validator are quite straightforward. This is the first of the two stages on the way to a Full Validator; [completing the archive](#complete-history-archive) follows. First, you'll need to stop your `stellar-core` instance:
164169

165170
```bash
166171
systemctl stop stellar-core # modify this if not using systemctl
@@ -193,11 +198,15 @@ As you allow your node to join the network again, you can watch it start publish
193198
2019-04-25T12:30:43.275 GDUQJ [History INFO] Publishing 1 queued checkpoints [16895-16895]: Awaiting 0/0 prerequisites of: publish-000041ff
194199
```
195200

196-
At this stage your validator is successfully publishing its history, which enables other users to join the network using your archive.
201+
At this stage your validator is publishing each new checkpoint as the network advances, which lets other nodes catch up from your archive as far back as the point where you started publishing. Everything before that point is still missing, and filling it in is the second stage.
197202

198203
## Complete History Archive
199204

200-
The [stellar-archivist](https://github.com/stellar/rs-stellar-archivist) command line tool scans, repairs, and mirrors history archives. Using the [SDF package repositories](https://github.com/stellar/packages), you can install it by running:
205+
This is the second stage. A [Full Validator](../README.mdx#full-validator) publishes an archive reaching back to the network's first ledger. The [stellar-archivist](https://github.com/stellar/rs-stellar-archivist) command line tool gives you what you need to close the gap.
206+
207+
### Installing stellar-archivist
208+
209+
Using the [SDF package repositories](https://github.com/stellar/packages), you can install it by running:
201210

202211
```bash
203212
apt-get install stellar-archivist-rs
@@ -207,12 +216,40 @@ The tool is also published on [crates.io](https://crates.io/crates/stellar-archi
207216

208217
For detailed usage, please run `stellar-archivist-rs --help` (or `stellar-archivist --help` if installed from crates.io).
209218

219+
### Where to run stellar-archivist
220+
221+
stellar-archivist is a standalone tool. It does not link against or communicate with `stellar-core`; all it needs is read and write access to the archive itself. That access can be a local filesystem path (`file://`), an HTTP(S) source, or a cloud object store (`s3://`, `gcs://`, `azblob://`, `b2://`), so you can run it from any machine that can reach your archive.
222+
223+
:::caution
224+
225+
Run stellar-archivist on a separate host from your validator to avoid resource contention.
226+
227+
Do **not** stop your validator to work on its archive. stellar-archivist operates on the archive files directly and does not interfere with live publishing. Your node should stay online and keep publishing checkpoints throughout.
228+
229+
:::
230+
231+
### Choosing a command
232+
233+
| Your goal | Command |
234+
| --- | --- |
235+
| Find out whether an archive is complete, and where the gaps are | [`scan`](#scanning-an-archive) — read-only |
236+
| Fix missing or corrupt files in an archive you already have | [`repair`](#repairing-an-archive) |
237+
| Build an archive from scratch, or backfill a range of history | [`mirror`](#mirroring-an-archive) |
238+
239+
All three accept `--low` and `--high` to restrict the work to a ledger range.
240+
241+
### Checking file contents with `--verify`
242+
243+
By default, every command checks only that files exist. Adding `--verify` also checks that their contents are sound, which requires significantly more CPU and memory.
244+
245+
Leave it off for routine completeness checks and for fixing missing files. Turn it on to find and fix corruption, where a file exists but its contents are wrong.
246+
210247
### Scanning an archive
211248

212-
`scan` is read-only. It reports files that are missing, and when `--verify` is passed, it also checks file content integrity.
249+
`scan` is read-only. It walks the archive and reports the files that are missing, which is how you find out whether your archive has a gap and exactly where it is.[^1]
213250

214251
```bash
215-
stellar-archivist-rs scan file:///mnt/xvdf/stellar-core-archive/node_001 --verify --report report.json
252+
stellar-archivist-rs scan file:///mnt/xvdf/stellar-core-archive/node_001 --report report.json
216253
```
217254

218255
```log
@@ -274,13 +311,12 @@ error: Archive issues found
274311

275312
### Repairing an archive
276313

277-
`repair` scans and fixes an archive in place: it identifies missing or broken files at the destination and re-fetches them from a known-good source — such as the SDF public history archive. `--verify` validates file integrity at both the destination and the source.
314+
Once a scan has told you what is broken, `repair` fixes it in place: it identifies missing or broken files at the destination and re-fetches them from a known-good source — such as the SDF public history archive.
278315

279316
```bash
280317
stellar-archivist-rs repair \
281318
https://history.stellar.org/prd/core-testnet/core_testnet_001 \
282-
file:///mnt/xvdf/stellar-core-archive/node_001 \
283-
--verify
319+
file:///mnt/xvdf/stellar-core-archive/node_001
284320
```
285321

286322
```log
@@ -296,21 +332,21 @@ Repair runs in stages and reports each one separately: a main pass that audits t
296332

297333
:::tip
298334

299-
A report written by `scan` can be fed straight back in as a repair plan (with the `--plan` flag). A repair with a pre-existing plan skips the main pass, because the plan already says what is broken.
335+
The JSON report written by `scan` can be handed straight back to `repair` as a plan, using `--plan`. Repair then re-fetches exactly those files.
300336

301337
```bash
302338
stellar-archivist-rs repair \
303339
https://history.stellar.org/prd/core-testnet/core_testnet_001 \
304340
file:///mnt/xvdf/stellar-core-archive/node_001 \
305-
--verify --plan report.json
341+
--plan report.json
306342
```
307343

308344
:::
309345

310346
A final scan confirms the archive has been repaired:
311347

312348
```bash
313-
stellar-archivist-rs scan file:///mnt/xvdf/stellar-core-archive/node_001 --verify
349+
stellar-archivist-rs scan file:///mnt/xvdf/stellar-core-archive/node_001
314350
```
315351

316352
```log
@@ -328,13 +364,15 @@ Add `--dry-run` to report what would be repaired without writing anything. Combi
328364

329365
### Mirroring an archive
330366

331-
`mirror` copies a source archive to a destination. Use it to onboard a fresh archive, or to backfill a range of history your validator did not publish itself. For example, run the following command against an empty destination:
367+
`mirror` copies a source archive to a destination. It is the command to reach for when backfilling: run it once to build a complete archive from scratch, or scope it with `--low` and `--high` to fill in just the range of history your validator did not publish itself.
368+
369+
For example, run the following command against an empty destination:
332370

333371
```bash
334372
stellar-archivist-rs mirror \
335373
https://history.stellar.org/prd/core-testnet/core_testnet_001 \
336374
file:///mnt/xvdf/stellar-core-archive/node_001 \
337-
--high 1023 --verify
375+
--high 1023
338376
```
339377

340378
```log
@@ -347,16 +385,16 @@ stellar-archivist-rs mirror \
347385
2026-08-20T17:09:23.851784Z INFO Updated destination .well-known to checkpoint 1023 (0x000003ff)
348386
```
349387

350-
:::tip
388+
Afterwards, the destination contains a new archive covering history up to checkpoint 1023.
351389

352-
Mirroring is resumable. Running it again against the same destination with a higher `--high` picks up from the destination's current checkpoint and copies only what is missing, unless you pass `--overwrite`.
390+
:::tip
353391

354-
:::
392+
Mirroring is resumable. Running it again against the same destination with a higher `--high` picks up from the destination's current checkpoint and copies only what is missing.
355393

356-
Once the archive is complete, start your Stellar Core instance again.
394+
Files already present at the destination are skipped rather than re-fetched. Use `--overwrite` to re-fetch and replace them instead.
357395

358-
```bash
359-
systemctl start stellar-core
360-
```
396+
:::
361397

362398
You should now have a complete history archive being written by your full validator. Congratulations!
399+
400+
[^1]: The sample output on this page was captured with `--verify`, which is why it reports hash mismatches alongside the missing files.

0 commit comments

Comments
 (0)