You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Dec 22, 2025. It is now read-only.
By integrating the consensus layer into the EVM-node, Caplin eliminates the need for separate disk storage, reducing overall system complexity and improving overall efficiency. OtterSync, a new syncing algorithm, further enhances performance by shifting 98% of the computation to network bandwidth, reducing sync times and improving chain tip performance, disk footprint, and decentralization.
Caplin is enabled by default, at which point an external consensus layer is no longer needed.
@@ -14,6 +12,14 @@ Caplin is enabled by default, at which point an external consensus layer is no l
14
12
./build/bin/erigon
15
13
```
16
14
17
-
Caplin also has an archive mode for historical states and blocks, which can be enabled with the `--caplin.archive` flag.
15
+
Caplin also has an archive mode for historical states, blocks, and blobs. These can be enabled with the following flags:
16
+
17
+
-`--caplin.states-archive`: Enables the storage and retrieval of historical state data, allowing access to past states of the blockchain for debugging, analytics, or other use cases.
18
+
-`--caplin.blocks-archive`: Enables the storage of historical block data, making it possible to query or analyze full block history.
19
+
-`--caplin.blobs-archive`: Enables the storage of historical blobs, ensuring access to additional off-chain data that might be required for specific applications.
20
+
21
+
In addition, Caplin can backfill recent blobs for an op-node or other uses with the new flag:
22
+
23
+
-`--caplin.blobs-immediate-backfill`: Backfills the last 18 days' worth of blobs to quickly populate historical blob data for operational needs or analytics.
18
24
19
25
Caplin can also be used for [block production](/advanced/block-prod.md#using-caplin-as-validator), aka **staking**.
@@ -11,155 +11,36 @@ While all Erigon components are separable and can be run on different machines,
11
11
12
12
</div>
13
13
14
-
Like many other Erigon components (txpool, sentry, rpc daemon) `downloader` can be running [integrated into Erigon](#start-erigon-with-snapshots-support) or as a [separate process](#running-downloader-as-a-separate-process).
14
+
For a comprehensive understanding of the Downloader's functionality, configuration, and usage, please refer to the embedded file you can find in your compiled Erigon folder at `./cmd/downloader/README.md`. This file covers the following key topics:
15
15
16
-
## Start Erigon with snapshots support
16
+
1.**Snapshots overview**: An introduction to snapshots, their benefits, and how they are created and used in Erigon.
17
+
2.**Starting Erigon with snapshots support**: Instructions on how to start Erigon with snapshots support, either by default or as a separate process.
18
+
3.**Creating new networks or bootnodes**: A guide on how to create new networks or bootnodes, including creating new snapshots and starting the Downloader.
19
+
4.**Architecture**: An overview of the Downloader's architecture, including how it works with Erigon and the different ways .torrent files can be created.
20
+
5.**Utilities**: A list of available utilities, including `torrent_cat`, `torrent_magnet`, and `torrent_clean`.
21
+
6.**Remote manifest verify**: Instructions on how to verify that remote webseeds have available manifests and all manifested files are available.
22
+
7.**Faster rsync**: Tips on how to use `rsync` for faster file transfer.
23
+
8.**Release details**: Information on how to start automatic commits of new hashes to the `master` branch.
24
+
9.**Creating a seedbox**: A guide on how to create a seedbox to support a new network or type of snapshots.
17
25
26
+
Some of the key sections in the documentation include:
18
27
19
-
Downloader is running by default inside Erigon, where it can be configured with the `--snapshots` flag:
28
+
***How to create new snapshots**: Instructions on how to create new snapshots, including using the `seg` command and creating .torrent files.
29
+
***How to start the Downloader**: Instructions on how to start the Downloader, either as a separate process or as part of Erigon.
30
+
***How to verify .seg files**: Instructions on how to verify that .seg files have the same checksum as the current .torrent files.
20
31
21
-
```bash
22
-
./build/bin/erigon --snapshots
23
-
```
24
-
25
-
The `--snapshots` flag is compatible with the `--prune.mode` flag.
26
-
27
-
When you start the Erigon node with the --snapshots flag, it allows you to control the behavior of the Downloader component, such as:
28
-
29
-
- Enabling or disabling the Downloader component
30
-
- Configuring the Downloader to use specific snapshots or directories
31
-
- Controlling the seeding and downloading of snapshots
32
-
33
-
## Running downloader as a separate process
34
-
35
-
Before using a separate downloader process the executable must be built:
36
-
37
-
```bash
38
-
cd erigon
39
-
make downloader
40
-
```
41
-
Downloader can then be started as independent process, for example:
-`--downloader.api.addr` is for internal communication with Erigon;
48
-
-`--torrent.port=42068` is for public BitTorrent protocol listen;
49
-
- The default download speed is 16mb/sec. You can increase/limit the network usage by adding the flags `--torrent.download.rate=512mb --torrent.upload.rate=512mb`
50
-
- On startup Erigon sends list of `.torrent` files to Downloader and waits for 100% download completion
- Use `--snap.keepblocks=true` to not delete retired blocks from DB.
55
-
- Any network/chain can start with snapshot sync:
56
-
- The node will only download snapshots that are registered in the next repository, which can be found at <https://github.com/erigontech/erigon-snapshot>.
57
-
- The node will periodically clear out old blocks from the database by copying them into snapshots of 1,000-block size. It will then merge these snapshots into larger ranges until it reaches snapshots of 500,000 blocks. Once this process is complete, the node will automatically begin creating new snapshots.".
58
-
59
-
## Creation of a new network or bootnode
60
-
61
-
When creating a new network or bootnode, it's possible that new snapshots may need to be created and seeded. To do this, you'll need to create the snapshots using the following command:
62
-
63
-
```bash
64
-
erigon snapshots retire --datadir=<your_datadir>
65
-
```
66
-
67
-
This will create .torrent files that the Downloader can use to seed the new snapshots. The output format is compatible with <https://github.com/ledgerwatch/erigon-snapshot>.
68
-
69
-
To change the size of the snapshots, add the following flags:
70
-
71
-
```bash
72
-
--from=0 --to=1_000_000 --segment.size=500_000
73
-
```
74
-
75
-
Then, use the downloader command to rebuild the snapshots:
The Downloader will automatically seed the new snapshots.
82
-
83
-
Note that while Erigon is not required to seed snapshots, running Erigon with the --snapshots flag will also enable seeding.
84
-
85
-
## Additional info
32
+
By referring to the embedded documentation file, you can gain a deeper understanding of the Downloader's capabilities and how to effectively utilize it in your Erigon setup.
86
33
87
-
When creating snapshots, you don't need a fully-synced Erigon node. In fact, you only need to complete a few initial stages of syncing. This allows you to start seeding snapshots without having to wait for a full sync.
88
-
89
-
To take advantage of this, you can use the following flags to start syncing, but only seed the first few stages:
90
-
91
-
- `STOP AFTER STAGE=Senders`: this flag tells Erigon to stop syncing after reaching the "Senders" stage. This allows you to seed the snapshots without waiting for the full sync.
92
-
- `./build/bin/erigon --snapshots=false --datadir=<your_datadir>`: this command starts the Erigon node in snapshot-only mode, which allows it to seed the snapshots without waiting for the full sync.
93
-
94
-
However, for security reasons, it's highly recommended to have a fully-synced Erigon node. This ensures that all data is up-to-date and secure.
95
-
96
-
### Indexing Snapshots Before Seeding
97
-
98
-
Before seeding snapshots, they must be indexed by Erigon. While indexing is not required for seeding, it's recommended to run the following command to ensure that snapshots are properly indexed:
99
-
100
-
```bash
101
-
./build/bin/erigon snapshots index --datadir=<your_datadir>
102
-
```
103
-
104
-
This step is not required for seeding, but it's essential for using snapshots effectively.
105
-
106
-
## Architecture
107
-
108
-
Snapshots can be created in four ways:
109
-
110
-
1. **Automated creation**: Erigon can create new .torrent files using the downloader.Download RPC method.
111
-
2. **Manual creation**: Erigon can create new .seg files, which are then scanned by the Downloader to create .torrent files.
112
-
3. **Manual seeding**: .torrent files can be manually copied from another server or restored from backup.
113
-
4. **Manual indexing**: .seg files can be manually copied and indexed by the Downloader.
114
-
115
-
**Erigon**'s role in the snapshot creation process:
116
-
- Connects to the Downloader and shares the list of hashes.
117
-
- Waits for the download of all snapshots.
118
-
- Creates `.idx` files (secondary indices) when .seg files are available.
119
-
- Switches to normal staged sync after downloading all snapshots.
120
-
- Ensures that snapshot downloading happens only once, even if new Erigon versions are released.
121
-
122
-
**Downloader**'s role in the snapshot creation process:
123
-
- Reads .torrent files and downloads the contents described by them.
124
-
- Uses a list of trackers to download files (see ./trackers/embed.go).
125
-
- Automatically seeds the downloaded files.
126
-
127
-
# Technical Details
128
-
129
-
To prevent attacks, `.idx` files are created with random seeds, ensuring that each node has a unique `.idx` file (but the same `.seg` file).
130
-
If `.seg` files are manually added or removed, the `<your_datadir>/snapshots/db` folder must also be updated.
131
-
132
-
## Verifying .seg file checksums
133
-
134
-
Use the `downloader --verify` command to check the checksums of the `.seg` files. Compare the checksums with those of the current `.torrent` files.
135
-
136
-
## Faster Rsync
137
-
138
-
Use the following command to perform a faster rsync:
Copy file name to clipboardExpand all lines: src/advanced/options.md
+7-12Lines changed: 7 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,11 +10,8 @@ The `--help` flag listing is reproduced below for your convenience.
10
10
```bash
11
11
./build/bin/erigon --help
12
12
```
13
-
# Commands
14
13
15
-
16
-
17
-
# Commands
14
+
## Commands
18
15
19
16
```
20
17
NAME:
@@ -24,7 +21,7 @@ USAGE:
24
21
erigon [command] [flags]
25
22
26
23
VERSION:
27
-
3.00.0-alpha6-f22317ef
24
+
3.00.0-alpha7-34714c0c
28
25
29
26
COMMANDS:
30
27
init Bootstrap and initialize a new genesis block
@@ -34,12 +31,10 @@ COMMANDS:
34
31
help, h Shows a list of commands or help for one command
35
32
36
33
GLOBAL OPTIONS:
37
-
--datadir value Data directory for the databases (default: /root/.local/share/erigon)
38
-
--ethash.dagdir value Directory to store the ethash mining DAGs (default: /root/.local/share/erigon-ethash)
34
+
--datadir value Data directory for the databases (default: /home/bloxster/.local/share/erigon)
35
+
--ethash.dagdir value Directory to store the ethash mining DAGs (default: /home/bloxster/.local/share/erigon-ethash)
39
36
--externalcl Enables the external consensus layer (default: false)
40
37
--txpool.disable Experimental external pool and block producer, see ./cmd/txpool/readme.md for more info. Disabling internal txpool and block producer. (default: false)
41
-
--txpool.locals value Comma separated accounts to treat as locals (no flush, priority inclusion)
42
-
--txpool.nolocals Disables price exemptions for locally submitted transactions (default: false)
43
38
--txpool.pricelimit value Minimum gas price (fee cap) limit to enforce for acceptance into the pool (default: 1)
44
39
--txpool.pricebump value Price bump percentage to replace an already existing transaction (default: 10)
45
40
--txpool.blobpricebump value Price bump percentage to replace existing (type-3) blob transaction (default: 100)
@@ -48,9 +43,7 @@ GLOBAL OPTIONS:
48
43
--txpool.totalblobpoollimit value Total limit of number of all blobs in txs within the txpool (default: 480)
49
44
--txpool.globalslots value Maximum number of executable transaction slots for all accounts (default: 10000)
50
45
--txpool.globalbasefeeslots value Maximum number of non-executable transactions where only not enough baseFee (default: 30000)
51
-
--txpool.accountqueue value Maximum number of non-executable transaction slots permitted per account (default: 64)
52
46
--txpool.globalqueue value Maximum number of non-executable transaction slots for all accounts (default: 30000)
53
-
--txpool.lifetime value Maximum amount of time non-executable transaction are queued (default: 3h0m0s)
54
47
--txpool.trace.senders value Comma separated list of addresses, whose transactions will traced in transaction pool with debug printing
55
48
--txpool.commit.every value How often transactions should be committed to the storage (default: 15s)
56
49
--prune.distance value Keep state history for the latest N blocks (default: everything) (default: 0)
@@ -227,7 +220,7 @@ GLOBAL OPTIONS:
227
220
--beacon.api.cors.allow-credentials set the cors' allow credentials (default: false)
228
221
--beacon.api.port value sets the port to listen for beacon api requests (default: 5555)
229
222
--beacon.api.read.timeout value Sets the seconds for a read time out in the beacon api (default: 5)
230
-
--beacon.api.write.timeout value Sets the seconds for a write time out in the beacon api (default: 5)
223
+
--beacon.api.write.timeout value Sets the seconds for a write time out in the beacon api (default: 31536000)
231
224
--beacon.api.protocol value Protocol for beacon API (default: "tcp")
232
225
--beacon.api.ide.timeout value Sets the seconds for a write time out in the beacon api (default: 25)
233
226
--caplin.backfilling sets whether backfilling is enabled for caplin (default: false)
@@ -247,6 +240,8 @@ GLOBAL OPTIONS:
247
240
--sync.loop.break.after value Sets the last stage of the sync loop to run
248
241
--sync.parallel-state-flushing Enables parallel state flushing (default: true)
249
242
--chaos.monkey Enable 'chaos monkey' to generate spontaneous network/consensus/etc failures. Use ONLY for testing (default: false)
243
+
--shutter Enable the Shutter encrypted transactions mempool (defaults to false) (default: false)
244
+
--shutter.keyper.bootnodes value [ --shutter.keyper.bootnodes value ] Use to override the default keyper bootnodes (defaults to using the bootnodes from the embedded config)
250
245
--pprof Enable the pprof HTTP server (default: false)
251
246
--pprof.addr value pprof HTTP server listening interface (default: "127.0.0.1")
252
247
--pprof.port value pprof HTTP server listening port (default: 6060)
0 commit comments