Skip to content

downloader, snapshots: add erigon snapshots verify subcommand #19972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
anacrolix wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

downloader, snapshots: add erigon snapshots verify subcommand #19972

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f9b64f9
downloader, snapshots: add erigon snapshots verify subcommand
anacrolix Mar 18, 2026
83dfbc5
downloader: extract newSnapStorage, reuse in snapshot verify
anacrolix Mar 18, 2026
95cecf8
downloader, snapshots: parallel verification with JSON-lines output
anacrolix Mar 18, 2026
0b4b6a2
downloader: comprehensive snapshot set, per-snapshot progress output
anacrolix Mar 18, 2026
d2df7b3
downloader: output human-readable text from VerifySnapshotDownloads
anacrolix Mar 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions cmd/utils/app/snapshots_cmd.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -546,6 +546,31 @@ var snapshotCommand = cli.Command{
},
},
},
{
Name: "verify",
Description: "verify snapshot downloads against .torrent files and preverified.toml; findings are written as JSON lines to stdout",

Copilot AI Apr 8, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This subcommand help text says findings are written as "JSON lines", but VerifySnapshotDownloads prints human-readable formatted text (and a final summary line). Update the description (or change output format) so CLI help matches actual behavior.

Suggested change
Description: "verify snapshot downloads against .torrent files and preverified.toml; findings are written as JSON lines to stdout",
Description: "verify snapshot downloads against .torrent files and preverified.toml; findings are written as human-readable text to stdout",

Copilot uses AI. Check for mistakes.
Action: func(cliCtx *cli.Context) (err error) {
dirs := datadir.New(cliCtx.String(utils.DataDirFlag.Name))
workers := cliCtx.Int("workers")
summary, err := downloader.VerifySnapshotDownloads(cliCtx.Context, dirs, workers, os.Stdout)
if err != nil {
return fmt.Errorf("verify snapshot downloads: %w", err)
}
log.Root().Info("[snapshots verify] done",
"snapshots", summary.Total,
"errors", summary.Errors,
"warnings", summary.Warnings,
)
if summary.Errors > 0 {
return fmt.Errorf("%d error(s) found during snapshot verification", summary.Errors)
}
return nil
},
Flags: joinFlags([]cli.Flag{
&utils.DataDirFlag,
&cli.IntFlag{Name: "workers", Value: 0, Usage: "parallel workers for torrent verification (0 = GOMAXPROCS/2)"},
}),
},
{
Name: "preverified",
Action: func(cliCtx *cli.Context) (err error) {
Expand Down
25 changes: 14 additions & 11 deletions db/downloader/downloader.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ import (
"io"
"io/fs"
"iter"
"log/slog"
"math"
"net"
"net/http"
Expand Down Expand Up @@ -1293,6 +1294,18 @@ func openMdbx(
return dbCfg.Open(ctx)
}

// newSnapStorage creates the file storage client used for snapshot torrents.
// Part-file-only piece completion is used: this requires hashing incomplete files at start-up,
// but the expectation is that files should not be partial.
// See also: https://github.com/erigontech/erigon/pull/10074
func newSnapStorage(snapDir string, logger *slog.Logger) storage.ClientImplCloser {
return storage.NewFileOpts(storage.NewFileClientOpts{
ClientBaseDir: snapDir,
UsePartFiles: g.Some(true),
Logger: logger,
})
}

// This used to return the MDBX database. Instead, that's opened separately now and should be passed
// in if it's revived.
func newTorrentClient(
Expand All @@ -1304,17 +1317,7 @@ func newTorrentClient(
torrentClient *torrent.Client,
err error,
) {
// Possibly File storage needs more optimization for handles, will test. Should reduce GC
// pressure and improve scheduler handling.
// See also: https://github.com/erigontech/erigon/pull/10074
// We're using part-file-only piece completion. This requires hashing incomplete files at
// start-up, but the expectation is that files should not be partial. Also managing separate
// piece completion complicates user management of snapshots.
m = storage.NewFileOpts(storage.NewFileClientOpts{
ClientBaseDir: snapDir,
UsePartFiles: g.Some(true),
Logger: cfg.Slogger.With("names", "storage"),
})
m = newSnapStorage(snapDir, cfg.Slogger.With("names", "storage"))
cfg.DefaultStorage = m

defer func() {
Expand Down
Loading
Loading