db/state, snapcfg: download erigondb.toml via torrent; move settings resolution out of Open() (#19587)

## Summary

- Move `ResolveErigonDBSettings` out of `Aggregator.Open()` so callers
resolve settings explicitly before constructing the aggregator, enabling
the downloader to provide the real `erigondb.toml` before domains are
configured.
- Whitelist `erigondb.toml` in the header-chain download phase so the
torrent downloader delivers it alongside headers/bodies.
- After the header-chain download completes, `ReloadErigonDBSettings()`
re-reads the file and propagates any stepSize changes to all
Domain/InvertedIndex instances.
- Add `WithErigonDBSettings()` builder on `AggOpts` so all call sites
pass pre-resolved settings.

Three runtime scenarios handled:
1. **Legacy datadir** (has `preverified.toml`, no `erigondb.toml`):
writes `erigondb.toml` with legacy settings (step_size=1,562,500) on
startup.
2. **Fresh sync with downloader**: starts with defaults, downloader
delivers real `erigondb.toml` during header-chain phase, settings are
reloaded and propagated.
3. **Fresh sync with `--no-downloader`**: writes defaults to disk
immediately since no downloader will provide the file.

## Test plan

- [x] Scenario 1 (legacy datadir): confirmed log `Creating erigondb.toml
with LEGACY settings step_size=1562500` and file written on startup
- [x] Scenario 2 (fresh + downloader): confirmed log `erigondb stepSize
changed, propagating` after header-chain download delivers
`erigondb.toml` with step_size=390625
- [x] Scenario 3 (fresh + `--no-downloader`): confirmed log
`Initializing erigondb.toml with DEFAULT settings (nodownloader)` and
file written immediately

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Author: wmitsuda

.claude/skills/erigondb-sync-integration-test-plan/SKILL.md

@@ -0,0 +1,118 @@
+---
+name: erigondb-sync-integration-test-plan
+description: Integration test plan for erigondb.toml settings resolution across 3 runtime scenarios (legacy, fresh+downloader, fresh+no-downloader)
+allowed-tools: Bash, Read
+user-invocable: false
+---
+
+# erigondb.toml Integration Test Plan
+
+Verify that `erigondb.toml` settings are correctly resolved across three distinct runtime scenarios: legacy datadirs, fresh sync with downloader, and fresh sync without downloader.
+
+Use this plan after any changes to the erigondb.toml resolution logic (creation, defaults, legacy detection, downloader delivery) to confirm all paths still work.
+
+## Parameters
+
+Two values must be provided by the caller:
+
+| Parameter | Description | Example |
+|-----------|-------------|---------|
+| **chain** | Chain name passed to `--chain=` | `hoodi`, `mainnet` |
+| **legacy datadir** | Path to an existing Erigon datadir that has `preverified.toml` but **no** `erigondb.toml` | `~/eth-nodes/erigon33-hoodi-stable` |
+
+All scenarios use `--prune.mode=archive`.
+
+## Pre-requisites
+
+1. **Built binary** — invoke `/erigon-build` if `./build/bin/erigon` does not exist.
+2. **Legacy datadir** — the user-provided path must contain `snapshots/preverified.toml` and must **not** contain `snapshots/erigondb.toml`.
+3. **Port availability** — use `/erigon-network-ports` for reference. Each scenario uses a different port offset (+100, +200, +300) via `/erigon-ephemeral` conflict detection.
+
+## Scenario A: Legacy datadir
+
+**Goal**: Starting on a legacy datadir (has `preverified.toml`, no `erigondb.toml`) creates `erigondb.toml` with legacy settings (`step_size = 1562500`).
+
+### Steps
+
+1. Create an ephemeral clone of the user's legacy datadir using `/erigon-ephemeral` (Mode B: clone). This gives an isolated copy so the original is untouched.
+2. Confirm pre-conditions on the clone:
+   - `snapshots/preverified.toml` exists
+   - `snapshots/erigondb.toml` does **not** exist
+3. Start erigon with port offset **+100** (via `/erigon-ephemeral` port conflict detection):
+   ```
+   ./build/bin/erigon \
+     --datadir=<clone-path> \
+     --chain=<chain> \
+     --prune.mode=archive \
+     <port flags at +100 offset>
+   ```
+4. Wait for startup (~15s). Check logs for a message indicating legacy settings detection (e.g., `Creating erigondb.toml with LEGACY settings`).
+5. Verify the generated file:
+   ```bash
+   cat <clone-path>/snapshots/erigondb.toml
+   ```
+   Expected: `step_size = 1562500`
+6. Kill the process and clean up the ephemeral datadir.
+
+## Scenario B: Fresh sync with downloader
+
+**Goal**: A fresh datadir (no `preverified.toml`, no `erigondb.toml`) starts with code defaults (`step_size = 1562500`), then the downloader delivers the network's `erigondb.toml` during the header-chain phase, which may have different settings (e.g., `step_size = 390625` for hoodi).
+
+### Steps
+
+1. Create an empty ephemeral datadir using `/erigon-ephemeral` (Mode A: empty).
+2. Start erigon with port offset **+200**:
+   ```
+   ./build/bin/erigon \
+     --datadir=<empty-path> \
+     --chain=<chain> \
+     --prune.mode=archive \
+     <port flags at +200 offset>
+   ```
+3. Check early logs for a message indicating defaults are being used (e.g., `erigondb.toml not found, using defaults`) with `step_size=1562500`.
+4. Wait for the header-chain download phase to complete (~30-120s depending on network). Watch logs for:
+   - `[1/6 OtterSync] Downloader completed header-chain`
+   - `Reading DB settings from existing erigondb.toml`
+   - `erigondb stepSize changed, propagating` (if the network's settings differ from the code defaults)
+5. Verify the delivered file:
+   ```bash
+   cat <empty-path>/snapshots/erigondb.toml
+   ```
+   The values come from the network's published `erigondb.toml` (check the chain's webseed for the canonical values).
+6. Kill the process and clean up the ephemeral datadir.
+
+## Scenario C: Fresh sync with `--no-downloader`
+
+**Goal**: A fresh datadir with `--no-downloader` immediately writes `erigondb.toml` with code defaults (`step_size = 1562500`).
+
+### Steps
+
+1. Create an empty ephemeral datadir using `/erigon-ephemeral` (Mode A: empty).
+2. Start erigon with port offset **+300** and `--no-downloader`:
+   ```
+   ./build/bin/erigon \
+     --datadir=<empty-path> \
+     --chain=<chain> \
+     --prune.mode=archive \
+     --no-downloader \
+     <port flags at +300 offset>
+   ```
+3. Check logs for a message indicating defaults were written immediately (e.g., `Initializing erigondb.toml with DEFAULT settings`) with `step_size=1562500`.
+4. Verify the file exists immediately:
+   ```bash
+   cat <empty-path>/snapshots/erigondb.toml
+   ```
+   Expected: `step_size = 1562500` (code defaults, since no downloader to provide network settings)
+5. Kill the process and clean up the ephemeral datadir.
+
+## Success Criteria
+
+| Scenario | Condition | Expected step_size | File timing |
+|----------|-----------|-------------------|-------------|
+| A (legacy) | `Creating erigondb.toml with LEGACY settings` log | 1,562,500 | Written immediately on startup |
+| B (fresh+downloader) | `erigondb.toml not found, using defaults` then `erigondb stepSize changed, propagating` | Code default 1,562,500 then network value (chain-dependent) | After downloader delivers it |
+| C (fresh+no-downloader) | `Initializing erigondb.toml with DEFAULT settings (nodownloader)` log | 1,562,500 | Written immediately on startup |
+
+## Cleanup
+
+After all scenarios complete, ensure all ephemeral datadirs and processes are cleaned up. Use `/erigon-ephemeral` Step 5 (cleanup) for each instance, or Step 6 (leftover detection) to find any stragglers.

cmd/integration/commands/stages.go

@@ -1133,7 +1133,11 @@ func allSnapshots(ctx context.Context, db kv.RoDB, logger log.Logger) (*freezebl
 		blockReader := freezeblocks.NewBlockReader(_allSnapshotsSingleton, _allBorSnapshotsSingleton)
 		txNums := blockReader.TxnumReader()
 
-		_aggSingleton = dbstate.New(dirs).Logger(logger).MustOpen(ctx, db)
+		var erigonDBSettings *dbstate.ErigonDBSettings
+		if erigonDBSettings, err = dbstate.ResolveErigonDBSettings(dirs, logger, false); err != nil {
+			return
+		}
+		_aggSingleton = dbstate.New(dirs).Logger(logger).WithErigonDBSettings(erigonDBSettings).MustOpen(ctx, db)
 
 		_aggSingleton.SetProduceMod(snapCfg.ProduceE3)
 

cmd/rpcdaemon/cli/config.go

@@ -439,7 +439,11 @@ func RemoteServices(ctx context.Context, cfg *httpcfg.HttpCfg, logger log.Logger
 			return nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, err
 		}
 
-		agg, err := dbstate.New(cfg.Dirs).Logger(logger).Open(ctx, rawDB)
+		erigonDBSettings, err := dbstate.ResolveErigonDBSettings(cfg.Dirs, logger, false)
+		if err != nil {
+			return nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, err
+		}
+		agg, err := dbstate.New(cfg.Dirs).Logger(logger).WithErigonDBSettings(erigonDBSettings).Open(ctx, rawDB)
 		if err != nil {
 			return nil, nil, nil, nil, nil, nil, nil, ff, nil, nil, fmt.Errorf("create aggregator: %w", err)
 		}

cmd/state/commands/opcode_tracer.go

@@ -436,7 +436,11 @@ func OpcodeTracer(genesis *types.Genesis, blockNum uint64, chaindata string, num
 	rawChainDb := mdbx.MustOpen(dirs.Chaindata)
 	defer rawChainDb.Close()
 
-	agg, err := dbstate.New(dirs).Logger(logger).Open(context.Background(), rawChainDb)
+	erigonDBSettings, err := dbstate.ResolveErigonDBSettings(dirs, logger, false)
+	if err != nil {
+		return err
+	}
+	agg, err := dbstate.New(dirs).Logger(logger).WithErigonDBSettings(erigonDBSettings).Open(context.Background(), rawChainDb)
 	if err != nil {
 		return err
 	}

cmd/utils/app/snapshots_cmd.go

@@ -2898,7 +2898,11 @@ func dbCfg(label kv.Label, path string) mdbx.MdbxOpts {
 		Accede(true) // integration tool: open db without creation and without blocking erigon
 }
 func openAgg(ctx context.Context, dirs datadir.Dirs, chainDB kv.RwDB, logger log.Logger) *state.Aggregator {
-	agg, err := state.New(dirs).SanityOldNaming().Logger(logger).Open(ctx, chainDB)
+	erigonDBSettings, err := state.ResolveErigonDBSettings(dirs, logger, false)
+	if err != nil {
+		panic(err)
+	}
+	agg, err := state.New(dirs).SanityOldNaming().Logger(logger).WithErigonDBSettings(erigonDBSettings).Open(ctx, chainDB)
 	if err != nil {
 		panic(err)
 	}

cmd/utils/app/squeeze_cmd.go

@@ -141,7 +141,11 @@ func squeezeStorage(ctx context.Context, dirs datadir.Dirs, logger log.Logger) e
 	ac := agg.BeginFilesRo()
 	defer ac.Close()
 
-	aggOld, err := state.New(dirsOld).Logger(logger).Open(ctx, db)
+	erigonDBSettingsOld, err := state.ResolveErigonDBSettings(dirsOld, logger, false)
+	if err != nil {
+		panic(err)
+	}
+	aggOld, err := state.New(dirsOld).Logger(logger).WithErigonDBSettings(erigonDBSettingsOld).Open(ctx, db)
 	if err != nil {
 		panic(err)
 	}
@@ -183,7 +187,11 @@ func squeezeStorage(ctx context.Context, dirs datadir.Dirs, logger log.Logger) e
 func squeezeCode(ctx context.Context, dirs datadir.Dirs, logger log.Logger) error {
 	db := dbCfg(dbcfg.ChainDB, dirs.Chaindata).MustOpen()
 	defer db.Close()
-	agg := state.New(dirs).Logger(logger).MustOpen(ctx, db)
+	erigonDBSettings, err := state.ResolveErigonDBSettings(dirs, logger, false)
+	if err != nil {
+		return err
+	}
+	agg := state.New(dirs).Logger(logger).WithErigonDBSettings(erigonDBSettings).MustOpen(ctx, db)
 	defer agg.Close()
 	agg.SetCompressWorkers(estimate.CompressSnapshot.Workers())
 

cmd/utils/app/step_cmd.go

@@ -27,7 +27,7 @@ func stepRebase(cliCtx *cli.Context) error {
 	defer ticker.Stop()
 
 	dirs := datadir.Open(cliCtx.String("datadir"))
-	settings, err := state.CreateOrReadErigonDBSettings(dirs, logger)
+	settings, err := state.ResolveErigonDBSettings(dirs, logger, true)
 	if err != nil {
 		return err
 	}

db/snapcfg/util.go

@@ -154,6 +154,10 @@ func (p Preverified) Typed(types []snaptype.Type) Preverified {
 			bestVersions.Set(p.Name, p)
 			continue
 		}
+		if p.Name == "erigondb.toml" {
+			bestVersions.Set(p.Name, p)
+			continue
+		}
 
 		v, name, ok := strings.Cut(p.Name, "-")
 		if !ok {

db/snapshotsync/snapshotsync.go

@@ -412,7 +412,7 @@ func SyncSnapshots(
 				continue
 			}
 			if headerchain &&
-				!(strings.Contains(p.Name, "headers") || strings.Contains(p.Name, "bodies") || p.Name == "salt-blocks.txt") {
+				!(strings.Contains(p.Name, "headers") || strings.Contains(p.Name, "bodies") || p.Name == "salt-blocks.txt" || p.Name == "erigondb.toml") {
 				continue
 			}
 			if !syncCfg.KeepExecutionProofs && isStateHistory(p.Name) && strings.Contains(p.Name, kv.CommitmentDomain.String()) {

db/state/aggregator.go

@@ -98,6 +98,11 @@ type Aggregator struct {
 	produce bool
 
 	checker *DependencyIntegrityChecker
+
+	// Domain configuration state: ConfigureDomains() is a no-op once configured is true.
+	configured   bool
+	savedSalt    *uint32
+	disableFsync bool
 }
 
 func newAggregator(ctx context.Context, dirs datadir.Dirs, reorgBlockDepth uint64, db kv.RoDB, logger log.Logger) (*Aggregator, error) {
@@ -234,23 +239,90 @@ func (a *Aggregator) reloadSalt() error {
 	}
 
 	for _, d := range a.d {
-		d.salt.Store(salt)
+		if d != nil {
+			d.salt.Store(salt)
+		}
 	}
 
 	for _, ii := range a.iis {
-		ii.salt.Store(salt)
+		if ii != nil {
+			ii.salt.Store(salt)
+		}
 	}
 
 	return nil
 }
 
-func (a *Aggregator) reloadErigonDBSettings() error {
-	settings, err := CreateOrReadErigonDBSettings(a.dirs, a.logger)
+// ReloadErigonDBSettings re-reads erigondb.toml from disk and updates the in-memory stepSize.
+// If domains are already configured and stepSize changed, it propagates the new value to all
+// Domain/InvertedIndex instances.
+func (a *Aggregator) ReloadErigonDBSettings(noDownloader bool) error {
+	oldStepSize := a.stepSize
+	oldStepsInFrozenFile := a.stepsInFrozenFile
+
+	settings, err := ResolveErigonDBSettings(a.dirs, a.logger, noDownloader)
 	if err != nil {
 		return err
 	}
 	a.stepSize = settings.StepSize
 	a.stepsInFrozenFile = settings.StepsInFrozenFile
+
+	if a.configured && (a.stepSize != oldStepSize || a.stepsInFrozenFile != oldStepsInFrozenFile) {
+		a.logger.Info("erigondb stepSize changed, propagating to domains/IIs",
+			"old_step_size", oldStepSize, "new_step_size", a.stepSize,
+			"old_steps_in_frozen_file", oldStepsInFrozenFile, "new_steps_in_frozen_file", a.stepsInFrozenFile)
+		// stepSize lives only on InvertedIndex; Domain and History access it through embedding.
+		// Update the InvertedIndex embedded in each domain's History, plus standalone IIs.
+		for _, d := range a.d {
+			if d != nil && d.History != nil && d.History.InvertedIndex != nil {
+				d.History.InvertedIndex.setStepSize(a.stepSize, a.stepsInFrozenFile)
+			}
+		}
+		for _, ii := range a.iis {
+			if ii != nil {
+				ii.setStepSize(a.stepSize, a.stepsInFrozenFile)
+			}
+		}
+	}
+	return nil
+}
+
+// ConfigureDomains creates all Domain and InvertedIndex instances. This is a no-op if domains
+// are already configured. It requires stepSize > 0 (which NewInvertedIndex enforces via panic).
+func (a *Aggregator) ConfigureDomains() error {
+	if a.configured {
+		return nil
+	}
+	if a.stepSize == 0 {
+		return fmt.Errorf("cannot configure domains: stepSize is 0")
+	}
+	// AdjustReceipt mutates the global statecfg.Schema; must run before Configure().
+	if err := statecfg.AdjustReceiptCurrentVersionIfNeeded(a.dirs, a.logger); err != nil {
+		return err
+	}
+	if err := statecfg.Configure(statecfg.Schema, a, a.dirs, a.savedSalt, a.logger); err != nil {
+		return err
+	}
+	a.configured = true
+
+	if a.disableFsync {
+		for _, d := range a.d {
+			if d != nil {
+				d.DisableFsync()
+			}
+		}
+		for _, ii := range a.iis {
+			if ii != nil {
+				ii.DisableFsync()
+			}
+		}
+	}
+
+	func() {
+		a.dirtyFilesLock.Lock()
+		defer a.dirtyFilesLock.Unlock()
+		a.recalcVisibleFiles(a.dirtyFilesEndTxNumMinimax())
+	}()
 	return nil
 }
 
@@ -430,13 +502,19 @@ func (a *Aggregator) Close() {
 func (a *Aggregator) closeDirtyFiles() {
 	wg := &sync.WaitGroup{}
 	for _, d := range a.d {
+		if d == nil {
+			continue
+		}
 		wg.Add(1)
 		go func() {
 			defer wg.Done()
 			d.Close()
 		}()
 	}
 	for _, ii := range a.iis {
+		if ii == nil {
+			continue
+		}
 		wg.Add(1)
 		go func() {
 			defer wg.Done()
@@ -464,6 +542,9 @@ func (a *Aggregator) SetCompressWorkers(i int) {
 		return
 	}
 	for _, d := range a.d {
+		if d == nil {
+			continue
+		}
 		d.CompressCfg.Workers = i
 		if d.History != nil {
 			d.History.CompressorCfg.Workers = i
@@ -1332,6 +1413,9 @@ func (a *Aggregator) FirstTxNumOfStep(step kv.Step) uint64 { // could have some
 }
 
 func (a *Aggregator) dirtyFilesEndTxNumMinimax() uint64 {
+	if a.d[kv.AccountsDomain] == nil || a.d[kv.StorageDomain] == nil || a.d[kv.CodeDomain] == nil {
+		return 0
+	}
 	m := min(
 		a.d[kv.AccountsDomain].dirtyFilesEndTxNumMinimax(),
 		a.d[kv.StorageDomain].dirtyFilesEndTxNumMinimax(),