Config File Issue: CometBFT Not Reading Seeds/Persistent Peers

[mojtaba-esk]

Summary of Bug

Problem

The TestStateSyncMocha test was failing because CometBFT wasn't picking up seeds and persistent peers that were written to config/config.toml via config.Modify() in the PostInit hook.

Symptoms

• CometBFT logs showed: "Adding persistent peers addrs=["...self-peer..."]" (only seeing itself)
• Repeated errors: "Couldn't connect to any seeds"
• Node couldn't connect to the Mocha network, causing state sync to fail
• Config file was verified to be written correctly (we read it back from the container)

What We Tried

1. Verified config file was written correctly by reading it directly from the container
2. Added extensive logging to trace peer discovery and config writing
3. Tried different config modification approaches
4. Verified the config file path and format

Workaround

We switched to passing seeds and persistent peers via CLI flags (--p2p.seeds and --p2p.persistent_peers) when starting the node, which works reliably. CLI args override config file values in CometBFT.

Root Cause (Unknown)

We never identified the exact root cause. Possible explanations:

1. Timing issue: CometBFT might read the config file at startup before PostInit runs, or there's a race condition
2. Config caching: CometBFT might cache the config at some point and not re-read it after our modifications
3. Path/context issues: The config might be written to a different location than where CometBFT reads it from

Related Code

• test/docker-e2e/e2e_state_sync_test.go - The test that was failing
• test/docker-e2e/networks/config.go - Mocha config with seeds/peers
• Uses config.Modify() from github.com/celestiaorg/tastora/framework/testutil/config

Current Status

Using CLI args as a workaround. The config file approach should be investigated to understand why it didn't work.

@chatton might have more context on how PostInit handles the things.

Version

v6.2.5

Steps to Reproduce

1. Revert the CLI args workaround in test/docker-e2e/e2e_state_sync_test.go:

   • Remove the --p2p.seeds and --p2p.persistent_peers CLI flags from startArgs
   • Add back code to write seeds/peers to config/config.toml via config.Modify() in configureStateSyncClient or PostInit

2. Run the test:

cd ./test/docker-e2e && go test -v ./... -run=TestCelestiaTestSuite/TestStateSyncMocha

3. Observe the failure:

   • CometBFT logs show: "Adding persistent peers addrs=["...self-peer..."]" (only itself)
   • Repeated errors: "Couldn't connect to any seeds"
   • Test fails with "context deadline exceeded" as the node can't sync

4. Verify config was written (optional):

   • Check container logs or exec into the container to confirm config/config.toml contains the seeds/peers
   • The config file will have the values, but CometBFT won't use them

Expected behavior: CometBFT should read seeds/peers from the config file and connect to the Mocha network.

Actual behavior: CometBFT only sees itself as a peer and can't connect to any seeds.