Integration tests for erigondb.toml multiple reloading codepaths

[wmitsuda]

this is a follow up from: #19587

In the past, the DB settings (as of now, stepsize and max merge steps) used to be a hardcoded setting. But we quickly moved for exposing them as a hardcoded parameterized setting for experiments, and then now to push it to production on 3.4.

Problem: there are now multiple code paths and legacy vs new sync configurations scenarios, summarizing them here:

• Legacy datadir (3.3), no erigondb.toml, needs to assume legacy settings on migration and force write that file
• new sync on 3.4: the setting will come from downloader during the header-chain phase.
• already synced node (doesn't matter if from 3.3 or 3.4), erigondb.toml already exists, respect such setting
• --no-downloader will write erigondb.toml at start using the embedded hardcoded setting.

the critical scenario is the (2), since we can't assume any setting until it is downloaded and hot reloaded. assume the process can be interrupted between process launch and header-chain download.

also, in order to support such hot reloading, I had to instrument SharedDomains to allow such thing and make it a little flexible since some checkings are done at object construction (i.e. had to mock instead of assuming 0).

it all works now, but it seems to me that such settings lifecycle has become more "important" and fragile (anyone who touch ottersync lifecycle code can potentially break it). in the past downloader was quite a "dumb" process.

I think now we need to build some protection around it in the form of integration tests. Ideally we would have automated tests for all scenarios above using a real chain, but that's not scalable, the real chains are ever growing databases.

I'd like to propose to support some synthetic chain with shorter steps sizes, such we can have it already constructed and deployed in some test R2 bucket and use it to test those lifecycles.