Skip to content

Commit a5d5400

Browse files
Merge #6304: backport: merge bitcoin#23280 (coalesce chainstate loading sequence between {,non-}unittest codepaths)
1974651 refactor: move remaining `LogPrintf` usage outside (Kittywhiskers Van Gogh) 04dbaa8 style-only: Remove redundant scope in *Chainstate (Kittywhiskers Van Gogh) 872158d Remove all #include // for * comments (Kittywhiskers Van Gogh) 09ab629 test/setup: Use LoadChainstate (Kittywhiskers Van Gogh) 459f339 node/chainstate: extract Dash post-`InitializeChainstate` logic (Kittywhiskers Van Gogh) c06e074 node/chainstate: Add options for in-memory DBs (Kittywhiskers Van Gogh) 52bb35d node/caches: Remove intermediate variables (Kittywhiskers Van Gogh) 4ab1827 node/caches: Extract cache calculation logic (Kittywhiskers Van Gogh) d7f1e23 validation: VerifyDB only needs Consensus::Params (Kittywhiskers Van Gogh) c405492 node/chainstate: Decouple from ShutdownRequested (Kittywhiskers Van Gogh) fdf803d node/chainstate: Decouple from GetTime (Kittywhiskers Van Gogh) f7aef8d init: Delay RPC block notif until warmup finished (Kittywhiskers Van Gogh) 94c0ceb Move -checkblocks LogPrintf to AppInitMain (Kittywhiskers Van Gogh) d3345ee node/chainstate: Reduce coupling of LogPrintf (Kittywhiskers Van Gogh) a141f5d node/chainstate: Decouple from concept of uiInterface (Kittywhiskers Van Gogh) 913411e Split off VerifyLoadedChainstate (Kittywhiskers Van Gogh) 53231ca node/chainstate: Remove do/while loop (Kittywhiskers Van Gogh) 2ea1bbc Move init logistics message for BAD_GENESIS_BLOCK to init.cpp (Kittywhiskers Van Gogh) 29c7362 Move mempool nullptr Assert out of LoadChainstate (Kittywhiskers Van Gogh) 7071282 node/chainstate: Decouple from concept of NodeContext (Kittywhiskers Van Gogh) ee9d3dd node/chainstate: Decouple from ArgsManager (Kittywhiskers Van Gogh) d7419e4 node/chainstate: Decouple from stringy errors (Kittywhiskers Van Gogh) 9ab08c4 node/chainstate: Decouple from GetTimeMillis (Kittywhiskers Van Gogh) 2455c06 node: Extract chainstate loading sequence (Kittywhiskers Van Gogh) 620146b chore: sync chainstate loading logic with upstream (Kittywhiskers Van Gogh) Pull request description: ## Additional Information * Dependent on #6296 * Dependent on #6443 * As one of the backport's intentions were to unify code between `init.cpp` and `setup_common.cpp`, Dash-specific initialization code (to the extent that it can be moved out non-disruptively) has been spun out into `DashChainstateSetup{,Close}()` so it can also be used in `setup_common.cpp` and `validation_chainstatemanager_tests.cpp`. This is also why `DashTestSetup{,Close}()` (now `DashPostChainstateSetup{,Close}()`) was introduced in [dash#5531](#5531). * `DashChainstateSetup{,Close}()` (as defined in `node/chainstate.cpp`) cannot take `NodeContext` in because `node/chainstate.cpp` is used in `bitcoin-chainstate`, which doesn't include `NodeContext` ([source](https://github.com/bitcoin/bitcoin/pull/24304/files#diff-4cb884d03ebb901069e4ee5de5d02538c40dd9b39919c615d8eaa9d364bbbd77R795-R798)), this is reflected by neither `LoadChainstate` nor `VerifyLoadedChainstate` taking in `NodeContext`. * To make it less onerous to use in unit tests, `DashChainstateSetup{,Close}()` has been overloaded with a variant that accepts `NodeContext`. * To remove `LogPrintf` usage in `node/chainstate.cpp`, index enablement reporting has been pulled out of chainstate loading and BLS scheme reporting has been abstracted out using `notify_bls_state`. ## Breaking Changes None expected. ## Checklist - [x] I have performed a self-review of my own code - [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)** - [x] I have added or updated relevant unit/integration/functional/e2e tests - [x] I have made corresponding changes to the documentation **(note: N/A)** - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_ ACKs for top commit: UdjinM6: utACK 1974651 PastaPastaPasta: utACK 1974651 Tree-SHA512: 8ebfce48dccc6a867339aff9374a4cb8dc7b02b0c17432db2b97c982523d0d9589a87e6527a103f314bf32be176486311ef6dcde1c4d5cdccc1eeb1f80bbb040
2 parents 7530f3d + 1974651 commit a5d5400

13 files changed

+821
-343
lines changed

src/Makefile.am

+4
Original file line numberDiff line numberDiff line change
@@ -270,6 +270,8 @@ BITCOIN_CORE_H = \
270270
netgroup.h \
271271
netmessagemaker.h \
272272
node/blockstorage.h \
273+
node/caches.h \
274+
node/chainstate.h \
273275
node/coin.h \
274276
node/coinstats.h \
275277
node/connection_types.h \
@@ -502,6 +504,8 @@ libbitcoin_server_a_SOURCES = \
502504
netgroup.cpp \
503505
net_processing.cpp \
504506
node/blockstorage.cpp \
507+
node/caches.cpp \
508+
node/chainstate.cpp \
505509
node/coin.cpp \
506510
node/coinstats.cpp \
507511
node/connection_types.cpp \

src/init.cpp

+157-293
Large diffs are not rendered by default.

src/node/caches.cpp

+32
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
// Copyright (c) 2021 The Bitcoin Core developers
2+
// Distributed under the MIT software license, see the accompanying
3+
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
4+
5+
#include <node/caches.h>
6+
7+
#include <txdb.h>
8+
#include <util/system.h>
9+
#include <validation.h>
10+
11+
CacheSizes CalculateCacheSizes(const ArgsManager& args, size_t n_indexes)
12+
{
13+
int64_t nTotalCache = (args.GetArg("-dbcache", nDefaultDbCache) << 20);
14+
nTotalCache = std::max(nTotalCache, nMinDbCache << 20); // total cache cannot be less than nMinDbCache
15+
nTotalCache = std::min(nTotalCache, nMaxDbCache << 20); // total cache cannot be greater than nMaxDbcache
16+
CacheSizes sizes;
17+
sizes.block_tree_db = std::min(nTotalCache / 8, nMaxBlockDBCache << 20);
18+
nTotalCache -= sizes.block_tree_db;
19+
sizes.tx_index = std::min(nTotalCache / 8, args.GetBoolArg("-txindex", DEFAULT_TXINDEX) ? nMaxTxIndexCache << 20 : 0);
20+
nTotalCache -= sizes.tx_index;
21+
sizes.filter_index = 0;
22+
if (n_indexes > 0) {
23+
int64_t max_cache = std::min(nTotalCache / 8, max_filter_index_cache << 20);
24+
sizes.filter_index = max_cache / n_indexes;
25+
nTotalCache -= sizes.filter_index * n_indexes;
26+
}
27+
sizes.coins_db = std::min(nTotalCache / 2, (nTotalCache / 4) + (1 << 23)); // use 25%-50% of the remainder for disk cache
28+
sizes.coins_db = std::min(sizes.coins_db, nMaxCoinsDBCache << 20); // cap total coins db cache
29+
nTotalCache -= sizes.coins_db;
30+
sizes.coins = nTotalCache; // the rest goes to in-memory cache
31+
return sizes;
32+
}

src/node/caches.h

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
// Copyright (c) 2021 The Bitcoin Core developers
2+
// Distributed under the MIT software license, see the accompanying
3+
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
4+
5+
#ifndef BITCOIN_NODE_CACHES_H
6+
#define BITCOIN_NODE_CACHES_H
7+
8+
#include <cstddef>
9+
#include <cstdint>
10+
11+
class ArgsManager;
12+
13+
struct CacheSizes {
14+
int64_t block_tree_db;
15+
int64_t coins_db;
16+
int64_t coins;
17+
int64_t tx_index;
18+
int64_t filter_index;
19+
};
20+
CacheSizes CalculateCacheSizes(const ArgsManager& args, size_t n_indexes = 0);
21+
22+
#endif // BITCOIN_NODE_CACHES_H

src/node/chainstate.cpp

+328
Large diffs are not rendered by default.

src/node/chainstate.h

+162
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
// Copyright (c) 2021 The Bitcoin Core developers
2+
// Distributed under the MIT software license, see the accompanying
3+
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
4+
5+
#ifndef BITCOIN_NODE_CHAINSTATE_H
6+
#define BITCOIN_NODE_CHAINSTATE_H
7+
8+
#include <cstdint>
9+
#include <functional>
10+
#include <memory>
11+
#include <optional>
12+
#include <string>
13+
14+
class CActiveMasternodeManager;
15+
class CChainstateHelper;
16+
class CCreditPoolManager;
17+
class CDeterministicMNManager;
18+
class CEvoDB;
19+
class CGovernanceManager;
20+
class ChainstateManager;
21+
class CMasternodeMetaMan;
22+
class CMasternodeSync;
23+
class CMNHFManager;
24+
class CSporkManager;
25+
class CTxMemPool;
26+
struct LLMQContext;
27+
28+
namespace llmq {
29+
class CChainLocksHandler;
30+
class CInstantSendManager;
31+
class CQuorumSnapshotManager;
32+
}
33+
34+
namespace Consensus {
35+
struct Params;
36+
}
37+
38+
enum class ChainstateLoadingError {
39+
ERROR_LOADING_BLOCK_DB,
40+
ERROR_BAD_GENESIS_BLOCK,
41+
ERROR_BAD_DEVNET_GENESIS_BLOCK,
42+
ERROR_TXINDEX_DISABLED_WHEN_GOV_ENABLED,
43+
ERROR_ADDRIDX_NEEDS_REINDEX,
44+
ERROR_SPENTIDX_NEEDS_REINDEX,
45+
ERROR_TIMEIDX_NEEDS_REINDEX,
46+
ERROR_PRUNED_NEEDS_REINDEX,
47+
ERROR_LOAD_GENESIS_BLOCK_FAILED,
48+
ERROR_CHAINSTATE_UPGRADE_FAILED,
49+
ERROR_REPLAYBLOCKS_FAILED,
50+
ERROR_LOADCHAINTIP_FAILED,
51+
ERROR_GENERIC_BLOCKDB_OPEN_FAILED,
52+
ERROR_COMMITING_EVO_DB,
53+
ERROR_UPGRADING_EVO_DB,
54+
ERROR_UPGRADING_SIGNALS_DB,
55+
SHUTDOWN_PROBED,
56+
};
57+
58+
/** This sequence can have 4 types of outcomes:
59+
*
60+
* 1. Success
61+
* 2. Shutdown requested
62+
* - nothing failed but a shutdown was triggered in the middle of the
63+
* sequence
64+
* 3. Soft failure
65+
* - a failure that might be recovered from with a reindex
66+
* 4. Hard failure
67+
* - a failure that definitively cannot be recovered from with a reindex
68+
*
69+
* Currently, LoadChainstate returns a std::optional<ChainstateLoadingError>
70+
* which:
71+
*
72+
* - if has_value()
73+
* - Either "Soft failure", "Hard failure", or "Shutdown requested",
74+
* differentiable by the specific enumerator.
75+
*
76+
* Note that a return value of SHUTDOWN_PROBED means ONLY that "during
77+
* this sequence, when we explicitly checked shutdown_requested() at
78+
* arbitrary points, one of those calls returned true". Therefore, a
79+
* return value other than SHUTDOWN_PROBED does not guarantee that
80+
* shutdown_requested() hasn't been called indirectly.
81+
* - else
82+
* - Success!
83+
*/
84+
std::optional<ChainstateLoadingError> LoadChainstate(bool fReset,
85+
ChainstateManager& chainman,
86+
CGovernanceManager& govman,
87+
CMasternodeMetaMan& mn_metaman,
88+
CMasternodeSync& mn_sync,
89+
CSporkManager& sporkman,
90+
std::unique_ptr<CActiveMasternodeManager>& mn_activeman,
91+
std::unique_ptr<CChainstateHelper>& chain_helper,
92+
std::unique_ptr<CCreditPoolManager>& cpoolman,
93+
std::unique_ptr<CDeterministicMNManager>& dmnman,
94+
std::unique_ptr<CEvoDB>& evodb,
95+
std::unique_ptr<CMNHFManager>& mnhf_manager,
96+
std::unique_ptr<llmq::CChainLocksHandler>& clhandler,
97+
std::unique_ptr<llmq::CInstantSendManager>& isman,
98+
std::unique_ptr<llmq::CQuorumSnapshotManager>& qsnapman,
99+
std::unique_ptr<LLMQContext>& llmq_ctx,
100+
CTxMemPool* mempool,
101+
bool fPruneMode,
102+
bool is_addrindex_enabled,
103+
bool is_governance_enabled,
104+
bool is_spentindex_enabled,
105+
bool is_timeindex_enabled,
106+
bool is_txindex_enabled,
107+
const Consensus::Params& consensus_params,
108+
const std::string& network_id,
109+
bool fReindexChainState,
110+
int64_t nBlockTreeDBCache,
111+
int64_t nCoinDBCache,
112+
int64_t nCoinCacheUsage,
113+
bool block_tree_db_in_memory,
114+
bool coins_db_in_memory,
115+
std::function<bool()> shutdown_requested = nullptr,
116+
std::function<void()> coins_error_cb = nullptr);
117+
118+
/** Initialize Dash-specific components during chainstate initialization */
119+
void DashChainstateSetup(ChainstateManager& chainman,
120+
CGovernanceManager& govman,
121+
CMasternodeMetaMan& mn_metaman,
122+
CMasternodeSync& mn_sync,
123+
CSporkManager& sporkman,
124+
std::unique_ptr<CActiveMasternodeManager>& mn_activeman,
125+
std::unique_ptr<CChainstateHelper>& chain_helper,
126+
std::unique_ptr<CCreditPoolManager>& cpoolman,
127+
std::unique_ptr<CDeterministicMNManager>& dmnman,
128+
std::unique_ptr<CEvoDB>& evodb,
129+
std::unique_ptr<CMNHFManager>& mnhf_manager,
130+
std::unique_ptr<llmq::CQuorumSnapshotManager>& qsnapman,
131+
std::unique_ptr<LLMQContext>& llmq_ctx,
132+
CTxMemPool* mempool,
133+
bool fReset,
134+
bool fReindexChainState,
135+
const Consensus::Params& consensus_params);
136+
137+
void DashChainstateSetupClose(std::unique_ptr<CChainstateHelper>& chain_helper,
138+
std::unique_ptr<CCreditPoolManager>& cpoolman,
139+
std::unique_ptr<CDeterministicMNManager>& dmnman,
140+
std::unique_ptr<CMNHFManager>& mnhf_manager,
141+
std::unique_ptr<llmq::CQuorumSnapshotManager>& qsnapman,
142+
std::unique_ptr<LLMQContext>& llmq_ctx,
143+
CTxMemPool* mempool);
144+
145+
enum class ChainstateLoadVerifyError {
146+
ERROR_BLOCK_FROM_FUTURE,
147+
ERROR_CORRUPTED_BLOCK_DB,
148+
ERROR_EVO_DB_SANITY_FAILED,
149+
ERROR_GENERIC_FAILURE,
150+
};
151+
152+
std::optional<ChainstateLoadVerifyError> VerifyLoadedChainstate(ChainstateManager& chainman,
153+
CEvoDB& evodb,
154+
bool fReset,
155+
bool fReindexChainState,
156+
const Consensus::Params& consensus_params,
157+
unsigned int check_blocks,
158+
unsigned int check_level,
159+
std::function<int64_t()> get_unix_time_seconds,
160+
std::function<void(bool)> notify_bls_state = nullptr);
161+
162+
#endif // BITCOIN_NODE_CHAINSTATE_H

src/rpc/blockchain.cpp

+1-1
Original file line numberDiff line numberDiff line change
@@ -1642,7 +1642,7 @@ static RPCHelpMan verifychain()
16421642

16431643
CChainState& active_chainstate = chainman.ActiveChainstate();
16441644
return CVerifyDB().VerifyDB(
1645-
active_chainstate, Params(), active_chainstate.CoinsTip(), *CHECK_NONFATAL(node.evodb), check_level, check_depth);
1645+
active_chainstate, Params().GetConsensus(), active_chainstate.CoinsTip(), *CHECK_NONFATAL(node.evodb), check_level, check_depth);
16461646
},
16471647
};
16481648
}

src/test/evo_deterministicmns_tests.cpp

+2-2
Original file line numberDiff line numberDiff line change
@@ -816,8 +816,8 @@ void FuncVerifyDB(TestChainSetup& setup)
816816

817817
// Verify db consistency
818818
LOCK(cs_main);
819-
BOOST_REQUIRE(CVerifyDB().VerifyDB(chainman.ActiveChainstate(), Params(), chainman.ActiveChainstate().CoinsTip(),
820-
*(setup.m_node.evodb), 4, 2));
819+
BOOST_REQUIRE(CVerifyDB().VerifyDB(chainman.ActiveChainstate(), Params().GetConsensus(),
820+
chainman.ActiveChainstate().CoinsTip(), *(setup.m_node.evodb), 4, 2));
821821
}
822822

823823
BOOST_AUTO_TEST_SUITE(evo_dip3_activation_tests)

0 commit comments

Comments
 (0)