Implement persistent reputation database for collator protocol (#7751) #10917

AlexandruCihodaru · 2026-01-27T20:47:27Z

Implements persistent storage for the experimental collator protocol's reputation database.

Changes:

Adds PersistentDb wrapper that persists the in-memory reputation DB to disk
Periodic persistence every 10 minutes (30s in test mode)
Immediate persistence on slashes and parachain deregistration
Loads existing state on startup with lookback for missed blocks

Implementation:

PersistentDb wraps the existing Db and adds persistence on top:

- All reputation logic (scoring, decay, LRU) stays in `Db`
- Persistence layer handles disk I/O and serialization
- Per-para data stored in parachains_db

Tests:

basic_persistence.rs: Validates persistence across restarts and startup lookback
pruning.rs: Validates automatic cleanup on parachain deregistration

Signed-off-by: Alexandru Cihodaru <[email protected]>

Implements disk persistence for the experimental collator protocol's reputation system to preserve peer scores across validator restarts. Key changes: - Add PersistentDb wrapper that persists in-memory Db to disk - Implement periodic persistence (10 min prod, 30 sec test mode) - Add immediate persistence for slashes and para pruning - Create serialization layer for ScoreEntry and reputation data - Add ReputationConfig for database column configuration Signed-off-by: Alexandru Cihodaru <[email protected]>

Signed-off-by: Alexandru Cihodaru <[email protected]>

Tests added: - basic_persistence.rs: Validates that reputation data persists across validator restarts and that the startup lookback mechanism correctly processes missed blocks during downtime. Tests both large gaps (20+ blocks) and small gaps (<20 blocks). - pruning.rs: Verifies that reputation data is correctly pruned when parachains are deregistered. Tests para cleanup via sudo, session boundary detection, immediate persistence of pruned state, and correct loading of only remaining paras after restart. Signed-off-by: Alexandru Cihodaru <[email protected]>

paritytech-workflow-stopper · 2026-01-27T20:53:48Z

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/21413586873
Failed job name: fmt

alindima

LGTM, good job!

alindima · 2026-01-30T08:30:20Z

...t/node/network/collator-protocol/src/validator_side_experimental/peer_manager/persistence.rs

+// along with Polkadot.  If not, see <http://www.gnu.org/licenses/>.
+
+//! Serialization types for disk persistence of collator reputation data.
+


all the db-related file would be better off in a separate module within the peer manager (including the in-memory db). And the in-memory db should be renamed from just plain db.
But let's do the renames in a follow-up PR, to not make review more difficult right now

alindima · 2026-01-30T08:35:03Z

...node/network/collator-protocol/src/validator_side_experimental/peer_manager/persistent_db.rs

+
+		let last_finalized = instance.inner.processed_finalized_block_number().await;
+
+		// Use info level for clear observability in tests and production


outdated comment

alindima · 2026-01-30T08:36:07Z

...node/network/collator-protocol/src/validator_side_experimental/peer_manager/persistent_db.rs

+		// Load metadata
+		if let Some(meta) = self.load_metadata()? {
+			// We need to directly access the inner fields to restore state
+			// This requires making Db fields pub(super) or adding setter methods


remove this comment

alindima · 2026-01-30T08:38:50Z

...node/network/collator-protocol/src/validator_side_experimental/peer_manager/persistent_db.rs

+
+	/// Persist all in-memory data to disk.
+	///
+	/// This should be called periodically by the main loop (currently, every 10 minutes).


optimisation: it should only be done for the paras that have been "dirtied" (had any change).

alindima · 2026-01-30T08:39:13Z

...node/network/collator-protocol/src/validator_side_experimental/peer_manager/persistent_db.rs

+	}
+
+	/// Persist a single para's data to disk (called immediately after slash).
+	fn persist_para(&self, para_id: &ParaId) -> Result<(), PersistenceError> {


nit: for slash we only need to persist one peer's reputation. The way it was previously stored (in the old PR), this was possible

alindima · 2026-01-30T09:52:59Z

polkadot/parachain/test-parachains/undying/src/lib.rs

 	);

 	if block_data.experimental_send_approved_peer {
+		// Create a valid PeerId in multihash format: [hash_code, digest_size, ...digest_bytes]


just double checking: this was added for test purposes, but the code can handle these invalid peerids gracefully right?

alindima · 2026-01-30T09:54:04Z

polkadot/zombienet-sdk-tests/tests/functional/reputation_persistence/basic_persistence.rs

+// Copyright (C) Parity Technologies (UK) Ltd.
+// SPDX-License-Identifier: Apache-2.0
+
+//! Test: Basic Persistence on Graceful Shutdown with Startup Lookback Verification


the name of the test is not expressive enough. What reputation is this about?

Also, this tests persistence on restart more than anything

alindima · 2026-01-30T10:09:46Z

polkadot/zombienet-sdk-tests/tests/functional/reputation_persistence/pruning.rs

@@ -0,0 +1,266 @@
+// Copyright (C) Parity Technologies (UK) Ltd.
+// SPDX-License-Identifier: Apache-2.0
+


if you want to have the tests running in CI you need to add them to the github actions config

alindima · 2026-01-30T10:26:33Z

polkadot/zombienet-sdk-tests/tests/functional/reputation_persistence/pruning.rs

+//! 5. **Deregister parachain 2001 using sudo**
+//! 6. Wait for session boundary (triggers pruning check)
+//! 7. Verify pruning logs show para 2001 was pruned
+//! 8. Wait for periodic persistence (pruned state written to disk)


the pruned state is written immediately, is it not?

alindima · 2026-01-30T10:27:36Z

polkadot/zombienet-sdk-tests/tests/functional/reputation_persistence/pruning.rs

+	);
+
+	log::info!("Verifying para 2000 continues normal operation (para 2001 is deregistered)");
+	assert_para_throughput(&validator0_client_after, 5, [(ParaId::from(PARA_ID_1), 4..6)])


can you also assert that para 2001 has throughput 0

AlexandruCihodaru added 4 commits January 27, 2026 22:29

Add a new column to parachains_db for collators reputation

d52e2df

Signed-off-by: Alexandru Cihodaru <[email protected]>

Update undying collator to send valid PeerId in ApprovedPeer UMP signal

2991942

Signed-off-by: Alexandru Cihodaru <[email protected]>

AlexandruCihodaru requested review from alindima, sandreim and tdimitrov January 27, 2026 20:47

alindima reviewed Jan 30, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement persistent reputation database for collator protocol (#7751) #10917

Implement persistent reputation database for collator protocol (#7751) #10917

Uh oh!

AlexandruCihodaru commented Jan 27, 2026

Uh oh!

paritytech-workflow-stopper bot commented Jan 27, 2026

Uh oh!

alindima left a comment

Uh oh!

alindima Jan 30, 2026

Uh oh!

alindima Jan 30, 2026

Uh oh!

alindima Jan 30, 2026

Uh oh!

alindima Jan 30, 2026

Uh oh!

alindima Jan 30, 2026

Uh oh!

alindima Jan 30, 2026

Uh oh!

alindima Jan 30, 2026

Uh oh!

alindima Jan 30, 2026

Uh oh!

alindima Jan 30, 2026

Uh oh!

alindima Jan 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		// along with Polkadot. If not, see <http://www.gnu.org/licenses/>.

		//! Serialization types for disk persistence of collator reputation data.


		let last_finalized = instance.inner.processed_finalized_block_number().await;

		// Use info level for clear observability in tests and production

		@@ -0,0 +1,266 @@
		// Copyright (C) Parity Technologies (UK) Ltd.
		// SPDX-License-Identifier: Apache-2.0

Implement persistent reputation database for collator protocol (#7751) #10917

Are you sure you want to change the base?

Implement persistent reputation database for collator protocol (#7751) #10917

Uh oh!

Conversation

AlexandruCihodaru commented Jan 27, 2026

Uh oh!

paritytech-workflow-stopper bot commented Jan 27, 2026

Uh oh!

alindima left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants