Add support for smoldot #1214

sigurpol · 2026-01-08T10:18:05Z

Connect using the embedded smoldot light client with --smoldot as altenative to WebSocket RPC connection via --wss.
The two options are mutually exclusive.
This PR can be merged without risk of regressions for the deployed miner and can help us testing smoldot variant in these obscure times where we are facing issues with multiple RPC nodes (see paritytech/polkadot-sdk#10719).

A small nightly test has been added where we just check basic connectivity towards Polkadot, Kusama, Westend and Paseo AssetHub via smoldot.
See an example here

Related issue: #1141

In a follow-up PR, we can add a script to actually update chain specs and an associated GH workflow to periodically update them (e.g. to run once per week or so)

This ensures faster sync at startup

chainspecs/kusama.json

src/opt.rs

README.md

jsdw

LGTM, nice one!

sigurpol · 2026-01-09T08:28:19Z

I noticed issues while finally entering in snapshot / signed phase in the election on WAH / PAH. It seems that with smoldot we have issues with block pinning with the ChainHead API. The miner fetches snapshot pages over multiple blocks and what happens is that we see a unknown or unpinned block error - which I interpret as the miner trying to query storage from a block which is no longer pinned. We don't have this issue when connecting via RPC, so I would assume that smoldot unpins block much quickly. If I get things rigtht, ChainHead backend by default never unpins (max_block_life= usize::MAX) , is there a way to configure smoldot's pin limit somehow? Or how can we tackle the case of the miner that needs to fetch ~30 pages of voter snapshot over multiple blocks?

2026-01-08T19:36:56.725125Z DEBUG polkadot-staking-miner: Phase           
  transition: Off → Snapshot(31) - stopping era pruning                     
  2026-01-08T19:36:56.725292Z DEBUG polkadot-staking-miner: Exiting Off     
  phase, stopping pruning session                                           
  2026-01-08T19:36:56.917957Z TRACE polkadot-staking-miner: Processing      
  block=11146499 round=65, phase=Snapshot(31)                               
  2026-01-08T19:36:56.918007Z TRACE polkadot-staking-miner: Sent block      
  #11146499 to miner                                                        
  2026-01-08T19:36:56.918042Z TRACE polkadot-staking-miner: Processing      
  block #11146499 (round 65, phase Snapshot(31))                            
  2026-01-08T19:37:09.700129Z TRACE polkadot-staking-miner: Target snapshot 
  with len 1256, hash:                                                      
  Ok(0xc69b5de2c4f547313942e02866a326d6349eded79c1e234ab8699732236ffe67)    
  2026-01-08T19:37:10.040487Z TRACE polkadot-staking-miner: Processing      
  block=11146500 round=65, phase=Snapshot(30)                               
  2026-01-08T19:37:10.040511Z TRACE polkadot-staking-miner: Sent block      
  #11146500 to miner                                                        
  2026-01-08T19:37:10.040535Z TRACE polkadot-staking-miner: Processing      
  block #11146500 (round 65, phase Snapshot(30))                            
  2026-01-08T19:37:13.576398Z TRACE polkadot-staking-miner: Processing      
  block=11146501 round=65, phase=Snapshot(29)                               
  2026-01-08T19:37:13.576445Z TRACE polkadot-staking-miner: Sent block      
  #11146501 to miner                                                        
  2026-01-08T19:37:17.017627Z  WARN polkadot-staking-miner: Block           
  processing failed, continuing: Subxt(Rpc(ClientError(User(UserError {     
  code: -32801, message: "unknown or unpinned block", data: None }))))      
  2026-01-08T19:37:17.017670Z TRACE polkadot-staking-miner: Processing      
  block #11146501 (round 65, phase Snapshot(29))                            
  2026-01-08T19:37:17.823748Z TRACE polkadot-staking-miner: Processing      
  block=11146502 round=65, phase=Snapshot(28)                               
  2026-01-08T19:37:17.823783Z TRACE polkadot-staking-miner: Sent block      
  #11146502 to miner                                                        
  2026-01-08T19:37:21.301938Z  WARN polkadot-staking-miner: Block           
  processing failed, continuing: Subxt(Rpc(ClientError(User(UserError {     
  code: -32801, message: "unknown or unpinned block", data: None }))))      
  2026-01-08T19:37:21.301969Z TRACE polkadot-staking-miner: Processing      
  block #11146502 (round 65, phase Snapshot(28))                            
  2026-01-08T19:37:23.016445Z TRACE polkadot-staking-miner: Processing      
  block=11146503 round=65, phase=Snapshot(27)                               
  2026-01-08T19:37:23.016539Z TRACE polkadot-staking-miner: Sent block      
  #11146503 to miner                                                        
  2026-01-08T19:37:30.673796Z TRACE polkadot-staking-miner: Voter snapshot  
  page=29 len=704, hash=Err(Subxt(Rpc(ClientError(User(UserError { code:    
  -32801, message: "unknown or unpinned block", data: None })))))           
  2026-01-08T19:37:30.673910Z  WARN polkadot-staking-miner: Block           
  processing failed, continuing: Subxt(Rpc(ClientError(User(UserError {     
  code: -32801, message: "unknown or unpinned block", data: None }))))      
  2026-01-08T19:37:30.673924Z TRACE polkadot-staking-miner: Processing      
  block #11146503 (round 65, phase Snapshot(27))                            
  2026-01-08T19:37:30.721856Z TRACE polkadot-staking-miner: Processing      
  block=11146504 round=65, phase=Snapshot(26)                               
  2026-01-08T19:37:30.721889Z TRACE polkadot-staking-miner: Sent block      
  #11146504 to miner                                                        
  2026-01-08T19:37:33.691744Z  WARN polkadot-staking-miner: Block           
  processing failed, continuing: Subxt(Rpc(ClientError(User(UserError {     
  code: -32801, message: "unknown or unpinned block", data: None }))))      
  2026-01-08T19:37:33.691785Z TRACE polkadot-staking-miner: Processing      
  block #11146504 (round 65, phase Snapshot(26))

sigurpol · 2026-01-09T08:52:47Z

I noticed issues while finally entering in snapshot / signed phase in the election on WAH / PAH. It seems that with smoldot we have issues with block pinning with the ChainHead API. The miner fetches snapshot pages over multiple blocks and what happens is that we see a unknown or unpinned block error - which I interpret as the miner trying to query storage from a block which is no longer pinned. We don't have this issue when connecting via RPC, so I would assume that smoldot unpins block much quickly. If I get things rigtht, ChainHead backend by default never unpins (max_block_life= usize::MAX) , is there a way to configure smoldot's pin limit somehow? Or how can we tackle the case of the miner that needs to fetch ~30 pages of voter snapshot over multiple blocks?

2026-01-08T19:36:56.725125Z DEBUG polkadot-staking-miner: Phase           
  transition: Off → Snapshot(31) - stopping era pruning                     
  2026-01-08T19:36:56.725292Z DEBUG polkadot-staking-miner: Exiting Off     
  phase, stopping pruning session                                           
  2026-01-08T19:36:56.917957Z TRACE polkadot-staking-miner: Processing      
  block=11146499 round=65, phase=Snapshot(31)                               
  2026-01-08T19:36:56.918007Z TRACE polkadot-staking-miner: Sent block      
  #11146499 to miner                                                        
  2026-01-08T19:36:56.918042Z TRACE polkadot-staking-miner: Processing      
  block #11146499 (round 65, phase Snapshot(31))                            
  2026-01-08T19:37:09.700129Z TRACE polkadot-staking-miner: Target snapshot 
  with len 1256, hash:                                                      
  Ok(0xc69b5de2c4f547313942e02866a326d6349eded79c1e234ab8699732236ffe67)    
  2026-01-08T19:37:10.040487Z TRACE polkadot-staking-miner: Processing      
  block=11146500 round=65, phase=Snapshot(30)                               
  2026-01-08T19:37:10.040511Z TRACE polkadot-staking-miner: Sent block      
  #11146500 to miner                                                        
  2026-01-08T19:37:10.040535Z TRACE polkadot-staking-miner: Processing      
  block #11146500 (round 65, phase Snapshot(30))                            
  2026-01-08T19:37:13.576398Z TRACE polkadot-staking-miner: Processing      
  block=11146501 round=65, phase=Snapshot(29)                               
  2026-01-08T19:37:13.576445Z TRACE polkadot-staking-miner: Sent block      
  #11146501 to miner                                                        
  2026-01-08T19:37:17.017627Z  WARN polkadot-staking-miner: Block           
  processing failed, continuing: Subxt(Rpc(ClientError(User(UserError {     
  code: -32801, message: "unknown or unpinned block", data: None }))))      
  2026-01-08T19:37:17.017670Z TRACE polkadot-staking-miner: Processing      
  block #11146501 (round 65, phase Snapshot(29))                            
  2026-01-08T19:37:17.823748Z TRACE polkadot-staking-miner: Processing      
  block=11146502 round=65, phase=Snapshot(28)                               
  2026-01-08T19:37:17.823783Z TRACE polkadot-staking-miner: Sent block      
  #11146502 to miner                                                        
  2026-01-08T19:37:21.301938Z  WARN polkadot-staking-miner: Block           
  processing failed, continuing: Subxt(Rpc(ClientError(User(UserError {     
  code: -32801, message: "unknown or unpinned block", data: None }))))      
  2026-01-08T19:37:21.301969Z TRACE polkadot-staking-miner: Processing      
  block #11146502 (round 65, phase Snapshot(28))                            
  2026-01-08T19:37:23.016445Z TRACE polkadot-staking-miner: Processing      
  block=11146503 round=65, phase=Snapshot(27)                               
  2026-01-08T19:37:23.016539Z TRACE polkadot-staking-miner: Sent block      
  #11146503 to miner                                                        
  2026-01-08T19:37:30.673796Z TRACE polkadot-staking-miner: Voter snapshot  
  page=29 len=704, hash=Err(Subxt(Rpc(ClientError(User(UserError { code:    
  -32801, message: "unknown or unpinned block", data: None })))))           
  2026-01-08T19:37:30.673910Z  WARN polkadot-staking-miner: Block           
  processing failed, continuing: Subxt(Rpc(ClientError(User(UserError {     
  code: -32801, message: "unknown or unpinned block", data: None }))))      
  2026-01-08T19:37:30.673924Z TRACE polkadot-staking-miner: Processing      
  block #11146503 (round 65, phase Snapshot(27))                            
  2026-01-08T19:37:30.721856Z TRACE polkadot-staking-miner: Processing      
  block=11146504 round=65, phase=Snapshot(26)                               
  2026-01-08T19:37:30.721889Z TRACE polkadot-staking-miner: Sent block      
  #11146504 to miner                                                        
  2026-01-08T19:37:33.691744Z  WARN polkadot-staking-miner: Block           
  processing failed, continuing: Subxt(Rpc(ClientError(User(UserError {     
  code: -32801, message: "unknown or unpinned block", data: None }))))      
  2026-01-08T19:37:33.691785Z TRACE polkadot-staking-miner: Processing      
  block #11146504 (round 65, phase Snapshot(26))

I think this code is potentially problematic

pub async fn storage_at_head(api: &Client) -> Result<Storage, Error> {
	let hash = get_latest_finalized_head(api.chain_api()).await?;
	storage_at(Some(hash), api.chain_api()).await
}

we just extract the hash and we drop the underlying BlockRef pointer handling the pinning mechanism

  pub struct BlockRef<H> {
      hash: H,
      _pointer: Option<Arc<dyn BlockRefT>>,  // if none we can unpin!!!
  }

so we create a storage item with just hash with block ref with pointer None.

What we could do instead:

  pub async fn storage_at_head(api: &Client) -> Result<Storage, Error> {
      // Single call that preserves the BlockRef with its pinning _pointer
      api.chain_api().storage().at_latest().await
  }

What happens: at_latest() internally calls latest_finalized_block_ref() and passes the entire BlockRef (including _pointer) to the storage client. The block stays pinned for the lifetime of the returned Storage object and smoldot shouldn't force any unpin (if I get it right). I'll try to test this variant. Why I think it might work:

subxt's max_block_life defaults to usize::MAX (no age-based force-unpinning)
smoldot's chainHead seems to have no hard limit on pinned blocks
at_latest() preserves the BlockRef pinning pointer

jsdw · 2026-01-09T10:32:27Z

I'll try to test this variant.

Aah yup, this is a good thing to be aware of; always hold onto block refs if you want to do something later with the block, to help keep it pinned as long as possible!

subxt's max_block_life defaults to usize::MAX (no age-based force-unpinning)

This means that Subxt will not try to unpin any blocks so long as they are held onto (ie the BlockRefs are kept around).

However, Smoldot will refuse to keep blocks in memory indefinitely. Having a quick look I think Smoldot will allow 32 blocks to be pinned before dropping the subscription (see here and here).

So basically, if we pin too many blocks on the Subxt side, Smoldot will end the chainHead subscription, leading to all blocks being "unpinned" on the Subxt side and the subscription being restarted. Given that you mentioned ~30 blocks worth of history being needed, I think you'll be cutting it very close and things will be prone to failing on occasion if blocks aren't quite unpinned fast enough.

Or how can we tackle the case of the miner that needs to fetch ~30 pages of voter snapshot over multiple blocks?

I can think of a few options now:

Would it be possible to download and cache all of the information you need from blocks rather than relying on Smoldot (or any other node) keeping the blocks in memory (pinned) for long enough?
You could revert to using the LegacyBackend instead of the ChainHeadBackend, which isn't bound by this pinning stuff and may give you access to older blocks, but not certain how much older. Less ideal because it would be nicer to avoid the legacy APIs if at all possible!
You could open an issue in the Smoldot repository to discuss making the number of pinned blocks configurable. If this was possible then Subxt could expose such an option too and we could just have a greater number of blocks pinned here to avoid hitting any issues.

sigurpol · 2026-01-09T11:13:23Z

So basically, if we pin too many blocks on the Subxt side, Smoldot will end the chainHead subscription, leading to all blocks being "unpinned" on the Subxt side and the subscription being restarted. Given that you mentioned ~30 blocks worth of history being needed, I think you'll be cutting it very close and things will be prone to failing on occasion if blocks aren't quite unpinned fast enough.

We indeed hit this since we store way too many BlockRef while reading the snapshot and submitting a solution. I think that now that I know we are severely constrained on the number of pinnable blocks, I can try to do something for that so basically this:

Would it be possible to download and cache all of the information you need from blocks rather than relying on Smoldot (or any other node) keeping the blocks in memory (pinned) for long enough?

Keep you posted!

kianenigma · 2026-01-09T15:58:24Z

Is this BlockRef what is returned from storage_at_head, and the issue is that we keep it in scope for too long? I suppose we can remove the usage of this altogether. It would make each storage query a bit slower as it has to setup more things, but conceptually it should all work fine.

sigurpol · 2026-01-09T17:18:54Z

Is this BlockRef what is returned from storage_at_head, and the issue is that we keep it in scope for too long? I suppose we can remove the usage of this altogether. It would make each storage query a bit slower as it has to setup more things, but conceptually it should all work fine.

We have two orthogonal problems: 1. we need to avoid that smoldot can unpin the block too early (and it can do so normally if we have a null pointer in the block ref - this is the issue the current version of the code has ) 2. we can't keep around too many blocks since smoldot has a limit of 32 -> the current code can be optimized to require MUCH less BlockRef than it's doing now. I am experimenting to reduce significantly the number of these BLockRef during snapshot and signed phase while keeping them around (i.e. with a valid pointer) long enough. You are suggesting another alternative which could also be explored.

sigurpol · 2026-01-12T11:08:18Z

Going back to your suggestion @kianenigma , I think we can make the miner much more robust vs BLockRef: During Signed phase, all snapshot data is guaranteed to be available and immutable so what we could do is

do nothing during snapshot phase
once we enter in signed phase, we fetch all snapshots in one shot, mine and submit

so we massively reduce the number to BlockRef needed (basically down to 1 or so)

sigurpol added 3 commits January 8, 2026 11:13

Add initial support for smoldot

5ea7112

Update chain specs

a39b5a8

update chains specs from PAPI

7d8f615

This ensures faster sync at startup

sigurpol changed the title ~~[DRAFT] Add initial support for smoldot~~ Add support for smoldot Jan 8, 2026

Add support for Paseo

2de1135

sigurpol requested review from Ank4n, andreitrand, jsdw, kianenigma and seadanda January 8, 2026 12:59

improve smoldot nightly test

d5cef79

Ank4n reviewed Jan 8, 2026

View reviewed changes

chainspecs/kusama.json Show resolved Hide resolved

Ank4n reviewed Jan 8, 2026

View reviewed changes

src/opt.rs Outdated Show resolved Hide resolved

Ank4n approved these changes Jan 8, 2026

View reviewed changes

sigurpol added 4 commits January 8, 2026 16:10

Rename SmoldotNetwork to include AssetHub suffix

7256032

clippy

6bf7704

update nightly job

2658694

Update README.md

f7e8f6b

jsdw reviewed Jan 8, 2026

View reviewed changes

README.md Show resolved Hide resolved

Update README.md

63d1592

jsdw approved these changes Jan 8, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for smoldot #1214

Add support for smoldot #1214

Uh oh!

sigurpol commented Jan 8, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsdw left a comment •

edited

Loading

Uh oh!

sigurpol commented Jan 9, 2026

Uh oh!

sigurpol commented Jan 9, 2026 •

edited

Loading

Uh oh!

jsdw commented Jan 9, 2026

Uh oh!

sigurpol commented Jan 9, 2026 •

edited

Loading

Uh oh!

kianenigma commented Jan 9, 2026

Uh oh!

sigurpol commented Jan 9, 2026

Uh oh!

sigurpol commented Jan 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Add support for smoldot #1214

Are you sure you want to change the base?

Add support for smoldot #1214

Uh oh!

Conversation

sigurpol commented Jan 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsdw left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sigurpol commented Jan 9, 2026

Uh oh!

sigurpol commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jsdw commented Jan 9, 2026

Uh oh!

sigurpol commented Jan 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kianenigma commented Jan 9, 2026

Uh oh!

sigurpol commented Jan 9, 2026

Uh oh!

sigurpol commented Jan 12, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sigurpol commented Jan 8, 2026 •

edited

Loading

jsdw left a comment •

edited

Loading

sigurpol commented Jan 9, 2026 •

edited

Loading

sigurpol commented Jan 9, 2026 •

edited

Loading