participant fails with performing synchronizer connection for global failed with an unhandled tried to dequeue while pending wasn't empty

[nicu-da]

2026-01-20T11:43:16.247+0100 [⋮] DEBUG - c.d.c.h.MutableHealthComponent:participant=sv3Participant (---) - Refreshing state of sequencer-client from Not Initialized to Ok() 
2026-01-20T11:43:16.248+0100 [⋮] ERROR - c.d.c.p.s.SynchronizerConnectionsManager:participant=sv3Participant (f148fc4e7eac00f72201b7bd59b785f4---6eadb5e7252db39f) - performing synchronizer connection for global failed with an unhandled error java.lang.IllegalArgumentException: requirement failed: tried to dequeue while pending wasn't empty: List((Traced(SignedTopologyTransaction(
  TopologyTransaction(
    PartyToParticipant(
      partyId = DSO-b01a7a4a-b01a7a4a::1220fd37cb59...,
      threshold = 2,
      participants = Map(PAR::sv1::1220b0924723... -> Submission, PAR::sv3::12206a7480b3... -> Submission, PAR::sv4::122009eb4868... -> Submission)
    ),
    serial = 5,
    operation = Replace,
    hash = SHA-256:872df4336afe...
  ),
  signatures = 12206a7480b3...,
  proposal
))(TraceContext(trace id=8dd569220bdb800c723b2760988c7439, W3C context=W3CTraceContext(parent=00-8dd569220bdb800c723b2760988c7439-a8a3f0e4245bfa41-01))),Future(<not completed>)))
	at scala.Predef$.require(Predef.scala:337)
	at com.digitalasset.canton.topology.SynchronizerOutboxQueue.$anonfun$dequeue$1(SynchronizerOutboxQueue.scala:72)

logs

failing CI https://github.com/hyperledger-labs/splice/actions/runs/21167878657/job/60877108439

[matthiasS-da]

Starting investigation...

[matthiasS-da]

Could you please clarify the commit hash?

[nicu-da]

3.5.0-snapshot.20260120.17809.0.vd0a14ac4

[daravep]

Did a quick peek between meetings. What happens is the following:

• While there is stuff being flushed to the synchronizer, the sync domain connect disconnects.
• The retry logic restarts it.
• We create a new “outbox” but we reuse the synchronizerOutboxQueue = state.synchronizerOutboxQueue,
• We then call startup again. But this startup starts pushing txs into an outbox queue which is already running.

So the issue is that the lifecycle of these two objects doesn’t align.

Solution options are:

• synchronise the flush of the synchronizer outbox queue with the startup
• create new objects (but increasing the chance of duplicate submissions - which are okay, just cost money)

[matthiasS-da]

Thanks for the input @daravep. I confirm your analysis. Logging is excellent in these classes!

I'll try to understand the trade-offs of the various solution options.

[matthiasS-da]

Talked to @gerolf-da. He'll provide a fix.

[gerolf-da]

I have a fix for the issue. I also included a replication of the exact issue, to be sure that the fix solves it: https://github.com/DACH-NY/canton/pull/30314. I requested a review from @matthiasS-da, since he knows how it gets to the situation.