Large number of triple generation timeouts on testnet with a few slow participants

[gilcu3]

Background

On testnet (epoch with 14 participants, 9-party triple-gen subsets), one node logged 192 ManyTriples { count: 64 } timeouts in a ~6 minute window, plus remote aborts showing peers timing out on the same batches:

ERROR mpc_node::tracking: task failed; description: background triple generation;
task_id: ManyTriples { start: UniqueId(..), count: 64 }; error msg: Timeout

Per-party message counters in the failure traces show a consistent picture:

• 3 participants deliver ~1000 messages fewer than the fastest peer by the 60s deadline; 2 more are ~600 behind. The remaining peers are within ~300 of each other.
• The batch needs ~4800 messages per peer to complete; the fastest peers finish with only ~5% headroom, so a below-average participant in the subset blows the 60s budget.
• With 5 of 13 peers slow-ish, most random 9-party subsets contain at least one, so a large fraction of batches fail and triple generation throughput drops far below capacity.
• Per-peer counters are near-identical across failed tasks, indicating a persistent per-peer throughput/latency limit (RTT-paced rounds), not transient load.

The timeout/batch-size combination leaves no headroom for participants that are slightly slower than the rest.

Acceptance Criteria

• Either the triple generation timeout is increased or the triple batch size is reduced, so that batches complete with comfortable headroom on testnet.

[metalurgical]

Both the acceptance criteria can work here, however it can also be a bandaid for a problem that can re-occur.

It would a be a good opportunity to collect comprehensive metrics and timeline profiling/traces to understand exactly how the system is currently functioning against how it is thought it should function.

I'd suggest having a look at how much interleaving is occurring between task, what is truly parallel, what ends up as parallel but is really sequential, what is being interrupted or takes priority, context switching, how tasks are being distributed across cores, etc. Then taking that and comparing it against the network details (latency, retries, hops, etc). Results could be surprising.

[gilcu3]

We discussed it in the team, and it seems a no-brainer to increase the timeout parameter, which can be done independently by the nodes.

[metalurgical]

Naturally :).

Have handled problems like these before, thought I'd share some insights.

Observed for more than a while, going well.

[gilcu3]

We are modifying the parameter network wide (internal thread). Remaining on this issue is to change the default throughout the codebase and making the triple buckets large enough so that they cover the current runtimes