Improving cancellations in the parallel builder.

[bertmiller]

### Summary
We need a robust way to handle cancellations (or updates) in the parallel builder, where drain_new_orders() returns None. Because sub-components (ConflictFinder, ResultsAggregator, etc.) store partial merges and simulation states, a single order removal is insufficient. Instead, we must fully reset them all, wiping stale references. However, the builder also spawns threads (worker pool, aggregator, block-building), which complicates concurrency.

Goals

1. Full Reset: When cancellations occur, we must clear:

• ConflictFinder
• ConflictTaskGenerator's existing_groups / task_queue
• ConflictResolvingPool's task_queue
• SimulationCache
• ResultsAggregator’s best_results
  Then re-ingest only orders and let the system proceed as if new (e.g. find conflicts, resolve them, build blocks).

2. Thread Safety: Ensure no data races or partial updates if aggregator/block-building threads hold references.
3. Maintain High Performance: Avoid large overhead from locking or repeated resets.

Core Considerations

1. Reset Mechanism: All submodules (conflict finder, aggregator, simulation cache, etc) need a consistent way to clear and re-ingest orders—no partial or stale references.
2. Concurrency Approach:
   *. Single-Thread Manager: Central struct (ParallelBuilder) mutates data, with worker threads taking only short-lived snapshots. Simple to reset, but limits fully parallel reads.
   *. Multi-Thread Shared State: Modules live behind locks (Arc<Mutex<...>>) or receive message-based commands. More flexible for continuous concurrency, but demands locking or event loops.
3. Data Usage Pattern: Decide if aggregator/block-building hold references 24/7 (requiring concurrency control) or just use ephemeral tasks (easier single-thread reset).
4. Performance vs. Simplicity: Locking or message-passing ensures safety for truly parallel modules, but increases complexity and overhead. A single-thread manager is simpler but less concurrent.

Please weigh in on the core considerations.

[grasphoper]

Hi @bertmiller . I had some ideas about this (a lot of text, but with pics!)

IMO, the current approach is so close to being a Single-Thread Manager one that it makes sense to implement it.

Also, from what I understand, the hardest work is being done by conflict_resolving_pool and block_building_result_assembler, which will remain running in their own threads.

My intuition is that this approach might be even faster as well compared to a Multi-Thread Shared State one

I drew 2 diagrams of what the current approach looks like and the changes to enable cancellations.

Current approach:

If we moved results_aggregator from a tokio thread to the common ParallelBuilder loop (currently run_order_intake), we'd be able to manage the cancellations relatively easily, only sacrificing this relatively lightweight added workload to our "main" thread.

We could then change the best_results DashMap to a HashMap and send these to block_building_result_assembler via a channel. This could help our "main" thread avoid locking from DashMap operations and also enable event-based-block-building in block_building_result_assembler

The proposed design would change to this:

How to clean state:

• we notice None on drain_new_orders in run_order_intake loop
• we clean all the entities in the loop (all except conflict_resolving_pool,
  simulation_cache, block_building_results_assembler)
• clean the task_queue, as we have it in conflict_task_generator
• at this point, we might have some ongoing tasks being executed by conflict_resolving_pool, but since we have results_aggregator available in the same loop, we can just record some blacklist: HashSet<GroupId> to skip these results later.
• simulation_cache is keyed by the Vec<GroupId>, so without the cleaned
  group_ids in our task_queue from now on, old entries are not an issue? I
  feel like we can just let it be
• Do we need to cancel the best_block here?(perhaps not) We could send a command to
  our new best_results channel to do that

Another thing is, I feel like if we knew OrderId of the orders that need cancellations, we could handle the cancellation without resetting the entities fully. I'll leave that for further discussion if relevant.

One last thing:

When we create conflict_resolving_pool:

• We spawn a thread when calling conflict_resolving_pool.start() like so:

  thread::spawn(move || {
      builder.conflict_resolving_pool.start();
  });

  It seems like we don't need to do that, as .start just spawns a task on a rayon ThreadPool and immediately returns anyway.

• Inside .start(), we only call self.thread_pool.spawn(..) once, thus only creating a single task for our ThreadPool to work on. I think we should be spawning as many tasks as we have pool workers here.