RFC: Rollup Boost HA Design

[0xKitsune]

The protocol team at TFH has been exploring high availability designs for Rollup Boost and put together a design doc outlining a few options we’ve considered. Our goal is to align on an approach that’s compatible with the existing OP Stack HA deployment and prioritizes the liveliness and fault tolerance guarantees it offers.

Our mental model is that Rollup Boost should be maximally simple, robust and inherit all of the guarantees from the existing sequencer architecture.

We are currently running Rollup Boost on World Chain Sepolia using a 1:1 Rollup Boost <-> Builder setup. While this is our current approach, this doc isn’t meant to be prescriptive but rather to kick off a discussion on the best way to run Rollup Boost in an HA configuration.

Feel free to let us know if anything needs clarification or if there’s something specific that would be helpful to add. Along with the other L2s planning to run Rollup Boost, we’d love to get OP’s thoughts on a design direction and how to continually iterate to align with the broader vision for the OP Stack.

Here is the link to the design document

[dmarzzz]

Adding HA Technical Design Document from @angel-ding-cb here https://hypnotic-ash-de1.notion.site/TDD-Rollup-Boost-Integration-with-HA-Sequencer-1d0c9d820ca380348f21e44a5442feaf

[alfonso-op]

Thanks for drafting, some thoughts:

• The simplicity of the 1:1 rollup-boost <> builder design is appealing. FWIW we will be adopting a similar architecture for Interop (associating a supervisor with each sequencer instance), so the ability to treat each sequencer instance as a single self-contained deployable unit is a nice property and lends to operational simplicity
• One other edge case failure scenario that should be considered for the 1:1 design - if all builders are unhealthy, a sequencer must still be selected as leader but should leverage the default execution client
• The Builder Load Balancer design seems preferable to the n Block Builders design - supports horizontal scaling of the builder without requiring modifying the config of each rollup-boost instance, improved separation of concerns (isolation of routing and block selection logic)
• General comment - in all designs, I'd be concerned about adding additional latency when builders are unhealthy. In particular, one of the failure scenarios alludes to a timeout in rollup-boost before it decides to fallback to the default execution client in this scenario. If the builders stay unhealthy for an extended period of time, we probably don’t want to rely on a repeated timeout expiry for each subsequent block before leveraging the default execution client for block building. Rather, it may be beneficial to have a way for rollup-boost (or another component) to check for builder health outside of the block production hot path and have rollup-boost check this state to decide whether to use rbuilder or the default execution client
• +1 on having an emergency shutoff switch

[jelias2]

Commentary on the solutions

• 1<>1 Rollup Boost to Builder Deployments
  • Health checks should probably include checking the builder sync status using optimism_SyncStatus to the builder op-node. Not just checking the rollup-boost for healthz
    • In this setup failures or out of sync builders should cause op-conductor to mark a sequencer as unhealthy or degraded
• N Block Builders
  • Does rollup-boost support multiplexing to multiple builders or does this functionailty need to be built out?
  • If a builder fails in this setup how will be it marked as unhealthy, and how will the multiplexing terminate sending calls to that builder?
  • Cons: Live operations to disable/enable a single builder require operations across all three rollup-boost instances
  • Upgrading a builder will may cause noise since it would be difficult to remove a builder from all 3 rollup-boosts at once
• Builder Load Balancer
  • Added network call may introduce a negligible of latency
  • Pro: op-conductor logic stays focused on sequencers and builder logic can be separated out to the loadbalancer to determine builder healthiness

Overall thoughts

• In general rollup-boost since it is only a proxy can be ran in an HA setup, ex. Multiple pods per instance to help ensure rollup-boost redundancy.
• Emergency Shutoff switch already exists by calling rollup-boost debug_SetExecutionMode to disabled?
• Have the solutions N block builders or builder loadbalancer been tested yet anywhere?
  • The 1<>1 solution appears to be working on internal environments at OP Labs
• Since building a block takes multiple Engine calls to the builder, how can it be ensured if multiplexing or using a loadbalancer would retain stickiness to single builder if thats needed. Or is stickiness not necessary throughout the entire flow?
  • What happens if builder generate different payload IDs based on current TX pool? The TX pool may not be perfectly in-sync across builders if they are in different regions
  • This could result in error calls or missing payloads which would add operational overhead.

[0xKitsune]

For visibility, I opened a PR to make it easier to leave comments and suggest edits inline.

[angel-ding-cb]

Thanks for drafting the solution TDD!

We are also pretty much aligned on running a 1:1 Rollup Boost to Builder Setup. A couple of thoughts for now:

1. I don't know if we need to health check on the rollup boost since if the rollup boost is down, the op-node will be down as well. This should trigger a leadership transfer already as the health check on the op-node fails. But agreed with @jelias2 that we need to health check the builder.
2. What are people's thoughts on tolerating flashblock reorgs during leadership transfer? If we want 100% inclusion of partial flashblocks, we will need to commit flashblocks to op-conductor in the raft consensus layer.

[0xKitsune]

Re @alfonso-op's comments

One other edge case failure scenario that should be considered for the 1:1 design - if all builders are unhealthy, a sequencer must still be selected as leader but should leverage the default execution client

If the preferred design requires that the builder health is incorporated into leader elections:

• op-conductor could be extended to directly check the health of the builder
• op-conductor could forward health checks through rollup-boost, and rollup-boost could report on the health of both the builder and the default execution client without needing to modify op-conductor.

In either approach, this raises the edge case you mentioned: if 2 or more builders are unhealthy, op-conductor still needs to select a healthy leader. One idea our team explored was having rollup-boost expose different responses based on builder and default execution client health. For example:

• 200: builder and execution client are healthy
• 206: builder unhealthy, execution client healthy
• 503: both are unhealthy

IIUC this isnt how op-conductor works today, however with this approach op-conductor could prioritize fully healthy sequencers, then fall back to partially healthy ones if no fully healthy options remain. Let me know if you have any other thoughts on potential ways to solve for this.

---

The Builder Load Balancer design seems preferable to the n Block Builders design - supports horizontal scaling of the builder without requiring modifying the config of each rollup-boost instance, improved separation of concerns (isolation of routing and block selection logic).

Agreed that the 1:1 approach seems to be the most straightforward. However if considering other designs, the load balancer design seems preferable to the n builders in that it supports horizontal scaling of the builder without requiring config changes in each rollup-boost instance and improves separation of concerns. However, it seems there are significant tradeoffs depending on the technical requirements of the system.

If the medium term goal is to support multiple builders operated by independent parties, then using a centralized load balancer introduces operational overhead controlled by a single organization (which may not be the sequencer operator). In that case, the n builders approach may be more aligned with decentralization goals.

On the other hand, a load balancer simplifies things operationally in the short term:

• Easier horizontal scaling without changing sequencer-side config
• Centralized control over builder selection and routing logic

However it does introduce another moving part to provision, configure, and maintain. Embedding the routing logic directly in rollup-boost may result in a simpler system overall.

It’s worth considering a couple key questions:

• Is the intention to support n builders operated independently?
  If yes, the n builders design may be more future proof.
• Are all builders managed by a single operator, and is operational simplicity the primary goal?
  If so, 1:1 may be the best option in the short/medium term. Using a load balancer could be explored in the medium term as well.

---

General comment - in all designs, I'd be concerned about adding additional latency when builders are unhealthy. Rather, it may be beneficial to have a way for rollup-boost (or another component) to check for builder health outside of the block production hot path and have rollup-boost check this state to decide whether to use rbuilder or the default execution client

We could implement something along these lines, or alternatively enforce strict timing requirements and fall back to the default execution client if the builder doesn’t respond in time.

[0xKitsune]

Re: @jelias2's comments

1<>1 Rollup Boost to Builder Deployments

Health checks should probably include checking the builder sync status using optimism_SyncStatus to the builder op-node. Not just checking the rollup-boost for healthz
- In this setup failures or out of sync builders should cause op-conductor to mark a sequencer as unhealthy or degraded

If builder health will be incorporated in the op-conductor health checks, there needs to be a reliable way to distinguish between when the sequencer is:

• Healthy
  • Default execution client is healthy
  • Builder is healthy
• Partially Healthy
  • Default client is healthy
  • Builder is unhealthy
    Unhealthy
• Default client is unhealthy

Without this, if 2 or more builders are down, op-conductor may have an issue electing a new leader.

---

N Block Builders

Does rollup-boost support multiplexing to multiple builders or does this functionailty need to be built out?

Logic to support n builders is not currently implemented, this would have to be built out

If a builder fails in this setup how will be it marked as unhealthy, and how will the multiplexing terminate sending calls to that builder?

Calls to the builder will still be sent. In the case of FCUs or newPayload requests, the builder response is ignored. When sending a getPayload request, if the builder does not return a valid block in time, the call is dropped.

Cons: Live operations to disable/enable a single builder require operations across all three rollup-boost instances

Yes, this is true, but I would not expect this operation to occur frequently. The operational overhead can be minimized with proper runbooks and procedures for adding or removing builders. Alternatively, a builder operator could choose do deploy their resources behind their own load balancer and scale horizontally as needed, reducing the need to add/remove endpoints.

Upgrading a builder will may cause noise since it would be difficult to remove a builder from all 3 rollup-boosts at once

This does not have to happen atomically. Chain operators can add/remove the endpoint from one sequencer instance at a time. Lmk if im overlooking something here.

---

Builder Load Balancer

Added network call may introduce a negligible of latency

Yes this will depend on the topology of the network, but this could be the case.

---

Overall thoughts

Emergency Shutoff switch already exists by calling rollup-boost debug_SetExecutionMode to disabled?

Yes this logic exists in the current version of rollup-boost but can be simplified.

Have the solutions N block builders or builder loadbalancer been tested yet anywhere?
The 1<>1 solution appears to be working on internal environments at OP Labs

At the moment, the n block builders/load balancer approaches are only in the design phase. Our team has focused on a 1:1 solution as well but given the recent discussions around alternative design paths from different L2 teams, this document is an attempt to discuss and align on an approach.

Since building a block takes multiple Engine calls to the builder, how can it be ensured if multiplexing or using a loadbalancer would retain stickiness to single builder if thats needed. Or is stickiness not necessary throughout the entire flow?

In the designs where there are multiple builders, requests would be forwarded to all builders. With this approach, it is not necessary for a request to be routed to the same builder during the block construction lifecycle

[0xKitsune]

Re: @angel-ding-cb's comments:

What are people's thoughts on tolerating flashblock reorgs during leadership transfer? If we want 100% inclusion of partial flashblocks, we will need to commit flashblocks to op-conductor in the raft consensus layer.

This doc largely doesn’t factor in Flashblocks atm since the spec is still early/evolving, but I agree this is something that will need to be thought through.

My current mental model is that since Flashblocks are essentially a preconf stream, most of the time the preconfirmed transactions will end up in the published block as originally streamed but it should not be treated as a strict guarantee. Leadership transfers that drop or reorder Flashblocks seem similar to normal chain reorgs. In the happy case, everything lands but the protocol should be resilient to cases where it doesn’t.

Its worth noting that my thoughts around Flashblocks are still early though so happy to hear alternative viewpoints!

[0x00101010]

Relaying some of my comments on the PR here

1. Unfortunately choice 2 (n builder) & 3 (load balancing) won't work due to that we have to decide which builder's payload you're gonna potentially accept, otherwise, it's not feasible to decide which one's flashblocks to stream to the public.
2. 1:1 sequencer / builder mapping is what we decided to go with (ideally colocating to minimize latencies)
3. a conductor proxy will need to be added for solution 1 so that only leader sequencer sends flashblocks to the public via web socket (proxy from rollup-boost) @angel-ding-cb from our team is working on it
4. to absolutely prevent flashblocks reorg, they'll need to be committed into raft consensus so that during leadership transfer, it can be picked up and used to produce the actual block (this is not prioritized yet)

[0xKitsune]

Thanks for reviewing/commenting! Ill check out the PR comments, a few thoughts in the meantime:

1:1 sequencer / builder mapping is what we decided to go with (ideally colocating to minimize latencies)

Agreed, this is what we are currently running as well and prefer at the moment.

Unfortunately choice 2 (n builder) & 3 (load balancing) won't work due to that we have to decide which builder's payload you're gonna potentially accept, otherwise, it's not feasible to decide which one's flashblocks to stream to the public.

I think this ultimately depends on how the Flashblocks behavior is defined. In the current FB draft spec, rollup-boost connects to a builder via WS stream and incrementally receives each Flashblock payload. At the end of the block construction window, rollup-boost builds the new payload from the Flashblocks it has locally and returns this to op-node. In this case after the stream is established, rollup-boost will not need to know which builder to route to.

However, this is just a draft and the design is still evolving. The current FB direction aims to support builders behind a load balancer but this may change.

a conductor proxy will need to be added for solution 1 so that only leader sequencer sends flashblocks to the public via web socket (proxy from rollup-boost) @angel-ding-cb from our team is working on it

Is this repo public? Would love to check it out.

[0x00101010]

Is this repo public? Would love to check it out.

It'll be committed to op-conductor, will let you know when the PR is ready

I think this ultimately depends on how the Flashblocks behavior is defined. In the current FB draft spec, rollup-boost connects to a builder via WS stream and incrementally receives each Flashblock payload. At the end of the block construction window, rollup-boost builds the new payload from the Flashblocks it has locally and returns this to op-node. In this case after the stream is established, rollup-boost will not need to know which builder to route to.

The issue here is that we'll need to pre-determine which builder to establish connection to before deciding on the actual payload, the current 2 and 3 choices will decide on that after it gets all payloads from builders (close to block sealing)

[0xKitsune]

Thanks everyone for the feedback! Based on the discussion, the design doc has been updated to focus on the 1:1 Rollup Boost <-> Builder deployment model as the primary HA strategy. We have also updated the doc to address comments as well as added sections detailing:

• Builder and execution client health checks (including partial health states)
• Execution modes to support dry-run deployments and emergency shutoff

Let us know if you have additional thoughts and if there is anything else we should consider/clarify further.