Re-think timeouts for getPayload

[michaelsproul]

Description

Presently Lighthouse uses a 2 second timeout for getPayload, both when requesting a blinded block and when requesting a full block.

lighthouse/beacon_node/execution_layer/src/engine_api/http.rs

Line 35 in 38514c0

pub const ENGINE_GET_PAYLOAD_TIMEOUT: Duration = Duration::from_secs(2);

This means that if the EL is slow we could slow down blinded block proposals to the point where they exceed the 4s attestation deadline. We wait for the maximum of the builder/EL response times here:

lighthouse/beacon_node/execution_layer/src/lib.rs

Lines 675 to 692 in 38514c0

	// Wait for the builder *and* local EL to produce a payload (or return an error).
	let ((relay_result, relay_duration), (local_result, local_duration)) = tokio::join!(
	timed_future(metrics::GET_BLINDED_PAYLOAD_BUILDER, async {
	builder
	.get_builder_header::<T, Payload>(slot, parent_hash, &pubkey)
	.await
	}),
	timed_future(metrics::GET_BLINDED_PAYLOAD_LOCAL, async {
	self.get_full_payload_caching::<Payload>(
	parent_hash,
	timestamp,
	prev_randao,
	suggested_fee_recipient,
	forkchoice_update_params,
	)
	.await
	})
	);

For the builder's payload, we will then also need to wait for the block to be un-blinded after signing, which adds yet more latency. With a 500ms offset from the start of the slot (due to e.g. attestation packing) plus 2 seconds of EL response time (worst-case) plus 1s for the VC to sign, unblind, and for the BN+relay to broadcast, we are cutting very close to the 4s deadline.

Steps to resolve

The engine API spec uses a 1s timeout for getPayload at all times, which would be more consistent. However I think it could be advantageous to retain the longer timeout for full blocks and just use the shorter 1s timeout for blinded blocks. I think this could be implemented in a straight-forward way by passing the timeout to get_full_payload_caching. This might be worth implementing after the capella branch is merged, as it also makes a lot of changes to these code paths.

[michaelsproul]

On the other hand, if the BN is not using a builder and is configured as the only BN for a validator, it would be ideal to wait as long as possible (up to 4s). This would be better managed by the VC, so perhaps we should set long timout on the BN (12s) and let the VC decide how long it wants to wait.

A user on Discord missed a block proposal recently because their only BN timed out on the local EL.