fixed fec sets #5771
fixed fec sets #5771
Conversation
|
Currently seeing ~7% padding overhead running with 4eb25e2 This is similar to what Jump has observed using the same entry coalesce bytes target. |
|
As far as what is driving the need for padding, here are some logs that shed light: There are a couple of "bad" cases that lead to more padding:
A few potential options to get better here:
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #5771 +/- ##
=========================================
- Coverage 82.9% 82.9% -0.1%
=========================================
Files 830 830
Lines 376347 376498 +151
=========================================
+ Hits 312282 312386 +104
- Misses 64065 64112 +47 🚀 New features to boost your workflow:
|
|
Running the same experiment ( This code:
Master code:
|
|
On mainnet over the last 2 weeks, the overhead from variable coding size has been around 2%: On mainnet over the last 2 weeks, the overhead from padding out the last data shreds has averaged around ~10 data shreds per slot or ~1.2% overhead: The final source of overhead would be the padding out of the last FEC set to 32 shreds. We don't have metrics for this, but my assumption is that we're padding half the FEC set on average, which would be 16 shreds, and this would result in ~1.92% padding overhead. So overall, my understanding is mainnet has ~5% padding overhead. This lines up w/ some other measurements analyzing zero content data in the 5-6% range |
|
Took a sample of 100k entries on mainnet, and the workload is much more favorable for packing fixed batches than the synthetic testing. This is because most entries contain a single tx (76%) or none (10.5%) and are only hundreds of bytes in size. Looks like most of the high tx count entries are a result of votes being batched together (inferring based on AVG tx size being right around 352B for larger entries) |
bw-solana
force-pushed
the
fixed_fec_set
branch
from
21892c7 to
7c32454
Compare
There was a problem hiding this comment.
suggest moving the shredding logic to another file. merkle.rs is already huge enough.
|
|
||
| // Wait up to `ENTRY_COALESCE_DURATION` to try to coalesce entries into a 32 shred batch | ||
| let data_shred_bytes = | ||
| ShredData::capacity(Some((6, true, false))).expect("Failed to get capacity") as u64; |
There was a problem hiding this comment.
this is all constant, no need recomputing this every time
| ) { | ||
| // Fetch the next entry. | ||
| let Ok((try_bank, (entry, tick_height))) = receiver.recv_deadline( | ||
| coalesce_start + max_coalesce_time(serialized_batch_byte_count, max_batch_byte_count), |
There was a problem hiding this comment.
suggest we try to maybe wake up some 5 ms early to avoid OS being annoying and waking this thread 5 ms too late instead.
bw-solana
force-pushed
the
fixed_fec_set
branch
from
f52cb83 to
b6d5360
Compare
|
Testing w/ latest entry coalescing policy is looking great so far. I'm seeing 5% padding, which is in line with our current padding (maybe slightly less?) w/ variable FEC sets. Main changes are to:
Picture shows a per slot breakdown of padding bytes and why we decided to exit the entry coalescing routine. Lots of tightly packed is 👍. Exiting due to reaching max size is okay. Exiting due to receive timeout is usually not good: Also note we appear to be maxing out CUs for the first block in a leader span, but the rest are light (seems to be due to demand) |
|
Padding bytes relatively small to data bytes: Broadcast below 350ms: Replay total elapsed time for a 12 slot sequence ( |
|
I think hyperoptimizing this is not necessary, as blocks become larger padding will disappear in the overall traffic. We just need to pack more TXs into the blocks in general. |
3 participants
Problem
We would like to move to fixed (32:32) FEC set sizes to simplify multiple areas of the code.
This PR is simply a draft to prove out the concept and is not meant to be merged.
If we decide to pursue this, we will need to:
Summary of Changes