Commit d0441d0
TxPool: map reduce and batch load (#18526)
Batches and deduplicates incoming transaction messages to reduce
redundant computation and MDBX transaction overhead.
### Architecture & Rationale
Two problems were identified in profiling:
1. **Blob verification allocations**: Duplicate transactions arriving
from multiple peers caused the same expensive blob verification to run
multiple times, leading to excessive allocations.
2. **MDBX transaction overhead**: Each incoming message opened its own
read transaction. Profiling showed this accounted for ~15% of CPU time.
The fix introduces:
- **LRU-based deduplication**: An LRU cache (4096 entries) filters
duplicate messages before they enter the processing batch. Uses
`maphash` on the raw message data to avoid string allocations (we dont
need to care about collision as it is a random seed for each instance
and this is not a consensus component anyways).
- **1-second batching window**: Instead of processing messages
immediately, they accumulate in a batch and are flushed every second (or
on stream close). A single MDBX read transaction is opened per batch and
reused for all messages in that batch.
This ensures each unique transaction is processed exactly once per batch
window, and the MDBX transaction cost is amortized across potentially
hundreds of messages.
---------
Co-authored-by: Giulio <monkeair@MacBook-Air-di-Giulio.local>1 parent 18f1bff commit d0441d0
1 file changed
+86
-14
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| 27 | + | |
26 | 28 | | |
27 | 29 | | |
28 | 30 | | |
| |||
176 | 178 | | |
177 | 179 | | |
178 | 180 | | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
179 | 249 | | |
180 | 250 | | |
181 | 251 | | |
| |||
187 | 257 | | |
188 | 258 | | |
189 | 259 | | |
| 260 | + | |
190 | 261 | | |
191 | 262 | | |
192 | | - | |
193 | | - | |
194 | | - | |
195 | | - | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
196 | 271 | | |
197 | | - | |
198 | | - | |
199 | | - | |
200 | | - | |
| 272 | + | |
201 | 273 | | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
202 | 279 | | |
203 | 280 | | |
204 | 281 | | |
205 | | - | |
| 282 | + | |
206 | 283 | | |
207 | 284 | | |
208 | 285 | | |
| |||
212 | 289 | | |
213 | 290 | | |
214 | 291 | | |
215 | | - | |
216 | | - | |
217 | | - | |
218 | | - | |
219 | | - | |
220 | 292 | | |
221 | 293 | | |
222 | 294 | | |
| |||
0 commit comments