Skip to content

Commit 8b1c45a

Browse files
committed
Optimize tx processing RAM
- Don't load blob fields that are not used - Process a smaller batch of transactions every time For example, we detected a Safe in production with 782KB used by `logs` and other fields for every row: - Before: 5000 × 782KB = 3.9GB - After: 500 × 782KB = 391MB Not even counting on the optimization not loading blob fields.
1 parent 2ccb6c8 commit 8b1c45a

File tree

2 files changed

+9
-2
lines changed

2 files changed

+9
-2
lines changed

config/settings/base.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -571,7 +571,7 @@
571571
"ETH_INTERNAL_TRACE_TXS_BATCH_SIZE", default=0
572572
) # Number of `trace_transaction` calls allowed in the same RPC batch call, as results can be quite big
573573
ETH_INTERNAL_TX_DECODED_PROCESS_BATCH = env.int(
574-
"ETH_INTERNAL_TX_DECODED_PROCESS_BATCH", default=5000
574+
"ETH_INTERNAL_TX_DECODED_PROCESS_BATCH", default=500
575575
) # Number of InternalTxDecoded to process together. Keep it low to be memory friendly
576576

577577
# Event indexing configuration (L2 and ERC20/721)

safe_transaction_service/history/models.py

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1325,7 +1325,14 @@ def pending_for_safes(self):
13251325
return (
13261326
self.not_processed()
13271327
.order_by_processing_queue()
1328-
.select_related("internal_tx", "internal_tx__ethereum_tx")
1328+
.select_related("internal_tx__ethereum_tx")
1329+
# Defer large blob fields not used during processing to reduce memory usage
1330+
.defer(
1331+
"internal_tx__data",
1332+
"internal_tx__code",
1333+
"internal_tx__output",
1334+
"internal_tx__ethereum_tx__data",
1335+
)
13291336
)
13301337

13311338
def pending_for_safe(self, safe_address: ChecksumAddress):

0 commit comments

Comments
 (0)