Skip to content

Use MassInsertBuilder for PutMessageSecrets, PutManySessions, PutManyLIDMappings#1101

Open
AlejoGarat wants to merge 1 commit into
tulir:mainfrom
AlejoGarat:fix/mass-insert-sessions-secrets
Open

Use MassInsertBuilder for PutMessageSecrets, PutManySessions, PutManyLIDMappings#1101
AlejoGarat wants to merge 1 commit into
tulir:mainfrom
AlejoGarat:fix/mass-insert-sessions-secrets

Conversation

@AlejoGarat
Copy link
Copy Markdown

@AlejoGarat AlejoGarat commented Mar 24, 2026

Problem

PutMessageSecrets, PutManySessions, and PutManyLIDMappings execute one db.Exec per row inside a DoTxn, resulting in N sequential round-trips to the database per call.

This is particularly noticeable during:

  • Pairing / initial sync: WhatsApp sends a large batch of message secrets and session keys at once, triggering dozens to hundreds of sequential inserts within a single transaction.
  • Mass reconnects: many users reconnecting simultaneously each trigger their own slow transactions, compounding the contention.

The DoTxn slow-transaction monitor (in go.mau.fi/util/dbutil) logs "Transaction still running" every 5s and "Transaction took long" on close for transactions exceeding 1s. These warnings appear frequently.

Solution

Use MassInsertBuilder (already available in go.mau.fi/util/dbutil and already used by PutAllContactNames and PutManyRedactedPhones) to reduce N round-trips to a single bulk INSERT per chunk of 500 rows.

Changes:

  • Add GetMassInsertValues() [4]any to MessageSecretInsert (in store/store.go)
  • Add GetMassInsertValues() [2]any to addressSessionTuple (in sqlstore/store.go)
  • LIDMapping already had GetMassInsertValues(); add the builder and refactor PutManyLIDMappings
  • PutManyLIDMappings keeps the per-row DELETE (needed for stale PN→LID cleanup) and batches only the INSERT

Before / After

Before (PutMessageSecrets with 200 inserts):

  • 200 × db.Exec(putMsgSecret, ...) inside a single transaction
  • 200 sequential round-trips to the database

After:

  • 1 × db.Exec with INSERT INTO whatsmeow_message_secrets VALUES ($1,$2,...), ($1,$6,...), ...
  • 1 round-trip for up to 500 rows (chunked if more)

PutMessageSecrets, PutManySessions, and PutManyLIDMappings were doing
one db.Exec per row inside a transaction, resulting in N sequential
round-trips to the database. This becomes a bottleneck during pairing
(initial key/session sync) and mass reconnects, causing the DoTxn
slow-transaction monitor to fire warnings at 1s and 5s intervals.

Replace the per-row loops with MassInsertBuilder (already used by
PutAllContactNames and PutManyRedactedPhones), reducing N round-trips
to a single bulk INSERT per chunk of 500 rows.

- Add GetMassInsertValues() to MessageSecretInsert and addressSessionTuple
- LIDMapping already had GetMassInsertValues(); add builder and refactor
- PutManyLIDMappings keeps the per-row DELETE (stale PN cleanup) and
  batches only the INSERT, which is where the bottleneck lies
@AlejoGarat AlejoGarat changed the title sqlstore: use MassInsertBuilder for PutMessageSecrets, PutManySessions, PutManyLIDMappings Use MassInsertBuilder for PutMessageSecrets, PutManySessions, PutManyLIDMappings Mar 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant