Bug Report: VTGates OOM on large transactions

[twthorn]

Overview of the Issue

The bug is caused by this line. We buffer with eventss the entire transaction before sending it. Very large transactions eg multi-GB can cause OOM errors. Example described here

it was from an 11GB size transaction (11 MB per row * 1000 rows). So it seems there was some issue with breaking this transaction up between tablet to vtgate and vtgate to client. We are still looking into it, but wanted to share early findings

Reproduction Steps

Create a transaction that exceeds the memory limit of the vtgate. It will cause it to OOM. If you set GOMEMLIMIT it will help (GC's the cloned ROW events) or use exclude keyspace from table name to avoid cloning.

Binary Version

19.0.7

Operating System and Environment details

vitess@vtgate-bedrock-cdc-dev-iad-7f96cfb9fb-8dxn8:/vt/bin$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Log Fragments

From VStream client: Network closed for unknown reason