Skip to content

mysql: add syncer.EventCacheCount setting#4413

Merged
dtunikov merged 4 commits into
mainfrom
feat/dbi-408/mysql-event-cache-count-env
Jun 12, 2026
Merged

mysql: add syncer.EventCacheCount setting#4413
dtunikov merged 4 commits into
mainfrom
feat/dbi-408/mysql-event-cache-count-env

Conversation

@dtunikov

Copy link
Copy Markdown
Collaborator

We saw OOM errors on mysql pipes with large rows
go-mysql has internal events channel where it stores binlog events, that is where this allocation is coming from
It reads packet header that contains info about payload length in bytes and allocates memory for reading next events from the binlog
I think it's expected driver behavior and i'm not sure we should fix smth in that regard
But
There is one thing we could configure additionally to our PEERDB_CDC_CHANNEL_BUFFER_SIZE, we could set EventCacheCount on mysql syncer level.
This way we can cap how many events mysql driver loads in memory. (we can't control events' size though, so if the events themselves are big -> it won't really save us from OOM), by default it equals 10240

mysql go driver simply read packets over network and tries to push events to it's internal channel (which is capped by EventCacheCount):

loop:
  data := b.c.ReadPacket()        // blocking socket read
  event := parse(data)
  s.AddEventToStreamer(event)     // blocking channel send → s.ch (cap = EventCacheCount)

@dtunikov dtunikov requested a review from a team as a code owner June 11, 2026 08:51
@claude

claude Bot commented Jun 11, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

Comment thread flow/connectors/mysql/ssh_keepalive_test.go Outdated
Comment thread flow/connectors/mysql/ssh_test.go Outdated
Comment thread flow/internal/dynamicconf.go Outdated
@dtunikov dtunikov enabled auto-merge (squash) June 12, 2026 09:31
dtunikov and others added 2 commits June 12, 2026 11:38
Co-authored-by: Ilia Demianenko <ilia.demianenko@clickhouse.com>
Co-authored-by: Ilia Demianenko <ilia.demianenko@clickhouse.com>
@dtunikov dtunikov merged commit f07ac54 into main Jun 12, 2026
15 checks passed
@dtunikov dtunikov deleted the feat/dbi-408/mysql-event-cache-count-env branch June 12, 2026 10:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants