Skip to content

PMM-12279 Enable perf_schema.memory_events collector and add memory panels#5649

Draft
theTibi wants to merge 2 commits into
mainfrom
PMM-12279
Draft

PMM-12279 Enable perf_schema.memory_events collector and add memory panels#5649
theTibi wants to merge 2 commits into
mainfrom
PMM-12279

Conversation

@theTibi

@theTibi theTibi commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Ticket number: PMM-12279

Feature build: SUBMODULES-0

What / why

PMM's bundled mysqld_exporter already ships a perf_schema.memory_events collector over performance_schema.memory_summary_global_by_event_name, exposing memory usage per event_name:

  • mysql_perf_schema_memory_events_used_bytes (= CURRENT_NUMBER_OF_BYTES_USED)
  • mysql_perf_schema_memory_events_alloc_bytes_total
  • mysql_perf_schema_memory_events_free_bytes_total

The collector was never enabled by PMM and nothing rendered it. This PR turns it on and graphs it, giving memory observability for PS/PXC (especially relevant in k8s).

Changes

  • Enable the collector in both synced generators (one lightweight global-summary query, LR resolution):
    • managed/services/agents/mysql.go — add --collect.perf_schema.memory_events to the LR args.
    • managed/services/victoriametrics/scrape_configs.go — add perf_schema.memory_events to the LR scrape options, keeping the two lists in sync.
  • Golden tests updated: mysql_test.go, scrape_configs_test.go, victoriametrics_test.go.
  • Dashboard — new "Performance Schema Memory" section in MySQL Performance Schema Details: current bytes used (topk by event), allocation rate, and free rate, plus a "Memory Event" filter variable. Panels degrade to "no data" when memory instruments are not enabled server-side (performance-schema-instrument='memory/%=ON').

Testing

Validated end-to-end against two monitored instances, Percona Server 8.0.46 and MySQL 9.1.0:

  • Both mysqld_exporter processes carry --collect.perf_schema.memory_events.
  • All three metrics present in VictoriaMetrics (~215–220 event series each); values match the source query (top consumer innodb/buf_buf_pool = 130.9 MB on both).
  • up=1 on all scrape jobs; no scrape errors and no parse/"unsupported" warnings on MySQL 9.1.
  • All three new dashboard panels render populated in the PMM UI for both versions.
  • TestMySQLdExporterConfig* and TestScrapeConfig pass; both packages build.

No regression on 8.0; works on the newer 9.x series.

If this PR adds, removes or alters one or more API endpoints, please review and update the relevant API documentation as well:

  • API Docs updated (n/a — no API endpoint changes)

If this PR is related to other PRs, contributions, or ongoing work in this or other repositories, please reference them here:

🤖 Generated with Claude Code

https://claude.ai/code/session_01TNZLMVAW9gHXcccq8uYBip


Generated by Claude Code

…anels

The bundled mysqld_exporter already ships a perf_schema.memory_events
collector over performance_schema.memory_summary_global_by_event_name,
exposing mysql_perf_schema_memory_events_used_bytes / _alloc_bytes_total /
_free_bytes_total per event_name. PMM never enabled it and no dashboard
rendered it.

- Enable the collector in both synced generators: the mysqld_exporter flag
  list (mysql.go, LR group) and the VM scrape-config generator
  (scrape_configs.go, LR options), keeping the two in sync.
- Update the golden tests (mysql_test.go, scrape_configs_test.go,
  victoriametrics_test.go) for the added collector.
- Add a "Performance Schema Memory" section to the MySQL Performance Schema
  Details dashboard: current bytes used (topk by event), allocation rate and
  free rate, plus a "Memory Event" filter variable. Panels degrade to
  no-data when memory instruments are not enabled server-side.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TNZLMVAW9gHXcccq8uYBip
@theTibi
theTibi requested review from a team and Nailya as code owners July 15, 2026 15:02
@theTibi
theTibi requested review from 4nte, JiriCtvrtka, matejkubinec and mattiasimonato and removed request for a team July 15, 2026 15:02
@theTibi
theTibi marked this pull request as draft July 15, 2026 15:03
@codecov

codecov Bot commented Jul 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 42.41%. Comparing base (31318c7) to head (66f04f4).
⚠️ Report is 34 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5649      +/-   ##
==========================================
- Coverage   43.59%   42.41%   -1.18%     
==========================================
  Files         415      431      +16     
  Lines       43134    34821    -8313     
  Branches        0      585     +585     
==========================================
- Hits        18804    14770    -4034     
+ Misses      22454    18568    -3886     
+ Partials     1876     1483     -393     
Flag Coverage Δ
admin 34.96% <ø> (+0.17%) ⬆️
agent ?
managed 43.80% <100.00%> (+0.82%) ⬆️
unittests 41.29% <ø> (?)
vmproxy ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

"--collect.info_schema.clientstats",
"--collect.info_schema.userstats",
"--collect.perf_schema.eventsstatements",
"--collect.perf_schema.memory_events",

@JiriCtvrtka JiriCtvrtka Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since collector is enabled globally, I think this collector fails on MariaDB versions before 10.5.2 because the required table does not exist. If so could we add message and include it in tests?

I just added message about compatibility on different MySQL dashboard and it looks like this:
Screenshot 2026-07-16 at 10 53 12

Because it is hard to tell if "NO DATA" are because of error or I am using unsupported version.

EDIT: My mistake, apparently we no longer officially support MariaDB.

"steppedLine": false,
"targets": [
{
"expr": "topk(10, rate(mysql_perf_schema_memory_events_free_bytes_total{service_name=~\"$service_name\", event_name=~\"$mem_event_name\"}[$interval]))",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if interval is set to 30s? This collector runs at low resolution, but the queries use rate(...[$interval]). When $interval is shorter than the scrape interval, rate() does not have enough samples and could returns no data.

"steppedLine": false,
"targets": [
{
"expr": "topk(10, rate(mysql_perf_schema_memory_events_alloc_bytes_total{service_name=~\"$service_name\", event_name=~\"$mem_event_name\"}[$interval]))",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it could be same case as #5649 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants