[receiver/mysqlreceiver] Add MySQL <8 and MariaDB multi-version support#47815
Open
cjksplunk wants to merge 59 commits intoopen-telemetry:mainfrom
Open
[receiver/mysqlreceiver] Add MySQL <8 and MariaDB multi-version support#47815cjksplunk wants to merge 59 commits intoopen-telemetry:mainfrom
cjksplunk wants to merge 59 commits intoopen-telemetry:mainfrom
Conversation
Detect the server version at runtime to handle MySQL <8 and all MariaDB versions, which lack the query_sample_text column in performance_schema.events_statements_summary_by_digest. - Add dbProduct/dbVersion types with capability predicates (isMySQL8Plus, supportsQuerySampleText) - Add getDBVersion() to the client interface with lazy caching - Add topQueryNoSampleText.tmpl — 5-column fallback query template - Branch getTopQueries() on supportsQuerySampleText() to select the appropriate template and scan set - Generate explain plans for query samples; include result as the mysql.query_plan attribute on db.server.query_sample events - Share a single plan cache between top-query and query-sample scrapers to avoid duplicate EXPLAIN calls - Only create the shared plan cache when at least one log event scraper is enabled, preventing goroutine leaks in default config Assisted-by: Claude Sonnet 4.6
…ties - TestVersionCompatibility: verifies getDBVersion() correctly identifies MySQL vs MariaDB product and version flags (isMySQL8Plus, supportsQuerySampleText), and that getTopQueries() selects the correct 6-column (MySQL 8+) or 5-column (MariaDB/<8) template by running without error against live containers - TestIntegrationLogScraper: end-to-end proof that scrapeTopQueryFunc and scrapeQuerySampleFunc work on MySQL 8.0.33, MariaDB 10.11, and MariaDB 11.4; validates shared plan cache interaction and structural log record attributes - Fix supportsQuerySampleText() boundary: QUERY_SAMPLE_TEXT was introduced in MySQL 8.0.3 RC, so the correct gate is isMySQL8Plus() (not 8.0.22); update client_test.go accordingly Assisted-by: Claude Sonnet 4.6
- Use assert.NoError (not require) in defers so shutdown failures don't panic after the test has already stopped (M1) - Log perf_schema setup errors in runPerfSchemaSetup instead of silently swallowing them (M2) - Add sync comment to topQueryNoSampleText.tmpl pointing to base template to prevent silent divergence (M3) - Introduce usesSampleText local in getTopQueries to avoid calling supportsQuerySampleText() twice (L6) Assisted-by: Claude Sonnet 4.6
Replace the bare *dbVersion pointer cache with sync.Once + result fields so concurrent calls to getDBVersion() are safe and SELECT VERSION() executes exactly once per connection. Also change the query() helper to take *sql.DB instead of mySQLClient by value — mySQLClient is now non-copyable (contains sync.Once) and query() only needed the sql.DB anyway. Update TestGetDBVersionCaching to prime the Once directly rather than setting the old cachedDBVersion field. Assisted-by: Claude Sonnet 4.6
…ability by version Assisted-by: Claude Sonnet 4.6
Adds connection-time version detection so the receiver selects the correct query template and replication status command for each server: - MySQL 8.0.22+: full 6-column top-query template (includes query_sample_text) and SHOW REPLICA STATUS - MySQL <8 and all MariaDB versions: 5-column fallback template (no query_sample_text) and SHOW SLAVE STATUS Key changes: - client.go: fetchDBVersion() called during Connect(); getDBVersion() returns cached dbVersion; version detection failure is non-fatal (safe fallback used instead of erroring out) - scraper.go: shared queryPlanCache passed through newMySQLScraper; scrapeTopQueries uses version-branched template; mysql.query_plan belongs only on db.server.top_query events, not query_sample events - metadata.yaml: mysql.query_plan removed from db.server.query_sample; mysql.query_plan.hash retained on both event types - templates/: topQuery.tmpl (MySQL 8+), topQueryNoSampleText.tmpl (MySQL <8/MariaDB), querySample.tmpl updated - integration_test.go: TestIntegrationLogScraper and TestVersionCompatibility tests for MySQL 8, MariaDB 10.11, MariaDB 11.4 - scraper_test.go: TestScrapeQuerySamplesExplainPlan updated to reflect that explainQuery is not called from scrapeQuerySampleFunc - expectedQuerySamples.yaml: removed stale mysql.query_plan attribute Assisted-by: Claude Sonnet 4.6
- Fix nil-pointer panic in getReplicaStatusStats when fetchDBVersion fails at Connect time: add version == nil guard (same pattern as supportsQuerySampleText) so the safe SHOW SLAVE STATUS fallback is used when dbVersion is zero-valued - Extract parseDBVersion from fetchDBVersion so version string parsing can be unit-tested without a live database; add TestParseDBVersion covering MySQL 8, MySQL 5.7/5.6, MySQL with suffix (-log), MariaDB 10.x/11.x, and malformed input - Rename TestSharedPlanCacheDeduplication to TestScrapeQuerySamplesNoExplain and fix misleading comment: scrapeQuerySampleFunc never calls explainQuery (explain runs only in the top-query scraper); the test was vacuously correct under the old name - README: split MySQL 5.6 into its own row showing query_sample as unsupported — performance_schema.user_variables_by_thread (used by querySample.tmpl for traceparent) was introduced in MySQL 5.7.3 Assisted-by: Claude Sonnet 4.6
…ed version on start Document that version detection in Connect() runs exactly once and is non-fatal by design; a failure at startup means the receiver operates with incorrect version information for its entire lifetime with no retry. Log the detected product, version, and supports_query_sample_text on successful detection, or warn when detection failed and the fallback is active. Assisted-by: Claude Sonnet 4.6
….3 and MariaDB <10.5.2 performance_schema.user_variables_by_thread is not available on MySQL <5.7.3 or MariaDB <10.5.2. Add a fallback query sample template that omits the join and returns an empty traceparent for those versions, preventing unknown-table errors on every scrape. Add supportsUserVariablesByThread() predicate to drive template selection and include it in the startup version log. Assisted-by: Claude Sonnet 4.6
…rtsUserVariablesByThread Extend TestDBVersionCapabilities with wantSupportsUserVariablesByThread covering MySQL 8, 5.7.44, 5.7.3 (boundary), 5.7.2 (below boundary), 5.6.51, MariaDB 10.11.6, 10.5.2 (boundary), 10.5.1 (below boundary), 11.4.2, and the zero-value case. Add wantUserVarsByThread assertions to TestIntegrationLogScraper and TestVersionCompatibility; add MySQL 5.7 case to TestIntegrationLogScraper as the key boundary version. Add getQuerySamples no-error assertion to TestVersionCompatibility to prove the correct template is selected per server version. Add EOL note to supportsUserVariablesByThread: MySQL 5.6 and 5.7 support is included for completeness but both versions are past end-of-life. Assisted-by: Claude Sonnet 4.6
Extract version-detection logging into logDetectedVersion() and add a Warn log when MySQL <8 is detected, indicating the version is past end-of-life and may not be supported in a future release. MariaDB versions are not subject to this check. Add TestLogDetectedVersion covering MySQL 8 (no warn), MySQL 5.7/5.6 (EOL warn), MariaDB 10.11/10.4 (no EOL warn), and unknown version (fallback warn). Assisted-by: Claude Sonnet 4.6
Set db.version and db.product on the instrumentation scope of all emitted ScopeLogs at scrape time. These are derived from the version detected at Connect and are available downstream via OTTL as instrumentation_scope.attributes["db.version"] and instrumentation_scope.attributes["db.product"]. Scope attributes are carried silently through the pipeline and ignored by default — users who need them can access them without any data model changes to resources or log records. Assisted-by: Claude Sonnet 4.6
The scraper already caches detectedVersion after start(). Push all version-based branching up to the scraper so the client interface becomes a pure query executor: - getTopQueries now takes supportsSampleText bool - getQuerySamples now takes supportsUserVarsByThread bool - getReplicaStatusStats now takes supportsReplicaStatus bool Scraper call sites resolve these flags from m.detectedVersion using the existing capability predicates. Add supportsReplicaStatus() predicate on dbVersion to keep the pattern consistent with supportsQuerySampleText() and supportsUserVariablesByThread(). Future metrics work that needs version-based decisions can use m.detectedVersion directly without touching the client interface. Assisted-by: Claude Sonnet 4.6
…lity refactor - TestDBVersionCapabilities: add wantSupportsReplicaStatus field and assertions across all cases, including boundary versions 8.0.22 (true) and 8.0.21 (false) - TestReplicaStatusQuery: replace inlined version logic with supportsReplicaStatus() — tests now cover the actual predicate the scraper calls - newTopQueryScraper: set s.detectedVersion from mc.getDBVersion() so that scraper-side capability flags match the mock version override; previously detectedVersion was zero value regardless of the mock Assisted-by: Claude Sonnet 4.6
- Fix TestScrapeQuerySamplesExplainPlan and TestScrapeQuerySamplesNoExplain to set detectedVersion to MySQL 8 (previously ran with zero value) - Remove redundant s.detectedVersion = v8 in TestScrapeTopQueryFuncScopeAttributes (now set by newTopQueryScraper via mc.getDBVersion()) - Add TestScrapeQuerySampleFuncFallbackVersion: exercises MySQL 5.6 path where supportsUserVariablesByThread() is false - TestVersionCompatibility: add wantReplicaStatus field and call getReplicaStatusStats to verify SHOW REPLICA/SLAVE STATUS template selection - TestIntegrationLogScraper: inject observer logger to assert logDetectedVersion EOL warn fires for MySQL 5.7 and not for MySQL 8/MariaDB; add scope attribute assertions (db.version, db.product) on both top-query and sample log output Assisted-by: Claude Sonnet 4.6
…y-collector-contrib into mysql-multi-version-support
… and changelog Remove inaccurate claims that EXPLAIN/query plans are available on db.server.query_sample events. Query plans are only emitted on db.server.top_query events (MySQL 8.0+ only via query_sample_text). Assisted-by: Claude Sonnet 4.6
…, and extend test coverage - Extract `dbVersion.isValid()` and `dbVersion.productString()` to eliminate repeated nil checks and product string conversions across predicates and scraper - Extract `emitLogsWithScopeAttrs()` to deduplicate the 3-line emit pattern shared by scrapeTopQueryFunc and scrapeQuerySampleFunc - Hoist cache key construction in scrapeTopQueries to a single `cacheKey` var - Replace inline scanRow conditional in getTopQueries with a closure defined once outside the loop - Add TestDBVersionHelperMethods, TestScrapeQuerySampleFuncScopeAttributes, and TestScrapeTopQueryFuncScanRowWithSampleText to cover the refactored paths; extend mockClient to capture explainQuery call args Assisted-by: Claude Sonnet 4.6
…k.peer.port on query samples performance_schema.threads.processlist_host contains only the client hostname or IP address — never a host:port pair. The original implementation called net.SplitHostPort() on this value, which always failed for real MySQL/MariaDB instances and logged a spurious error: "Failed to parse processlistHost value: missing port in address <ip>" The test fixture worked around this by using a fabricated "192.168.1.80:1234" value that no real MySQL instance would produce. The fix queries the separate processlist_port column from performance_schema.threads, which correctly provides the client port as an independent integer value. Both templates (querySample.tmpl and querySampleNoUserVars.tmpl) now select COALESCE(thread.processlist_port, 0), the querySample struct gains a processlistPort field, and scraper.go uses both fields directly without any parsing. Assisted-by: Claude Sonnet 4.6
… port is unavailable performance_schema.threads.processlist_host contains only the client hostname or IP — never host:port. The previous commit attempted to query a separate processlist_port column, but that column does not exist in any supported MySQL or MariaDB version. Per MySQL documentation (https://dev.mysql.com/doc/refman/8.0/en/performance-schema-threads-table.html and the equivalent MySQL 5.7 page), the PROCESSLIST_HOST column deliberately omits the port number for TCP/IP connections: "the PROCESSLIST_HOST column does not include the port number for TCP/IP connections. To obtain this information from the Performance Schema, enable the socket instrumentation... and examine the socket_instances table." MariaDB's performance_schema.threads documentation (https://mariadb.com/kb/en/performance-schema-threads-table/) likewise lists no port column and makes no provision for obtaining it from this table. Socket instrumentation (required to get the port via socket_instances) is disabled by default and carries non-trivial overhead, making it unsuitable as a general-purpose solution. client.port and network.peer.port are therefore emitted as 0 (not populated). The previous implementation called net.SplitHostPort() on processlist_host, which always failed for real MySQL/MariaDB instances, producing a spurious error log on every scrape interval. The test fixture had fabricated a "192.168.1.80:1234" value that no real instance would produce. Assisted-by: Claude Sonnet 4.6
…rmation_schema.PROCESSLIST performance_schema.threads.processlist_host returns only a bare hostname/IP with no port — this is deliberate per MySQL and MariaDB documentation. Port information is not available in that table. information_schema.PROCESSLIST.HOST returns "host:port" for TCP/IP connections (e.g. "192.168.1.80:58061"), which is the correct source for both address and port. The two tables are joined on PROCESSLIST_ID = ID. The query sample templates now LEFT JOIN information_schema.PROCESSLIST and select pl.HOST in place of thread.processlist_host. scraper.go parses the result with net.SplitHostPort; Unix socket connections (no port) fall back gracefully to address-only with port 0. Assisted-by: Claude Sonnet 4.6
…esslist (MySQL 8.0.22+) client.port and network.peer.port on db.server.query_sample events were always 0 because performance_schema.threads.PROCESSLIST_HOST is host-only by design. performance_schema.processlist.HOST returns "host:port" for TCP/IP connections and is available on MySQL 8.0.22+. A LEFT JOIN on PROCESSLIST_ID = ID is added to the query sample templates when the supportsProcesslist() capability predicate is true. SUBSTRING_INDEX splits host and port in SQL; the port is scanned into a new clientPort field on the querySample struct and passed directly to RecordDbServerQuerySampleEvent. client.port and network.peer.port remain 0 on MariaDB (all versions) and MySQL <8.0.22 because the only other source that exposes host:port is information_schema.PROCESSLIST, which acquires a global mutex while iterating active threads — the same mutex held by SHOW PROCESSLIST. This has negative performance consequences on busy systems, was deprecated in MySQL 8.0, removed in MySQL 9.0, and was already removed from this receiver in a prior change. Assisted-by: Claude Sonnet 4.6
…o mysql-multi-version-support
….1 fallback support Remove the querySampleNoUserVars.tmpl template and associated code paths that handled servers lacking performance_schema.user_variables_by_thread. The minimum supported versions are now MySQL 5.7.x and MariaDB 10.5.x, both of which have this table unconditionally. Removes supportsUserVariablesByThread() predicate, the supportsUserVarsByThread parameter from getQuerySamples, and the fallback template embed. Updates README, tests, and integration test accordingly. Assisted-by: Claude Sonnet 4.6
…om COMPATIBILITY.md Remove the now-deleted predicate and querySampleNoUserVars.tmpl references. Also correct supportsQuerySampleText minimum to 8.0.3 (was 8.0+). Assisted-by: Claude Sonnet 4.6
…-version-support - README.md: collapse redundant MariaDB minor-version rows to 10.5.x–10.11.x and 11.x (LTS: 11.4, 11.8) - COMPATIBILITY.md: fill in Tested Platforms table with real live-test data (MySQL 8.4.7, 5.7.44, MariaDB 10.5.28, 11.8.2 — AWS RDS, 2026-04-21); add missing MariaDB 11.8 row; fix stale "all MariaDB 10.x" references - chloggen: correct minimum versions to MySQL 5.7.3+ and MariaDB 10.5.2+; remove erroneous MySQL 5.6 entry Assisted-by: Claude Sonnet 4.6
Assisted-by: Claude Sonnet 4.6
…plunk/opentelemetry-collector-contrib into mysql-multi-version-support
chloggen requires receiver/mysql, not receiver/mysqlreceiver. Assisted-by: Claude Sonnet 4.6
…o mysql-multi-version-support
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Extends the MySQL receiver to detect the database product and version at connect time and gate collection behavior accordingly, enabling use against MySQL 5.7.x and all supported MariaDB versions in addition to the existing MySQL 8+ support.
Version detection (
fetchDBVersion) runs once atConnect()time via a singleSELECT VERSION()query. Failure is non-fatal — the receiver falls back to MySQL <8 defaults and surfaces any connection error on the first scrape. The detected product (MySQLorMariaDB) and version (semver) are cached for the lifetime of the receiver instance.Capability predicates gate three behaviors:
supportsQuerySampleText()topQueryNoSampleText.tmpl(noquery_sample_textcolumn); EXPLAIN is skippedsupportsProcesslist()client.port/network.peer.portremain0;information_schema.PROCESSLISTis not used as fallback (global mutex, removed in MySQL 9.0)supportsReplicaStatus()SHOW SLAVE STATUSTimer wait tiers (
querySample.tmpl) — resolved in order:TIMER_WAITfor completed waits (all versions)statement.TIMER_WAITis updated only at yield points, not continuously)thread.processlist_timeinteger-second fallback (MySQL 5.7+, all MariaDB)Scope attributes —
db.versionanddb.productare stamped on the instrumentation scope of every emitted log record so consumers can identify the source product and version without parsing event attributes.EOL warning — a startup log warning is emitted when a MySQL version past end-of-life is detected.
New file
COMPATIBILITY.mddocuments every version-gated predicate, the timer wait tier logic, version detection semantics, and a tested-platforms matrix.Minimum supported versions: MySQL 5.7.3+, 8.0+; MariaDB 10.5.2+, 11.x.
Link to tracking issue
Fixes #47302
Testing
client_test.go):TestDBVersionCapabilitiescovers all three predicates across MySQL 8.0.27, 8.0.22, 8.0.21, 8.0.3, 8.0.2, 8.0.0, 5.7.44, MariaDB 10.11.6, 11.4.2, and the zero value.TestParseDBVersioncovers MySQL plain, MySQL with-logsuffix, MariaDB, MariaDB with5.5.5-compat prefix, and Debian-decorated MariaDB strings.TestGetDBVersionCachingandTestFetchDBVersionTimeoutcover the one-shot caching and startup-stall safeguard.scraper_test.go): mock client extended withdbVersionOverrideto simulate MySQL <8 and MariaDB; existing scraper tests updated to match new 14-field query sample fixture format.integration_test.go): newTestMySQLMultiVersionCapabilitiesruns against Docker images for MySQL 8.4, MySQL 5.7, MariaDB 10.5, and MariaDB 11.4. Each container is started, scraped, and verified for the correct capability flags, EOL warnings, and log record content.COMPATIBILITY.mdtested-platforms table.Documentation
receiver/mysqlreceiver/README.md: updated Prerequisites section with a supported-versions table; added Query plan availability by version section; addedevents_waits_currentenablement guide.receiver/mysqlreceiver/COMPATIBILITY.md: new file — capability predicate reference, timer wait tier table, version detection semantics, tested-platforms matrix,events_waits_currentconsumer setup guide.