Collection of fixes for macOS - #5654
Open
DaveGosselin-MariaDB wants to merge 12 commits into
Open
DaveGosselin-MariaDB wants to merge 12 commits into
DaveGosselin-MariaDB wants to merge 12 commits into
Conversation
DaveGosselin-MariaDB
force-pushed
the
11.4-macos-testing-fixes
branch
from
September 9, 2026 18:49
cffcb3a to
343dc55
Compare
gkodinov
approved these changes
Sep 11, 2026
gkodinov
left a comment
gkodinov
left a comment
Member
There was a problem hiding this comment.
Looks mostly good. Some small improvements suggested below.
Please get the innodb team to review as well.
dr-m
reviewed
Sep 11, 2026
Comment on lines
+269
to
+271
| /** the number of bytes allocated for tmp_buf, which is tmp_buf_size | ||
| rounded up to a multiple of the large page size */ | ||
| size_t tmp_buf_alloc_size; |
Contributor
There was a problem hiding this comment.
This would increase sizeof recv_sys. Is this really needed on all platforms (certainly not on Windows), or at all? What damage is done if the core dump exclusion is only requested on the requested size, and anything after it is allowed to be included in core dumps?
Does macOS even support any madvise(2) option that would allow the memory to be omitted from a core dump?
What if we just allocate recv_sys.tmp_buf as regular memory on macOS? It will be freed after crash recovery is completed or aborted.
Member
Author
There was a problem hiding this comment.
This change is not really needed on Windows, but the imbalance is not specific to macOS. Likely any platform where the rounding happens is exposed to this problem.
on macOS ut_dontdump and ut_dodump are empty functions because their bodies compile only under DBUG_OFF && HAVE_MADVISE && MADV_DODUMP; while macOS has madvise(2), it has no "advice" that omits memory from a core dump. MADV_DONTDUMP and MADV_DODUMP don't exist on mac.
Allocating tmp_buf as regular memory could work. The tradeoff is scope. It could also remove the only macOS dependency on large pages. That means that the fix likely becomes an #ifdef __APPLE__ in log0recv.cc. Doing it on every platform is smaller.
At lower_case_table_names=2 this returns nothing. CREATE DATABASE Db1; CREATE FUNCTION Db1.f1(a INT) RETURNS INT RETURN a; SELECT ROUTINE_NAME FROM information_schema.ROUTINES WHERE ROUTINE_SCHEMA='Db1'; mysql.proc records the function's database as db1, in lower case. Creating a routine lower-cases its database name whenever lower_case_table_names is anything but 0, at sql/sp_head.h:121. The datadir, SCHEMATA and DATABASE() all keep Db1. CALL Db1.f1() still works, because calling a routine lower-cases the database name too and then searches mysql.proc for db1. The query above never lower-cases it. It searches for Db1, and mysql.proc.db collates utf8mb3_bin, so the comparison runs byte for byte and no row matches. At setting 1 the server lower-cases the filter value as well, at sql/sql_show.cc:4394, and lower-cases every name it stores, so the query and the table always agree. Setting 2 lower-cases the routine's copy and nothing else. The fix lower-cases the filter value before the search. Sorting the same query brings the row back. SELECT ROUTINE_NAME FROM information_schema.ROUTINES WHERE ROUTINE_SCHEMA='Db1' ORDER BY ROUTINE_NAME; The sort keeps the filter from reaching that search. The server reads all of mysql.proc instead, then applies the WHERE to ROUTINE_SCHEMA, which compares case insensitively. That shape answered correctly all along. The same search fills PARAMETERS and backs SHOW FUNCTION STATUS, SHOW PROCEDURE STATUS, SHOW PACKAGE STATUS and SHOW PACKAGE BODY STATUS. Every one returned nothing for Db1. mariadb-dump lists routines with SHOW FUNCTION STATUS WHERE Db=..., at client/mysqldump.cc:2859, which is the main.mysqldump failure. Setting 0 keeps Db1 and db1 as two databases holding two routines. A case sensitive volume confirms both stay distinct before and after this change. beb9a54 (MDEV-20609) added the search in 10.11.1. main.lowercase_routines runs both query shapes.
With lower_case_table_names 0 the server can have databases Db1 and db1, each with a function f1. A single statement naming both databases, like SELECT Db1.f1(), db1.f1(), reported that db1.f1 does not exist. The set of routines a statement uses compared its entries without regard to case. Only one routine was loaded but the reference to the other found nothing. The set now compares its entries exactly, as the routine cache and the lock manager already do.
macOS declares select() in sys/select.h, which the HAVE_SELECT probe did not include. clang rejects a call to an undeclared function, so the probe failed and HAVE_SELECT was left undefined. my_sleep() then took its last fallback, a busy loop on time() that rounds the requested interval up to a whole second. Every sub-second sleep in the server became a one second spin on a CPU, which is what made rpl.rpl_perfschema_applier_status_by_worker, rpl.rpl_shutdown_sighup and rpl.rpl_semi_sync_shutdown_await_ack fail.
macOS puts the data directory on a case insensitive file system, so lower_case_table_names is 2 and both tests recorded an answer that only holds for 0. period.i_s_notembedded looked up I_S.PERIODS and I_S.KEY_PERIOD_USAGE by the schema name TEST. That comparison follows the table name comparison, so it finds the table under 1 and 2 and finds nothing under 0. Those four queries move to the new test period.i_s_case_sensitive, which requires lower_case_table_names=0. The win rdiff of period.i_s_notembedded covered the same difference and is no longer needed. atomic.drop_db_long_names generated table and view names in upper case and compared the DROP statements that DDL recovery writes to the binary log. Under 2 the names come back from the directory in lower case. Generating them in lower case to begin with gives the same names on every setting. Lower case also changes where the view name sorts against its table name for the letters after v, which moves one view between two of the recorded DROP VIEW statements.
Introduces a new MTR include, not_mac.inc, which when included at the top of a test, prevents that test from running on macOS. sys_vars.sysvars_readonly_debug is the first user. It expects the server to fault when a read only sysvar is written behind the sysvar interface. That protection needs the ro_after_init section, which a linker script places and ld64 has no option to take, so HAVE_RO_AFTER_INIT stays undefined on macOS. Without it no variable is moved into the read only root either, so neither of the two assignments is refused.
Its default value depends on the operating system, ON where the log can be memory mapped and OFF elsewhere, so the recorded row only holds on some platforms. The other variables whose default depends on the operating system are already excluded the same way.
DaveGosselin-MariaDB
force-pushed
the
11.4-macos-testing-fixes
branch
from
September 14, 2026 20:44
343dc55 to
f8e8b99
Compare
dr-m
reviewed
Sep 15, 2026
| # Some operating systems or file systems do not support sparse files. | ||
| # For example, tmpfs on FreeBSD does not support them. | ||
| # On Microsoft Windows, sparse files have to be created in a special way. | ||
| --source include/big_test.inc |
Contributor
There was a problem hiding this comment.
The comment was a justification for disabling the test on most builders. The test itself mightnot be big, only the logical file size is. You removed the justification but not the guard that you would replace with a more accurate one have_sparse_files.inc.
Please remove the big_test.inc and wait for the complete results in the grid view since the test would be enabled in more environments. Also try to check the execution time in a few environments. I think that anything over 45 or 60 seconds should be considered to be a "big test". However, we must keep in mind that "big tests" only run on very few environments.
Member
Author
There was a problem hiding this comment.
Thanks, yes, my mistake. After I push the latest changes I'll check the grid view for any log_upgrade test failures and sample the execution time to be sure that this is not a "big test" in terms of run time.
Member
Author
There was a problem hiding this comment.
I sampled a few environments in buildbot where the log_upgrade and log_upgrade_101_flags tests ran, here are the results. Times as reported by MTR. I think it's safe to remove the 'big test' flag based on these.
| Environment | log_upgrade | log_upgrade_101_flags |
|---|---|---|
| amd64-ubuntu-2204-bigtest | 2673 | 2909 |
| amd64-windows | 1607 | 1638 |
| aarch64-macos | 3624 | 3967 |
The test replaces the number of buffer pool blocks with a fixed value so that the message is stable. The pattern only accepted 5.., and macOS builds without a futex use SUX_LOCK_GENERIC, which enlarges buf_block_t enough to bring the count down into 4...
The injected deadlock reaches the client as ER_GET_ERRNO carrying errno 11, and the text comes from my_strerror(). 11 is EAGAIN on Linux and EDEADLK on macOS, so the message reads "Resource temporarily unavailable" on one and "Resource deadlock avoided" on the other. Replace the quoted text so the test does not depend on it.
The client reports why it could not load client_ed25519, and macOS names every path that dlopen() tried. Two expressions are added, one for the chunk that holds the start of that message and one for the chunks that continue it. Whether the message arrives in one chunk or several depends on the vardir, because the path appears four times in the dlopen text. With --vardir /Volumes/<repo>/var the line is 417 bytes and fits the 512 byte buffer that --exec output is read in. With the default vardir it does not. Both expressions stop at a newline. reg_replace compiles with REG_DOTALL, so an unrestricted .* runs past the line terminator whenever the whole message reaches the replacement in one chunk, and the error line then joins the line after it.
main.large_pages fails on macOS with "Warning: Memory not freed: 16375" at shutdown and no accompanying safemalloc report. The residual stays at 16375 whether innodb_buffer_pool_size is 8M or 128M, and dropping --large-pages makes it go away. recv_sys_t::find_checkpoint() asks for tmp_buf_size, which is MTR_SIZE_MAX + 9, or 1048585 bytes. my_large_malloc() rounds that up to a multiple of the large page size and charges the rounded figure to global_memory_used, while recv_sys_t::tmp_free() credits back the 1048585 that was requested. The page size here is 16384, 1048585 rounds up to 1064960, and the difference is the 16375 reported. The caller cannot see the rounded figure because ut_malloc_dontdump() takes the size by value and, with a null ut_new_pfx_t, has nowhere to report what it allocated. ut_malloc_dontdump_size() writes the size back, and recv_sys_t keeps it in tmp_buf_alloc_size for the free. tmp_buf_size remains the capacity that parse() asserts against. Only macOS rounds up. my_get_large_page_sizes() has no huge page interface to consult there, so its fallback branch reports the ordinary page size as the only large page size and the plain mmap() always succeeds. On Linux the candidate is 2 MiB, the MAP_HUGETLB mapping fails with ENOMEM when no huge pages are reserved, and the retry loop settles on large_page_size == 0, which records the request unrounded. No memory was lost either way, since munmap() rounds its length up to a whole page. The counter was wrong, and the counter is what MTR checks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
innodb.log_upgrade and innodb.log_upgrade_101_flags build 8GB redo log files by seeking past the end of an empty file and writing a single byte. That needs a filesystem which leaves the skipped range unallocated. HFS on macOS allocates every block of it instead, so the write fails with ENOSPC and the test reports a perl failure. include/have_sparse_files.inc probes a directory the caller names, writing one byte 64MB into an empty file there and comparing the allocated block count against that offset. Both tests name the vardir tmp, where they build their redo log files. The offset stays above 16MB since APFS allocates the whole range for a file smaller than that rather than recording a hole.
The wait timer can have a granularity coarser than the time an uncontended read lock is held, so the recorded duration of one lock can be zero, which reads back as NULL. This can cause the test to fail with a false negative. Take the lock twenty more times at each measurement point, with the extra statements silent so the recorded result does not change. The mutex part of the test already works this way, since one SELECT produces ten THR_LOCK::mutex events. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DaveGosselin-MariaDB
force-pushed
the
11.4-macos-testing-fixes
branch
from
September 15, 2026 14:12
f8e8b99 to
12dedca
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Each commit in this collection fixes a test failure specific to macOS.
Three commits, "MDEV-33616: Charge and credit the same size for the recovery buffer", "MDEV-33616: Only one of two routines named in a statement is found", and "MDEV-33616: Routines of a mixed case database are not listed" change the server code.
Each commit has its own explanation in its commit message.