K8SPS-866: stop pt-heartbeat from starting during a clone - #1518
Draft
hors wants to merge 5 commits into
Draft
Conversation
When a new replica joins, MySQL copies the data with a clone. While the clone is running, the sys_operator database does not exist yet. The pt-heartbeat sidecar used to start too early and kept crashing with "Unknown database 'sys_operator'". Now the sidecar asks the database directly. It starts pt-heartbeat only after: - the clone is no longer "In Progress", and - the sys_operator database is present. If the data is still not ready when the timeout is reached, the sidecar exits instead of running against an incomplete database. How long to wait is now a real setting: spec.mysql.cloneTimeoutSeconds. It defaults to 21600 (6 hours), which is generous for very large (multi-TiB) datasets. The same value is passed to both the bootstrap clone (BOOTSTRAP_CLONE_TIMEOUT) and the sidecar wait (CLONE_TIMEOUT_SECONDS), so there is one setting to change instead of two hidden ones. api changes: add spec.mysql.cloneTimeoutSeconds
When a new replica joins, MySQL copies the data with a clone. While the clone is running, the sys_operator database does not exist yet. The pt-heartbeat sidecar used to start too early and kept crashing with "Unknown database 'sys_operator'". Now the sidecar asks the database directly and starts pt-heartbeat only after the clone is no longer "In Progress" and the sys_operator database is present. It waits for as long as that takes - a clone of a large dataset can run for hours, and waiting costs nothing. There is no wait timeout: a genuinely stuck replica is already surfaced by the mysql container's own readiness, so a timeout only added noise (a slow clone looked like a crash loop) and a value nobody could tune for multi-TiB datasets. The sidecar stops only on shutdown. A clone also finishes with a mandatory mysqld restart to finalize the data. The datadir looks ready a few seconds before that restart, so pt-heartbeat can start just in time to hit it and exit once with "Server shutdown in progress". To avoid a container restart, pt-heartbeat now runs under a small retry loop: if it exits shortly after starting, the script waits for MySQL to come back and starts it again in place. A pt-heartbeat that ran for a while before exiting, or that keeps exiting, is treated as a real failure and the container is allowed to restart. The clone operation itself still has a timeout (BOOTSTRAP_CLONE_TIMEOUT) as a safety net so a hung clone can abort and retry instead of hanging forever. Its default is generous (6h) and it stays configurable through spec.mysql.env for the rare very large dataset.
hors
changed the title
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The updated heartbeat entrypoint’s mysql output parsing/quoting can cause incorrect state detection (including infinite waiting) and the new 6h clone timeout can still be undercut by the existing 1h driver read timeout during CLONE INSTANCE.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR adjusts the pt-heartbeat sidecar startup behavior during replica bootstrap/cloning so it no longer crash-loops while the clone is still “In Progress” and required schemas (e.g., sys_operator) are not yet available, and it makes bootstrap clone operations less likely to fail on large datasets by increasing the default clone timeout.
Changes:
- Increase the default bootstrap clone timeout (BOOTSTRAP_CLONE_TIMEOUT) from 1h to 6h.
- Gate pt-heartbeat startup on MySQL clone state +
sys_operatorexistence, and add a bounded retry loop to avoid a container restart during the post-clone mysqld restart. - Gate emission of legacy
CLONE_TIMEOUT_SECONDSenv var to CR versions< 1.3.0, and add a unit test to ensure the gating behavior.
File summaries
| File | Description |
|---|---|
| pkg/mysql/mysql.go | Introduces a 6h default clone timeout constant and gates legacy heartbeat env injection by CR version. |
| pkg/mysql/mysql_test.go | Adds coverage verifying CLONE_TIMEOUT_SECONDS env is present pre-1.3.0 and dropped from 1.3.0+. |
| cmd/internal/db/db.go | Updates DB default clone timeout to 6h for bootstrap clone operations. |
| cmd/internal/db/db_test.go | Updates expected default clone timeout values to 6h. |
| build/heartbeat-entrypoint.sh | Reworks heartbeat startup gating to query clone state + schema presence and adds in-process retry logic around pt-heartbeat to avoid restart churn. |
Review details
Suppressed comments (1)
build/heartbeat-entrypoint.sh:136
- The MySQL wait loop uses an unquoted environment assignment (
MYSQL_PWD=${MYSQL_PASSWORD}) before invoking mysql. If the password contains spaces or shell-special characters, this can break the retry check and cause an unnecessary restart.
until [ "$shutdown_requested" -eq 1 ] || MYSQL_PWD=${MYSQL_PASSWORD} $MYSQL_CMDLINE -P$MYSQL_ADMIN_PORT -e 'SELECT 1;' >/dev/null 2>&1; do
sleep 2
- Files reviewed: 5/5 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
52
to
+55
| CLONE_STATUS=$(MYSQL_PWD=${MYSQL_PASSWORD} $MYSQL_CMDLINE -P$MYSQL_ADMIN_PORT -e 'SELECT STATE FROM performance_schema.clone_status;' | sed -n -e '2p' | tr -d '\n') | ||
| if [[ $CLONE_STATUS == "Completed" || -z $CLONE_IN_PROGRESS ]]; then | ||
| echo '[INFO] Clone completed, starting pt-heartbeat' | ||
| break | ||
| if [[ $CLONE_STATUS != "In Progress" ]]; then | ||
| HAS_SYS_OPERATOR=$(MYSQL_PWD=${MYSQL_PASSWORD} $MYSQL_CMDLINE -P$MYSQL_ADMIN_PORT -e "SELECT SCHEMA_NAME FROM information_schema.SCHEMATA WHERE SCHEMA_NAME='sys_operator';" | sed -n -e '2p' | tr -d '\n') | ||
| if [[ $HAS_SYS_OPERATOR == "sys_operator" ]]; then |
Comment on lines
60
to
62
| if p.CloneTimeoutSeconds == 0 { | ||
| p.CloneTimeoutSeconds = defs.DefaultCloneTimeoutSecondsSeconds // 1 hour for clone operations (large databases can take time) | ||
| p.CloneTimeoutSeconds = defs.DefaultCloneTimeoutSeconds // generous default; large databases can take hours to clone | ||
| } |
egegunes
reviewed
Aug 31, 2026
Comment on lines
+49
to
+53
| // DefaultCloneTimeoutSeconds is the default for the bootstrap clone timeout | ||
| // (BOOTSTRAP_CLONE_TIMEOUT) when it is not set via spec.mysql.env. It is | ||
| // generous (6h) because multi-TiB clones can run for hours; it is a safety | ||
| // net that lets a genuinely hung clone abort and retry, not a tuning knob. | ||
| DefaultCloneTimeoutSeconds = 21600 |
Contributor
There was a problem hiding this comment.
do we really need this safety net? we'll increase this to 6 hours and someone will definitely complain how 6 hours is not enough for their dataset size.
i think we need to disable timeout for clone operations and if we are worried that it might hung, find a better way to detect it
Collaborator
Author
There was a problem hiding this comment.
So, a fixed clone timeout is always a guess if too short aborts a real clone
of a large/slow dataset, too long delays noticing a hung one. Instead of
a fixed timeout, let's allow bootstrap watches to monitor the clone's progress. WDYT?
Collaborator
commit: 8e2edad |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a new async replica joins, MySQL copies the data with a clone. Two
problems around that clone are fixed here.
While the clone runs, the sys_operator database does not exist yet. The
pt-heartbeat sidecar started too early and kept crashing with
"Unknown database 'sys_operator'". It decided the data was ready by
looking at the clone.lock file, but a native InnoDB clone does not
reliably create that file, so the check was wrong.
Now the sidecar asks the database directly and starts pt-heartbeat only
after the clone is no longer "In Progress" and sys_operator is present.
It waits for as long as that takes - a large clone can run for hours and
waiting costs nothing; a genuinely stuck replica is already shown by the
mysql container's own readiness. There is no wait timeout to tune.
A clone also finishes with a mandatory mysqld restart to finalize the
data. The datadir looks ready for a few seconds before that restart, so
pt-heartbeat could start just in time to hit it and exit once with
"Server shutdown in progress". pt-heartbeat now runs under a small retry
loop: if it exits shortly after starting, the script waits for MySQL to
come back and starts it again in place, so a fresh replica no longer
shows a container restart. A pt-heartbeat that ran for a while and then
exits is treated as a real failure and the container is allowed to
restart.
A fixed clone timeout is always a guess: too short aborts a real clone
of a large/slow dataset, too long delays noticing a hung one. Instead of
a fixed timeout, the bootstrap now watches the clone's progress: it polls
how many bytes the clone has transferred (performance_schema.clone_progress)
and aborts the clone only if it makes no progress for a while
(BOOTSTRAP_CLONE_STALL_TIMEOUT, default 900s / 15 min). A clone that keeps
moving runs as long as it needs; a frozen one is aborted, which fails the
startup probe and lets kubelet restart the container and retry. This all
happens inside the pod, so it works with the operator down.
Set BOOTSTRAP_CLONE_STALL_TIMEOUT to 0 (via spec.mysql.env) to turn the
watchdog off and let the clone run with no progress check at all. The
startup probe timeout (which caps how long the clone may run) is raised
to 7 days so a long but progressing clone is not killed by kubelet.
All of the new clone behaviour is enabled only from crVersion 1.3.0.
Older clusters keep their previous behaviour, so upgrading just the
operator does not change their pod template or restart their pods.
CHECKLIST
Jira
Needs Doc) and QA (Needs QA)?Tests
Config/Logging/Testability