Skip to content

Commit b09b523

Browse files
jaideepkarandejaideepkarande
committed
PXC-5203: Cluster disruption or deadlock with LOCK TABLES/INSTANCE FOR BACKUP misuse
https://perconadev.atlassian.net/browse/PXC-5203 A session that ran its own DDL/DML while holding LOCK TABLES FOR BACKUP was replicated via TOI before the local backup-lock check rejected it, so peers applied the DDL while the originator failed it locally -- causing a Galera "inconsistency detected" vote and node eviction. Resolution: Similar issue was already fixed for "LOCK INSTANCE FOR BACKUP" extended the condition for "LOCK TABLES FOR BACKUP"
1 parent 287713b commit b09b523

3 files changed

Lines changed: 155 additions & 1 deletion

File tree

‎mysql-test/suite/galera/r/pxc_backup_lock_toi_reject.result‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
#
2+
# 1. Create table t1 on node_1, wait for it on node_2.
3+
#
4+
CREATE TABLE t1 (id INT PRIMARY KEY, a INT, KEY k_a (a)) ENGINE=InnoDB;
5+
INSERT INTO t1 VALUES (1, 10), (2, 20), (3, 30);
6+
#
7+
# 2. LOCK TABLES FOR BACKUP, ALTER TABLE will fail,
8+
# Assert transaction is not replicated.
9+
#
10+
LOCK TABLES FOR BACKUP;
11+
ALTER TABLE t1 DROP KEY k_a;
12+
ERROR HY000: Can't execute the query because you have a conflicting backup lock
13+
include/assert.inc [k_a index must still exist on node_1 (statement rejected locally)]
14+
include/assert.inc [k_a index must still exist on node_2 -- nothing was replicated]
15+
include/assert.inc [Cluster must still have both nodes (no inconsistency vote / node eviction)]
16+
UNLOCK TABLES;
17+
#
18+
# 3. After UNLOCK, the same ALTER TABLE succeeds and replicates normally.
19+
#
20+
ALTER TABLE t1 DROP KEY k_a;
21+
include/assert.inc [k_a index must be dropped on node_2 after UNLOCK TABLES]
22+
#
23+
# 4. LOCK INSTANCE FOR BACKUP with same-session DDL is replicated.
24+
#
25+
ALTER TABLE t1 ADD KEY k_a (a);
26+
LOCK INSTANCE FOR BACKUP;
27+
ALTER TABLE t1 DROP KEY k_a;
28+
UNLOCK INSTANCE;
29+
include/assert.inc [k_a index must be dropped on node_1 -- LOCK INSTANCE FOR BACKUP allows same-session DDL]
30+
#
31+
# 5. Cleanup.
32+
#
33+
DROP TABLE t1;

‎mysql-test/suite/galera/t/pxc_backup_lock_toi_reject.test‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
################################################################################
2+
#
3+
# PXC-5203: A session holding LOCK TABLES FOR BACKUP that then runs its own
4+
# DDL used to have the DDL replicated via Galera TOI *before* the local
5+
# backup-lock check rejected it. Peers applied the DDL successfully while the
6+
# originating node failed the statement locally -- a certification-vs-local
7+
# result divergence that made Galera raise an "Inconsistency detected" vote
8+
# and kick the node out of the cluster.
9+
#
10+
# This test proves the DDL is now rejected immediately, before it ever
11+
# reaches TOI/replication, so peers never see it and no divergence occurs.
12+
# It also proves LOCK INSTANCE FOR BACKUP (the ticket's "no problem" case)
13+
# keeps working exactly as before: same-session DDL still succeeds and
14+
# still replicates normally.
15+
#
16+
################################################################################
17+
# 1. Create table t1 on node_1, wait for it on node_2.
18+
# 2. LOCK TABLES FOR BACKUP + same-session ALTER TABLE fails immediately with
19+
# ER_CANT_EXECUTE_WITH_BACKUP_LOCK; the index change never reaches node_2.
20+
# 3. After UNLOCK, the same ALTER TABLE succeeds and replicates normally.
21+
# 4. LOCK INSTANCE FOR BACKUP with same-session DDL is replicated.
22+
# 5. Cleanup.
23+
################################################################################
24+
--source include/galera_cluster.inc
25+
26+
--echo #
27+
--echo # 1. Create table t1 on node_1, wait for it on node_2.
28+
--echo #
29+
30+
--connection node_1
31+
CREATE TABLE t1 (id INT PRIMARY KEY, a INT, KEY k_a (a)) ENGINE=InnoDB;
32+
INSERT INTO t1 VALUES (1, 10), (2, 20), (3, 30);
33+
34+
--connection node_2
35+
--let $wait_condition = SELECT COUNT(*) = 1 FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 't1'
36+
--source include/wait_condition.inc
37+
38+
--echo #
39+
--echo # 2. LOCK TABLES FOR BACKUP, ALTER TABLE will fail,
40+
--echo # Assert transaction is not replicated.
41+
--echo #
42+
43+
--connection node_1
44+
LOCK TABLES FOR BACKUP;
45+
--error ER_CANT_EXECUTE_WITH_BACKUP_LOCK
46+
ALTER TABLE t1 DROP KEY k_a;
47+
48+
--let $assert_text = k_a index must still exist on node_1 (statement rejected locally)
49+
--let $assert_cond = COUNT(*) = 1 FROM information_schema.statistics WHERE TABLE_NAME="t1" AND INDEX_NAME="k_a" AND TABLE_SCHEMA=DATABASE()
50+
--source include/assert.inc
51+
52+
--connection node_2
53+
--let $assert_text = k_a index must still exist on node_2 -- nothing was replicated
54+
--let $assert_cond = COUNT(*) = 1 FROM information_schema.statistics WHERE TABLE_NAME="t1" AND INDEX_NAME="k_a" AND TABLE_SCHEMA=DATABASE()
55+
--source include/assert.inc
56+
57+
--let $assert_text = Cluster must still have both nodes (no inconsistency vote / node eviction)
58+
--let $assert_cond = (SELECT VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME="wsrep_cluster_size") = 2
59+
--source include/assert.inc
60+
61+
--connection node_1
62+
UNLOCK TABLES;
63+
64+
--echo #
65+
--echo # 3. After UNLOCK, the same ALTER TABLE succeeds and replicates normally.
66+
--echo #
67+
68+
ALTER TABLE t1 DROP KEY k_a;
69+
70+
--connection node_2
71+
--let $assert_text = k_a index must be dropped on node_2 after UNLOCK TABLES
72+
--let $assert_cond = COUNT(*) = 0 FROM information_schema.statistics WHERE TABLE_NAME="t1" AND INDEX_NAME="k_a" AND TABLE_SCHEMA=DATABASE()
73+
--source include/assert.inc
74+
75+
--echo #
76+
--echo # 4. LOCK INSTANCE FOR BACKUP with same-session DDL is replicated.
77+
--echo #
78+
79+
--connection node_1
80+
ALTER TABLE t1 ADD KEY k_a (a);
81+
82+
--connection node_2
83+
--let $wait_condition = SELECT COUNT(*) = 1 FROM information_schema.statistics WHERE TABLE_NAME="t1" AND INDEX_NAME="k_a" AND TABLE_SCHEMA=DATABASE()
84+
--source include/wait_condition.inc
85+
86+
--connection node_1
87+
LOCK INSTANCE FOR BACKUP;
88+
ALTER TABLE t1 DROP KEY k_a;
89+
UNLOCK INSTANCE;
90+
91+
--let $assert_text = k_a index must be dropped on node_1 -- LOCK INSTANCE FOR BACKUP allows same-session DDL
92+
--let $assert_cond = COUNT(*) = 0 FROM information_schema.statistics WHERE TABLE_NAME="t1" AND INDEX_NAME="k_a" AND TABLE_SCHEMA=DATABASE()
93+
--source include/assert.inc
94+
95+
--connection node_2
96+
--let $wait_condition = SELECT COUNT(*) = 0 FROM information_schema.statistics WHERE TABLE_NAME="t1" AND INDEX_NAME="k_a" AND TABLE_SCHEMA=DATABASE()
97+
--source include/wait_condition.inc
98+
99+
--echo #
100+
--echo # 5. Cleanup.
101+
--echo #
102+
103+
--connection node_1
104+
DROP TABLE t1;

‎sql/wsrep_mysqld.cc‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3061,6 +3061,23 @@ int wsrep_to_isolation_begin(THD *thd, const char *db_, const char *table_,
30613061
return -1;
30623062
}
30633063

3064+
/*
3065+
Acquire Percona's LOCK TABLES FOR BACKUP lock.
3066+
LOCK TABLES FOR BACKUP (Percona's BACKUP_TABLES lock) forbids the holding
3067+
session from running its own DDL/DML (see the abort_if_acquired() guards
3068+
in sql_alter_instance.cc, sql_tablespace.cc, sql_base.cc). Unlike
3069+
LOCK INSTANCE FOR BACKUP below, this rejection must happen here, before
3070+
TOI replication, otherwise the statement would already be certified and
3071+
applied on peers by the time the local abort_if_acquired() check inside
3072+
the statement executor rejects it, causing the nodes to diverge and
3073+
Galera to raise an inconsistency vote.
3074+
*/
3075+
if (thd->backup_tables_lock.abort_if_acquired() ||
3076+
thd->backup_tables_lock.acquire_protection(
3077+
thd, MDL_TRANSACTION, thd->variables.lock_wait_timeout)) {
3078+
return -1;
3079+
}
3080+
30643081
/*
30653082
Acquire an intention exclusive lock to protect against others setting the
30663083
global read_only and error out if the server is already read only.
@@ -3642,4 +3659,4 @@ std::string wsrep_fix_received_query(const char *query, size_t query_len) {
36423659
}
36433660

36443661
return query_ret;
3645-
}
3662+
}

0 commit comments

Comments
 (0)