Symptom
InnoDB deadlocks on production between two batched Event::delete transactions:
|
Holds |
Waits for |
TXN 1 (SELECT * FROM Storage WHERE Id=2 FOR UPDATE) |
Event_Summaries MonitorId=8 |
Storage Id=2 |
TXN 2 (Month delete trigger UPDATE Event_Summaries) |
Storage Id=2 |
Event_Summaries MonitorId=8 |
Root cause
scripts/ZoneMinder/lib/ZoneMinder/Event.pm delete() deletes the Events row first — the delete triggers cascade into the bucket tables and X-lock the per-monitor Event_Summaries row — and only afterwards calls $storage->lock_and_load() (SELECT ... Storage FOR UPDATE).
Storage is shared across monitors, but Event_Summaries is per-monitor. In a caller-managed batch (AutoCommit off) that deletes events spanning several monitors on one storage area, the coarse Storage lock ends up held while a finer Event_Summaries lock for another monitor is requested — the reverse of a concurrent batch's order. Classic ABBA cycle.
The Storage row was never included in the documented "canonical lock-acquisition order" (Events -> buckets -> Event_Summaries); lock_and_load was bolted on at the end, inverting it.
Fix
Lock the coarse shared Storage row first, before the event/bucket/Event_Summaries deletes, giving a consistent global order Storage -> Events -> buckets -> Event_Summaries. Replace the lock_and_load + save read-modify-write with a single atomic UPDATE Storage SET DiskSpace = GREATEST(COALESCE(DiskSpace,0) - ?, 0) WHERE Id = ?. Align MoveTo similarly.
Symptom
InnoDB deadlocks on production between two batched
Event::deletetransactions:SELECT * FROM Storage WHERE Id=2 FOR UPDATE)Event_SummariesMonitorId=8StorageId=2UPDATE Event_Summaries)StorageId=2Event_SummariesMonitorId=8Root cause
scripts/ZoneMinder/lib/ZoneMinder/Event.pmdelete()deletes the Events row first — the delete triggers cascade into the bucket tables and X-lock the per-monitorEvent_Summariesrow — and only afterwards calls$storage->lock_and_load()(SELECT ... Storage FOR UPDATE).Storageis shared across monitors, butEvent_Summariesis per-monitor. In a caller-managed batch (AutoCommit off) that deletes events spanning several monitors on one storage area, the coarse Storage lock ends up held while a finer Event_Summaries lock for another monitor is requested — the reverse of a concurrent batch's order. Classic ABBA cycle.The
Storagerow was never included in the documented "canonical lock-acquisition order" (Events -> buckets -> Event_Summaries);lock_and_loadwas bolted on at the end, inverting it.Fix
Lock the coarse shared
Storagerow first, before the event/bucket/Event_Summaries deletes, giving a consistent global orderStorage -> Events -> buckets -> Event_Summaries. Replace thelock_and_load+saveread-modify-write with a single atomicUPDATE Storage SET DiskSpace = GREATEST(COALESCE(DiskSpace,0) - ?, 0) WHERE Id = ?. AlignMoveTosimilarly.