Concurrent map access and a lock-ordering deadlock in the CloudEventSource event emitter

[PrazwalR]

Report

pkg/eventemitter/eventemitter.go has two concurrency defects in the CloudEventSource emit path.

1. eventHandlersCache is read without its lock

The map is guarded by eventHandlersCacheLock, and the writers take it correctly:

// createEventHandlers
e.eventHandlersCacheLock.Lock()
...
e.eventHandlersCache[eventHandlerKey] = eventHandler

// clearEventHandlersCache
delete(e.eventHandlersCache, eventHandlerKey)

But four reads take no lock at all:

Line	Function	Access
371	emitEventByHandler	e.eventHandlersCache[eventData.HandlerKey]
380	emitEventByHandler	for key, handler := range e.eventHandlersCache
404	emitEventByHandler	e.eventHandlersCache[eventData.HandlerKey]
416	emitErrorHandle	e.eventHandlersCache[eventData.HandlerKey]

Emit and checkIfEventHandlersExist do take RLock, so the omission looks accidental rather than deliberate.

These run on the startEventLoop goroutine (and on the handler callback goroutines spawned at line 395), while createEventHandlers / clearEventHandlersCache run on the CloudEventSource reconciler goroutine. Any CloudEventSource create, update or delete that overlaps event emission is a concurrent map read and write.

That is a Go runtime throw, not a panic — recover() cannot catch it, so the whole operator process dies. This is the same class as the map races already fixed in pkg/fallback (#7838 / #7843).

2. defer RUnlock inside a loop deadlocks the emit loop against the reconcilers

for key, handler := range e.eventHandlersCache {
    e.eventFilterCacheLock.RLock()
    defer e.eventFilterCacheLock.RUnlock()   // released only at function return

The deferred unlock means the read lock is still held on the next iteration. With two or more registered CloudEventSources:

1. The event loop acquires RLock in iteration 1 and keeps it.
2. createEventHandlers takes eventHandlersCacheLock.Lock() then blocks on eventFilterCacheLock.Lock(), waiting for the reader to drain.
3. Iteration 2 calls RLock() again. Go's sync.RWMutex bars new readers while a writer is waiting, so the event loop blocks — while still holding the read lock the writer is waiting on.

Neither goroutine can proceed. The writer still holds eventHandlersCacheLock, so Emit (which takes RLock on it) blocks forever too — and Emit is called synchronously from the ScaledObject, ScaledJob, TriggerAuthentication and ClusterTriggerAuthentication reconcilers. Those controller workers wedge one by one while the operator keeps passing its liveness probe.

Expected Behavior

Emitting events concurrently with CloudEventSource reconciles is safe: no runtime throw, and no deadlock between the emit loop and the reconcilers.

Actual Behavior

1. fatal error: concurrent map read and map write terminates the operator process, unrecoverable.
2. With two or more CloudEventSources, the emit loop and createEventHandlers can deadlock permanently, silently wedging every reconciler that calls Emit.

Steps to Reproduce the Problem

Both depend on goroutine scheduling, so they show up under load rather than on demand. What is directly verifiable is the structure:

1. grep -n "eventHandlersCache\[" pkg/eventemitter/eventemitter.go — the reads at lines 371, 404 and 416, plus the range at 380.
2. Compare with createEventHandlers and clearEventHandlersCache, which hold eventHandlersCacheLock for the corresponding writes.
3. grep -n "defer e.eventFilterCacheLock.RUnlock()" pkg/eventemitter/eventemitter.go — the deferred unlock sits inside the for loop.

KEDA Version

main

Kubernetes Version

None

Platform

Any

Anything else?

Reachable on any cluster with at least one CloudEventSource; the deadlock needs two or more.

Would you be open to contributing a fix?

Yes — I have a fix that takes both locks once before the loop, in the same order createEventHandlers uses, and routes the point lookups through a guarded helper. go test -race ./pkg/eventemitter/ passes with it.

[PrazwalR]

I'd like to pick this up — could I be assigned? I have a fix that takes both locks once before the loop, in the same order createEventHandlers uses, and routes the point lookups through a guarded helper. go test -race ./pkg/eventemitter/ passes with it.