fix(overlays): never block on modelsLock from under a model's effectLock

The effect update threads can reach PixelOverlayModelSub::setState() while
holding the model's effectLock (an effect stopping calls setState on its
model), and an unresolved submodel parent then made foundParent() block on
modelsLock via getModel(). Every HTTP and command path takes those locks
the other way - modelsLock for the lookup, then effectLock to inspect or
replace the running effect - so the two directions deadlock. Same shape as
37c8f92, one lock over, and worse: the effectLock side can run on the
channel output thread via updateRunningEffects(), wedging output as well as
the API. An unresolved parent is reachable at runtime because deleting a
model resets every child's cached parent to null.

Add PixelOverlayManager::tryGetModel(), a try_lock lookup, and use it for
parent resolution. Paths that already hold modelsLock (all the manager and
command paths) still resolve exactly as before - the recursive try_lock
always succeeds. A contended effect-side resolve now returns unresolved and
retries on the next setState instead of blocking, which leaves
modelsLock -> effectLock as the only ordering in the subsystem, so the
cycle can no longer close.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Author: dkulp

src/overlays/PixelOverlay.cpp

@@ -388,6 +388,14 @@ PixelOverlayModel* PixelOverlayManager::getModel(const std::string& name) {
     std::unique_lock<std::recursive_mutex> lock(modelsLock);
     return getModelLocked(name);
 }
+bool PixelOverlayManager::tryGetModel(const std::string& name, PixelOverlayModel*& model) {
+    std::unique_lock<std::recursive_mutex> lock(modelsLock, std::try_to_lock);
+    if (!lock.owns_lock()) {
+        return false;
+    }
+    model = getModelLocked(name);
+    return true;
+}
 PixelOverlayModel* PixelOverlayManager::getModelLocked(const std::string& name) {
     auto a = models.find(name);
     if (a != models.end()) {

src/overlays/PixelOverlay.h

@@ -40,6 +40,15 @@ class PixelOverlayManager {
 
     void addModel(Json::Value config);
     PixelOverlayModel* getModel(const std::string& name);
+    // Non-blocking lookup for callers that may hold a model's effectLock (the
+    // effect update threads).  modelsLock -> effectLock is the sanctioned
+    // order everywhere else; blocking on modelsLock from under an effectLock
+    // is the reverse edge and deadlocks against any HTTP/command path.  This
+    // try-acquires modelsLock instead: returns false with model untouched if
+    // the lock is contended (caller retries on a later call), true with the
+    // lookup result (possibly nullptr for an unknown name) otherwise.  A
+    // thread already holding modelsLock always succeeds (recursive).
+    bool tryGetModel(const std::string& name, PixelOverlayModel*& model);
 
     void addModelListener(const std::string& name, const std::string& id, std::function<void(PixelOverlayModel*)> listener);
     void removeModelListener(const std::string& name, const std::string& id);

src/overlays/PixelOverlayModelSub.cpp

@@ -145,19 +145,30 @@ PixelOverlayModelSub::~PixelOverlayModelSub() {
 
 // Resolve the parent model, caching the pointer.
 //
-// MUST NOT be called from the channel output thread. It takes modelsLock (via
-// getModel()), and the output thread reaches doOverlay() from
-// PixelOverlayManager::doOverlays() with activeModelsLock already held. Every
-// HTTP handler takes those two the other way round -- modelsLock for the model
-// lookup, then activeModelsLock when setState() reaches modelStateChanged() --
-// so acquiring them in this order deadlocks the two threads against each other.
-// setState() below is the only place that resolves, and it does so before the
-// model can become visible to the output thread at all.
+// MUST NOT be called from the channel output thread. The output thread reaches
+// doOverlay() from PixelOverlayManager::doOverlays() with activeModelsLock
+// already held, and every HTTP handler takes modelsLock before
+// activeModelsLock (via setState() -> modelStateChanged()), so a lookup from
+// that side would invert them. setState() below is the only caller, and the
+// resolution itself uses tryGetModel(): setState() can also run on the effect
+// update threads (an effect stopping calls model->setState() under this
+// model's effectLock), where BLOCKING on modelsLock is the reverse of the
+// HTTP/command paths' modelsLock -> effectLock order and deadlocks outright --
+// the same shape 37c8f92a8 fixed one lock over. The try-acquire never blocks:
+// from the manager paths (which already hold modelsLock recursively) it always
+// succeeds, and a contended effect-side resolve just stays unresolved until
+// the next setState().
 bool PixelOverlayModelSub::foundParent() {
     if (parent)
         return true;
 
-    parent = PixelOverlayManager::INSTANCE.getModel(config["Parent"].asString());
+    PixelOverlayModel* p = nullptr;
+    if (!PixelOverlayManager::INSTANCE.tryGetModel(config["Parent"].asString(), p)) {
+        // modelsLock contended and not ours; do not log -- the parent may be
+        // perfectly valid, we just can't look it up without risking deadlock.
+        return false;
+    }
+    parent = p;
 
     if (parent) {
         return true;