New eclipselink dead lock scenario ... enhancement and unit test (#2350) (#2449)

Fixes #2094
This is enhancement for case of possible deadlock which should happens in org.eclipse.persistence.internal.sessions.AbstractSession#getCacheKeyFromTargetSessionForMerge method if used org.eclipse.persistence.internal.identitymaps.CacheKey instance is locked by another thread.
There is new system or persistence property eclipselink.concurrency.manager.allow.getcachekeyformerge.mode which can control org.eclipse.persistence.internal.sessions.AbstractSession#getCacheKeyFromTargetSessionForMerge logic to get org.eclipse.persistence.internal.identitymaps.CacheKey instance and object value behind this.
There are two allowed values:

    ORIGIN (DEFAULT) - There is infinite java.lang.Object.wait() call in case of some conditions during time when object/entity referred from org.eclipse.persistence.internal.identitymaps.CacheKey is locked and modified by another thread. In some cases it should leads into deadlock.
    WAITLOOP - Merge manager will try in the loop with timeout wait cacheKey.wait(ConcurrencyUtil.SINGLETON.getAcquireWaitTime()); fetch object/entity from org.eclipse.persistence.internal.identitymaps.CacheKey. If fetch will be successful object/entity loop finish and continue
    with remaining code. If not java.lang.InterruptedException is thrown and caught and used org.eclipse.persistence.internal.identitymaps.CacheKey instance status is set into invalidation state. This strategy avoid deadlock issue, but there should be impact to the performance.


(cherry picked from commit dd61247)

Signed-off-by: Radek Felcman <radek.felcman@oracle.com>

Author: Radek Felcman

docs/docs.jpa/src/main/asciidoc/persistenceproperties_ref.adoc

@@ -1,6 +1,6 @@
 ///////////////////////////////////////////////////////////////////////////////
 
-    Copyright (c) 2022, 2024 Oracle and/or its affiliates. All rights reserved.
+    Copyright (c) 2022, 2025 Oracle and/or its affiliates. All rights reserved.
 
     This program and the accompanying materials are made available under the
     terms of the Eclipse Public License v. 2.0, which is available at
@@ -213,6 +213,7 @@ EclipseLink includes the following persistence property extensions for
 concurrency management:
 
 * link:#concurrency-manager-allow-concurrencyexception[`concurrency.manager.allow.concurrencyexception`]
+* link:#concurrency-manager-allow-getcachekeyformerge-mode[`concurrency.manager.allow.getcachekeyformerge.mode`]
 * link:#concurrency-manager-allow-interruptedexception[`concurrency.manager.allow.interruptedexception`]
 * link:#concurrency-manager-allow-readlockstacktrace[`concurrency.manager.allow.readlockstacktrace`]
 * link:#concurrency-manager-build-object-complete-waittime[`concurrency.manager.build.object.complete.waittime`]
@@ -286,6 +287,7 @@ The following lists the EclipseLink persistence property
 * link:#compositeunitmember[`composite-unit.member`]
 * link:#compositeunitproperties[`composite-unit.properties`]
 * link:#concurrency-manager-allow-concurrencyexception[`concurrency.manager.allow.concurrencyexception`]
+* link:#concurrency-manager-allow-getcachekeyformerge-mode[`concurrency.manager.allow.getcachekeyformerge.mode`]
 * link:#concurrency-manager-allow-interruptedexception[`concurrency.manager.allow.interruptedexception`]
 * link:#concurrency-manager-allow-readlockstacktrace[`concurrency.manager.allow.readlockstacktrace`]
 * link:#concurrency-manager-build-object-complete-waittime[`concurrency.manager.build.object.complete.waittime`]
@@ -3103,6 +3105,55 @@ persistence.xml_*
 
 '''''
 
+=== concurrency.manager.allow.getcachekeyformerge.mode
+
+This property control in `org.eclipse.persistence.internal.sessions.AbstractSession#getCacheKeyFromTargetSessionForMerge(java.lang.Object, org.eclipse.persistence.internal.descriptors.ObjectBuilder, org.eclipse.persistence.descriptors.ClassDescriptor, org.eclipse.persistence.internal.sessions.MergeManager)`
+strategy how `org.eclipse.persistence.internal.identitymaps.CacheKey` will be fetched from shared cache.
+
+
+*Values*
+
+link:#concurrency.manager.allow.getcachekeyformerge.mode[Table 5-30]
+describes this persistence property's values.
+
+[[concurrency.manager.allow.getcachekeyformerge.mode]]
+
+*_Table 5-30 Valid Values for
+concurrency.manager.allow.getcachekeyformerge.mode_*
+
+|=======================================================================
+|*Value* |*Description*
+|ORIGIN |(Default) There is infinite `java.lang.Object.wait()` call in case
+of some conditions during time when object/entity referred from
+`org.eclipse.persistence.internal.identitymaps.CacheKey` is locked
+and modified by another thread. In some cases it should leads into deadlock.
+
+|WAITLOOP |Merge manager will try in the loop with timeout wait `cacheKey.wait(ConcurrencyUtil.SINGLETON.getAcquireWaitTime());`
+fetch object/entity from `org.eclipse.persistence.internal.identitymaps.CacheKey`.
+If fetch will be successful object/entity loop finish and continue with remaining code.
+If not `java.lang.InterruptedException` is thrown and caught and used `org.eclipse.persistence.internal.identitymaps.CacheKey`
+instance status is set into invalidation state. This strategy avoid deadlock issue,
+but there should be impact to the performance.
+|=======================================================================
+
+*Examples*
+
+link:#concurrency.manager.allow.getcachekeyformerge.mode[Example
+5-24] shows how to use this property in the `persistence.xml` file.
+
+[[concurrency.manager.allow.getcachekeyformerge.mode]]
+
+*_Example 5-24 Using concurrency.manager.allow.getcachekeyformerge.mode in
+persistence.xml_*
+
+[source,oac_no_warn]
+----
+<property name="eclipselink.concurrency.manager.allow.getcachekeyformerge.mode" value="WAITLOOP" />
+
+----
+
+'''''
+
 === concurrency.manager.allow.concurrencyexception
 
 See valid values table.

foundation/org.eclipse.persistence.core/src/main/java/org/eclipse/persistence/config/MergeManagerOperationMode.java

@@ -0,0 +1,42 @@
+/*
+ * Copyright (c) 2025 Oracle and/or its affiliates. All rights reserved.
+ *
+ * This program and the accompanying materials are made available under the
+ * terms of the Eclipse Public License v. 2.0 which is available at
+ * http://www.eclipse.org/legal/epl-2.0,
+ * or the Eclipse Distribution License v. 1.0 which is available at
+ * http://www.eclipse.org/org/documents/edl-v10.php.
+ *
+ * SPDX-License-Identifier: EPL-2.0 OR BSD-3-Clause
+ */
+
+// Contributors:
+//     Oracle - initial API and implementation
+package org.eclipse.persistence.config;
+
+/**
+ * INTERNAL:
+ * <p>
+ * <b>Purpose</b>: It is data model behind {@linkplain  org.eclipse.persistence.config.SystemProperties#CONCURRENCY_MANAGER_ALLOW_GET_CACHE_KEY_FOR_MERGE_MODE}
+ * or {@linkplain  org.eclipse.persistence.config.PersistenceUnitProperties#CONCURRENCY_MANAGER_ALLOW_GET_CACHE_KEY_FOR_MERGE_MODE}.
+ */
+public final class MergeManagerOperationMode {
+
+    /**
+     * {@code ORIGIN} (DEFAULT) - There is infinite {@linkplain java.lang.Object#wait()} call in case of some conditions during time when object/entity referred from
+     * {@code org.eclipse.persistence.internal.identitymaps.CacheKey} is locked and modified by another thread. In some cases it should leads into deadlock.
+     */
+    public static final String ORIGIN = "ORIGIN";
+
+    /**
+     * {@code WAITLOOP} - Merge manager will try in the loop with timeout wait {@code cacheKey.wait(ConcurrencyUtil.SINGLETON.getAcquireWaitTime());}
+     * fetch object/entity from {@linkplain org.eclipse.persistence.internal.identitymaps.CacheKey}. If fetch will be successful object/entity loop finish and continue
+     * with remaining code. If not @{code java.lang.InterruptedException} is thrown and caught and used {@linkplain org.eclipse.persistence.internal.identitymaps.CacheKey} instance
+     * status is set into invalidation state. This strategy avoid deadlock issue, but there should be impact to the performance.
+     */
+    public static final String WAITLOOP = "WAITLOOP";
+
+    private MergeManagerOperationMode() {
+        // no instance please
+    }
+}

foundation/org.eclipse.persistence.core/src/main/java/org/eclipse/persistence/config/PersistenceUnitProperties.java

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 1998, 2024 Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1998, 2025 Oracle and/or its affiliates. All rights reserved.
  * Copyright (c) 1998, 2024 IBM Corporation. All rights reserved.
  *
  * This program and the accompanying materials are made available under the
@@ -4045,6 +4045,23 @@ public class PersistenceUnitProperties {
      */
     public static final String CONCURRENCY_MANAGER_ALLOW_INTERRUPTED_EXCEPTION  = "eclipselink.concurrency.manager.allow.interruptedexception";
 
+    /**
+     * <p>
+     * This property control in {@link org.eclipse.persistence.internal.sessions.AbstractSession#getCacheKeyFromTargetSessionForMerge(java.lang.Object, org.eclipse.persistence.internal.descriptors.ObjectBuilder, org.eclipse.persistence.descriptors.ClassDescriptor, org.eclipse.persistence.internal.sessions.MergeManager)}
+     * strategy how {@code org.eclipse.persistence.internal.identitymaps.CacheKey} will be fetched from shared cache.
+     * <p>
+     * <b>Allowed Values</b> (case-sensitive String)<b>:</b>
+     * <ul>
+     * <li>{@code ORIGIN} (DEFAULT) - There is infinite {@linkplain java.lang.Object#wait()} call in case of some conditions during time when object/entity referred from
+     * {@code org.eclipse.persistence.internal.identitymaps.CacheKey} is locked and modified by another thread. In some cases it should leads into deadlock.
+     * <li>{@code WAITLOOP} - Merge manager will try in the loop with timeout wait {@code cacheKey.wait(ConcurrencyUtil.SINGLETON.getAcquireWaitTime());}
+     * fetch object/entity from {@linkplain org.eclipse.persistence.internal.identitymaps.CacheKey}. If fetch will be successful object/entity loop finish and continue
+     * with remaining code. If not @{code java.lang.InterruptedException} is thrown and caught and used {@linkplain org.eclipse.persistence.internal.identitymaps.CacheKey} instance
+     * status is set into invalidation state. This strategy avoid deadlock issue, but there should be impact to the performance.
+     * </ul>
+     */
+    public static final String CONCURRENCY_MANAGER_ALLOW_GET_CACHE_KEY_FOR_MERGE_MODE  = "eclipselink.concurrency.manager.allow.getcachekeyformerge.mode";
+
     /**
      * <p>
      * This property control (enable/disable) if <code>ConcurrencyException</code> fired when dead-lock diagnostic is enabled.

foundation/org.eclipse.persistence.core/src/main/java/org/eclipse/persistence/config/SystemProperties.java

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 1998, 2024 Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 1998, 2025 Oracle and/or its affiliates. All rights reserved.
  * Copyright (c) 2024 Contributors to the Eclipse Foundation. All rights reserved.
  *
  * This program and the accompanying materials are made available under the
@@ -158,6 +158,23 @@ public class SystemProperties {
      */
     public static final String CONCURRENCY_MANAGER_ALLOW_INTERRUPTED_EXCEPTION  = "eclipselink.concurrency.manager.allow.interruptedexception";
 
+    /**
+     * <p>
+     * This property control in {@link org.eclipse.persistence.internal.sessions.AbstractSession#getCacheKeyFromTargetSessionForMerge(java.lang.Object, org.eclipse.persistence.internal.descriptors.ObjectBuilder, org.eclipse.persistence.descriptors.ClassDescriptor, org.eclipse.persistence.internal.sessions.MergeManager)}
+     * strategy how {@code org.eclipse.persistence.internal.identitymaps.CacheKey} will be fetched from shared cache.
+     * <p>
+     * <b>Allowed Values</b> (case-sensitive String)<b>:</b>
+     * <ul>
+     * <li>{@code ORIGIN} (DEFAULT) - There is infinite {@linkplain java.lang.Object#wait()} call in case of some conditions during time when object/entity referred from
+     * {@code org.eclipse.persistence.internal.identitymaps.CacheKey} is locked and modified by another thread. In some cases it should leads into deadlock.
+     * <li>{@code WAITLOOP} - Merge manager will try in the loop with timeout wait {@code cacheKey.wait(ConcurrencyUtil.SINGLETON.getAcquireWaitTime());}
+     * fetch object/entity from {@linkplain org.eclipse.persistence.internal.identitymaps.CacheKey}. If fetch will be successful object/entity loop finish and continue
+     * with remaining code. If not @{code java.lang.InterruptedException} is thrown and caught and used {@linkplain org.eclipse.persistence.internal.identitymaps.CacheKey} instance
+     * status is set into invalidation state. This strategy avoid deadlock issue, but there should be impact to the performance.
+     * </ul>
+     */
+    public static final String CONCURRENCY_MANAGER_ALLOW_GET_CACHE_KEY_FOR_MERGE_MODE  = "eclipselink.concurrency.manager.allow.getcachekeyformerge.mode";
+
     /**
      * <p>
      * This property control (enable/disable) if {@code ConcurrencyException} fired when dead-lock diagnostic is enabled.

foundation/org.eclipse.persistence.core/src/main/java/org/eclipse/persistence/internal/helper/ConcurrencyManager.java

@@ -19,8 +19,6 @@
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.Date;
-import java.util.HashMap;
-import java.util.HashSet;
 import java.util.IdentityHashMap;
 import java.util.Iterator;
 import java.util.List;
@@ -864,7 +862,7 @@ public void releaseAllLocksAcquiredByThread(DeferredLockManager lockManager) {
      * @return Never null if the read lock manager does not yet exist for the current thread. otherwise its read log
      *         manager is returned.
      */
-    protected static ReadLockManager getReadLockManager(Thread thread) {
+    public static ReadLockManager getReadLockManager(Thread thread) {
         Map<Thread, ReadLockManager> readLockManagers = getReadLockManagers();
         return readLockManagers.get(thread);
     }
@@ -1122,4 +1120,28 @@ public Lock getInstanceLock() {
     public Condition getInstanceLockCondition() {
         return this.instanceLockCondition;
     }
+
+    /**
+     * Check if {@code org.eclipse.persistence.internal.helper.ConcurrencyManager} or child like {@code org.eclipse.persistence.internal.identitymaps.CacheKey} is currently being owned for writing
+     * and if that owning thread happens to be the current thread doing the check.
+     *
+     * @return {@code false} means either the thread is currently not owned by any other thread for writing purposes. Or otherwise if is owned by some thread
+     * but the thread is not the current thread. {@code false} is returned if and only if instance is being owned by some thread
+     * and that thread is not the current thread, it is some other competing thread.
+     */
+    public boolean isAcquiredForWritingAndOwnedByDifferentThread() {
+        instanceLock.lock();
+        try {
+            if (!this.isAcquired()) {
+                return false;
+            }
+            if (this.activeThread == null) {
+                return false;
+            }
+            Thread currentThread = Thread.currentThread();
+            return this.activeThread != currentThread;
+        } finally {
+            instanceLock.unlock();
+        }
+    }
 }