Description
Found a deadlock. Here's what happens, as near as I can make out:
D is invoked to change e.g. an instance variable V on owner O. It locks on itself, and starts to make a Shadow of O. It calls getFields() on O's class. This has not yet happened, so the fields are not cached, and the Class delegates to DebugifyingClassLoader, attempting to lock it.
In the meantime, though, another thread is loading classes. DebugifyingClassLoader is locked to load the classes, and tries to call Debugify.debugifyClass(). This attempts to lock D, which results in deadlock.
Here's a trace of the deadlock, slightly anonymized:
Thread [main] (Suspended)
owns: Object (id=93)
owns: DebugifyingClassLoader (id=84)
waited by: Thread [qtp1199720022-30-acceptor-0@771ed183-ServerConnector@1f9bf19f{SSL,[ssl, http/1.1]}{0.0.0.0:4439}] (Suspended)
waiting for: Class<T> (com.lambda.Debugger.D) (id=85)
owned by: Thread [qtp1199720022-30-acceptor-0@771ed183-ServerConnector@1f9bf19f{SSL,[ssl, http/1.1]}{0.0.0.0:4439}] (Suspended)
Debugify.debugifyClass(JavaClass, String) line: 293
DebugifyingClassLoader.findClass(String, boolean) line: 183
DebugifyingClassLoader.loadClass(String, boolean) line: 112
DebugifyingClassLoader(ClassLoader).loadClass(String) line: 357
Class<T>.getDeclaredFields0(boolean) line: not available [native method]
Class<T>.privateGetDeclaredFields(boolean) line: 2583
Class<T>.getDeclaredFields() line: 1916
ClassReflectionHelperImpl$8.run() line: 166
ClassReflectionHelperImpl$8.run() line: 162
AccessController.doPrivileged(PrivilegedAction<T>) line: not available [native method]
ClassReflectionHelperImpl.secureGetDeclaredFields(Class<?>) line: 162
ClassReflectionHelperImpl.getDeclaredFieldWrappers(Class<?>) line: 198
ClassReflectionHelperImpl.getAllFieldWrappers(Class<?>) line: 215
ClassReflectionHelperImpl$4.compute(Class<?>) line: 109
ClassReflectionHelperImpl$4.compute(Object) line: 105
LRUHybridCache$OriginThreadAwareFuture$1.call() line: 115
LRUHybridCache$OriginThreadAwareFuture$1.call() line: 111
FutureTask<V>.run() line: 266
LRUHybridCache$OriginThreadAwareFuture.run() line: 173
LRUHybridCache<K,V>.compute(K) line: 292
ClassReflectionHelperImpl.getAllFields(Class<?>) line: 333
Utilities.findInitializerFields(Class<?>, ServiceLocatorImpl, Collector) line: 1410
DefaultClassAnalyzer.getFields(Class<T>) line: 113
JerseyClassAnalyzer.getFields(Class<T>) line: 244
Utilities.getInitFields(Class<?>, ClassAnalyzer, Collector) line: 251
ClazzCreator<T>.initialize(ActiveDescriptor<?>, String, Collector) line: 157
ClazzCreator<T>.initialize(ActiveDescriptor<?>, Collector) line: 182
SystemDescriptor<T>.internalReify(Class<?>, Collector) line: 723
SystemDescriptor<T>.reify(Class<?>, Collector) line: 678
ServiceLocatorImpl.reifyDescriptor(Descriptor, Injectee) line: 416
Utilities.createService(ActiveDescriptor<T>, Injectee, ServiceLocatorImpl, ServiceHandle<T>, Class<?>) line: 2029
ServiceHandleImpl<T>.getService(ServiceHandle<T>) line: 105
ServiceHandleImpl<T>.getService() line: 87
FactoryCreator<T>.create(ServiceHandle<?>, SystemDescriptor<?>) line: 117
SystemDescriptor<T>.create(ServiceHandle<?>) line: 471
SingletonContext$1.compute(ContextualInput<Object>) line: 82
SingletonContext$1.compute(Object) line: 70
Cache$OriginThreadAwareFuture$1.call() line: 97
FutureTask<V>.run() line: 266
Cache$OriginThreadAwareFuture.run() line: 154
Cache<K,V>.compute(K) line: 199
SingletonContext.findOrCreate(ActiveDescriptor<T>, ServiceHandle<?>) line: 121
Utilities.createService(ActiveDescriptor<T>, Injectee, ServiceLocatorImpl, ServiceHandle<T>, Class<?>) line: 2064
ServiceLocatorImpl.internalGetService(Type, String, Unqualified, Annotation...) line: 711
ServiceLocatorImpl.getService(Class<T>, Annotation...) line: 653
... main methods ...
NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]
NativeMethodAccessorImpl.invoke(Object, Object[]) line: 62
DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
Method.invoke(Object, Object...) line: 498
Debugger.runTarget(Class, Object[]) line: 1141
Debugger$1.run() line: 1199
Thread.run() line: 748
Thread [qtp1199720022-30-acceptor-0@771ed183-ServerConnector@1f9bf19f{SSL,[ssl, http/1.1]}{0.0.0.0:4439}] (Suspended)
owns: Class<T> (com.lambda.Debugger.D) (id=85)
waiting for: DebugifyingClassLoader (id=84)
Class<T>.getDeclaredFields0(boolean) line: not available [native method]
Class<T>.privateGetDeclaredFields(boolean) line: 2583
Class<T>.privateGetPublicFields(Set<Class<?>>) line: 2614
Class<T>.getFields() line: 1557
Shadow.createShadowInternal(Object, boolean) line: 1160
Shadow.createShadow(Object, boolean) line: 1025
Shadow.get(Object, boolean) line: 1907
Shadow.get(Object) line: 1901
D.changeIVA(Object, Object, int, String, TraceLine) line: 389
ManagedSelector$Accept.<init>(ManagedSelector, SelectableChannel, Object) line: 556
ServerConnector$ServerConnectorManager(SelectorManager).accept(SelectableChannel, Object) line: 200
ServerConnector$ServerConnectorManager(SelectorManager).accept(SelectableChannel) line: 184
ServerConnector.accepted(SocketChannel) line: 362
ServerConnector.accept(int) line: 353
AbstractConnector$Acceptor.run() line: 603
QueuedThreadPool.runJob(Runnable) line: 672
QueuedThreadPool$2.run() line: 590
Thread.run() line: 748
I'm using EclipseLink and Jetty and a few other heavy-loading things.
Now, I think I've maybe fixed the problem - it locked consistently before, and no longer locks. However, it's pretty ugly, I'm not sure it's covering all cases, it could theoretically cause some other kind of problem, and I may have applied it in more places than is necessary. Basically, I've swapped out most of D's synchronization with first synchronizing on the DebugifyingClassLoader, and THEN on D. This ensures same-order locking (in the case I saw), preventing deadlock. Still, note the caveats.
I'm making a pull request with this and a few other fixes; please look it over and see if it's acceptable.