[Bug] GraphQL exception when deleting custom target

[andrewazores]

Looks good. I did encounter one error when deleting the custom target, but it seemed to succeed anyway and I had no problem recreating the target.

2025-11-13 14:53:34,145 INFO  [io.qua.htt.access-log] (executor-thread-24) 99.233.6.209 - - [13/Nov/2025:14:53:34 +0000] "DELETE /api/v4/targets/3 HTTP/1.1" 204 -
2025-11-13 14:53:34,148 ERROR [io.sma.graphql] (executor-thread-23) SRGQL012000: Data Fetching Error: java.lang.NullPointerException: Cannot load from object array because "results" is null
	at org.hibernate.persister.entity.AbstractEntityPersister.initLazyProperties(AbstractEntityPersister.java:1629)
	at org.hibernate.persister.entity.AbstractEntityPersister.initializeLazyPropertiesFromDatastore(AbstractEntityPersister.java:1581)
	at org.hibernate.persister.entity.AbstractEntityPersister.initializedLazyField(AbstractEntityPersister.java:1497)
	at org.hibernate.persister.entity.AbstractEntityPersister.initializeLazyProperty(AbstractEntityPersister.java:1450)
	at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.lambda$loadAttribute$0(LazyAttributeLoadingInterceptor.java:109)
	at org.hibernate.bytecode.enhance.spi.interceptor.EnhancementHelper.performWork(EnhancementHelper.java:210)
	at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.loadAttribute(LazyAttributeLoadingInterceptor.java:79)
	at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.fetchAttribute(LazyAttributeLoadingInterceptor.java:75)
	at org.hibernate.bytecode.enhance.spi.interceptor.LazyAttributeLoadingInterceptor.handleRead(LazyAttributeLoadingInterceptor.java:58)
	at org.hibernate.bytecode.enhance.spi.interceptor.AbstractInterceptor.readObject(AbstractInterceptor.java:155)
	at io.cryostat.discovery.DiscoveryNode.$$_hibernate_read_target(DiscoveryNode.java)
	at io.cryostat.discovery.DiscoveryNode.getTarget(DiscoveryNode.java)
	at io.cryostat.graphql.RootNode$DiscoveryNodeFilter.lambda$test$2(RootNode.java:82)
	at java.base/java.util.function.Predicate.lambda$and$0(Predicate.java:69)
	at java.base/java.util.function.Predicate.lambda$and$0(Predicate.java:69)
	at java.base/java.util.function.Predicate.lambda$and$0(Predicate.java:69)
	at java.base/java.util.function.Predicate.lambda$and$0(Predicate.java:69)
	at java.base/java.util.function.Predicate.lambda$and$0(Predicate.java:69)
	at java.base/java.util.function.Predicate.lambda$and$0(Predicate.java:69)
	at io.cryostat.graphql.RootNode$DiscoveryNodeFilter.test(RootNode.java:123)
	at io.cryostat.graphql.TargetNodes.lambda$getTargetNodes$1(TargetNodes.java:85)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178)
	at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
	at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1939)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
	at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
	at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
	at io.cryostat.graphql.TargetNodes.getTargetNodes(TargetNodes.java:86)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at io.smallrye.graphql.execution.datafetcher.helper.ReflectionInvoker.invoke(ReflectionInvoker.java:97)
	at io.smallrye.graphql.execution.datafetcher.DefaultDataFetcher.invokeAndTransform(DefaultDataFetcher.java:33)
	at io.quarkus.smallrye.graphql.runtime.spi.datafetcher.QuarkusDefaultDataFetcher.invokeAndTransform(QuarkusDefaultDataFetcher.java:45)
	at io.smallrye.graphql.execution.datafetcher.AbstractDataFetcher.get(AbstractDataFetcher.java:70)
	at graphql.execution.ExecutionStrategy.invokeDataFetcher(ExecutionStrategy.java:533)
	at graphql.execution.ExecutionStrategy.fetchField(ExecutionStrategy.java:497)
	at graphql.execution.ExecutionStrategy.fetchField(ExecutionStrategy.java:438)
	at graphql.execution.ExecutionStrategy.resolveFieldWithInfo(ExecutionStrategy.java:397)
	at graphql.execution.ExecutionStrategy.getAsyncFieldValueInfo(ExecutionStrategy.java:335)
	at graphql.execution.AsyncExecutionStrategy.execute(AsyncExecutionStrategy.java:57)
	at graphql.execution.Execution.executeOperation(Execution.java:180)
	at graphql.execution.Execution.execute(Execution.java:116)
	at graphql.GraphQL.execute(GraphQL.java:546)
	at graphql.GraphQL.lambda$parseValidateAndExecute$13(GraphQL.java:476)
	at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187)
	at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2341)
	at graphql.GraphQL.parseValidateAndExecute(GraphQL.java:471)
	at graphql.GraphQL.lambda$executeAsync$9(GraphQL.java:429)
	at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187)
	at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2341)
	at graphql.GraphQL.executeAsync(GraphQL.java:418)
	at io.smallrye.graphql.execution.ExecutionService.lambda$writeAsync$0(ExecutionService.java:248)
	at io.smallrye.context.impl.wrappers.SlowContextualSupplier.get(SlowContextualSupplier.java:21)
	at io.smallrye.mutiny.operators.uni.builders.UniCreateFromCompletionStage.subscribe(UniCreateFromCompletionStage.java:24)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:35)
	at io.smallrye.mutiny.groups.UniSubscribe.withSubscriber(UniSubscribe.java:51)
	at io.smallrye.mutiny.groups.UniSubscribe.with(UniSubscribe.java:110)
	at io.smallrye.mutiny.groups.UniSubscribe.with(UniSubscribe.java:88)
	at io.smallrye.graphql.execution.ExecutionService.writeAsync(ExecutionService.java:250)
	at io.smallrye.graphql.execution.ExecutionService.execute(ExecutionService.java:209)
	at io.smallrye.graphql.execution.ExecutionService.executeAsync(ExecutionService.java:148)
	at io.quarkus.smallrye.graphql.runtime.SmallRyeGraphQLExecutionHandler.doRequest(SmallRyeGraphQLExecutionHandler.java:367)
	at io.quarkus.smallrye.graphql.runtime.SmallRyeGraphQLExecutionHandler.handlePost(SmallRyeGraphQLExecutionHandler.java:125)
	at io.quarkus.smallrye.graphql.runtime.SmallRyeGraphQLExecutionHandler.doHandle(SmallRyeGraphQLExecutionHandler.java:86)
	at io.quarkus.smallrye.graphql.runtime.SmallRyeGraphQLAbstractHandler.handleWithIdentity(SmallRyeGraphQLAbstractHandler.java:96)
	at io.quarkus.smallrye.graphql.runtime.SmallRyeGraphQLAbstractHandler.handle(SmallRyeGraphQLAbstractHandler.java:77)
	at io.quarkus.smallrye.graphql.runtime.SmallRyeGraphQLAbstractHandler.handle(SmallRyeGraphQLAbstractHandler.java:31)
	at io.quarkus.smallrye.graphql.runtime.SmallRyeGraphQLCompressionHandler.handle(SmallRyeGraphQLCompressionHandler.java:23)
	at io.quarkus.smallrye.graphql.runtime.SmallRyeGraphQLCompressionHandler.handle(SmallRyeGraphQLCompressionHandler.java:7)
	at io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$5(ContextImpl.java:205)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:270)
	at io.vertx.core.impl.ContextImpl$1.execute(ContextImpl.java:221)
	at io.vertx.core.impl.WorkerTask.run(WorkerTask.java:56)
	at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:645)
	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
	at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

Originally posted by @ebaron in #1122 (review)

[ebaron]

FYI, I deleted the custom target again and got a different exception this time:

2025-11-13 15:07:46,673 ERROR [io.sma.graphql] (executor-thread-38) SRGQL012000: Data Fetching Error: jakarta.persistence.EntityNotFoundException: Unable to find io.cryostat.discovery.DiscoveryNode with id 68
	at io.quarkus.hibernate.orm.runtime.boot.FastBootEntityManagerFactoryBuilder$JpaEntityNotFoundDelegate.handleEntityNotFound(FastBootEntityManagerFactoryBuilder.java:255)
	at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:209)
	at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:240)
	at org.hibernate.event.internal.DefaultLoadEventListener.doOnLoad(DefaultLoadEventListener.java:109)
	at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:68)
	at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:151)
	at org.hibernate.internal.SessionImpl.fireLoadNoChecks(SessionImpl.java:1287)
	at org.hibernate.internal.SessionImpl.internalLoad(SessionImpl.java:1047)
	at org.hibernate.sql.results.graph.entity.internal.EntitySelectFetchInitializer.initialize(EntitySelectFetchInitializer.java:234)
	at org.hibernate.sql.results.graph.entity.internal.AbstractBatchEntitySelectFetchInitializer.resolveInstanceFromIdentifier(AbstractBatchEntitySelectFetchInitializer.java:120)
	at org.hibernate.sql.results.graph.entity.internal.AbstractBatchEntitySelectFetchInitializer.resolveInstance(AbstractBatchEntitySelectFetchInitializer.java:115)
	at org.hibernate.sql.results.graph.entity.internal.AbstractBatchEntitySelectFetchInitializer.resolveInstance(AbstractBatchEntitySelectFetchInitializer.java:37)
	at org.hibernate.sql.results.internal.StandardRowReader.coordinateInitializers(StandardRowReader.java:234)
	at org.hibernate.sql.results.internal.StandardRowReader.readRow(StandardRowReader.java:136)
	at org.hibernate.internal.ScrollableResultsImpl.prepareCurrentRow(ScrollableResultsImpl.java:131)
	at org.hibernate.internal.ScrollableResultsImpl.next(ScrollableResultsImpl.java:49)
	at org.hibernate.query.internal.ScrollableResultsIterator.hasNext(ScrollableResultsIterator.java:31)
	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
	at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1939)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
	at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622)
	at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627)
	at io.cryostat.graphql.TargetNodes.getTargetNodes(TargetNodes.java:86)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at io.smallrye.graphql.execution.datafetcher.helper.ReflectionInvoker.invoke(ReflectionInvoker.java:97)
	at io.smallrye.graphql.execution.datafetcher.DefaultDataFetcher.invokeAndTransform(DefaultDataFetcher.java:33)
	at io.quarkus.smallrye.graphql.runtime.spi.datafetcher.QuarkusDefaultDataFetcher.invokeAndTransform(QuarkusDefaultDataFetcher.java:45)
	at io.smallrye.graphql.execution.datafetcher.AbstractDataFetcher.get(AbstractDataFetcher.java:70)
	at graphql.execution.ExecutionStrategy.invokeDataFetcher(ExecutionStrategy.java:533)
	at graphql.execution.ExecutionStrategy.fetchField(ExecutionStrategy.java:497)
	at graphql.execution.ExecutionStrategy.fetchField(ExecutionStrategy.java:438)
	at graphql.execution.ExecutionStrategy.resolveFieldWithInfo(ExecutionStrategy.java:397)
	at graphql.execution.ExecutionStrategy.getAsyncFieldValueInfo(ExecutionStrategy.java:335)
	at graphql.execution.AsyncExecutionStrategy.execute(AsyncExecutionStrategy.java:57)
	at graphql.execution.Execution.executeOperation(Execution.java:180)
	at graphql.execution.Execution.execute(Execution.java:116)
	at graphql.GraphQL.execute(GraphQL.java:546)
	at graphql.GraphQL.lambda$parseValidateAndExecute$13(GraphQL.java:476)
	at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187)
	at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2341)
	at graphql.GraphQL.parseValidateAndExecute(GraphQL.java:471)
	at graphql.GraphQL.lambda$executeAsync$9(GraphQL.java:429)
	at java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187)
	at java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2341)
	at graphql.GraphQL.executeAsync(GraphQL.java:418)
	at io.smallrye.graphql.execution.ExecutionService.lambda$writeAsync$0(ExecutionService.java:248)
	at io.smallrye.context.impl.wrappers.SlowContextualSupplier.get(SlowContextualSupplier.java:21)
	at io.smallrye.mutiny.operators.uni.builders.UniCreateFromCompletionStage.subscribe(UniCreateFromCompletionStage.java:24)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:35)
	at io.smallrye.mutiny.groups.UniSubscribe.withSubscriber(UniSubscribe.java:51)
	at io.smallrye.mutiny.groups.UniSubscribe.with(UniSubscribe.java:110)
	at io.smallrye.mutiny.groups.UniSubscribe.with(UniSubscribe.java:88)
	at io.smallrye.graphql.execution.ExecutionService.writeAsync(ExecutionService.java:250)
	at io.smallrye.graphql.execution.ExecutionService.execute(ExecutionService.java:209)
	at io.smallrye.graphql.execution.ExecutionService.executeAsync(ExecutionService.java:148)
	at io.quarkus.smallrye.graphql.runtime.SmallRyeGraphQLExecutionHandler.doRequest(SmallRyeGraphQLExecutionHandler.java:367)
	at io.quarkus.smallrye.graphql.runtime.SmallRyeGraphQLExecutionHandler.handlePost(SmallRyeGraphQLExecutionHandler.java:125)
	at io.quarkus.smallrye.graphql.runtime.SmallRyeGraphQLExecutionHandler.doHandle(SmallRyeGraphQLExecutionHandler.java:86)
	at io.quarkus.smallrye.graphql.runtime.SmallRyeGraphQLAbstractHandler.handleWithIdentity(SmallRyeGraphQLAbstractHandler.java:96)
	at io.quarkus.smallrye.graphql.runtime.SmallRyeGraphQLAbstractHandler.handle(SmallRyeGraphQLAbstractHandler.java:77)
	at io.quarkus.smallrye.graphql.runtime.SmallRyeGraphQLAbstractHandler.handle(SmallRyeGraphQLAbstractHandler.java:31)
	at io.quarkus.smallrye.graphql.runtime.SmallRyeGraphQLCompressionHandler.handle(SmallRyeGraphQLCompressionHandler.java:23)
	at io.quarkus.smallrye.graphql.runtime.SmallRyeGraphQLCompressionHandler.handle(SmallRyeGraphQLCompressionHandler.java:7)
	at io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$5(ContextImpl.java:205)
	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:270)
	at io.vertx.core.impl.ContextImpl$1.execute(ContextImpl.java:221)
	at io.vertx.core.impl.WorkerTask.run(WorkerTask.java:56)
	at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:645)
	at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
	at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
	at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
	at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

[andrewazores]

#1122 (comment)

It could be related to the various transaction boundary separation changes, if the operation was previously completed by a single thread within one transaction and now could be spread across two (or more) transactions and potentially different threads as well.

[ebaron]

One more time and no exception occurred. I'm guessing this is just a timing thing of some thread trying to do an operation while the target is deleted.

[andrewazores]

The DataFetchingErrors indicate that the GraphQL response to the custom target deletion request should be coming back as an error response. That isn't too critical, because after deleting a custom target we don't really need to get back a response body describing the deleted target node, and obviously the web-client isn't actually using that response body for anything either (or even trying to read it). It's still an API bug, just not one that is likely to matter much.

[andrewazores]

I can't seem to reproduce this error on my cluster after 5 attempts at creating and deleting a custom target, so that probably reinforces further that this is a thread timing problem or a problem that the operation gets split across multiple threads. The normal case with a successful GraphQL response should have an HTTP 200 OK response status code, and a response body like this:

{
    "data": {
        "targetNodes": [
            {
                "target": {
                    "id": 1,
                    "report": {
                        "lastUpdated": 0,
                        "aggregate": {
                            "count": 0,
                            "max": -1.0
                        }
                    }
                }
            }
        ]
    }
}

[andrewazores]

Looking at it again/trying to reproduce it I don't even think it's as bad as I was describing it. I'm pretty sure what's actually happening is that the Topology view is just trying to re-render and update its graphical state, but it's out of sync with the backend because of the deletion operation it just requested. Because the deletion operation is just a plain HTTP DELETE request, and the GraphQL query is a separate operation the frontend fires to update the view, so if both of those requests are in-flight at the same time then the execution order matters and the GraphQL query can end up becoming invalid.