[Bug] Test suite flakiness since #968

[andrewazores]

Current Behavior

Since PR #968 the occurrence of test failures due to timeouts or incorrect cleanup (which may be due to previous timeouts) has increased significantly. Actual usage shows that everything is behaving well, and #968 should have the expected effect of reducing request latencies, so it's likely that the expected performance increase has simply exacerbated the already existing flakiness seen in these tests.

Expected Behavior

No response

Steps To Reproduce

No response

Environment

- OS:
- Environment:
- Version:

Anything else?

No response

[andrewazores]

Example of such a timeout failure:

Error:  Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 134.6 s <<< FAILURE! -- in itest.RecordingWorkflowBucketedMetadataTest
Error:  itest.RecordingWorkflowBucketedMetadataTest.testWorkflow -- Time elapsed: 68.09 s <<< ERROR!
java.util.concurrent.TimeoutException
	at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1960)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2095)
	at itest.RecordingWorkflowBucketedMetadataTest.testWorkflow(RecordingWorkflowBucketedMetadataTest.java:216)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at io.quarkus.test.junit.QuarkusTestExtension.runExtensionMethod(QuarkusTestExtension.java:1000)
	at io.quarkus.test.junit.QuarkusTestExtension.interceptTestMethod(QuarkusTestExtension.java:848)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	Suppressed: io.smallrye.mutiny.TimeoutException
		at io.smallrye.mutiny.operators.uni.UniBlockingAwait.await(UniBlockingAwait.java:65)
		at io.smallrye.mutiny.groups.UniAwait.atMost(UniAwait.java:65)
		at io.cryostat.targets.TargetConnectionManager.executeConnectedTask(TargetConnectionManager.java:206)
		at io.cryostat.targets.TargetConnectionManager.executeConnectedTask(TargetConnectionManager.java:202)
		at io.cryostat.targets.TargetConnectionManager_ClientProxy.executeConnectedTask(Unknown Source)
		at io.cryostat.recordings.RecordingHelper.listActiveRecordingsImpl(RecordingHelper.java:280)
		at io.cryostat.recordings.RecordingHelper.lambda$listActiveRecordings$1(RecordingHelper.java:259)
		at io.quarkus.narayana.jta.QuarkusTransactionImpl.callInTheirTx(QuarkusTransactionImpl.java:173)
		at io.quarkus.narayana.jta.QuarkusTransactionImpl.callJoinExisting(QuarkusTransactionImpl.java:77)
		at io.quarkus.narayana.jta.QuarkusTransactionImpl.call(QuarkusTransactionImpl.java:33)
		at io.quarkus.narayana.jta.TransactionRunnerImpl.call(TransactionRunnerImpl.java:34)
		at io.cryostat.recordings.RecordingHelper.listActiveRecordings(RecordingHelper.java:259)
		at io.cryostat.recordings.RecordingHelper_ClientProxy.listActiveRecordings(Unknown Source)
		at io.cryostat.recordings.ActiveRecordings.list(ActiveRecordings.java:91)
		at io.cryostat.recordings.ActiveRecordings_Subclass.list$$superforward(Unknown Source)
		at io.cryostat.recordings.ActiveRecordings_Subclass$$function$$6.apply(Unknown Source)
		at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:73)
		at io.quarkus.arc.impl.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:62)
		at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorBase.invokeInOurTx(TransactionalInterceptorBase.java:136)
		at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorBase.invokeInOurTx(TransactionalInterceptorBase.java:107)
		at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorRequired.doIntercept(TransactionalInterceptorRequired.java:38)
		at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorBase.intercept(TransactionalInterceptorBase.java:61)
		at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorRequired.intercept(TransactionalInterceptorRequired.java:32)
		at io.quarkus.narayana.jta.runtime.interceptor.TransactionalInterceptorRequired_Bean.intercept(Unknown Source)
		at io.quarkus.arc.impl.InterceptorInvocation.invoke(InterceptorInvocation.java:42)
		at io.quarkus.arc.impl.AroundInvokeInvocationContext.perform(AroundInvokeInvocationContext.java:30)
		at io.quarkus.arc.impl.InvocationContexts.performAroundInvoke(InvocationContexts.java:27)
		at io.cryostat.recordings.ActiveRecordings_Subclass.list(Unknown Source)
		at io.cryostat.recordings.ActiveRecordings$quarkusrestinvoker$list_c8fc83b539ec35f0405f17e2cb62eec8d4468d64.invoke(Unknown Source)
		at org.jboss.resteasy.reactive.server.handlers.InvocationHandler.handle(InvocationHandler.java:29)
		at io.quarkus.resteasy.reactive.server.runtime.QuarkusResteasyReactiveRequestContext.invokeHandler(QuarkusResteasyReactiveRequestContext.java:141)
		at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
		at io.quarkus.vertx.core.runtime.VertxCoreRecorder$15.runWith(VertxCoreRecorder.java:645)
		at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2651)
		at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2630)
		at org.jboss.threads.EnhancedQueueExecutor.runThreadBody(EnhancedQueueExecutor.java:1622)
		at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1589)
		at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
		at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.base/java.lang.Thread.run(Thread.java:1583)