feat(reports): configure reports for presigned recording transfer #1102
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/build_test |
|
|
|
/build_test |
|
|
Seems like a strange failure, and I'm not sure how/why that would be connected to this PR. |
|
/build_test |
|
|
andrewazores
force-pushed
the
reports-presign
branch
from
1ba46f7 to
dd7e2ab
Compare
7 tasks
andrewazores
force-pushed
the
reports-presign
branch
from
dd7e2ab to
eec0a48
Compare
|
I'm actually not sure if that Endpoints synchronization message above is a real failure. Running the tests locally with other dependencies satisfied results in a passed scorecard suite, even with that message printed. I wonder if that's also just an artefact of the container shutdown/cleanup process. $ export REPORTS_IMG=quay.io/andrewazores/cryostat-reports:presign-ssl-certs-1
$ export IMAGE_NAMESPACE=quay.io/andrewazores
$ export OPERATOR_VERSION=4.1.0-reports-presign-31 # yea, 31, don't judge
$ SKIP_TESTS=true make generate manifests manager oci-build bundle bundle-build ; podman image prune -f ; podman push $IMAGE_NAMESPACE/cryostat-operator:$OPERATOR_VERSION ; podman push $IMAGE_NAMESPACE/cryostat-operator-bundle:$OPERATOR_VERSION
$ make test-scorecard 2>&1 | tee scorecard.log |
ebaron
reviewed
May 1, 2025
There was a problem hiding this comment.
I got this when testing using the PR description:
Reports pod:
2025-05-01 18:49:51,964 INFO [io.cry.rep.ReportResource] (executor-thread-1) Attempting to download presigned recording from https://cryostat-sample-storage.cryostat-operator-system.svc.cluster.local:8333/archivedrecordings/9Y5wflb2w0C3dysM-Wg5c3urPYFliz0u3A_SWvt5xUo=/quarkus-cryostat-agent-6f5746dc9f-c9wg2_test_20250501T184951Z.jfr?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20250501T184951Z&X-Amz-SignedHeaders=host&X-Amz-Credential=cryostat/20250501/us-east-1/s3/aws4_request&X-Amz-Expires=60&X-Amz-Signature=446cf11f5fe77486003c331b27e2359907ccce0ecd12d625eab886b79cf0921f
2025-05-01 18:49:52,182 INFO [io.qua.htt.access-log] (executor-thread-1) 10.128.0.50 - - [01/May/2025:18:49:52 +0000] "POST /remote_report HTTP/1.1" 500 72
Cryostat pod:
2025-05-01 18:49:52,185 WARN [com.git.ben.caf.cac.LocalAsyncCache] (vert.x-eventloop-thread-2) Exception thrown during asynchronous load: java.util.concurrent.CompletionException: org.jboss.resteasy.reactive.ClientWebApplicationException: Received: 'Internal Server Error, status code 500' when invoking REST Client method: 'io.cryostat.reports.ReportSidecarService#generatePresigned'
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:874)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
at io.smallrye.context.impl.wrappers.SlowContextualConsumer.accept(SlowContextualConsumer.java:21)
at io.smallrye.mutiny.helpers.UniCallbackSubscriber.onFailure(UniCallbackSubscriber.java:62)
at io.smallrye.mutiny.operators.uni.UniOperatorProcessor.onFailure(UniOperatorProcessor.java:55)
at io.smallrye.mutiny.operators.uni.UniOperatorProcessor.onFailure(UniOperatorProcessor.java:55)
at io.smallrye.mutiny.operators.uni.UniOnItemConsume$UniOnItemComsumeProcessor.onFailure(UniOnItemConsume.java:65)
at io.smallrye.mutiny.operators.uni.UniOnFailureTransform$UniOnFailureTransformProcessor.onFailure(UniOnFailureTransform.java:64)
at io.smallrye.mutiny.operators.uni.builders.UniCreateFromCompletionStage$CompletionStageUniSubscription.forwardResult(UniCreateFromCompletionStage.java:58)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
at org.jboss.resteasy.reactive.client.impl.RestClientRequestContext.handleUnrecoverableError(RestClientRequestContext.java:378)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.handleException(AbstractResteasyReactiveContext.java:329)
at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:175)
at org.jboss.resteasy.reactive.client.impl.RestClientRequestContext$1.lambda$execute$0(RestClientRequestContext.java:324)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:270)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:252)
at io.vertx.core.impl.ContextInternal.lambda$runOnContext$0(ContextInternal.java:50)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.jboss.resteasy.reactive.ClientWebApplicationException: Received: 'Internal Server Error, status code 500' when invoking REST Client method: 'io.cryostat.reports.ReportSidecarService#generatePresigned'
at org.jboss.resteasy.reactive.client.impl.RestClientRequestContext.unwrapException(RestClientRequestContext.java:205)
... 14 more
Caused by: jakarta.ws.rs.WebApplicationException: Internal Server Error, status code 500
at io.smallrye.context.impl.wrappers.SlowContextualSupplier.get(SlowContextualSupplier.java:21)
at io.smallrye.mutiny.operators.uni.builders.UniCreateFromCompletionStage.subscribe(UniCreateFromCompletionStage.java:24)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.operators.uni.UniOnFailureTransform.subscribe(UniOnFailureTransform.java:31)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.operators.uni.UniOnItemConsume.subscribe(UniOnItemConsume.java:30)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.performInnerSubscription(UniOnItemTransformToUni.java:81)
at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.onItem(UniOnItemTransformToUni.java:57)
at io.smallrye.mutiny.operators.uni.builders.UniCreateFromItemSupplier.subscribe(UniCreateFromItemSupplier.java:29)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni.subscribe(UniOnItemTransformToUni.java:25)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.groups.UniSubscribe.withSubscriber(UniSubscribe.java:51)
at io.smallrye.mutiny.groups.UniSubscribe.with(UniSubscribe.java:110)
at io.smallrye.mutiny.operators.uni.UniSubscribeToCompletionStage.subscribe(UniSubscribeToCompletionStage.java:30)
at io.smallrye.mutiny.groups.UniSubscribe.asCompletionStage(UniSubscribe.java:174)
at io.smallrye.mutiny.groups.UniSubscribe.asCompletionStage(UniSubscribe.java:162)
at io.smallrye.mutiny.Uni.subscribeAsCompletionStage(Uni.java:141)
at io.quarkus.cache.runtime.caffeine.CaffeineCacheImpl$4$1.apply(CaffeineCacheImpl.java:122)
at io.quarkus.cache.runtime.caffeine.CaffeineCacheImpl$4$1.apply(CaffeineCacheImpl.java:116)
at com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncAsMapView.lambda$computeIfAbsent$0(LocalAsyncCache.java:366)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2677)
at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1916)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2675)
at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2658)
at com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncAsMapView.computeIfAbsent(LocalAsyncCache.java:365)
at com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncAsMapView.computeIfAbsent(LocalAsyncCache.java:293)
at io.quarkus.cache.runtime.caffeine.CaffeineCacheImpl$4.get(CaffeineCacheImpl.java:115)
at io.quarkus.cache.runtime.caffeine.CaffeineCacheImpl$4.get(CaffeineCacheImpl.java:109)
at io.smallrye.context.impl.wrappers.SlowContextualSupplier.get(SlowContextualSupplier.java:21)
at io.smallrye.mutiny.operators.uni.builders.UniCreateFromCompletionStage.subscribe(UniCreateFromCompletionStage.java:24)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.operators.uni.UniEmitOn.subscribe(UniEmitOn.java:22)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.operators.uni.UniOnItemConsume.subscribe(UniOnItemConsume.java:30)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.operators.uni.UniFailOnTimeout.subscribe(UniFailOnTimeout.java:36)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.operators.uni.UniOnItemConsume.subscribe(UniOnItemConsume.java:30)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.operators.uni.UniOnTermination.subscribe(UniOnTermination.java:21)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.groups.UniSubscribe.withSubscriber(UniSubscribe.java:51)
at io.smallrye.mutiny.groups.UniSubscribe.with(UniSubscribe.java:110)
at io.smallrye.mutiny.operators.uni.UniSubscribeToCompletionStage.subscribe(UniSubscribeToCompletionStage.java:30)
at io.smallrye.mutiny.groups.UniSubscribe.asCompletionStage(UniSubscribe.java:174)
at io.smallrye.mutiny.groups.UniSubscribe.asCompletionStage(UniSubscribe.java:162)
at io.smallrye.mutiny.Uni.subscribeAsCompletionStage(Uni.java:141)
at io.cryostat.recordings.LongRunningRequestGenerator_onMessage_Invoker_izeA4IbSRjunhFJnQw1q0NJka-4.invoke(Unknown Source)
at io.cryostat.recordings.LongRunningRequestGenerator_onMessage_LazyInvoker_izeA4IbSRjunhFJnQw1q0NJka-4.invoke(Unknown Source)
at io.quarkus.vertx.runtime.EventConsumerInvoker.invokeBean(EventConsumerInvoker.java:79)
at io.quarkus.vertx.runtime.EventConsumerInvoker.invoke(EventConsumerInvoker.java:51)
at io.quarkus.vertx.runtime.VertxEventBusConsumerRecorder$3$1$2.call(VertxEventBusConsumerRecorder.java:158)
at io.quarkus.vertx.runtime.VertxEventBusConsumerRecorder$3$1$2.call(VertxEventBusConsumerRecorder.java:154)
at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$4(ContextImpl.java:192)
at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:270)
at io.vertx.core.impl.ContextImpl$1.execute(ContextImpl.java:221)
at io.vertx.core.impl.WorkerTask.run(WorkerTask.java:56)
at io.quarkus.vertx.core.runtime.VertxCoreRecorder$14.runWith(VertxCoreRecorder.java:635)
at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
at org.jboss.threads.DelegatingRunnable.run(DelegatingRunnable.java:11)
at org.jboss.threads.ThreadLocalResettingRunnable.run(ThreadLocalResettingRunnable.java:11)
... 2 more
|
Was that with the PR body description or the steps in the last most recent comment? |
It was the body description |
|
Oh whoops, forgot to override REPORTS_IMG. Is that still needed, or has that change been merged? |
|
That change hasn't been merged yet on the -reports side. |
ebaron
previously approved these changes
May 1, 2025
andrewazores
force-pushed
the
reports-presign
branch
from
eec0a48 to
f74d5f0
Compare
7 tasks
|
/build_test |
|
|
ebaron
previously approved these changes
May 2, 2025
|
@ebaron I'm trying to rebase/resync this with current I tried updating the tests either way - to create secrets using |
|
This PR/issue depends on: |
ebaron
reviewed
Jun 17, 2025
There was a problem hiding this comment.
I tried updating the tests either way - to create secrets using Data or StringData, with what looks like the same failed result. Not sure what I am missing here, is there anything obvious to you?
I would stick with using Data instead of StringData. All you should need to do is add the new secret usage to the NewReportsPodAnnotations method:
diff --git a/internal/test/resources.go b/internal/test/resources.go
index 61113b81..5e4f9051 100644
--- a/internal/test/resources.go
+++ b/internal/test/resources.go
@@ -2385,10 +2385,13 @@ func (r *TestResources) NewStoragePodAnnotations() map[string]string {
func (r *TestResources) NewReportsPodAnnotations() map[string]string {
annotations := map[string]string{}
- secrets := []*corev1.Secret{}
+ secrets := []*corev1.Secret{
+ r.NewStorageSecret(),
+ }
if r.TLS {
secrets = append(secrets,
r.NewCertSecret(r.NewReportsCert()),
+ r.NewCertSecret(r.NewStorageCert()),
)
}
|
/build_test |
|
/build_test |
|
/build_test |
|
|
|
|
|
|
|
/build_test |
|
|
ebaron
approved these changes
Jun 17, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Welcome to Cryostat! 👋
Before contributing, make sure you have:
mainbranch[chore, ci, docs, feat, fix, test]git commit -S -m "YOUR_COMMIT_MESSAGE"Related to cryostatio/cryostat#856
Depends on #1101
Based on #1101
Depends on cryostatio/cryostat-reports#345
Description of the change:
Adjusts the report deployment configuration to account for the Cryostat change above.
Rather than Cryostat acting as a network pipe middleman to stream JFR files out of storage and into the report generator(s), Cryostat now simply points the report generator to an archived recording file storage URL. The report generator retrieves the file on its own. This requires a few extra pieces of configuration on the report generator side to tell it what the storage container location is, as well as needing to relax the storage network policy to allow these requests, and so that the report generator correctly trusts the storage SSL/TLS cert (TODO).
Motivation for the change:
Support the new reports presigned transfer feature, which should reduce overall network bandwidth used for report generations and decrease report generation latency.
How to manually test:
REPORTS_IMG=quay.io/andrewazores/cryostat-reports:presign-ssl-certs-1reportOptions: { replicas: 1 }make sample_appin a target namespaceAttempting to download presigned recording from ...