ramenctl validate application fails to report noobaa bucket download test failed

[netzzer]

Version v0.19.0 ramenctl "validate application" fails to report one managed cluster noobaa bucket download test failed in ODF 4.21 RDR test environment.

% tree ~/Downloads/out-busybox-ceph-as/validate-application.data/s3
/Users/nir/Downloads/out-busybox-ceph-as/validate-application.data/s3
├── s3profile-hyper3-ocs-storagecluster
└── s3profile-hyper4-ocs-storagecluster
    └── busybox-cephfs-as
        └── busybox-cephfs-as-placement-drpc
            └── v1alpha1.VolumeReplicationGroup
                └── a

out-busybox-ceph-as.tar.gz

[nirs]

ramenctl validate-application Diagnosis

Application: busybox-cephfs-as-placement-drpc / openshift-gitops
Clusters: hub, hyper3 (primary), hyper4 (secondary)
DR Policy: hyper3-hyper4-5m | Replication: VolSync + CephFS consistency groups
ramenctl: v0.19.0 (commit 2081c30)
Report time: 2026-04-20T10:09:42 PDT

Result

Status: Failed — 21 ok, 2 problem

Failed Check	Cluster	Condition
DRPC Protected	hub	VRG on hyper3 progressing on protecting workload resources (VRG Kube object protect error), retrying till ClusterDataProtected condition is met
VRG ClusterDataProtected	hyper3	VRG Kube object protect error

Root Cause: InvalidBucketState on hyper3 S3 bucket

The ramen-dr-cluster-operator on hyper3 cannot upload the VRG object to S3 bucket odrbucket-36bceb61c09c via s3profile-hyper3-ocs-storagecluster. Every reconcile attempt fails with:

code: InvalidBucketState, message: The request is not valid with the current state of the bucket.

This has been failing persistently since at least Apr 18, 03:29 UTC — confirmed in both current.log and previous.log of the operator pod (ramen-dr-cluster-operator-595587b66-r7ksr), meaning it survived a pod restart.

The ramenctl gather itself also received an HTTP 502 Bad Gateway when trying to download from the hyper3 S3 endpoint, further confirming the S3 service is unhealthy.

S3 gather detail

ramenctl gathered S3 data from both profiles in parallel. Both profiles could list objects in the bucket, but only hyper4's endpoint could download:

Profile	Endpoint	List	Download	Duration
s3profile-hyper4-ocs-storagecluster	s3-openshift-storage.apps.hyper4.vmw.ibmfusion.eu	1 object	1/1	1.06s
s3profile-hyper3-ocs-storagecluster	s3-openshift-storage.apps.hyper3.vmw.ibmfusion.eu	1 object	0/1 (502)	2.31s

The downloaded VRG object was saved only under s3/s3profile-hyper4-ocs-storagecluster/. No data was retrieved from hyper3's profile.

ramenctl bug: The download failure (0/1 objects, HTTP 502 after 3 retries) is logged as a WARN at s3/s3.go:256 but does not propagate to the report. ramenctl marks the hyper3 profile as gathered: ok and the gather step as status: passed even though no data was actually retrieved. It should fail the gather step or set gathered: ok: false when the download count (0) is less than the listed object count (1). The bug is in the S3 gather logic — likely s3/s3.go or application/command.go — where a non-zero download failure count is not treated as an error.

Failure Chain

Step	Component	What happens
1	NooBaa / S3 gateway (hyper3)	Bucket odrbucket-36bceb61c09c returns InvalidBucketState on PutObject. ramenctl gather also got HTTP 502 from s3-openshift-storage.apps.hyper3.vmw.ibmfusion.eu.
2	ramen-dr-cluster-operator (hyper3)	vrg_vrgobject.go:50 fails to upload VRG object. Emits VrgUploadFailed warning event.
3	VRG on hyper3	ClusterDataProtected condition → False, reason VolumeReplicationGroupObjectCaptureError.
4	DRPC on hub	Protected condition → False — "progressing on protecting workload resources, retrying till ClusterDataProtected condition is met".

Secondary Issues (hyper4)

These are non-blocking and self-resolving but indicate instability in the VolSync replication pipeline on the secondary cluster:

Error	Component	Impact
Images are not ready to use	ReplicationGroupDestination (cephfscg)	Volume snapshots on hyper4 not ready yet — transient during sync cycles
object has been modified	RGD / PVC reconciler	Optimistic locking conflicts on busybox-pvc and RGD — retried automatically
Failed to PrecreateDestPVCIfEnabled	RGD reconciler	PVC conflict during pre-creation — cascaded from the above conflict

Recommended Actions

1. Fix the S3 bucket on hyper3

Check the NooBaa system on hyper3 (openshift-storage namespace). The InvalidBucketState error typically means the OBC/bucket claim is in a bad state — the bucket may have been deleted, the backing store may be unhealthy, or there is a NooBaa operator issue.

oc get bucketclass,backingstore -n openshift-storage
oc get obc -A | grep odrbucket
oc describe obc <obc-name> -n openshift-storage
oc logs -n openshift-storage deploy/noobaa-core -c noobaa-core --tail=200

2. Verify the S3 route is healthy

The HTTP 502 suggests NooBaa S3 endpoint pods may be crashing or overloaded.

oc get pods -n openshift-storage -l app=noobaa-endpoint
oc get route s3 -n openshift-storage
curl -k https://s3-openshift-storage.apps.hyper3.vmw.ibmfusion.eu

3. Monitor hyper4 RGD convergence

The "Images not ready" and conflict errors on hyper4 are transient. Once the S3 issue on hyper3 is fixed and VRG protection succeeds, verify that the secondary VRG and RGD on hyper4 stabilize.

4. Re-run validation

After fixing the S3 bucket, re-run:

ramenctl validate application <app-name> -o out-busybox-ceph-as

All Validation Checks

#	Check	Cluster	Result
1	DRPC action state	hub	OK
2	DRPC condition: Available	hub	OK
3	DRPC condition: PeerReady	hub	OK
4	DRPC condition: Protected	hub	FAIL
5	DRPC phase (Deployed)	hub	OK
6	DRPC progression (Completed)	hub	OK
7	DRPC deleted	hub	OK
8	VRG condition: DataReady	hyper3	OK
9	VRG condition: ClusterDataReady	hyper3	OK
10	VRG condition: ClusterDataProtected	hyper3	FAIL
11	VRG condition: KubeObjectsReady	hyper3	OK
12	VRG condition: NoClusterDataConflict	hyper3	OK
13	VRG deleted	hyper3	OK
14	VRG state (Primary)	hyper3	OK
15	PVC busybox-pvc: phase (Bound)	hyper3	OK
16	PVC busybox-pvc: ReplicationSourceSetup	hyper3	OK
17	PVC busybox-pvc: deleted	hyper3	OK
18	VRG (secondary) condition: NoClusterDataConflict	hyper4	OK
19	VRG (secondary) deleted	hyper4	OK
20	VRG (secondary) state (Secondary)	hyper4	OK
21	S3 profiles gathered	hub	OK
22	S3 profile: s3profile-hyper4-ocs-storagecluster	hub	OK
23	S3 profile: s3profile-hyper3-ocs-storagecluster	hub	OK

Key Timestamps

Event	Time (UTC)
Application created	2026-04-13T17:35:33
ClusterDataProtected last True	2026-04-14T16:15:59
ClusterDataProtected → False	2026-04-18T03:29:30
DRPC Protected → False	2026-04-18T03:32:01
Last successful group sync	2026-04-20T17:05:19
Validation report run	2026-04-20T10:09:42 PDT

[nirs]

@parikshithb can you take this?

Seems that the issue is that we don't report an error if we fail to download objects from s3, and later we mark the s3 profiles are "ok". This is misleading the user that s3 is accessible when in this case the s3 endpoint fail to upload or download.

Wrong report showing the s3 is ok: