The noobaa operator is crashing with panic: runtime error: invalid memory address or nil pointer dereference [recovered]

[rijesh-purayil]

Environment info

• NooBaa Operator Version: noobaa-operator:5.15.2
• Platform: minikube v1.33.1 on Ubuntu 22.04.4 LTS and OpenShift 4.14

Actual behavior

1. When we apply noobaaccount, the noobaa operator will crash with the error.

root@ubuntu-011:~# oc create -f noobaaacount_cr.yaml
noobaaaccount.noobaa.io/default created
root@ubuntu-011:~# 

root@ubuntu-011:~# oc get pods
NAME                                               READY   STATUS             RESTARTS      AGE
noobaa-core-0                                      1/1     Running            0             2d13h
noobaa-db-pg-0                                     1/1     Running            0             2d13h
noobaa-default-backing-store-noobaa-pod-4e5fdaa7   1/1     Running            0             2d13h
noobaa-endpoint-6d78f84dcd-8dgvg                   1/1     Running            0             2d13h
noobaa-operator-5b57c57687-tnk46                   0/1     CrashLoopBackOff   4 (60s ago)   2d13h
root@ubuntu-011:~#

time="2024-06-14T03:08:18Z" level=info msg="✅ Exists: BackingStore \"noobaa-default-backing-store\"\n"
time="2024-06-14T03:08:18Z" level=info msg="✅ Exists: Secret \"noobaa-admin\"\n"
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1f48c44]

goroutine 1673 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116 +0x1e5
panic({0x22d49c0?, 0x4299610?})
	/opt/homebrew/Cellar/go/1.22.2/libexec/src/runtime/panic.go:770 +0x132
github.com/noobaa/noobaa-operator/v5/pkg/noobaaaccount.(*Reconciler).CreateNooBaaAccount(0xc0011ab5e0)
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/pkg/noobaaaccount/reconciler.go:336 +0xe4
github.com/noobaa/noobaa-operator/v5/pkg/noobaaaccount.(*Reconciler).ReconcilePhaseConfiguring(0xc0011ab5e0)
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/pkg/noobaaaccount/reconciler.go:266 +0x9b
github.com/noobaa/noobaa-operator/v5/pkg/noobaaaccount.(*Reconciler).ReconcilePhases(0xc0011ab5e0)
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/pkg/noobaaaccount/reconciler.go:154 +0x36
github.com/noobaa/noobaa-operator/v5/pkg/noobaaaccount.(*Reconciler).Reconcile(0xc0011ab5e0)
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/pkg/noobaaaccount/reconciler.go:111 +0x105
github.com/noobaa/noobaa-operator/v5/pkg/controller/noobaaaccount.Add.func1({0xc000101808?, 0xc000d0e980?}, {{{0xc001053f78?, 0xc001588d20?}, {0xc001053f70?, 0xc001588cf0?}}})
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/pkg/controller/noobaaaccount/noobaaaccount_controller.go:33 +0xd6
sigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile(0x7f4347e125b8?, {0x2e5fc48?, 0xc00110d3b0?}, {{{0xc001053f78?, 0xc000d0e980?}, {0xc001053f70?, 0xc001588d50?}}})
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/vendor/sigs.k8s.io/controller-runtime/pkg/reconcile/reconcile.go:111 +0x3d
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x2e65b18?, {0x2e5fc48?, 0xc00110d3b0?}, {{{0xc001053f78?, 0xb?}, {0xc001053f70?, 0x0?}}})
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0005e8c80, {0x2e5fc80, 0xc0007f6af0}, {0x23ed400, 0xc001510ea0})
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316 +0x3bc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0005e8c80, {0x2e5fc80, 0xc0007f6af0})
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266 +0x1c9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 209
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:223 +0x50c
root@ubuntu-011:~# 

Expected behavior

1. The noobaaccunt CR https://github.com/noobaa/noobaa-operator/blob/master/deploy/crds/noobaa.io_v1alpha1_noobaaaccount_cr.yaml needs an update, this one will not get applied.

root@ubuntu-011:~# oc create -f noobaaacount_cr.yaml
The NooBaaAccount "default" is invalid: spec.allow_bucket_creation: Required value
root@ubuntu-011:~#

2. The https://github.com/noobaa/noobaa-operator/blob/master/deploy/crds/noobaa.io_noobaaaccounts.yaml has only one property required, this is also need inspection.

Steps to reproduce

1. nooba install using CLI is installing nooba perfectly. This do not apply noobaaaccount CR.
2. This time the pods can see in Running state without restart as shown below.

root@ubuntu-011:~# oc get pods
NAME                                               READY   STATUS    RESTARTS   AGE
noobaa-core-0                                      1/1     Running   0          2d13h
noobaa-db-pg-0                                     1/1     Running   0          2d13h
noobaa-default-backing-store-noobaa-pod-4e5fdaa7   1/1     Running   0          2d13h
noobaa-endpoint-6d78f84dcd-8dgvg                   1/1     Running   0          2d13h
noobaa-operator-5b57c57687-tnk46                   1/1     Running   0          2d13h
root@ubuntu-011:~#

root@ubuntu-011:~# oc get noobaa
NAME     S3-ENDPOINTS                     STS-ENDPOINTS                    IMAGE                                PHASE   AGE
noobaa   ["https://192.168.49.2:32420"]   ["https://192.168.49.2:30220"]   noobaa/noobaa-core:master-20230920   Ready   2d13h
root@ubuntu-011:~#

3. Then apply the noobaaaccount_cr as shown below.

root@ubuntu-011:~# cat noobaaacount_cr.yaml
apiVersion: noobaa.io/v1alpha1
kind: NooBaaAccount
metadata:
  name: default
spec: 
  allow_bucket_creation: true
  default_resource: noobaa-default-backing-store 
root@ubuntu-011:~#

root@ubuntu-011:~# oc create -f noobaaacount_cr.yaml
noobaaaccount.noobaa.io/default created
root@ubuntu-011:~#

4. Then when you look at the pods status, we can see nooba operator pods started restarting.

root@ubuntu-011:~# oc get pods
NAME                                               READY   STATUS             RESTARTS      AGE
noobaa-core-0                                      1/1     Running            0             2d13h
noobaa-db-pg-0                                     1/1     Running            0             2d13h
noobaa-default-backing-store-noobaa-pod-4e5fdaa7   1/1     Running            0             2d13h
noobaa-endpoint-6d78f84dcd-8dgvg                   1/1     Running            0             2d13h
noobaa-operator-5b57c57687-tnk46                   0/1     CrashLoopBackOff   4 (60s ago)   2d13h
root@ubuntu-011:~#

root@ubuntu-011:~# kubectl get noobaaaccount
NAME      PHASE   AGE
default           116m
root@ubuntu-011:~#

More information - Screenshots / Logs / Other output

To satisfy nooba operator. the following CR (with force_md5_etag: no) needs to be applied, then the operator will not crash and the noobaccount will get into ready state.

root@ubuntu-011:~# cat noobaaacount_cr.yaml 
apiVersion: noobaa.io/v1alpha1
kind: NooBaaAccount
metadata:
  name: default
spec: 
  allow_bucket_creation: true
  default_resource: noobaa-default-backing-store 
  force_md5_etag: no
root@ubuntu-011:~#

root@ubuntu-011:~# oc delete -f noobaaacount_cr.yaml 
noobaaaccount.noobaa.io "default" deleted
root@ubuntu-011:~#

And then operator will get settled, no more new restarts.

root@ubuntu-011:~# kubectl get pods -w
NAME                                               READY   STATUS    RESTARTS         AGE
noobaa-core-0                                      1/1     Running   0                2d15h
noobaa-db-pg-0                                     1/1     Running   0                2d15h
noobaa-default-backing-store-noobaa-pod-4e5fdaa7   1/1     Running   0                2d15h
noobaa-endpoint-6d78f84dcd-8dgvg                   1/1     Running   0                2d15h
noobaa-operator-5b57c57687-tnk46                   1/1     Running   28 (8m32s ago)   2d15h

Then can see noobaaaccount ready.

root@ubuntu-011:~# oget noobaaaccountl 
NAME      PHASE   AGE
default   Ready   2m31s
root@ubuntu-011:~#

I will attach the nooba operator log during crash.

[rijesh-purayil]

Operator log during crash.

root@ubuntu-011:~# oc logs -f noobaa-operator-5b57c57687-tnk46
time="2024-06-14T03:07:56Z" level=info msg="CLI version: 5.15.2\n"
time="2024-06-14T03:07:56Z" level=info msg="noobaa-image: noobaa/noobaa-core:master-20230920\n"
time="2024-06-14T03:07:56Z" level=info msg="operator-image: noobaa/noobaa-operator:5.15.2\n"
time="2024-06-14T03:07:56Z" level=info msg="Registering Components."
time="2024-06-14T03:07:57Z" level=info msg="❌ CRD Missing: : no matches for kind \"CephCluster\" in version \"ceph.rook.io/v1\"\n"
time="2024-06-14T03:07:57Z" level=info msg="OBC Provisioner - start.." provisioner=noobaa.noobaa.io/obc
time="2024-06-14T03:07:57Z" level=info msg="running noobaa provisioner noobaa.noobaa.io/obc" provisioner=noobaa.noobaa.io/obc
time="2024-06-14T03:07:57Z" level=info msg="Starting the Operator ..."
I0614 03:07:57.160219       1 manager.go:120] "msg"="starting provisioner" "logger"="objectbucket.io/provisioner-manager" "name"="noobaa.noobaa.io/obc"
time="2024-06-14T03:07:58Z" level=info msg="Create event detected for noobaa-default-backing-store (noobaa), queuing Reconcile"
time="2024-06-14T03:07:58Z" level=info msg="checking which bucketclasses to reconcile. mapping backingstore noobaa/noobaa-default-backing-store to bucketclasses"
time="2024-06-14T03:07:58Z" level=info msg="will reconcile these bucketclasses: [noobaa/noobaa-default-bucket-class]"
I0614 03:07:58.674908       1 request.go:697] Waited for 1.017705585s due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/batch/v1?timeout=32s
time="2024-06-14T03:07:58Z" level=info msg="Create event detected for noobaa-default-bucket-class (noobaa), queuing Reconcile"
time="2024-06-14T03:07:58Z" level=info msg="Start BucketClass Reconcile..." bucketclass=noobaa/noobaa-default-bucket-class
time="2024-06-14T03:07:58Z" level=info msg="✅ Exists: NooBaa \"noobaa\"\n"
time="2024-06-14T03:07:58Z" level=info msg="✅ Exists: BucketClass \"noobaa-default-bucket-class\"\n"
time="2024-06-14T03:07:58Z" level=info msg="SetPhase: Verifying" bucketclass=noobaa/noobaa-default-bucket-class
time="2024-06-14T03:07:58Z" level=info msg="validating placement policy &{Tiers:[{Placement: BackingStores:[noobaa-default-backing-store]}]}"
time="2024-06-14T03:07:58Z" level=info msg="✅ Exists: BackingStore \"noobaa-default-backing-store\"\n"
time="2024-06-14T03:07:58Z" level=info msg="validated placement policy successfully &{Tiers:[{Placement: BackingStores:[noobaa-default-backing-store]}]}"
time="2024-06-14T03:07:58Z" level=info msg="SetPhase: Configuring" bucketclass=noobaa/noobaa-default-bucket-class
time="2024-06-14T03:07:58Z" level=info msg="SetPhase: Ready" bucketclass=noobaa/noobaa-default-bucket-class
time="2024-06-14T03:07:58Z" level=info msg="✅ Done" bucketclass=noobaa/noobaa-default-bucket-class
time="2024-06-14T03:07:58Z" level=info msg="UpdateStatus: Done" bucketclass=noobaa/noobaa-default-bucket-class
I0614 03:08:08.697395       1 request.go:697] Waited for 10.647157001s due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/scheduling.k8s.io/v1?timeout=32s
time="2024-06-14T03:08:12Z" level=info msg="Create event detected for noobaa-default-backing-store-noobaa-pod-4e5fdaa7 (noobaa), queuing Reconcile"
time="2024-06-14T03:08:12Z" level=info msg="Create event detected for noobaa-default-backing-store-noobaa-pvc-4e5fdaa7 (noobaa), queuing Reconcile"
time="2024-06-14T03:08:14Z" level=info msg="Create event detected for noobaa-endpoint (noobaa), queuing Reconcile"
time="2024-06-14T03:08:14Z" level=info msg="Create event detected for noobaa (noobaa), queuing Reconcile"
time="2024-06-14T03:08:14Z" level=info msg="Create event detected for noobaa.noobaa.io (), queuing Reconcile"
time="2024-06-14T03:08:14Z" level=info msg="Create event detected for standard (), queuing Reconcile"
time="2024-06-14T03:08:15Z" level=info msg="Create event detected for backing-store-pv-pool-noobaa-default-backing-store (noobaa), queuing Reconcile"
time="2024-06-14T03:08:15Z" level=info msg="Create event detected for noobaa-admin (noobaa), queuing Reconcile"
time="2024-06-14T03:08:15Z" level=info msg="Create event detected for noobaa-db (noobaa), queuing Reconcile"
time="2024-06-14T03:08:15Z" level=info msg="Create event detected for noobaa-endpoints (noobaa), queuing Reconcile"
time="2024-06-14T03:08:15Z" level=info msg="Create event detected for noobaa-operator (noobaa), queuing Reconcile"
time="2024-06-14T03:08:15Z" level=info msg="Create event detected for noobaa-root-master-key-backend (noobaa), queuing Reconcile"
time="2024-06-14T03:08:15Z" level=info msg="Create event detected for noobaa-root-master-key-volume (noobaa), queuing Reconcile"
time="2024-06-14T03:08:15Z" level=info msg="Create event detected for noobaa-server (noobaa), queuing Reconcile"
time="2024-06-14T03:08:15Z" level=info msg="Create event detected for noobaa-core (noobaa), queuing Reconcile"
time="2024-06-14T03:08:15Z" level=info msg="Create event detected for noobaa-db-pg (noobaa), queuing Reconcile"
time="2024-06-14T03:08:15Z" level=info msg="Create event detected for default (noobaa), queuing Reconcile"
time="2024-06-14T03:08:15Z" level=info msg="Create event detected for noobaa-default-backing-store (noobaa), queuing Reconcile"
time="2024-06-14T03:08:15Z" level=info msg="Start ..." noobaaaccount=noobaa/default
time="2024-06-14T03:08:15Z" level=info msg="✅ Exists: NooBaaAccount \"default\"\n"
time="2024-06-14T03:08:15Z" level=info msg="✅ Exists: NooBaa \"noobaa\"\n"
time="2024-06-14T03:08:15Z" level=info msg="SetPhase: Verifying" noobaaaccount=noobaa/default
time="2024-06-14T03:08:15Z" level=info msg="SetPhase: Connecting" noobaaaccount=noobaa/default
time="2024-06-14T03:08:15Z" level=info msg="Create event detected for noobaa-config (noobaa), queuing Reconcile"
time="2024-06-14T03:08:15Z" level=info msg="Create event detected for noobaa-postgres-config (noobaa), queuing Reconcile"
time="2024-06-14T03:08:15Z" level=info msg="Create event detected for noobaa-postgres-initdb-sh (noobaa), queuing Reconcile"
time="2024-06-14T03:08:15Z" level=info msg="Create event detected for noobaa-db-pg (noobaa), queuing Reconcile"
time="2024-06-14T03:08:15Z" level=info msg="Create event detected for noobaa-mgmt (noobaa), queuing Reconcile"
time="2024-06-14T03:08:16Z" level=info msg="Create event detected for s3 (noobaa), queuing Reconcile"
time="2024-06-14T03:08:16Z" level=info msg="Create event detected for sts (noobaa), queuing Reconcile"
time="2024-06-14T03:08:16Z" level=info msg="Create event detected for noobaa-operator (noobaa), queuing Reconcile"
time="2024-06-14T03:08:16Z" level=info msg="checking which namespaceStores to reconcile. mapping secret noobaa/noobaa-operator to namespaceStores"
time="2024-06-14T03:08:16Z" level=info msg="will reconcile these namespaceStores: []"
time="2024-06-14T03:08:16Z" level=info msg="Create event detected for noobaa-root-master-key-backend (noobaa), queuing Reconcile"
time="2024-06-14T03:08:16Z" level=info msg="checking which namespaceStores to reconcile. mapping secret noobaa/noobaa-root-master-key-backend to namespaceStores"
time="2024-06-14T03:08:16Z" level=info msg="Create event detected for noobaa (noobaa), queuing Reconcile"
time="2024-06-14T03:08:16Z" level=info msg="will reconcile these namespaceStores: []"
time="2024-06-14T03:08:16Z" level=info msg="Create event detected for noobaa-root-master-key-volume (noobaa), queuing Reconcile"
time="2024-06-14T03:08:16Z" level=info msg="checking which namespaceStores to reconcile. mapping secret noobaa/noobaa-root-master-key-volume to namespaceStores"
time="2024-06-14T03:08:16Z" level=info msg="will reconcile these namespaceStores: []"
time="2024-06-14T03:08:16Z" level=info msg="Create event detected for noobaa-server (noobaa), queuing Reconcile"
time="2024-06-14T03:08:16Z" level=info msg="checking which namespaceStores to reconcile. mapping secret noobaa/noobaa-server to namespaceStores"
time="2024-06-14T03:08:16Z" level=info msg="Start BackingStore Reconcile ..." backingstore=noobaa/noobaa-default-backing-store
time="2024-06-14T03:08:16Z" level=info msg="Start NooBaa system Reconcile ..." sys=noobaa/noobaa
time="2024-06-14T03:08:16Z" level=info msg="will reconcile these namespaceStores: []"
time="2024-06-14T03:08:16Z" level=info msg="Create event detected for backing-store-pv-pool-noobaa-default-backing-store (noobaa), queuing Reconcile"
time="2024-06-14T03:08:16Z" level=info msg="checking which namespaceStores to reconcile. mapping secret noobaa/backing-store-pv-pool-noobaa-default-backing-store to namespaceStores"
time="2024-06-14T03:08:16Z" level=info msg="✅ Exists: NooBaa \"noobaa\"\n"
time="2024-06-14T03:08:16Z" level=info msg="✅ Exists: NooBaa \"noobaa\"\n"
time="2024-06-14T03:08:16Z" level=info msg="will reconcile these namespaceStores: []"
time="2024-06-14T03:08:16Z" level=info msg="Create event detected for noobaa-admin (noobaa), queuing Reconcile"
time="2024-06-14T03:08:16Z" level=info msg="checking which namespaceStores to reconcile. mapping secret noobaa/noobaa-admin to namespaceStores"
time="2024-06-14T03:08:16Z" level=info msg="✅ Exists: NooBaa \"noobaa\"\n"
time="2024-06-14T03:08:16Z" level=info msg="Memory Usage: Phase \"Starting\" - Alloc = 12 MiB  Sys = 34 MiB  NumGC = 10" sys=noobaa/noobaa
time="2024-06-14T03:08:16Z" level=info msg="SetPhase: \"Verifying\"" sys=noobaa/noobaa
time="2024-06-14T03:08:16Z" level=info msg="Parsed image (NamedTagged) noobaa/noobaa-core:master-20230920" func=CheckSystemCR sys=noobaa/noobaa
time="2024-06-14T03:08:16Z" level=info msg="Using custom image \"noobaa/noobaa-core:master-20230920\"" func=CheckSystemCR sys=noobaa/noobaa
time="2024-06-14T03:08:16Z" level=info msg="SetPhase: \"Creating\"" sys=noobaa/noobaa
time="2024-06-14T03:08:16Z" level=info msg="✅ Exists: BackingStore \"noobaa-default-backing-store\"\n"
time="2024-06-14T03:08:16Z" level=info msg="will reconcile these namespaceStores: []"
time="2024-06-14T03:08:16Z" level=info msg="Create event detected for noobaa-db (noobaa), queuing Reconcile"
time="2024-06-14T03:08:16Z" level=info msg="checking which namespaceStores to reconcile. mapping secret noobaa/noobaa-db to namespaceStores"
time="2024-06-14T03:08:16Z" level=info msg="✅ Exists: Service \"noobaa-mgmt\"\n"
time="2024-06-14T03:08:16Z" level=info msg="will reconcile these namespaceStores: []"
time="2024-06-14T03:08:16Z" level=info msg="Create event detected for noobaa-endpoints (noobaa), queuing Reconcile"
time="2024-06-14T03:08:16Z" level=info msg="checking which namespaceStores to reconcile. mapping secret noobaa/noobaa-endpoints to namespaceStores"
time="2024-06-14T03:08:16Z" level=info msg="will reconcile these namespaceStores: []"
time="2024-06-14T03:08:16Z" level=info msg="✅ Exists: Secret \"noobaa-operator\"\n"
time="2024-06-14T03:08:16Z" level=info msg="✅ Exists: Secret \"backing-store-pv-pool-noobaa-default-backing-store\"\n"
time="2024-06-14T03:08:16Z" level=info msg="✅ Exists:  \"backing-store-pv-pool-noobaa-default-backing-store\"\n"
time="2024-06-14T03:08:16Z" level=info msg="✅ Exists: Secret \"noobaa-admin\"\n"
time="2024-06-14T03:08:16Z" level=info msg="✈️  RPC: system.read_system() Request: <nil>"
time="2024-06-14T03:08:16Z" level=warning msg="RPC: GetConnection creating connection to wss://noobaa-mgmt.noobaa.svc.cluster.local:443/rpc/ 0xc000ba2180"
time="2024-06-14T03:08:16Z" level=info msg="RPC: Connecting websocket (0xc000ba2180) &{RPC:0xc0008bfb30 Address:wss://noobaa-mgmt.noobaa.svc.cluster.local:443/rpc/ State:init WS:<nil> PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s cancelPings:<nil>}"
time="2024-06-14T03:08:16Z" level=info msg="ReconcileObject: Done - unchanged ServiceAccount noobaa " sys=noobaa/noobaa
time="2024-06-14T03:08:16Z" level=info msg="SetPhase: Verifying" backingstore=noobaa/noobaa-default-backing-store
time="2024-06-14T03:08:16Z" level=info msg="SetPhase: Connecting" backingstore=noobaa/noobaa-default-backing-store
time="2024-06-14T03:08:16Z" level=info msg="ReconcileObject: Done - unchanged ConfigMap noobaa-config " sys=noobaa/noobaa
time="2024-06-14T03:08:17Z" level=warning msg="Discovery of OAuth endpoints failed, got: Get \"https://openshift.default.svc/.well-known/oauth-authorization-server\": dial tcp: lookup openshift.default.svc on 10.96.0.10:53: no such host" sys=noobaa/noobaa
time="2024-06-14T03:08:17Z" level=info msg="ReconcileObject: Done - unchanged Secret noobaa-server " sys=noobaa/noobaa
time="2024-06-14T03:08:17Z" level=info msg="ReconcileObject: Done - unchanged Secret noobaa-db " sys=noobaa/noobaa
time="2024-06-14T03:08:17Z" level=info msg="✅ Exists: NooBaa \"noobaa\"\n"
time="2024-06-14T03:08:17Z" level=info msg="setKMSConditionType k8s" sys=noobaa/noobaa
time="2024-06-14T03:08:17Z" level=info msg="ReconcileObject: Done - unchanged Secret noobaa-root-master-key-volume " sys=noobaa/noobaa
time="2024-06-14T03:08:17Z" level=info msg="setKMSConditionStatus Sync" sys=noobaa/noobaa
time="2024-06-14T03:08:17Z" level=info msg="ReconcileKeyRotation, KMS Starting" sys=noobaa/noobaa
time="2024-06-14T03:08:17Z" level=info msg="ReconcileKeyRotation, KMS skip reconcile, single master root mode" sys=noobaa/noobaa
time="2024-06-14T03:08:17Z" level=info msg="✅ Exists: Service \"noobaa-mgmt\"\n"
time="2024-06-14T03:08:17Z" level=info msg="✅ Exists: Secret \"noobaa-operator\"\n"
time="2024-06-14T03:08:17Z" level=info msg="❌ CRD Missing: SecurityContextConstraints \"noobaa-db\"\n"
time="2024-06-14T03:08:17Z" level=info msg="ReconcileObject: Done - unchanged ServiceAccount noobaa-db " sys=noobaa/noobaa
time="2024-06-14T03:08:17Z" level=info msg="✅ Exists: Secret \"noobaa-admin\"\n"
time="2024-06-14T03:08:17Z" level=info msg="✈️  RPC: system.read_system() Request: <nil>"
time="2024-06-14T03:08:17Z" level=info msg="RPC: Connected websocket (0xc000ba2180) &{RPC:0xc0008bfb30 Address:wss://noobaa-mgmt.noobaa.svc.cluster.local:443/rpc/ State:init WS:<nil> PendingRequests:map[] NextRequestID:0 Lock:{state:9 sema:0} ReconnectDelay:0s cancelPings:<nil>}"
time="2024-06-14T03:08:17Z" level=info msg="ReconcileObject: Done - unchanged Role noobaa-db " sys=noobaa/noobaa
time="2024-06-14T03:08:17Z" level=info msg="✅ RPC: system.read_system() Response OK: took 96.7ms"
time="2024-06-14T03:08:17Z" level=info msg="SetPhase: Configuring" noobaaaccount=noobaa/default
time="2024-06-14T03:08:17Z" level=info msg="✅ RPC: system.read_system() Response OK: took 73.1ms"
time="2024-06-14T03:08:17Z" level=info msg="ReconcileObject: Done - unchanged RoleBinding noobaa-db " sys=noobaa/noobaa
time="2024-06-14T03:08:17Z" level=info msg="✈️  RPC: host.list_hosts() Request: {Query:{Pools:[noobaa-default-backing-store]}}"
time="2024-06-14T03:08:17Z" level=info msg="ReconcileObject: Done - unchanged ConfigMap noobaa-postgres-initdb-sh " sys=noobaa/noobaa
time="2024-06-14T03:08:17Z" level=info msg="ReconcileObject: Done - unchanged ConfigMap noobaa-postgres-config " sys=noobaa/noobaa
time="2024-06-14T03:08:17Z" level=info msg="ReconcileObject: Done - unchanged StatefulSet noobaa-db-pg " sys=noobaa/noobaa
time="2024-06-14T03:08:17Z" level=info msg="ReconcileObject: Done - unchanged Service noobaa-db-pg " sys=noobaa/noobaa
time="2024-06-14T03:08:17Z" level=info msg="✅ RPC: host.list_hosts() Response OK: took 10.8ms"
time="2024-06-14T03:08:17Z" level=info msg="SetPhase: Creating" backingstore=noobaa/noobaa-default-backing-store
time="2024-06-14T03:08:17Z" level=info msg="ReconcileObject: Done - unchanged Service noobaa-mgmt " sys=noobaa/noobaa
time="2024-06-14T03:08:17Z" level=info msg="✅ Exists: ServiceAccount \"default\"\n"
time="2024-06-14T03:08:17Z" level=info msg="SetPhase: Ready" backingstore=noobaa/noobaa-default-backing-store
time="2024-06-14T03:08:17Z" level=info msg="UpdateStatus: Done" backingstore=noobaa/noobaa-default-backing-store
time="2024-06-14T03:08:17Z" level=info msg="Start BackingStore Reconcile ..." backingstore=noobaa/noobaa-default-backing-store
time="2024-06-14T03:08:17Z" level=info msg="✅ Exists: NooBaa \"noobaa\"\n"
time="2024-06-14T03:08:18Z" level=info msg="Update event detected for noobaa-default-backing-store (noobaa), queuing Reconcile"
time="2024-06-14T03:08:18Z" level=info msg="checking which bucketclasses to reconcile. mapping backingstore noobaa/noobaa-default-backing-store to bucketclasses"
time="2024-06-14T03:08:18Z" level=info msg="ReconcileObject: Done - unchanged StatefulSet noobaa-core " sys=noobaa/noobaa
time="2024-06-14T03:08:18Z" level=info msg="✅ Exists: Service \"noobaa-mgmt\"\n"
time="2024-06-14T03:08:18Z" level=info msg="will reconcile these bucketclasses: [noobaa/noobaa-default-bucket-class]"
time="2024-06-14T03:08:18Z" level=info msg="checking which bucketclasses to reconcile. mapping backingstore noobaa/noobaa-default-backing-store to bucketclasses"
time="2024-06-14T03:08:18Z" level=info msg="✅ Exists: NooBaa \"noobaa\"\n"
time="2024-06-14T03:08:18Z" level=info msg="Start BucketClass Reconcile..." bucketclass=noobaa/noobaa-default-bucket-class
time="2024-06-14T03:08:18Z" level=info msg="will reconcile these bucketclasses: [noobaa/noobaa-default-bucket-class]"
time="2024-06-14T03:08:18Z" level=info msg="✅ Exists: Secret \"noobaa-operator\"\n"
time="2024-06-14T03:08:18Z" level=info msg="✅ Exists: BackingStore \"noobaa-default-backing-store\"\n"
time="2024-06-14T03:08:18Z" level=info msg="✅ Exists: Secret \"noobaa-admin\"\n"
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1f48c44]

goroutine 1673 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:116 +0x1e5
panic({0x22d49c0?, 0x4299610?})
	/opt/homebrew/Cellar/go/1.22.2/libexec/src/runtime/panic.go:770 +0x132
github.com/noobaa/noobaa-operator/v5/pkg/noobaaaccount.(*Reconciler).CreateNooBaaAccount(0xc0011ab5e0)
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/pkg/noobaaaccount/reconciler.go:336 +0xe4
github.com/noobaa/noobaa-operator/v5/pkg/noobaaaccount.(*Reconciler).ReconcilePhaseConfiguring(0xc0011ab5e0)
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/pkg/noobaaaccount/reconciler.go:266 +0x9b
github.com/noobaa/noobaa-operator/v5/pkg/noobaaaccount.(*Reconciler).ReconcilePhases(0xc0011ab5e0)
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/pkg/noobaaaccount/reconciler.go:154 +0x36
github.com/noobaa/noobaa-operator/v5/pkg/noobaaaccount.(*Reconciler).Reconcile(0xc0011ab5e0)
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/pkg/noobaaaccount/reconciler.go:111 +0x105
github.com/noobaa/noobaa-operator/v5/pkg/controller/noobaaaccount.Add.func1({0xc000101808?, 0xc000d0e980?}, {{{0xc001053f78?, 0xc001588d20?}, {0xc001053f70?, 0xc001588cf0?}}})
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/pkg/controller/noobaaaccount/noobaaaccount_controller.go:33 +0xd6
sigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile(0x7f4347e125b8?, {0x2e5fc48?, 0xc00110d3b0?}, {{{0xc001053f78?, 0xc000d0e980?}, {0xc001053f70?, 0xc001588d50?}}})
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/vendor/sigs.k8s.io/controller-runtime/pkg/reconcile/reconcile.go:111 +0x3d
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x2e65b18?, {0x2e5fc48?, 0xc00110d3b0?}, {{{0xc001053f78?, 0xb?}, {0xc001053f70?, 0x0?}}})
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0005e8c80, {0x2e5fc80, 0xc0007f6af0}, {0x23ed400, 0xc001510ea0})
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316 +0x3bc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0005e8c80, {0x2e5fc80, 0xc0007f6af0})
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266 +0x1c9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 209
	/Users/usrivast/dev/noobaa/origs/noobaa-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:223 +0x50c
root@ubuntu-011:~#

[guymguym]

fyi @nimrod-becker