[BUG] rhdh container segfaults on startup with CRC on Intel Mac

[reestr]

General information

• OS: macOS
• Hypervisor: vfkit
• Did you run crc setup before starting it (Yes/No)? Yes
• Running CRC on: Mac Pro

CRC version

CRC version: 2.33.0+c43b17
OpenShift version: 4.14.12
Podman version: 4.4.4

CRC status

DEBU CRC version: 2.33.0+c43b17
DEBU OpenShift version: 4.14.12
DEBU Podman version: 4.4.4
DEBU Running 'crc status'
CRC VM:          Running
OpenShift:       Running (v4.14.12)
RAM Usage:       8.573GB of 50.51GB
Disk Usage:      31.62GB of 214.1GB (Inside the CRC VM)
Cache Usage:     178.9GB
Cache Directory: /Users/trees/.crc/cache

CRC config

- consent-telemetry                     : no
- cpus                                  : 12
- disk-size                             : 200
- memory                                : 49152

Host Operating System

ProductName:		macOS
ProductVersion:		14.4.1
BuildVersion:		23E224

Steps to reproduce

1. From CRC VM, podman run registry.redhat.io/rhdh/rhdh-hub-rhel9:1.1-97

Expected

Container succesfully starts, only presenting javascript error when probing for plugins.

podman run -it registry.redhat.io/rhdh/rhdh-hub-rhel9:1.1-97 /bin/bash
Loading config from MergedConfigSource{FileConfigSource{path="/opt/app-root/src/app-config.yaml"}, FileConfigSource{path="/opt/app-root/src/app-config.example.yaml"}, FileConfigSource{path="/opt/app-root/src/app-config.example.production.yaml"}, EnvConfigSource{count=0}}
2024-03-27T19:46:29.984Z backstage info Found 0 new secrets in config that will be redacted
2024-03-27T19:46:30.103Z backstage info loaded dynamic backend plugin 'backstage-plugin-azure-devops-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-azure-devops-backend-dynamic'
2024-03-27T19:46:30.115Z backstage info loaded dynamic backend plugin 'backstage-plugin-catalog-backend-module-bitbucket-cloud-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-catalog-backend-module-bitbucket-cloud-dynamic'
2024-03-27T19:46:30.126Z backstage info loaded dynamic backend plugin 'backstage-plugin-catalog-backend-module-bitbucket-server-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-catalog-backend-module-bitbucket-server-dynamic'
2024-03-27T19:46:30.158Z backstage info loaded dynamic backend plugin 'backstage-plugin-catalog-backend-module-github-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-catalog-backend-module-github-dynamic'
2024-03-27T19:46:30.190Z backstage info loaded dynamic backend plugin 'backstage-plugin-catalog-backend-module-github-org-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-catalog-backend-module-github-org-dynamic'
2024-03-27T19:46:30.222Z backstage info loaded dynamic backend plugin 'backstage-plugin-catalog-backend-module-gitlab-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-catalog-backend-module-gitlab-dynamic'
2024-03-27T19:46:30.281Z backstage info loaded dynamic backend plugin 'backstage-plugin-jenkins-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-jenkins-backend-dynamic'
2024-03-27T19:46:30.915Z backstage info loaded dynamic backend plugin 'backstage-plugin-kubernetes-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-kubernetes-backend-dynamic'
2024-03-27T19:46:31.037Z backstage info loaded dynamic backend plugin 'backstage-plugin-scaffolder-backend-module-azure-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-scaffolder-backend-module-azure-dynamic'
2024-03-27T19:46:31.068Z backstage info loaded dynamic backend plugin 'backstage-plugin-scaffolder-backend-module-bitbucket-cloud-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-scaffolder-backend-module-bitbucket-cloud-dynamic'
2024-03-27T19:46:31.086Z backstage info loaded dynamic backend plugin 'backstage-plugin-scaffolder-backend-module-bitbucket-server-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-scaffolder-backend-module-bitbucket-server-dynamic'
2024-03-27T19:46:31.123Z backstage info loaded dynamic backend plugin 'backstage-plugin-scaffolder-backend-module-gerrit-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-scaffolder-backend-module-gerrit-dynamic'
2024-03-27T19:46:31.264Z backstage info loaded dynamic backend plugin 'backstage-plugin-scaffolder-backend-module-github-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-scaffolder-backend-module-github-dynamic'
2024-03-27T19:46:31.378Z backstage info loaded dynamic backend plugin 'backstage-plugin-scaffolder-backend-module-gitlab-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-scaffolder-backend-module-gitlab-dynamic'
2024-03-27T19:46:31.427Z backstage info loaded dynamic backend plugin 'backstage-plugin-sonarqube-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-sonarqube-backend-dynamic'
2024-03-27T19:46:31.917Z backstage info loaded dynamic backend plugin 'backstage-plugin-techdocs-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-techdocs-backend-dynamic'
2024-03-27T19:46:31.999Z backstage info loaded dynamic backend plugin 'immobiliarelabs-backstage-plugin-gitlab-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/immobiliarelabs-backstage-plugin-gitlab-backend-dynamic'
2024-03-27T19:46:32.001Z backstage info loaded dynamic backend plugin '@janus-idp/backstage-plugin-3scale-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-3scale-backend-dynamic/alpha'
2024-03-27T19:46:32.002Z backstage info loaded dynamic backend plugin '@janus-idp/backstage-plugin-aap-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-aap-backend-dynamic/alpha'
2024-03-27T19:46:32.031Z backstage info loaded dynamic backend plugin '@janus-idp/backstage-plugin-keycloak-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-keycloak-backend-dynamic/alpha'
2024-03-27T19:46:32.431Z backstage info loaded dynamic backend plugin '@janus-idp/backstage-plugin-ocm-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-ocm-backend-dynamic/alpha'
2024-03-27T19:46:32.433Z backstage info loaded dynamic backend plugin '@janus-idp/backstage-scaffolder-backend-module-quay-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-scaffolder-backend-module-quay-dynamic/alpha'
2024-03-27T19:46:32.485Z backstage info loaded dynamic backend plugin '@janus-idp/backstage-scaffolder-backend-module-regex-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-scaffolder-backend-module-regex-dynamic/alpha'
2024-03-27T19:46:32.533Z backstage info loaded dynamic backend plugin '@janus-idp/backstage-scaffolder-backend-module-servicenow-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-scaffolder-backend-module-servicenow-dynamic/alpha'
2024-03-27T19:46:32.577Z backstage info loaded dynamic backend plugin '@janus-idp/backstage-scaffolder-backend-module-sonarqube-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-scaffolder-backend-module-sonarqube-dynamic/alpha'
2024-03-27T19:46:32.640Z backstage info loaded dynamic backend plugin 'roadiehq-backstage-plugin-argo-cd-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/roadiehq-backstage-plugin-argo-cd-backend-dynamic'
2024-03-27T19:46:32.718Z backstage info loaded dynamic backend plugin 'roadiehq-scaffolder-backend-argocd-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/roadiehq-scaffolder-backend-argocd-dynamic'
2024-03-27T19:46:32.774Z backstage info loaded dynamic backend plugin 'roadiehq-scaffolder-backend-module-utils-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/roadiehq-scaffolder-backend-module-utils-dynamic'
2024-03-27T19:46:32.784Z rootHttpRouter info Listening on :7007
2024-03-27T19:46:32.803Z auth info Enabled Provider Factories : {}
2024-03-27T19:46:32.831Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-azure-devops' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-azure-devops'
2024-03-27T19:46:32.831Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-dynatrace' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-dynatrace'
2024-03-27T19:46:32.831Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-github-actions' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-github-actions'
2024-03-27T19:46:32.831Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-github-issues' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-github-issues'
2024-03-27T19:46:32.832Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-jenkins' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-jenkins'
2024-03-27T19:46:32.832Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-kubernetes' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-kubernetes'
2024-03-27T19:46:32.832Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-lighthouse' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-lighthouse'
2024-03-27T19:46:32.832Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-sonarqube' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-sonarqube'
2024-03-27T19:46:32.832Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-tech-radar' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-tech-radar'
2024-03-27T19:46:32.833Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-techdocs' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-techdocs'
2024-03-27T19:46:32.833Z scalprum info Loaded dynamic frontend plugin 'immobiliarelabs-backstage-plugin-gitlab' from 'file:///opt/app-root/src/dynamic-plugins-root/immobiliarelabs-backstage-plugin-gitlab'
2024-03-27T19:46:32.833Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-acr' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-acr'
2024-03-27T19:46:32.834Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-analytics-provider-segment' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-analytics-provider-segment'
2024-03-27T19:46:32.834Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-dynamic-plugins-info' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-dynamic-plugins-info'
2024-03-27T19:46:32.834Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-jfrog-artifactory' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-jfrog-artifactory'
2024-03-27T19:46:32.834Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-nexus-repository-manager' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-nexus-repository-manager'
2024-03-27T19:46:32.834Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-ocm' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-ocm'
2024-03-27T19:46:32.835Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-quay' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-quay'
2024-03-27T19:46:32.835Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-rbac' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-rbac'
2024-03-27T19:46:32.835Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-tekton' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-tekton'
2024-03-27T19:46:32.835Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-topology' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-topology'
2024-03-27T19:46:32.835Z scalprum info Loaded dynamic frontend plugin 'pagerduty-backstage-plugin' from 'file:///opt/app-root/src/dynamic-plugins-root/pagerduty-backstage-plugin'
2024-03-27T19:46:32.836Z scalprum info Loaded dynamic frontend plugin 'roadiehq-backstage-plugin-argo-cd' from 'file:///opt/app-root/src/dynamic-plugins-root/roadiehq-backstage-plugin-argo-cd'
2024-03-27T19:46:32.836Z scalprum info Loaded dynamic frontend plugin 'roadiehq-backstage-plugin-datadog' from 'file:///opt/app-root/src/dynamic-plugins-root/roadiehq-backstage-plugin-datadog'
2024-03-27T19:46:32.836Z scalprum info Loaded dynamic frontend plugin 'roadiehq-backstage-plugin-github-insights' from 'file:///opt/app-root/src/dynamic-plugins-root/roadiehq-backstage-plugin-github-insights'
2024-03-27T19:46:32.836Z scalprum info Loaded dynamic frontend plugin 'roadiehq-backstage-plugin-github-pull-requests' from 'file:///opt/app-root/src/dynamic-plugins-root/roadiehq-backstage-plugin-github-pull-requests'
2024-03-27T19:46:32.836Z scalprum info Loaded dynamic frontend plugin 'roadiehq-backstage-plugin-jira' from 'file:///opt/app-root/src/dynamic-plugins-root/roadiehq-backstage-plugin-jira'
2024-03-27T19:46:32.836Z scalprum info Loaded dynamic frontend plugin 'roadiehq-backstage-plugin-security-insights' from 'file:///opt/app-root/src/dynamic-plugins-root/roadiehq-backstage-plugin-security-insights'
2024-03-27T19:46:32.841Z proxy info [HPM] Proxy created: /quay/api  -> https://quay.io/
2024-03-27T19:46:32.842Z proxy info [HPM] Proxy rewrite rule created: "^/api/proxy/quay/api/?" ~> "/"
2024-03-27T19:46:32.843Z proxy warn skipped configuring /sonarqube due to Proxy target for route "/sonarqube" must be a string, but is of type undefined
2024-03-27T19:46:32.843Z proxy warn skipped configuring /jenkins/api due to Proxy target for route "/jenkins/api" must be a string, but is of type undefined
2024-03-27T19:46:32.844Z proxy warn skipped configuring /jira/api due to Proxy target for route "/jira/api" must be a string, but is of type undefined
2024-03-27T19:46:32.844Z proxy warn skipped configuring /acr/api due to Proxy target for route "/acr/api" must be a string, but is of type undefined
2024-03-27T19:46:32.844Z proxy warn skipped configuring /jfrog-artifactory/api due to Proxy target for route "/jfrog-artifactory/api" must be a string, but is of type undefined
2024-03-27T19:46:32.844Z proxy info [HPM] Proxy created: /pagerduty  -> https://api.pagerduty.com
2024-03-27T19:46:32.844Z proxy info [HPM] Proxy rewrite rule created: "^/api/proxy/pagerduty/?" ~> "/"
2024-03-27T19:46:32.845Z proxy warn skipped configuring /dynatrace due to Proxy target for route "/dynatrace" must be a string, but is of type undefined
2024-03-27T19:46:32.845Z proxy warn skipped configuring /nexus-repository-manager due to Proxy target for route "/nexus-repository-manager" must be a string, but is of type undefined
2024-03-27T19:46:32.851Z techdocs info Creating Local publisher for TechDocs
2024-03-27T19:46:32.852Z backstage info Resolving '@backstage/plugin-techdocs-backend/package.json' in the dynamic backend plugins
/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1781
              throw new errors.ForwardedError(
                    ^

ForwardedError: Module 'gitlabFillerProcessor' for plugin 'catalog' startup failed; caused by Error: Missing required config value at 'integrations.gitlab'
    at /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1781:21
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1780:13
    at async processNode (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1302:22)
    ... 7 lines matching cause stack trace ...
    at async BackstageBackend.start (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1881:5) {
  cause: Error: Missing required config value at 'integrations.gitlab'
      at ConfigReader.getConfigArray (/opt/app-root/src/node_modules/@backstage/config/dist/index.cjs.js:218:13)
      at ObservableConfigProxy.getConfigArray (/opt/app-root/src/node_modules/@backstage/config-loader/dist/index.cjs.js:1327:30)
      at new GitlabFillerProcessor (/opt/app-root/src/dynamic-plugins-root/immobiliarelabs-backstage-plugin-gitlab-backend-dynamic/dist/index.cjs.js:137:14)
      at Object.init [as func] (/opt/app-root/src/dynamic-plugins-root/immobiliarelabs-backstage-plugin-gitlab-backend-dynamic/dist/index.cjs.js:201:37)
      at /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1780:35
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async processNode (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1302:22)
      at async Promise.all (index 7)
      at async processMoreNodes (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1297:7)
      at async _DependencyGraph.parallelTopologicalTraversal (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1308:5)
      at async /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1777:9
      at async Promise.all (index 1)
      at async BackendInitializer.doStart_fn (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1757:3)
      at async BackendInitializer.start (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1611:5)
      at async BackstageBackend.start (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1881:5)
}

Node.js v18.19.0

Actual

Container does not start correctly, it instead segfaults.

Logs

No crc logs beyond starting:

tail -f .crc/crc.log
time="2024-03-28T10:55:20+01:00" level=debug msg="Running 'crc version'"
time="2024-03-28T10:55:21+01:00" level=debug msg="No new version available. The latest version is 2.33.0"
time="2024-03-28T10:55:42+01:00" level=debug msg="CRC version: 2.33.0+c43b17"
time="2024-03-28T10:55:42+01:00" level=debug msg="OpenShift version: 4.14.12"
time="2024-03-28T10:55:42+01:00" level=debug msg="Podman version: 4.4.4"
time="2024-03-28T10:55:42+01:00" level=debug msg="Running 'crc status'"
time="2024-03-28T10:56:05+01:00" level=debug msg="CRC version: 2.33.0+c43b17\n"
time="2024-03-28T10:56:05+01:00" level=debug msg="OpenShift version: 4.14.12\n"
time="2024-03-28T10:56:05+01:00" level=debug msg="Podman version: 4.4.4\n"
time="2024-03-28T10:56:05+01:00" level=debug msg="Running 'crc config view'"

Journal for the CRC VM shows the segfault:

Mar 28 08:17:36 crc-vlf7c-master-0 podman[89386]: 2024-03-28 08:17:36.87998515 +0000 UTC m=+0.249987179 container create 47f8ada02f5b3f9913c1f18a170b1cb
632e39eea7c0134b388190b00ccebf5f8 (image=registry.redhat.io/rhdh/rhdh-hub-rhel9:1.1-97, name=priceless_gould, distribution-scope=public, com.redhat.lice
nse_terms=https://www.redhat.com/agreements, maintainer=RHDH Team <rhdh-bot@redhat.com>, vendor=Red Hat, Inc., usage=, name=rhdh/rhdh-hub-rhel9, help=Fo
r more information visit https://github.com/sclorg/s2i-nodejs-container, description=Red Hat Developer Hub container, url=https://access.redhat.com/cont
ainers/#/registry.access.redhat.com/rhdh/rhdh-hub-rhel9/images/1.1-97, io.k8s.description=Red Hat Developer Hub container, vcs-type=git, io.buildah.version=1.29.0, io.k8s.display-name=Red Hat Developer Hub container, io.openshift.expose-services=, com.redhat.dev-mode=DEV_MODE:false, com.redhat.component=rhdh-hub-container, license=EPLv2, io.openshift.tags=rhdh,hub, build-date=2024-03-13T03:28:54, io.s2i.scripts-url=image:///usr/libexec/s2i, architecture=x86_64, version=1.1, io.openshift.s2i.scripts-url=image:///usr/libexec/s2i, com.redhat.dev-mode.port=DEBUG_PORT:5858, release=97, summary=Red Hat Developer Hub container, vcs-ref=d75bd51e4e5156722859d34de73a599a259971fc, com.redhat.deployments-dir=/opt/app-root/src)                                    Mar 28 08:17:36 crc-vlf7c-master-0 NetworkManager[1375]: <info>  [1711613856.8923] manager: (vethb004e9a6): new Veth device (/org/freedesktop/NetworkManager/Devices/217)                                                                                                                                       Mar 28 08:17:36 crc-vlf7c-master-0 kernel: cni-podman0: port 1(vethb004e9a6) entered blocking state                                                     Mar 28 08:17:36 crc-vlf7c-master-0 kernel: cni-podman0: port 1(vethb004e9a6) entered disabled state                                                     
Mar 28 08:17:36 crc-vlf7c-master-0 kernel: device vethb004e9a6 entered promiscuous mode                                                                 Mar 28 08:17:36 crc-vlf7c-master-0 NetworkManager[1375]: <info>  [1711613856.9010] device (vethb004e9a6): carrier: link connected                       
Mar 28 08:17:36 crc-vlf7c-master-0 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready                   
Mar 28 08:17:36 crc-vlf7c-master-0 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethb004e9a6: link becomes ready                                              
Mar 28 08:17:36 crc-vlf7c-master-0 kernel: cni-podman0: port 1(vethb004e9a6) entered blocking state                                                     
Mar 28 08:17:36 crc-vlf7c-master-0 kernel: cni-podman0: port 1(vethb004e9a6) entered forwarding state                                                   
Mar 28 08:17:36 crc-vlf7c-master-0 NetworkManager[1375]: <info>  [1711613856.9016] device (cni-podman0): carrier: link connected 
Mar 28 08:17:37 crc-vlf7c-master-0 systemd[1]: Started libpod-conmon-47f8ada02f5b3f9913c1f18a170b1cb632e39eea7c0134b388190b00ccebf5f8.scope.
Mar 28 08:17:37 crc-vlf7c-master-0 systemd[1]: Started libcrun container.                                                                               
Mar 28 08:17:37 crc-vlf7c-master-0 podman[89386]: 2024-03-28 08:17:37.224923003 +0000 UTC m=+0.594925037 container init 47f8ada02f5b3f9913c1f18a170b1cb6
32e39eea7c0134b388190b00ccebf5f8 (image=registry.redhat.io/rhdh/rhdh-hub-rhel9:1.1-97, name=priceless_gould, com.redhat.dev-mode=DEV_MODE:false, io.open
shift.s2i.scripts-url=image:///usr/libexec/s2i, build-date=2024-03-13T03:28:54, io.buildah.version=1.29.0, name=rhdh/rhdh-hub-rhel9, com.redhat.componen
t=rhdh-hub-container, distribution-scope=public, help=For more information visit https://github.com/sclorg/s2i-nodejs-container, vcs-ref=d75bd51e4e51567
22859d34de73a599a259971fc, com.redhat.dev-mode.port=DEBUG_PORT:5858, url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhdh/rhdh-hub
-rhel9/images/1.1-97, architecture=x86_64, vcs-type=git, maintainer=RHDH Team <rhdh-bot@redhat.com>, io.openshift.expose-services=, io.k8s.description=R
ed Hat Developer Hub container, io.k8s.display-name=Red Hat Developer Hub container, usage=, license=EPLv2, summary=Red Hat Developer Hub container, des
cription=Red Hat Developer Hub container, vendor=Red Hat, Inc., version=1.1, io.openshift.tags=rhdh,hub, io.s2i.scripts-url=image:///usr/libexec/s2i, re
lease=97, com.redhat.license_terms=https://www.redhat.com/agreements, com.redhat.deployments-dir=/opt/app-root/src)
Mar 28 08:17:37 crc-vlf7c-master-0 podman[89386]: 2024-03-28 08:17:37.266338227 +0000 UTC m=+0.636340272 container start 47f8ada02f5b3f9913c1f18a170b1cb
632e39eea7c0134b388190b00ccebf5f8 (image=registry.redhat.io/rhdh/rhdh-hub-rhel9:1.1-97, name=priceless_gould, distribution-scope=public, com.redhat.depl
oyments-dir=/opt/app-root/src, io.openshift.tags=rhdh,hub, build-date=2024-03-13T03:28:54, io.k8s.description=Red Hat Developer Hub container, io.opensh
ift.expose-services=, architecture=x86_64, description=Red Hat Developer Hub container, maintainer=RHDH Team <rhdh-bot@redhat.com>, url=https://access.r
edhat.com/containers/#/registry.access.redhat.com/rhdh/rhdh-hub-rhel9/images/1.1-97, com.redhat.dev-mode=DEV_MODE:false, com.redhat.dev-mode.port=DEBUG_
PORT:5858, io.buildah.version=1.29.0, com.redhat.license_terms=https://www.redhat.com/agreements, license=EPLv2, help=For more information visit https:/
/github.com/sclorg/s2i-nodejs-container, com.redhat.component=rhdh-hub-container, vcs-ref=d75bd51e4e5156722859d34de73a599a259971fc, vcs-type=git, io.k8s.display-name=Red Hat Developer Hub container, usage=, name=rhdh/rhdh-hub-rhel9, version=1.1, summary=Red Hat Developer Hub container, io.openshift.s2i.scripts-url=image:///usr/libexec/s2i, io.s2i.scripts-url=image:///usr/libexec/s2i, release=97, vendor=Red Hat, Inc.)                                    
Mar 28 08:17:37 crc-vlf7c-master-0 podman[89386]: 2024-03-28 08:17:37.266609278 +0000 UTC m=+0.636611306 container attach 47f8ada02f5b3f9913c1f18a170b1cb632e39eea7c0134b388190b00ccebf5f8 (image=registry.redhat.io/rhdh/rhdh-hub-rhel9:1.1-97, name=priceless_gould, io.s2i.scripts-url=image:///usr/libexec/s2i, io.openshift.expose-services=, com.redhat.dev-mode=DEV_MODE:false, com.redhat.license_terms=https://www.redhat.com/agreements, vcs-ref=d75bd51e4e5156722859d34de73a599a259971fc, com.redhat.dev-mode.port=DEBUG_PORT:5858, help=For more information visit https://github.com/sclorg/s2i-nodejs-container, vcs-type=git, io.k8s.description=Red Hat Developer Hub container, description=Red Hat Developer Hub container, io.buildah.version=1.29.0, io.openshift.tags=rhdh,hub, maintainer=RHDH Team <rhdh-bot@redhat.com>, name=rhdh/rhdh-hub-rhel9, distribution-scope=public, com.redhat.component=rhdh-hub-container, s
ummary=Red Hat Developer Hub container, io.openshift.s2i.scripts-url=image:///usr/libexec/s2i, com.redhat.deployments-dir=/opt/app-root/src, url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhdh/rhdh-hub-rhel9/images/1.1-97, version=1.1, build-date=2024-03-13T03:28:54, release=97, v
endor=Red Hat, Inc., io.k8s.display-name=Red Hat Developer Hub container, usage=, license=EPLv2, architecture=x86_64)
Mar 28 08:17:39 crc-vlf7c-master-0 priceless_gould[89543]: Loading config from MergedConfigSource{FileConfigSource{path="/opt/app-root/src/app-config.ya
ml"}, FileConfigSource{path="/opt/app-root/src/app-config.example.yaml"}, FileConfigSource{path="/opt/app-root/src/app-config.example.production.yaml"},
 EnvConfigSource{count=0}}                                                                                                                              
Mar 28 08:17:39 crc-vlf7c-master-0 kernel: node[89545]: segfault at 38c2b5fef588 ip 000038c2b5fef580 sp 00007ffc96abfa38 error 25
Mar 28 08:17:39 crc-vlf7c-master-0 kernel: Code: Unable to access opcode bytes at RIP 0x38c2b5fef556. 
Mar 28 08:17:39 crc-vlf7c-master-0 systemd[1]: Started Process Core Dump (PID 89557/UID 0).                  
Mar 28 08:17:41 crc-vlf7c-master-0 systemd-coredump[89558]: [_] Process 89545 (node) of user 1001 dumped core.
                                                                                                                                                                                                                    Stack trace of thread 1:                                                                                                                                #0  0x000038c2b5fef580 n/a (n/a + 0x0)                                                                                                                  #1  0x000038c2b6022c6b n/a (n/a + 0x0)                                                                                                                  #2  0x000038c2b601a9d0 n/a (n/a + 0x0)                                                                                                                  #3  0x000055bfa9eeac1f n/a (/usr/bin/node + 0x146cc1f)                                                                                                  #4  0x000055bfa9e76990 n/a (/usr/bin/node + 0x13f8990)                                                                                                  #5  0x000055bfa9e76990 n/a (/usr/bin/node + 0x13f8990)                                                                                                  #6  0x000055bfa9e76990 n/a (/usr/bin/node + 0x13f8990) 
                                                            #7  0x000055bfa9f42f31 n/a (/usr/bin/node + 0x14c4f31)                                     
                                                            #8  0x000055bfa9e9bdfb n/a (/usr/bin/node + 0x141ddfb)                                     
                                                            #9  0x000055bfa9e74ec3 n/a (/usr/bin/node + 0x13f6ec3)
                                                            #10 0x000055bfa96dd011 n/a (/usr/bin/node + 0xc5f011)
                                                            #11 0x000055bfa96ddc70 n/a (/usr/bin/node + 0xc5fc70)
                                                            #12 0x000055bfa96dddb4 n/a (/usr/bin/node + 0xc5fdb4)                                                                                                   #13 0x000055bfa970da39 n/a (/usr/bin/node + 0xc8fa39)                                                                                                   #14 0x000055bfa970dd63 n/a (/usr/bin/node + 0xc8fd63)                                                                                                   #15 0x000055bfa92c09b2 n/a (/usr/bin/node + 0x8429b2)                                                                                                   #16 0x000055bfa92c0bf5 n/a (/usr/bin/node + 0x842bf5)                                                                                                   #17 0x000055bfa941cb31 n/a (/usr/bin/node + 0x99eb31)                                                                                                   #18 0x000055bfa941e525 n/a (/usr/bin/node + 0x9a0525)                                                                                                   #19 0x000055bfa9e5532e n/a (/usr/bin/node + 0x13d732e)                                                                                                  #20 0x000055bfa9e67fe4 n/a (/usr/bin/node + 0x13e9fe4)    
                                                            #21 0x000055bfa9e55cb9 n/a (/usr/bin/node + 0x13d7cb9)                                                                                                  #22 0x000055bfa92c1a5a n/a (/usr/bin/node + 0x843a5a)
                                                            #23 0x000055bfa93efceb n/a (/usr/bin/node + 0x971ceb)
                                                            #24 0x000055bfa93f01a5 n/a (/usr/bin/node + 0x9721a5)
                                                            #25 0x000055bfa93510cd n/a (/usr/bin/node + 0x8d30cd)
                                                            #26 0x000055bfa9354e72 n/a (/usr/bin/node + 0x8d6e72)
                                                            #27 0x00007f8ebcac9eb0 n/a (/usr/lib64/libc.so.6 + 0x3feb0)
                                                            ELF object binary architecture: AMD x86-64
Mar 28 08:17:41 crc-vlf7c-master-0 systemd[1]: systemd-coredump@17-89557-0.service: Deactivated successfully.
Mar 28 08:17:41 crc-vlf7c-master-0 systemd[1]: systemd-coredump@17-89557-0.service: Consumed 1.669s CPU time.
Mar 28 08:17:41 crc-vlf7c-master-0 podman[89386]: 2024-03-28 08:17:41.554552588 +0000 UTC m=+4.924554625 container died 47f8ada02f5b3f9913c1f18a170b1cb632e39eea7c0134b388190b00ccebf5f8 (image=registry.redhat.io/rhdh/rhdh-hub-rhel9:1.1-97, name=priceless_gould, name=rhdh/rhdh-hub-rhel9, version=1.1, com.redhat.dev-mode.port=DEBUG_PORT:5858, release=97, vcs-ref=d75bd51e4e5156722859d34de73a599a259971fc, io.openshift.s2i.scripts-url=image:///usr/libexec/s2i, distribution-scope=public, architecture=x86_64, maintainer=RHDH Team <rhdh-bot@redhat.com>, io.openshift.expose-services=, io.buildah.version=1.29.0, com.redhat.license_terms=https://www.redhat.com/agreements, io.s2i.scripts-url=image:///usr/libexec/s2i, vcs-type=git, io.k8s.display-name=Red Hat Developer Hub container, summary=Red Hat Developer Hub container, license=EPLv2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhdh/rhdh-hub-rhel9/images/1.1-97, description=Red Hat Developer Hub container, com.redhat.deployments-dir=/opt/app-root/src, usage=, help=For more information visit https://github.com/sclorg/s2i-nodejs-container, io.k8s.description=Red Hat Developer Hub container, io.openshift.tags=rhdh,hub, build-date=2024-03-13T03:28:54, com.redhat.component=rhdh-hub-container, vendor=Red Hat, Inc., com.redhat.dev-mode=DEV_MODE:false)
Mar 28 08:17:41 crc-vlf7c-master-0 systemd[1]: libpod-47f8ada02f5b3f9913c1f18a170b1cb632e39eea7c0134b388190b00ccebf5f8.scope: Deactivated successfully.
Mar 28 08:17:41 crc-vlf7c-master-0 systemd[1]: libpod-47f8ada02f5b3f9913c1f18a170b1cb632e39eea7c0134b388190b00ccebf5f8.scope: Consumed 3.107s CPU time.
Mar 28 08:17:41 crc-vlf7c-master-0 kernel: cni-podman0: port 1(vethb004e9a6) entered disabled state
Mar 28 08:17:41 crc-vlf7c-master-0 kernel: device vethb004e9a6 left promiscuous mode
Mar 28 08:17:41 crc-vlf7c-master-0 kernel: cni-podman0: port 1(vethb004e9a6) entered disabled state
Mar 28 08:17:41 crc-vlf7c-master-0 podman[89679]: 2024-03-28 08:17:41.976224669 +0000 UTC m=+0.408813988 container cleanup 47f8ada02f5b3f9913c1f18a170b1cb632e39eea7c0134b388190b00ccebf5f8 (image=registry.redhat.io/rhdh/rhdh-hub-rhel9:1.1-97, name=priceless_gould, maintainer=RHDH Team <rhdh-bot@redhat.com>, version=1.1, build-date=2024-03-13T03:28:54, summary=Red Hat Developer Hub container, description=Red Hat Developer Hub container, release=97, com.redhat.deployments-dir=/opt/app-root/src, com.redhat.license_terms=https://www.redhat.com/agreements, io.openshift.expose-services=, com.redhat.dev-mode=DEV_MODE:false, vcs-ref=d75bd51e4e5156722859d34de73a599a259971fc, vcs-type=git, architecture=x86_64, com.redhat.component=rhdh-hub-container, io.buildah.version=1.29.0, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhdh/rhdh-hub-rhel9/images/1.1-97, usage=, name=rhdh/rhdh-hub-rhel9, com.redhat.dev-mode.port=DEBUG_PORT:5858, io.k8s.display-name=Red Hat Developer Hub container, io.openshift.s2i.scripts-url=image:///usr/libexec/s2i, help=For more information visit https://github.com/sclorg/s2i-nodejs-container, distribution-scope=public, io.openshift.tags=rhdh,hub, license=EPLv2, io.s2i.scripts-url=image:///usr/libexec/s2i, io.k8s.description=Red Hat Developer Hub container)
Mar 28 08:17:41 crc-vlf7c-master-0 systemd[1]: libpod-conmon-47f8ada02f5b3f9913c1f18a170b1cb632e39eea7c0134b388190b00ccebf5f8.scope: Deactivated successfully.

Podman output:
Journal for the CRC VM shows the segfault:

[root@crc-vlf7c-master-0 /]# podman start registry.redhat.io/rhdh/rhdh-hub-rhel9:1.1-97Error: no container with name or ID "registry.redhat.io/rhdh/rhdh-hub-rhel9:1.1-97" found: no such container[root@crc-vlf7c-master-0 /]# podman run registry.redhat.io/rhdh/rhdh-hub-rhel9:1.1-97Loading config from MergedConfigSource{FileConfigSource{path="/opt/app-root/src/app-config.yaml"}, FileConfigSource{path="/opt/app-root/src/app-config.example.yaml"}, FileConfigSource{path="/opt/app-root/src/app-config.example.production.yaml"}, EnvConfigSource{count=0}}[root@crc-vlf7c-master-0 /]# 

Please consider posting the output of crc start --log-level debug on http://gist.github.com/ and post the link in the issue.

This issue is specific to CRC on MacOS, there are reports of rhdh running as expected on CRC with Fedora 39.

I am not able to reproduce this issue using a Vanilla Fedora 39 Server install with Podman using Parallels (Apple Virtulization Framework) with the same hardware (Intel MacOS). RHDH runs as expected on this.

[gbraad]

is this running on an Intel or M-based mac?

[reestr]

is this running on an Intel or M-based mac?

Per Issue subject; Intel Mac. Right now there are no aarch images for RHDH.

[gbraad]

On M1 it would have to run emulated, which might trigger an issue. It seems the node process segfaults, but you mentioned this works 'inside a Fedora 39 VM with podman 4.9.3'. How do you run this VM? Qemu, vfkit, VMware/parallels, etc.

[reestr]

On M1 it would have to run emulated, which might trigger an issue. It seems the node process segfaults, but you mentioned this works 'inside a Fedora 39 VM with podman 4.9.3'. How do you run this VM? Qemu, vfkit, VMware/parallels, etc.

Parallels, using Apple Virtualization Framework as it's engine.

[cfergeau]

I was not able to reproduce this on my intel macbook, but it's still using macos12, so maybe that's why

$ podman run registry.redhat.io/rhdh/rhdh-hub-rhel9:1.1-97
[...]
2024-03-28T14:14:19.782Z backstage info Resolving '@backstage/plugin-techdocs-backend/package.json' in the dynamic backend plugins 
/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1781
              throw new errors.ForwardedError(
                    ^

ForwardedError: Module 'gitlabFillerProcessor' for plugin 'catalog' startup failed; caused by Error: Missing required config value at 'integrations.gitlab'
    at /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1781:21
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1780:13
    at async processNode (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1302:22)
    ... 7 lines matching cause stack trace ...
    at async BackstageBackend.start (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1881:5) {
  cause: Error: Missing required config value at 'integrations.gitlab'
      at ConfigReader.getConfigArray (/opt/app-root/src/node_modules/@backstage/config/dist/index.cjs.js:218:13)
      at ObservableConfigProxy.getConfigArray (/opt/app-root/src/node_modules/@backstage/config-loader/dist/index.cjs.js:1327:30)
      at new GitlabFillerProcessor (/opt/app-root/src/dynamic-plugins-root/immobiliarelabs-backstage-plugin-gitlab-backend-dynamic/dist/index.cjs.js:137:14)
      at Object.init [as func] (/opt/app-root/src/dynamic-plugins-root/immobiliarelabs-backstage-plugin-gitlab-backend-dynamic/dist/index.cjs.js:201:37)
      at /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1780:35
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async processNode (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1302:22)
      at async Promise.all (index 7)
      at async processMoreNodes (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1297:7)
      at async _DependencyGraph.parallelTopologicalTraversal (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1308:5)
      at async /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1777:9
      at async Promise.all (index 1)
      at async BackendInitializer.doStart_fn (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1757:3)
      at async BackendInitializer.start (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1611:5)
      at async BackstageBackend.start (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1881:5)
}

$ journalctl --grep 'dumped core'
-- Boot b29117ed08684f3bae03d2d380eaf10b --
-- Boot 40b7534aa06a4a4f978727ac5b0287e4 --
-- No entries --

[cfergeau]

I also can't reproduce after upgrading my intel macbook to macos13

Tim has run more tests, booting the crc VM using https://github.com/crc-org/vfkit/blob/main/contrib/scripts/start-crc-bundle.sh does not result in a crash, it's only when running the VM through CRC (and thus with the cluster running) that the crash occurs.

@reestr Another test I'd be curious about is to reproduce the bug with crc (crc start, reproduce the bug, crc stop), and then boot the image from ~/.crc/machine/crc/crc.img using https://gist.github.com/cfergeau/a0847af2986ec8526cb20f323ce00dce
You will need to use .crc/machines/crc/id_ecdsa to ssh into the VM, and then try to run the container which crashed before.

[praveenkumar]

I also can't reproduce it on my intel macbook to macos14.4

$ sw_vers                                                                                              
ProductName:		macOS
ProductVersion:		14.4.1
BuildVersion:		23E224

crc with openshift preset running as expected.

% crc   status
CRC VM:          Running
OpenShift:       Running (v4.14.12)
RAM Usage:       7.182GB of 10.96GB
Disk Usage:      25.21GB of 32.68GB (Inside the CRC VM)
Cache Usage:     72.18GB
Cache Directory: /Users/prkumar/.crc/cache

Get the expected output when running rhdh container

Loading config from MergedConfigSource{FileConfigSource{path="/opt/app-root/src/app-config.yaml"}, FileConfigSource{path="/opt/app-root/src/app-config.example.yaml"}, FileConfigSource{path="/opt/app-root/src/app-config.example.production.yaml"}, EnvConfigSource{count=0}}
2024-04-01T11:42:06.537Z backstage info Found 0 new secrets in config that will be redacted 
2024-04-01T11:42:07.414Z backstage info loaded dynamic backend plugin 'backstage-plugin-azure-devops-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-azure-devops-backend-dynamic' 
2024-04-01T11:42:07.479Z backstage info loaded dynamic backend plugin 'backstage-plugin-catalog-backend-module-bitbucket-cloud-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-catalog-backend-module-bitbucket-cloud-dynamic' 
2024-04-01T11:42:07.579Z backstage info loaded dynamic backend plugin 'backstage-plugin-catalog-backend-module-bitbucket-server-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-catalog-backend-module-bitbucket-server-dynamic' 
2024-04-01T11:42:07.795Z backstage info loaded dynamic backend plugin 'backstage-plugin-catalog-backend-module-github-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-catalog-backend-module-github-dynamic' 
2024-04-01T11:42:08.009Z backstage info loaded dynamic backend plugin 'backstage-plugin-catalog-backend-module-github-org-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-catalog-backend-module-github-org-dynamic' 
2024-04-01T11:42:08.172Z backstage info loaded dynamic backend plugin 'backstage-plugin-catalog-backend-module-gitlab-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-catalog-backend-module-gitlab-dynamic' 
2024-04-01T11:42:08.550Z backstage info loaded dynamic backend plugin 'backstage-plugin-jenkins-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-jenkins-backend-dynamic' 
2024-04-01T11:42:11.789Z backstage info loaded dynamic backend plugin 'backstage-plugin-kubernetes-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-kubernetes-backend-dynamic' 
2024-04-01T11:42:12.273Z backstage info loaded dynamic backend plugin 'backstage-plugin-scaffolder-backend-module-azure-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-scaffolder-backend-module-azure-dynamic' 
2024-04-01T11:42:12.460Z backstage info loaded dynamic backend plugin 'backstage-plugin-scaffolder-backend-module-bitbucket-cloud-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-scaffolder-backend-module-bitbucket-cloud-dynamic' 
2024-04-01T11:42:12.561Z backstage info loaded dynamic backend plugin 'backstage-plugin-scaffolder-backend-module-bitbucket-server-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-scaffolder-backend-module-bitbucket-server-dynamic' 
2024-04-01T11:42:12.749Z backstage info loaded dynamic backend plugin 'backstage-plugin-scaffolder-backend-module-gerrit-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-scaffolder-backend-module-gerrit-dynamic' 
2024-04-01T11:42:13.488Z backstage info loaded dynamic backend plugin 'backstage-plugin-scaffolder-backend-module-github-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-scaffolder-backend-module-github-dynamic' 
2024-04-01T11:42:14.017Z backstage info loaded dynamic backend plugin 'backstage-plugin-scaffolder-backend-module-gitlab-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-scaffolder-backend-module-gitlab-dynamic' 
2024-04-01T11:42:14.297Z backstage info loaded dynamic backend plugin 'backstage-plugin-sonarqube-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-sonarqube-backend-dynamic' 
2024-04-01T11:42:16.462Z backstage info loaded dynamic backend plugin 'backstage-plugin-techdocs-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-techdocs-backend-dynamic' 
2024-04-01T11:42:16.869Z backstage info loaded dynamic backend plugin 'immobiliarelabs-backstage-plugin-gitlab-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/immobiliarelabs-backstage-plugin-gitlab-backend-dynamic' 
2024-04-01T11:42:16.878Z backstage info loaded dynamic backend plugin '@janus-idp/backstage-plugin-3scale-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-3scale-backend-dynamic/alpha' 
2024-04-01T11:42:16.882Z backstage info loaded dynamic backend plugin '@janus-idp/backstage-plugin-aap-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-aap-backend-dynamic/alpha' 
2024-04-01T11:42:17.042Z backstage info loaded dynamic backend plugin '@janus-idp/backstage-plugin-keycloak-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-keycloak-backend-dynamic/alpha' 
2024-04-01T11:42:19.296Z backstage info loaded dynamic backend plugin '@janus-idp/backstage-plugin-ocm-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-ocm-backend-dynamic/alpha' 
2024-04-01T11:42:19.314Z backstage info loaded dynamic backend plugin '@janus-idp/backstage-scaffolder-backend-module-quay-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-scaffolder-backend-module-quay-dynamic/alpha' 
2024-04-01T11:42:19.505Z backstage info loaded dynamic backend plugin '@janus-idp/backstage-scaffolder-backend-module-regex-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-scaffolder-backend-module-regex-dynamic/alpha' 
2024-04-01T11:42:19.725Z backstage info loaded dynamic backend plugin '@janus-idp/backstage-scaffolder-backend-module-servicenow-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-scaffolder-backend-module-servicenow-dynamic/alpha' 
2024-04-01T11:42:19.896Z backstage info loaded dynamic backend plugin '@janus-idp/backstage-scaffolder-backend-module-sonarqube-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-scaffolder-backend-module-sonarqube-dynamic/alpha' 
2024-04-01T11:42:20.202Z backstage info loaded dynamic backend plugin 'roadiehq-backstage-plugin-argo-cd-backend-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/roadiehq-backstage-plugin-argo-cd-backend-dynamic' 
2024-04-01T11:42:20.528Z backstage info loaded dynamic backend plugin 'roadiehq-scaffolder-backend-argocd-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/roadiehq-scaffolder-backend-argocd-dynamic' 
2024-04-01T11:42:20.812Z backstage info loaded dynamic backend plugin 'roadiehq-scaffolder-backend-module-utils-dynamic' from 'file:///opt/app-root/src/dynamic-plugins-root/roadiehq-scaffolder-backend-module-utils-dynamic' 
2024-04-01T11:42:20.864Z rootHttpRouter info Listening on :7007 
2024-04-01T11:42:20.959Z auth info Enabled Provider Factories : {} 
2024-04-01T11:42:21.079Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-azure-devops' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-azure-devops'  
2024-04-01T11:42:21.084Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-dynatrace' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-dynatrace'  
2024-04-01T11:42:21.085Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-github-actions' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-github-actions'  
2024-04-01T11:42:21.086Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-github-issues' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-github-issues'  
2024-04-01T11:42:21.087Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-jenkins' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-jenkins'  
2024-04-01T11:42:21.088Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-kubernetes' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-kubernetes'  
2024-04-01T11:42:21.089Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-lighthouse' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-lighthouse'  
2024-04-01T11:42:21.090Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-sonarqube' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-sonarqube'  
2024-04-01T11:42:21.091Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-tech-radar' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-tech-radar'  
2024-04-01T11:42:21.093Z scalprum info Loaded dynamic frontend plugin 'backstage-plugin-techdocs' from 'file:///opt/app-root/src/dynamic-plugins-root/backstage-plugin-techdocs'  
2024-04-01T11:42:21.094Z scalprum info Loaded dynamic frontend plugin 'immobiliarelabs-backstage-plugin-gitlab' from 'file:///opt/app-root/src/dynamic-plugins-root/immobiliarelabs-backstage-plugin-gitlab'  
2024-04-01T11:42:21.095Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-acr' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-acr'  
2024-04-01T11:42:21.097Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-analytics-provider-segment' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-analytics-provider-segment'  
2024-04-01T11:42:21.098Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-dynamic-plugins-info' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-dynamic-plugins-info'  
2024-04-01T11:42:21.099Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-jfrog-artifactory' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-jfrog-artifactory'  
2024-04-01T11:42:21.100Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-nexus-repository-manager' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-nexus-repository-manager'  
2024-04-01T11:42:21.101Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-ocm' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-ocm'  
2024-04-01T11:42:21.102Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-quay' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-quay'  
2024-04-01T11:42:21.103Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-rbac' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-rbac'  
2024-04-01T11:42:21.104Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-tekton' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-tekton'  
2024-04-01T11:42:21.105Z scalprum info Loaded dynamic frontend plugin '@janus-idp/backstage-plugin-topology' from 'file:///opt/app-root/src/dynamic-plugins-root/janus-idp-backstage-plugin-topology'  
2024-04-01T11:42:21.106Z scalprum info Loaded dynamic frontend plugin 'pagerduty-backstage-plugin' from 'file:///opt/app-root/src/dynamic-plugins-root/pagerduty-backstage-plugin'  
2024-04-01T11:42:21.107Z scalprum info Loaded dynamic frontend plugin 'roadiehq-backstage-plugin-argo-cd' from 'file:///opt/app-root/src/dynamic-plugins-root/roadiehq-backstage-plugin-argo-cd'  
2024-04-01T11:42:21.108Z scalprum info Loaded dynamic frontend plugin 'roadiehq-backstage-plugin-datadog' from 'file:///opt/app-root/src/dynamic-plugins-root/roadiehq-backstage-plugin-datadog'  
2024-04-01T11:42:21.109Z scalprum info Loaded dynamic frontend plugin 'roadiehq-backstage-plugin-github-insights' from 'file:///opt/app-root/src/dynamic-plugins-root/roadiehq-backstage-plugin-github-insights'  
2024-04-01T11:42:21.110Z scalprum info Loaded dynamic frontend plugin 'roadiehq-backstage-plugin-github-pull-requests' from 'file:///opt/app-root/src/dynamic-plugins-root/roadiehq-backstage-plugin-github-pull-requests'  
2024-04-01T11:42:21.115Z scalprum info Loaded dynamic frontend plugin 'roadiehq-backstage-plugin-jira' from 'file:///opt/app-root/src/dynamic-plugins-root/roadiehq-backstage-plugin-jira'  
2024-04-01T11:42:21.117Z scalprum info Loaded dynamic frontend plugin 'roadiehq-backstage-plugin-security-insights' from 'file:///opt/app-root/src/dynamic-plugins-root/roadiehq-backstage-plugin-security-insights'  
2024-04-01T11:42:21.142Z proxy info [HPM] Proxy created: /quay/api  -> https://quay.io/ 
2024-04-01T11:42:21.148Z proxy info [HPM] Proxy rewrite rule created: "^/api/proxy/quay/api/?" ~> "/" 
2024-04-01T11:42:21.157Z proxy warn skipped configuring /sonarqube due to Proxy target for route "/sonarqube" must be a string, but is of type undefined 
2024-04-01T11:42:21.158Z proxy warn skipped configuring /jenkins/api due to Proxy target for route "/jenkins/api" must be a string, but is of type undefined 
2024-04-01T11:42:21.160Z proxy warn skipped configuring /jira/api due to Proxy target for route "/jira/api" must be a string, but is of type undefined 
2024-04-01T11:42:21.161Z proxy warn skipped configuring /acr/api due to Proxy target for route "/acr/api" must be a string, but is of type undefined 
2024-04-01T11:42:21.161Z proxy warn skipped configuring /jfrog-artifactory/api due to Proxy target for route "/jfrog-artifactory/api" must be a string, but is of type undefined 
2024-04-01T11:42:21.162Z proxy info [HPM] Proxy created: /pagerduty  -> https://api.pagerduty.com 
2024-04-01T11:42:21.162Z proxy info [HPM] Proxy rewrite rule created: "^/api/proxy/pagerduty/?" ~> "/" 
2024-04-01T11:42:21.166Z proxy warn skipped configuring /dynatrace due to Proxy target for route "/dynatrace" must be a string, but is of type undefined 
2024-04-01T11:42:21.166Z proxy warn skipped configuring /nexus-repository-manager due to Proxy target for route "/nexus-repository-manager" must be a string, but is of type undefined 
2024-04-01T11:42:21.200Z techdocs info Creating Local publisher for TechDocs 
2024-04-01T11:42:21.205Z backstage info Resolving '@backstage/plugin-techdocs-backend/package.json' in the dynamic backend plugins 
/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1781
              throw new errors.ForwardedError(
                    ^

ForwardedError: Module 'gitlabFillerProcessor' for plugin 'catalog' startup failed; caused by Error: Missing required config value at 'integrations.gitlab'
    at /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1781:21
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1780:13
    at async processNode (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1302:22)
    ... 7 lines matching cause stack trace ...
    at async BackstageBackend.start (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1881:5) {
  cause: Error: Missing required config value at 'integrations.gitlab'
      at ConfigReader.getConfigArray (/opt/app-root/src/node_modules/@backstage/config/dist/index.cjs.js:218:13)
      at ObservableConfigProxy.getConfigArray (/opt/app-root/src/node_modules/@backstage/config-loader/dist/index.cjs.js:1327:30)
      at new GitlabFillerProcessor (/opt/app-root/src/dynamic-plugins-root/immobiliarelabs-backstage-plugin-gitlab-backend-dynamic/dist/index.cjs.js:137:14)
      at Object.init [as func] (/opt/app-root/src/dynamic-plugins-root/immobiliarelabs-backstage-plugin-gitlab-backend-dynamic/dist/index.cjs.js:201:37)
      at /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1780:35
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async processNode (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1302:22)
      at async Promise.all (index 7)
      at async processMoreNodes (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1297:7)
      at async _DependencyGraph.parallelTopologicalTraversal (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1308:5)
      at async /opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1777:9
      at async Promise.all (index 1)
      at async BackendInitializer.doStart_fn (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1757:3)
      at async BackendInitializer.start (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1611:5)
      at async BackstageBackend.start (/opt/app-root/src/node_modules/@backstage/backend-app-api/dist/index.cjs.js:1881:5)
}

Node.js v18.19.0

[reestr]

I also can't reproduce it on my intel macbook to macos14.4

Can you please share which version of OpenShift/CoreOS crc is running?

[praveenkumar]

I also can't reproduce it on my intel macbook to macos14.4

Can you please share which version of OpenShift/CoreOS crc is running?

I am using the released crc 2.33 which comes with 4.14.12

$ crc version
CRC version: 2.33.0+c43b17
OpenShift version: 4.14.12
Podman version: 4.4.4

[praveenkumar]

After having a quick sync-up with @reestr following is the observation

• Running crc with OCP bundle but without kubelet service started, every run of rhdh container (using podman) succeed without any segfault.
• As soon as kubelet service started and then try to run rhdh container (using podman) resulted to segfault.
• We again stop the kubelet service and try to run rhdh container without any segfault.

Looks like something to do with kubelet service and rhdh container and it is only happen with @reestr`s system. On my mac every single time it succeed (no segfault)

[cfergeau]

@reestr summary was slightly different

quick write up of findings.
After building crc which doesn't start kubelet and using it with the existing installed bundle (4.14.12) we were seeing rhdh start successfully from podman on repeated tries. Then after starting the kublet service, attempting to use podman to start rhdh would always segfault. After stopping the kubelet service, we would then see rhdh consistently start without segfault.
I requested to then try the latest bundle, to see if there were any relevant fixes in RHCOS/OCP...
After starting this bundle with the kubelet disabled, the first attempt to start rhdh failed, after a number of tries it did successfully start. Starting the kubelet made no difference, rhdh would sometimes start but more often segfault.

[praveenkumar]

@reestr summary was slightly different

@cfergeau in terms of unreleased bundle (till then) yes, but with released bundle (4.14.12) it is same. With 4.15.3 there is other observation where even without kubelet start it failed 2-3 times but then succeed after that multiple time. But with kubelet started it only succeed once and failed multiple time.