Skip to content

Cinder cinder-volume fails to load RBDDriver due to missing rados/rbd Python libraries in container image #1147

@vish6760

Description

@vish6760

Describe the bug
When deploying Cinder with Ceph (RBD backend). The cinder-volume pods fail to initialize the RBDDriver. Logs show that the required Python module for Ceph (rados and rbd) is missing from the container image (ghcr.io/rackerlabs/genestack-images/cinder:2024.1-latest), and not sure why we are using heat images in the cinder overrides config at https://github.com/rackerlabs/genestack/blob/main/base-helm-configs/cinder/cinder-helm-overrides.yaml#L8

To Reproduce
Steps to reproduce the behavior:

  1. Updated the required override config for ceph in /etc/genestack/helm-configs/cinder/cinder-helm-overrides.yaml
  2. Run the Cinder deployment Script /opt/genestack/bin/install-cinder.sh
  3. Review the cinder-volume pod logs to identify the errors indicating missing Python modules.

Expected behavior
Cinder-volume images should have the Python module for Ceph (rados and rbd), and the drive should be initialized successfully.

Server (please complete the following information):

  • OS: Ubuntu 22.04

Additional context
Run the Cinder deployment Script /opt/genestack/bin/install-cinder.sh

(genestack) root@controller-01:~# /opt/genestack/bin/install-cinder.sh
"openstack-helm" already exists with the same configuration, skipping
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "metallb" chart repository
...Successfully got an update from the "kubeovn" chart repository
...Successfully got an update from the "fluent" chart repository
...Successfully got an update from the "mariadb-operator" chart repository
...Successfully got an update from the "openstack-helm-infra" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "openstack-helm" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈Happy Helming!⎈
Executing Helm command:
helm upgrade --install cinder openstack-helm/cinder --version 2024.2.409+13651f45-628a320c     --namespace=openstack     --timeout 120m -f /opt/genestack/base-helm-configs/cinder/cinder-helm-overrides.yaml -f /etc/genestack/helm-configs/global_overrides/endpoints.yaml -f /etc/genestack/helm-configs/cinder/cinder-helm-overrides.yaml --set endpoints.identity.auth.admin.password="$(kubectl --namespace openstack get secret keystone-admin -o jsonpath='{.data.password}' | base64 -d)" --set endpoints.identity.auth.cinder.password="$(kubectl --namespace openstack get secret cinder-admin -o jsonpath='{.data.password}' | base64 -d)" --set endpoints.oslo_db.auth.admin.password="$(kubectl --namespace openstack get secret mariadb -o jsonpath='{.data.root-password}' | base64 -d)" --set endpoints.oslo_db.auth.cinder.password="$(kubectl --namespace openstack get secret cinder-db-password -o jsonpath='{.data.password}' | base64 -d)" --set endpoints.oslo_cache.auth.memcache_secret_key="$(kubectl --namespace openstack get secret os-memcached -o jsonpath='{.data.memcache_secret_key}' | base64 -d)" --set conf.cinder.keystone_authtoken.memcache_secret_key="$(kubectl --namespace openstack get secret os-memcached -o jsonpath='{.data.memcache_secret_key}' | base64 -d)" --set conf.cinder.database.slave_connection="mysql+pymysql://cinder:$(kubectl --namespace openstack get secret cinder-db-password -o jsonpath='{.data.password}' | base64 -d)@mariadb-cluster-secondary.openstack.svc.cluster.local:3306/cinder" --set endpoints.oslo_messaging.auth.admin.password="$(kubectl --namespace openstack get secret rabbitmq-default-user -o jsonpath='{.data.password}' | base64 -d)" --set endpoints.oslo_messaging.auth.cinder.password="$(kubectl --namespace openstack get secret cinder-rabbitmq-password -o jsonpath='{.data.password}' | base64 -d)" --post-renderer /etc/genestack/kustomize/kustomize.sh --post-renderer-args cinder/overlay
Release "cinder" has been upgraded. Happy Helming!
NAME: cinder
LAST DEPLOYED: Mon Aug 25 23:54:08 2025
NAMESPACE: openstack
STATUS: deployed
REVISION: 1
TEST SUITE: None

cinder-volume container logs show:

2025-08-25 08:26:02.243 8 ERROR cinder.volume.manager Failed to initialize driver.: cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: rados and rbd python libraries not found
Traceback (most recent call last):
  File "/var/lib/openstack/lib/python3.12/site-packages/cinder/volume/manager.py", line 525, in _init_host
    self.driver.check_for_setup_error()
  File "/var/lib/openstack/lib/python3.12/site-packages/cinder/volume/drivers/rbd.py", line 535, in check_for_setup_error
    raise exception.VolumeBackendAPIException(data=msg)
cinder.exception.VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: rados and rbd python libraries not found

Pod image details:

(genestack) root@controller-01:~# kubectl describe pod cinder-volume-57cd58b46-47cd4 -n openstack |grep Image
    Image:         ghcr.io/rackerlabs/genestack-images/kubernetes-entrypoint:latest
    Image ID:      ghcr.io/rackerlabs/genestack-images/kubernetes-entrypoint@sha256:b03af9b08d8cb68f83b5c000b704039ab52411a8140ef0a96fce9db3b63fb930
    Image:         ghcr.io/rackerlabs/genestack-images/cinder:2024.1-latest
    Image ID:      ghcr.io/rackerlabs/genestack-images/cinder@sha256:7885951ba986ede47d0967a11e26ef40e55b81db2cefd19168f1b55db9d71d88
    Image:         ghcr.io/rackerlabs/genestack-images/cinder:2024.1-latest
    Image ID:      ghcr.io/rackerlabs/genestack-images/cinder@sha256:7885951ba986ede47d0967a11e26ef40e55b81db2cefd19168f1b55db9d71d88
    Image:         ghcr.io/rackerlabs/genestack-images/heat:2024.1-latest
    Image ID:      ghcr.io/rackerlabs/genestack-images/heat@sha256:8d49f629eb3486c6fd665aab312229052a3c94c2792f15bdd65a5e0d6cdd79c9
    Image:         ghcr.io/rackerlabs/genestack-images/cinder:2024.1-latest
    Image ID:      ghcr.io/rackerlabs/genestack-images/cinder@sha256:7885951ba986ede47d0967a11e26ef40e55b81db2cefd19168f1b55db9d71d88

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions