Summary
The openstack-cloud-controller charm's update-status hook crashes with an unhandled
httpx.ConnectError whenever the Kubernetes API is temporarily unreachable. The unit is then
permanently stuck in Juju error state and requires manual intervention (juju resolve) to recover.
This caused a CI pipeline failure in the canonicalk8s-openstack integration test run on 2026-06-26.
Affected component
Field | Value
Charm | openstack-cloud-controller
Channel | latest/stable
Revision | 48
Source file | src/charm.py
Method | _update_status → _check_node_provider_ids
Environment
Field | Value
k8s charm | 1.35/stable rev 1846 (k8s 1.35.0)
Cloud | OpenStack Caracal / Noble
Juju | 3.6.24
Base | ubuntu@24.04
Model | canonicalk8s-openstack on foundation-openstack
Observed behaviour
The update-status hook crashes with an uncaught exception:
unit-openstack-cloud-controller-0: 01:55:48 ERROR unit.openstack-cloud-controller/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
File ".../src/charm.py", line 147, in _update_status
nodes_without_provider_id = self._check_node_provider_ids()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../src/charm.py", line 126, in _check_node_provider_ids
for node in client.list(Node):
File ".../lightkube/core/generic_client.py", line 109, in __iter__
for rv, chunk in self._inner_iter:
....
httpx.ConnectError: [Errno 113] No route to host
unit-openstack-cloud-controller-0: 01:55:48 ERROR juju.worker.uniter.operation hook "update-status" (via hook dispatching script: dispatch) failed: exit status 1
The unit is then permanently locked in Juju error state:
openstack-cloud-controller/0* error idle hook failed: "update-status"
It does not self-recover. It must be manually resolved with juju resolve openstack-cloud-controller/0.
Expected behaviour
The _update_status / _check_node_provider_ids method should catch connection errors from the
Kubernetes API client (at minimum httpx.ConnectError, httpx.ConnectTimeout, and
httpcore.ConnectError) and set a transient waiting status instead of allowing the exception to
propagate and crash the hook. The k8s API being temporarily unreachable is a normal, expected
condition during cluster maintenance operations (leader replacement, snap upgrades, etc.).
Root cause
In src/charm.py, the _check_node_provider_ids method calls client.list(Node) via the
lightkube Kubernetes client with no error handling around network connectivity:
# charm.py ~line 126
def _check_node_provider_ids(self):
for node in client.list(Node): # <-- no try/except around this
...
When the k8s API endpoint is unreachable, lightkube → httpx → httpcore raises
httpx.ConnectError: [Errno 113] No route to host. This exception propagates all the way up to
ops.main() at charm.py:283, causing the hook to exit with status 1 and Juju to mark the unit
as errored.
Suggested fix
Wrap the Kubernetes API calls in _update_status (and any other hook handler that calls the k8s
API) to catch connectivity errors and set a waiting status:
from lightkube.core.exceptions import ApiError
import httpx
def _check_node_provider_ids(self):
try:
for node in client.list(Node):
...
except (httpx.ConnectError, httpx.ConnectTimeout, httpx.TimeoutException) as e:
raise ops.model.WaitingStatus(f"Kubernetes API temporarily unreachable: {e}")
Or, at the _update_status handler level:
def _update_status(self, event):
try:
nodes_without_provider_id = self._check_node_provider_ids()
...
except (httpx.ConnectError, httpx.ConnectTimeout) as e:
self.unit.status = ops.model.WaitingStatus(f"k8s API unreachable: {e}")
return
Impact
Unit permanently stuck in error state after any k8s API disruption
CI pipeline fails: juju.errors.JujuUnitError: Unit in error: openstack-cloud-controller/0
Downstream tests (e.g. test_extra_sans) that call wait_for_idle(status="active") fail even when the test itself is otherwise correct
Manual juju resolve required to unblock the pipeline
Additional context: secondary test design gap
The test_k8s test suite (test_remove_leader_control_plane) performs a leader control-plane
replacement but only waits for k8s/* and k8s-worker/* units to become healthy. It does not
check openstack-cloud-controller or cinder-csi health after the operation. As a result:
The update-status hook failure occurs during test_remove_leader_control_plane (~01:47–01:53)
test_k8s reports integration-k8s: OK without detecting the broken cloud-controller
The errored unit is silently handed off to the test_openstack suite
test_extra_sans (the fourth openstack test) is the first test to call wait_for_idle(status="active") on the full model and fails
Recommendation: test_k8s (and any test that modifies k8s topology on a model with OpenStack
integrations) should include a post-operation assertion that all units — including subordinates —
return to a healthy state.
To Reproduce
Steps to reproduce
- Deploy a canonicalk8s-openstack model with openstack-cloud-controller related to k8s
- Trigger any operation that temporarily takes the Kubernetes API offline:
- Remove the leader control-plane unit and add a replacement (juju remove-unit k8s/leader + juju add-unit k8s)
- Apply a config change that triggers a snap reinstall (juju config k8s extra-sans=...)
- Wait for Juju's periodic update-status hook to fire on openstack-cloud-controller/0 (every 5 minutes by default)
Environment
canonical k8s on charmed openstack caracal with noble
Relevant log output
Model Controller Cloud/Region Version SLA Timestamp
canonicalk8s-openstack foundation-openstack openstack_cloud/RegionOne 3.6.24 unsupported 01:54:38Z
App Version Status Scale Charm Channel Rev Exposed Message
cinder-csi v1.27.1 active 1 cinder-csi latest/stable 47 no Versions: cinder-csi-driver=v1.27.1
k8s 1.35.0 active 3 k8s 1.35/stable 1846 yes Ready
k8s-worker 1.35.0 active 3 k8s-worker 1.35/stable 1843 no Ready
openstack-cloud-controller v1.34.1 active 1 openstack-cloud-controller latest/stable 48 no Versions: openstack-cloud-controller-manager=v1.34.1
openstack-integrator yoga active 1 openstack-integrator latest/stable 123 no Ready
Unit Workload Agent Machine Public address Ports Message
k8s-worker/1 active idle 1 10.243.37.107 Ready
k8s-worker/2* active idle 2 10.243.37.111 Ready
k8s-worker/3 active idle 7 10.243.37.201 Ready
k8s/2* active idle 5 10.243.37.191 6443/tcp Ready
k8s/3 active idle 8 10.243.37.214 6443/tcp Ready
k8s/4 active idle 9 10.243.37.227 6443/tcp Ready
openstack-integrator/0* active idle 6 10.243.37.128 Ready
cinder-csi/0* blocked idle 10.243.37.128 Missing required certificates
openstack-cloud-controller/0* error idle 10.243.37.128 hook failed: "update-status"
Machine State Address Inst id Base AZ Message
1 started 10.243.37.107 f2c16312-2ceb-4555-a08c-2d7ee165214b ubuntu@24.04 nova ACTIVE
2 started 10.243.37.111 f84cb757-fd6a-4b6d-8056-d9e5083317fa ubuntu@24.04 nova ACTIVE
5 started 10.243.37.191 f8b70667-e942-46de-a1b3-682b8877581a ubuntu@24.04 nova ACTIVE
6 started 10.243.37.128 eedb69fc-aed5-4056-bce3-091c82fefb05 ubuntu@24.04 nova ACTIVE
7 started 10.243.37.201 d2a70b3d-9808-4164-bbcc-bb86b4ff07dd ubuntu@24.04 nova ACTIVE
8 started 10.243.37.214 1790859c-331e-4cbb-941b-9d0643666ce0 ubuntu@24.04 nova ACTIVE
9 started 10.243.37.227 84011a2d-4a18-415b-9a90-efeac2f1fd4c ubuntu@24.04 nova ACTIVE
Additional context
it fails to negotiate the version from keystone:
unit-openstack-integrator-0: 01:16:13 ERROR Failed to determine API version: unknown url type: ''
unit-openstack-integrator-0: 01:22:50 ERROR Failed to determine API version: unknown url type: ''
The charm attempted to negotiate the OpenStack API version from Keystone twice during deployment,
and failed both times with unknown url type: ''
The Keystone auth URL was empty because the
relation data was not yet fully populated at those points. The charm then silently fell back to a
hardcoded default of "yoga" and transitioned to active: Ready without ever successfully
detecting the actual cloud version. No further version-detection attempts were made.
Summary
The openstack-cloud-controller charm's update-status hook crashes with an unhandled
httpx.ConnectError whenever the Kubernetes API is temporarily unreachable. The unit is then
permanently stuck in Juju error state and requires manual intervention (juju resolve) to recover.
This caused a CI pipeline failure in the canonicalk8s-openstack integration test run on 2026-06-26.
Affected component
Field | Value
Charm | openstack-cloud-controller
Channel | latest/stable
Revision | 48
Source file | src/charm.py
Method | _update_status → _check_node_provider_ids
Environment
Field | Value
k8s charm | 1.35/stable rev 1846 (k8s 1.35.0)
Cloud | OpenStack Caracal / Noble
Juju | 3.6.24
Base | ubuntu@24.04
Model | canonicalk8s-openstack on foundation-openstack
Observed behaviour
The update-status hook crashes with an uncaught exception:
unit-openstack-cloud-controller-0: 01:55:48 ERROR unit.openstack-cloud-controller/0.juju-log Uncaught exception while in charm code:
Traceback (most recent call last):
The unit is then permanently locked in Juju error state:
It does not self-recover. It must be manually resolved with juju resolve openstack-cloud-controller/0.
Expected behaviour
The
_update_status / _check_node_provider_idsmethod should catch connection errors from theKubernetes API client (at minimum httpx.ConnectError, httpx.ConnectTimeout, and
httpcore.ConnectError) and set a transient waiting status instead of allowing the exception to
propagate and crash the hook. The k8s API being temporarily unreachable is a normal, expected
condition during cluster maintenance operations (leader replacement, snap upgrades, etc.).
Root cause
In src/charm.py, the _check_node_provider_ids method calls client.list(Node) via the
lightkube Kubernetes client with no error handling around network connectivity:
When the k8s API endpoint is unreachable, lightkube → httpx → httpcore raises
httpx.ConnectError: [Errno 113] No route to host. This exception propagates all the way up to
ops.main() at charm.py:283, causing the hook to exit with status 1 and Juju to mark the unit
as errored.
Suggested fix
Wrap the Kubernetes API calls in _update_status (and any other hook handler that calls the k8s
API) to catch connectivity errors and set a waiting status:
Or, at the _update_status handler level:
Impact
Unit permanently stuck in error state after any k8s API disruption
CI pipeline fails: juju.errors.JujuUnitError: Unit in error: openstack-cloud-controller/0
Downstream tests (e.g. test_extra_sans) that call wait_for_idle(status="active") fail even when the test itself is otherwise correct
Manual juju resolve required to unblock the pipeline
Additional context: secondary test design gap
The test_k8s test suite (test_remove_leader_control_plane) performs a leader control-plane
replacement but only waits for k8s/* and k8s-worker/* units to become healthy. It does not
check openstack-cloud-controller or cinder-csi health after the operation. As a result:
The update-status hook failure occurs during test_remove_leader_control_plane (~01:47–01:53)
test_k8s reports integration-k8s: OK without detecting the broken cloud-controller
The errored unit is silently handed off to the test_openstack suite
test_extra_sans (the fourth openstack test) is the first test to call wait_for_idle(status="active") on the full model and fails
Recommendation: test_k8s (and any test that modifies k8s topology on a model with OpenStack
integrations) should include a post-operation assertion that all units — including subordinates —
return to a healthy state.
To Reproduce
Steps to reproduce
Environment
canonical k8s on charmed openstack caracal with noble
Relevant log output
Additional context
it fails to negotiate the version from keystone:
The Keystone auth URL was empty because the
relation data was not yet fully populated at those points. The charm then silently fell back to a
hardcoded default of "yoga" and transitioned to active: Ready without ever successfully
detecting the actual cloud version. No further version-detection attempts were made.