Open
Description
Steps to reproduce
Prevent the promtail binary from downloading (somehow) when integrating the logging endpoint of postgresql-k8s with something else.
Deployment steps (only fails if promtail fails to download):
juju deploy postgresql-k8s
juju deploy loki-k8s
juju integrate postgresql-k8s:logging loki-k8s
Expected behavior
Should the download of the promtail binary fail, it should be retried.
Actual behavior
If the download of the promtail binary fails, postgresql-k8s stays in error due to Start service "promtail" (cannot start service: fork/exec /opt/promtail/promtail-static-amd64: no such file or directory).
The download of promtail does not appear to be retried, so the postgresql-k8s is forever in error.
Versions
Operating system:
ubuntu-22.04. Attached crashdump has more details
Juju CLI:
3.6.2. Attached crashdump has more details
Juju agent:
3.6.2. Attached crashdump has more details
Charm revision:
462. Attached crashdump has more details
microk8s:
N/A
Log output
juju-k8s-crashdump logs:
juju-logs.tar.gz
2025-02-08T01:00:21.130Z [container-agent] 2025-02-08 01:00:21 INFO juju.worker.uniter.operation runhook.go:186 ran "logging-relation-joined" hook (via hook dispatching script: dispatch)
2025-02-08T01:02:35.676Z [container-agent] 2025-02-08 01:02:35 WARNING juju-log logging:4: Promtail binary couldn't be downloaded - <urlopen error [Errno 110] Connection timed out>
2025-02-08T01:02:36.182Z [container-agent] 2025-02-08 01:02:36 INFO juju.worker.uniter.operation runhook.go:186 ran "logging-relation-changed" hook (via hook dispatching script: dispatch)
2025-02-08T01:02:37.332Z [container-agent] 2025-02-08 01:02:37 ERROR juju-log logging:4: Uncaught exception while in charm code:
2025-02-08T01:02:37.332Z [container-agent] Traceback (most recent call last):
2025-02-08T01:02:37.332Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/./src/charm.py", line 2239, in <module>
2025-02-08T01:02:37.332Z [container-agent] main(PostgresqlOperatorCharm, use_juju_for_storage=True)
2025-02-08T01:02:37.332Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/__init__.py", line 343, in __call__
2025-02-08T01:02:37.332Z [container-agent] return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
2025-02-08T01:02:37.332Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/_main.py", line 543, in main
2025-02-08T01:02:37.332Z [container-agent] manager.run()
2025-02-08T01:02:37.332Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/_main.py", line 529, in run
2025-02-08T01:02:37.332Z [container-agent] self._emit()
2025-02-08T01:02:37.332Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/_main.py", line 518, in _emit
2025-02-08T01:02:37.332Z [container-agent] _emit_charm_event(self.charm, self.dispatcher.event_name, self._juju_context)
2025-02-08T01:02:37.332Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/_main.py", line 134, in _emit_charm_event
2025-02-08T01:02:37.332Z [container-agent] event_to_emit.emit(*args, **kwargs)
2025-02-08T01:02:37.332Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/framework.py", line 347, in emit
2025-02-08T01:02:37.332Z [container-agent] framework._emit(event)
2025-02-08T01:02:37.332Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/framework.py", line 857, in _emit
2025-02-08T01:02:37.332Z [container-agent] self._reemit(event_path)
2025-02-08T01:02:37.332Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/framework.py", line 947, in _reemit
2025-02-08T01:02:37.332Z [container-agent] custom_handler(event)
2025-02-08T01:02:37.332Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/lib/charms/tempo_coordinator_k8s/v0/charm_tracing.py", line 1064, in wrapped_function
2025-02-08T01:02:37.332Z [container-agent] return callable(*args, **kwargs) # type: ignore
2025-02-08T01:02:37.332Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/lib/charms/loki_k8s/v1/loki_push_api.py", line 1946, in _on_relation_changed
2025-02-08T01:02:37.332Z [container-agent] container.restart(WORKLOAD_SERVICE_NAME)
2025-02-08T01:02:37.332Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/model.py", line 2399, in restart
2025-02-08T01:02:37.332Z [container-agent] self._pebble.restart_services(service_names)
2025-02-08T01:02:37.332Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/pebble.py", line 2219, in restart_services
2025-02-08T01:02:37.332Z [container-agent] return self._services_action('restart', services, timeout, delay)
2025-02-08T01:02:37.332Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/pebble.py", line 2244, in _services_action
2025-02-08T01:02:37.332Z [container-agent] raise ChangeError(change.err, change)
2025-02-08T01:02:37.332Z [container-agent] ops.pebble.ChangeError: cannot perform the following tasks:
2025-02-08T01:02:37.332Z [container-agent] - Start service "promtail" (cannot start service: fork/exec /opt/promtail/promtail-static-amd64: no such file or directory)
2025-02-08T01:02:37.332Z [container-agent] ----- Logs from task 0 -----
2025-02-08T01:02:37.332Z [container-agent] 2025-02-08T01:02:37Z INFO Service "promtail" already stopped.
2025-02-08T01:02:37.332Z [container-agent] ----- Logs from task 1 -----
2025-02-08T01:02:37.332Z [container-agent] 2025-02-08T01:02:37Z ERROR cannot start service: fork/exec /opt/promtail/promtail-static-amd64: no such file or directory
2025-02-08T01:02:37.332Z [container-agent] -----
2025-02-08T01:02:37.636Z [container-agent] 2025-02-08 01:02:37 ERROR juju.worker.uniter.operation runhook.go:180 hook "logging-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1
2025-02-08T01:02:37.638Z [container-agent] 2025-02-08 01:02:37 INFO juju.worker.uniter resolver.go:180 awaiting error resolution for "relation-changed" hook
2025-02-08T01:02:42.660Z [container-agent] 2025-02-08 01:02:42 INFO juju.worker.uniter resolver.go:180 awaiting error resolution for "relation-changed" hook
2025-02-08T01:02:43.929Z [container-agent] 2025-02-08 01:02:43 ERROR juju-log logging:4: Uncaught exception while in charm code:
2025-02-08T01:02:43.929Z [container-agent] Traceback (most recent call last):
2025-02-08T01:02:43.929Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/./src/charm.py", line 2239, in <module>
2025-02-08T01:02:43.929Z [container-agent] main(PostgresqlOperatorCharm, use_juju_for_storage=True)
2025-02-08T01:02:43.929Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/__init__.py", line 343, in __call__
2025-02-08T01:02:43.929Z [container-agent] return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
2025-02-08T01:02:43.929Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/_main.py", line 543, in main
2025-02-08T01:02:43.929Z [container-agent] manager.run()
2025-02-08T01:02:43.929Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/_main.py", line 529, in run
2025-02-08T01:02:43.929Z [container-agent] self._emit()
2025-02-08T01:02:43.929Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/_main.py", line 518, in _emit
2025-02-08T01:02:43.929Z [container-agent] _emit_charm_event(self.charm, self.dispatcher.event_name, self._juju_context)
2025-02-08T01:02:43.929Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/_main.py", line 134, in _emit_charm_event
2025-02-08T01:02:43.929Z [container-agent] event_to_emit.emit(*args, **kwargs)
2025-02-08T01:02:43.929Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/framework.py", line 347, in emit
2025-02-08T01:02:43.929Z [container-agent] framework._emit(event)
2025-02-08T01:02:43.929Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/framework.py", line 857, in _emit
2025-02-08T01:02:43.929Z [container-agent] self._reemit(event_path)
2025-02-08T01:02:43.929Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/framework.py", line 947, in _reemit
2025-02-08T01:02:43.929Z [container-agent] custom_handler(event)
2025-02-08T01:02:43.929Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/lib/charms/tempo_coordinator_k8s/v0/charm_tracing.py", line 1064, in wrapped_function
2025-02-08T01:02:43.929Z [container-agent] return callable(*args, **kwargs) # type: ignore
2025-02-08T01:02:43.929Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/lib/charms/loki_k8s/v1/loki_push_api.py", line 1946, in _on_relation_changed
2025-02-08T01:02:43.929Z [container-agent] container.restart(WORKLOAD_SERVICE_NAME)
2025-02-08T01:02:43.929Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/model.py", line 2399, in restart
2025-02-08T01:02:43.929Z [container-agent] self._pebble.restart_services(service_names)
2025-02-08T01:02:43.929Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/pebble.py", line 2219, in restart_services
2025-02-08T01:02:43.929Z [container-agent] return self._services_action('restart', services, timeout, delay)
2025-02-08T01:02:43.929Z [container-agent] File "/var/lib/juju/agents/unit-postgresql-k8s-0/charm/venv/ops/pebble.py", line 2244, in _services_action
2025-02-08T01:02:43.929Z [container-agent] raise ChangeError(change.err, change)
2025-02-08T01:02:43.929Z [container-agent] ops.pebble.ChangeError: cannot perform the following tasks:
2025-02-08T01:02:43.929Z [container-agent] - Start service "promtail" (cannot start service: fork/exec /opt/promtail/promtail-static-amd64: no such file or directory)
2025-02-08T01:02:43.929Z [container-agent] ----- Logs from task 0 -----
2025-02-08T01:02:43.929Z [container-agent] 2025-02-08T01:02:43Z INFO Service "promtail" already stopped.
2025-02-08T01:02:43.929Z [container-agent] ----- Logs from task 1 -----
2025-02-08T01:02:43.929Z [container-agent] 2025-02-08T01:02:43Z ERROR cannot start service: fork/exec /opt/promtail/promtail-static-amd64: no such file or directory
2025-02-08T01:02:43.929Z [container-agent] -----
2025-02-08T01:02:44.229Z [container-agent] 2025-02-08 01:02:44 ERROR juju.worker.uniter.operation runhook.go:180 hook "logging-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1
2025-02-08T01:02:44.232Z [container-agent] 2025-02-08 01:02:44 INFO juju.worker.uniter resolver.go:180 awaiting error resolution for "relation-changed" hook
Activity