Network is unreachable during the `model.deploy`

[Deezzir]

Description

I have deployed Canonical K8S 1.32/stable and am now testing my charm with pytest_operator and Python-libjuju:

juju bootstrap k8s k8s-controller
juju add-model test

Controller:

k8s-controller:
  details:
    controller-uuid: 6b044df4-7222-4e05-8bdd-7a185266a0b4
    api-endpoints: ['10.152.183.81:17070']
  models:
    controller:
      model-uuid: 17b948e6-9446-48e4-8113-ec8c8a5fbc6d
      unit-count: 1
    test:
      model-uuid: dbb4020d-67bd-4048-856b-5db11fd0d0ef

Run using tox -vve integration -- --model test fails during the deployment of a locally built charm:

Traceback (most recent call last):
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/_pytest/runner.py", line 341, in from_call
    result: TResult | None = func()
                             ^^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/_pytest/runner.py", line 242, in <lambda>
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/pluggy/_callers.py", line 139, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/_pytest/threadexception.py", line 92, in pytest_runtest_call
    yield from thread_exception_runtest_hook()
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/_pytest/threadexception.py", line 68, in thread_exception_runtest_hook
    yield
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/_pytest/unraisableexception.py", line 95, in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/_pytest/unraisableexception.py", line 70, in unraisable_exception_runtest_hook
    yield
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/_pytest/logging.py", line 846, in pytest_runtest_call
    yield from self._runtest_for(item, "call")
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/_pytest/logging.py", line 829, in _runtest_for
    yield
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/_pytest/capture.py", line 898, in pytest_runtest_call
    return (yield)
            ^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/pluggy/_callers.py", line 122, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/_pytest/skipping.py", line 257, in pytest_runtest_call
    return (yield)
            ^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/_pytest/runner.py", line 174, in pytest_runtest_call
    item.runtest()
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/_pytest/python.py", line 1627, in runtest
    self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/pluggy/_callers.py", line 182, in _multicall
    return outcome.get_result()
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/pluggy/_result.py", line 100, in get_result
    raise exc.with_traceback(exc.__traceback__)
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/_pytest/python.py", line 159, in pytest_pyfunc_call
    result = testfunction(**testargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/pytest_asyncio/plugin.py", line 529, in inner
    _loop.run_until_complete(task)
  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/tests/integration/test_s3_relation.py", line 43, in test_build_and_deploy
    await asyncio.gather(
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/juju/model/__init__.py", line 1989, in deploy
    identifier = await self.add_local_charm_dir(charm_dir, charm_series)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/juju/model/__init__.py", line 923, in add_local_charm_dir
    charm_url = await loop.run_in_executor(None, func)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/deezzir/Desktop/Repos/Canonical/velero-operator/.tox/integration/lib/python3.12/site-packages/juju/model/__init__.py", line 953, in add_local_charm
    conn.request("POST", path, charm_file, headers)
  File "/usr/lib/python3.12/http/client.py", line 1336, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.12/http/client.py", line 1382, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.12/http/client.py", line 1331, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.12/http/client.py", line 1091, in _send_output
    self.send(msg)
  File "/usr/lib/python3.12/http/client.py", line 1035, in send
    self.connect()
  File "/usr/lib/python3.12/http/client.py", line 1470, in connect
    super().connect()
  File "/usr/lib/python3.12/http/client.py", line 1001, in connect
    self.sock = self._create_connection(
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/socket.py", line 865, in create_connection
    raise exceptions[0]
  File "/usr/lib/python3.12/socket.py", line 850, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

It only occurs during the deployment step, and other tests that involve interacting with the model work fine. My current workaround is to build and deploy manually, and then run the other tests.

Also, looking at the vars, the library uses the correct IP, Port and model UUID in add_local_charm function: 10.152.183.81 17070 /model/dbb4020d-67bd-4048-856b-5db11fd0d0ef/charms?series=

Note: Deploying a non-local charm works fine as well, so the error only occurs during the deployment of a locally built charm

Urgency

Annoying bug in our test suite

Python-libjuju version

3.6.1.1

Juju version

3.6.5-genericlinux-amd64

Reproduce / Test

Simple test case that fails:

@pytest.mark.abort_on_fail
async def test_build_and_deploy(ops_test: OpsTest):
    charm = await ops_test.build_charm(".")
    model = get_model(ops_test)

    await asyncio.gather(
        model.deploy(charm, application_name=APP_NAME),
        model.wait_for_idle(apps=[APP_NAME], status="blocked", timeout=TIMEOUT),
    )

[dimaqq]

The circumstances (deploying locally built charm, but not a charmhub charm) is suggestive of a bug in Juju, its integration with CK8s or perhaps a timing issue, rather than this library.

Additionally microk8s tests pass, only the CK8s branch is affected: canonical/velero-operator#55

[SimonRichardson]

@Deezzir Can we get the controller logs for this? I wonder if we're causing a panic, which is shutting down the connection. This is likely only a older client concern, which is why we're not seeing this just from the juju CLI.