Open
Description
Long story short
The documentation claims that kubernetes_asyncio is supported; however, it does not appear to be used in the piggybacking code. Attempting to launch the operator with only kubernetes_asyncio installed results in authentication failures.
Kopf version
1.36.0
Kubernetes version
1.24
Python version
3.10.9
Code
@kopf.on.create(group="velero.io", version="v1", plural="backups")
async def create_fn(body, **kwargs):
logging.info(f"A handler is called with body: {body}")Logs
[2023-02-28 11:16:33,027] kopf._core.reactor.r [DEBUG ] Starting Kopf 1.36.0.
[2023-02-28 11:16:33,028] kopf._core.engines.a [INFO ] Initial authentication has been initiated.
[2023-02-28 11:16:33,028] kopf.activities.auth [DEBUG ] Activity 'login_with_kubeconfig' is invoked.
[2023-02-28 11:16:33,058] kopf.activities.auth [INFO ] Activity 'login_with_kubeconfig' succeeded.
[2023-02-28 11:16:33,058] kopf._core.engines.a [INFO ] Initial authentication has finished.
[2023-02-28 11:16:33,767] kopf._core.reactor.r [ERROR ] Resource observer has failed: ('forbidden: User "system:anonymous" cannot get path "/apis"', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'forbidden: User "system:anonymous" cannot get path "/apis"', 'reason': 'Forbidden', 'details': {}, 'code': 403})
Traceback (most recent call last):
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 148, in check_response
response.raise_for_status()
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1005, in raise_for_status
raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('REDACTED/apis')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 108, in guard
await coro
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_core/reactor/observation.py", line 113, in resource_observer
resources = await scanning.scan_resources(groups=group_filter, settings=settings, logger=logger)
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/scanning.py", line 31, in scan_resources
resources.update(await coro)
File "/nix/store/0pyymzxf7n0fzpaqnvwv92ab72v3jq8d-python3-3.10.9/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
return f.result() # May raise f.exception().
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/scanning.py", line 68, in _read_new_apis
rsp = await api.get('/apis', settings=settings, logger=logger)
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 111, in get
response = await request(
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/auth.py", line 45, in wrapper
return await fn(*args, **kwargs, context=context)
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 85, in request
await errors.check_response(response) # but do not parse it!
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 150, in check_response
raise cls(payload, status=response.status) from e
kopf._cogs.clients.errors.APIForbiddenError: ('forbidden: User "system:anonymous" cannot get path "/apis"', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'forbidden: User "system:anonymous" cannot get path "/apis"', 'reason': 'Forbidden', 'details': {}, 'code': 403})
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG ] Admission insights chain is cancelled.
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG ] Namespace observer is cancelled.
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG ] Credentials retriever is cancelled.
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG ] Admission webhook server is cancelled.
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG ] Admission validating configuration manager is cancelled.
[2023-02-28 11:16:33,768] kopf._core.reactor.o [DEBUG ] Streaming tasks stopping is skipped: no tasks given.
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG ] Multidimensional multitasker is cancelled.
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG ] Poster of events is cancelled.
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG ] Admission mutating configuration manager is cancelled.
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG ] Daemon killer is cancelled.
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG ] Root tasks are stopped: finishing normally; tasks left: set()
[2023-02-28 11:16:33,768] kopf._core.reactor.r [DEBUG ] Hung tasks stopping is skipped: no tasks given.
Traceback (most recent call last):
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 148, in check_response
response.raise_for_status()
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1005, in raise_for_status
raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('REDACTED/apis')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/nix/store/277vy3lj0y87drkzz1g7d5lfz6vkcqdx-python3.10-kopf-1.36.0/bin/.kopf-wrapped", line 9, in <module>
sys.exit(main())
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/cli.py", line 59, in wrapper
return fn(*args, **kwargs)
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/click/decorators.py", line 84, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/cli.py", line 113, in run
return running.run(
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_core/reactor/running.py", line 58, in run
loop.run_until_complete(operator(
File "/nix/store/0pyymzxf7n0fzpaqnvwv92ab72v3jq8d-python3-3.10.9/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_core/reactor/running.py", line 135, in operator
await run_tasks(operator_tasks, ignored=existing_tasks)
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_core/reactor/running.py", line 416, in run_tasks
await aiotasks.reraise(root_done | root_cancelled | hung_done | hung_cancelled)
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 238, in reraise
task.result() # can raise the regular (non-cancellation) exceptions.
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 108, in guard
await coro
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_core/reactor/observation.py", line 113, in resource_observer
resources = await scanning.scan_resources(groups=group_filter, settings=settings, logger=logger)
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/scanning.py", line 31, in scan_resources
resources.update(await coro)
File "/nix/store/0pyymzxf7n0fzpaqnvwv92ab72v3jq8d-python3-3.10.9/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
return f.result() # May raise f.exception().
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/scanning.py", line 68, in _read_new_apis
rsp = await api.get('/apis', settings=settings, logger=logger)
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 111, in get
response = await request(
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/auth.py", line 45, in wrapper
return await fn(*args, **kwargs, context=context)
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 85, in request
await errors.check_response(response) # but do not parse it!
File "/nix/store/6rvwfqz6vp817f1spxz3g8bmfsdh9kqp-python3-3.10.9-env/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 150, in check_response
raise cls(payload, status=response.status) from e
kopf._cogs.clients.errors.APIForbiddenError: ('forbidden: User "system:anonymous" cannot get path "/apis"', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'forbidden: User "system:anonymous" cannot get path "/apis"', 'reason': 'Forbidden', 'details': {}, 'code': 403})
Additional information
The "native" authentication doesn't seem to support short-lived tokens as provided by a typical EKS cluster. The Python clients have no issue dealing with this, however, as noted above it appears that kubernetes_asyncio is never queried. The issue is resolved by adding the kubernetes package as a dependency. However, I now need to have both kubernetes and kubernetes_asyncio available for the operator to work. One reason this is undesirable is it bloats the final container image.