Handle async do_complete in the debugger

[Carreau]

A number of methods of ipykernel can optionally return awaitable[T] instead of just T, this is the case for do_complete.

I think it's a mistake ; see ipython/ipykernel#1295 ; in particular because it's easy to forget / hard to properly type-check, and I'd like to make it mandatory in the long term to have await.

Spyder seem to not handle the case where do_completer return an awaitable (or more partiularly is do_complete is a coroutine function.

This tries to handle it – and as of course do_completer can be async, all caller must be async. So I try to do all the required updates.

Note: I also add explict imports in some test, to get better error message in case those deps are not installed.

[Carreau]

This is more to start a discussion than a "this must be merged". It's mostly to start some discussion also with upstream as we sort of disagree that having def f(...): -> T | Awaitable[T] is a good thing.

[Carreau]

Rebased. I think we agree upstream that things should mostly be async when possible. so +1 to merge this once tests are passing, and you agree.,

[ccordoba12]

Thanks @Carreau! A couple of small questions, the rest looks good to me.

[ccordoba12]

Why is this necessary now? It wasn't before.

[Carreau]

I do not remember.

I would say better error messages in pytest short summary:

- FAILED spyder_kernels/...::test_save_load_hdf5_files - assert '(None, "No m...ed \'h5py\'")' == "({'a': array...(4.5)}, None)"
+ FAILED spyder_kernels/...::test_save_load_hdf5_files - ModuleNotFoundError: No module named 'h5py'

and

- FAILED spyder_kernels/...::test_django_settings - assert "'settings':" in '{}'
+ FAILED spyder_kernels/...::test_django_settings - ModuleNotFoundError: No module named 'django'

[ccordoba12]

Ok, thanks for the clarification. Merging then.

[ccordoba12]

Thanks @Carreau!

[ccordoba12]

@Carreau, I tested this in Spyder and it actually breaks code completions for our debugger.

You started this PR by saying:

A number of methods of ipykernel can optinally return awaitable[T] instead of just T, this is the case for do_complete.

But that's not actually the case in IPykernel's main nor in the 6.x branch:

https://github.com/ipython/ipykernel/blob/cac11a08449d41ceb8a56e3095e78e00cb1626eb/ipykernel/ipkernel.py#L512-L532

So, I guess these changes are unnecessary and need to be reverted, right?

[Carreau]

But that's not actually the case in IPykernel's main nor in the 6.x branch:

ipython/ipykernel@cac11a0/ipykernel/ipkernel.py#L512-L532

So, I guess these changes are unnecessary and need to be reverted, right?

The code that call do_complete (and similar) accept both T|awaitable[T] and emit a warning that returning awaitable[T] might become mandatory in the future.

https://github.com/ipython/ipykernel/blob/cac11a08449d41ceb8a56e3095e78e00cb1626eb/ipykernel/kernelbase.py#L830-L838

Some subclass of ipykernel (I don't remember which) use async do_complete, maybe xeus maybe another...

I haven't changed the base methods because it would break downstream (like Spyder kernel): ipython/ipykernel#1339. This PR ensure that Spyder kernel is not broken when we change the base methods, but we need a release cycle for that.

Is there any chances you run with warnings turned into errors and this breaks the completions ?

But if you want to revert, feel free to do so.

[ccordoba12]

The code that call do_complete (and similar) accept both T|awaitable[T] and emit a warning that returning awaitable[T] might become mandatory in the future.

Ok, but the thing is through this call:

spyder-kernels/spyder_kernels/customize/spyderpdb.py

Line 425 in 2e19c2c

result = await self.shell.kernel._do_complete(code, cursor_pos)

we're calling IPykernel's do_complete directly:

spyder-kernels/spyder_kernels/console/kernel.py

Lines 451 to 453 in 2e19c2c

	def _do_complete(self, code, cursor_pos):
	"""Call parent class do_complete"""
	return super(SpyderKernel, self).do_complete(code, cursor_pos)

And since it's not async today, we're getting this error:

[SpyderKernelApp] ERROR | Exception in control handler:
Traceback (most recent call last):
  File "/home/carlos/miniconda3/envs/py310-pip/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 362, in process_control
    await result
  File "/home/carlos/miniconda3/envs/py310-pip/lib/python3.10/site-packages/ipykernel/kernelbase.py", line 831, in complete_request
    matches = await matches
  File "/home/carlos/Projects/spyder/spyder/external-deps/spyder-kernels/spyder_kernels/customize/spyderpdb.py", line 359, in do_complete
    return await self._complete_exclamation(code, cursor_pos)
  File "/home/carlos/Projects/spyder/spyder/external-deps/spyder-kernels/spyder_kernels/customize/spyderpdb.py", line 532, in _complete_exclamation
    result = await self.shell.kernel._do_complete(code, cursor_pos)
TypeError: object dict can't be used in 'await' expression

When googling for it, I found that do_complete in IPykernel should be async to fix it.

Or do you know another way in which we could solve this?

[Carreau]

Ah,

then you need to do like in this PR:

result = call()
if isawaitable(result):
    result = await result

this will handle both awaitable and non awaitable. This will work for both an async def function, or a sync function returning a future.

[Carreau]

#534 should help, I haven't tried.

[ccordoba12]

Thanks @Carreau! That fixes completions in our debugger with some minor tweaks.