VSCode unable to resolve imports with a custom `comtypes.client.gen_dir` path

[gexgd0419]

For example, I set comtypes.client.gen_dir to be comInterfaces, so that generated wrapper modules will appear in that directory, instead of the default location of comtypes.gen.

After generating the wrappers, I want to import the wrappers to get some type hints.

I cannot import from comtypes.gen, because the wrappers are not there, since I have moved them into comInterfaces.

I can import from comInterfaces, but then, VS code still cannot find the symbols inside the wrappers, because the generated wrappers still reference comtypes.gen._SomeLibraryID_ somehow.

Is there something I can tweak in comtypes to change the import paths inside those wrappers, so that they can work with static Python checkers? Or is there something I can change in the VS Code settings in order to fix this?

[junkmd]

Hi,

To get straight to the point, this is a tough one.

This package needs comtypes.client.gen_dir to be a path that the internal function _find_gen_dir can give.

comtypes/comtypes/client/_code_cache.py

Lines 32 to 49 in 6a580b3

	def _find_gen_dir():
	"""Create, if needed, and return a directory where automatically
	generated modules will be created.
	Usually, this is the directory 'Lib/site-packages/comtypes/gen'.
	If the above directory cannot be created, or if it is not a
	directory in the file system (when comtypes is imported from a
	zip-archive or a zipped egg), or if the current user cannot create
	files in this directory, an additional directory is created and
	appended to comtypes.gen.__path__ .
	For a Python script using comtypes, the additional directory is
	'%APPDATA%\\<username>\\Python\\Python25\\comtypes_cache'.
	For an executable frozen with py2exe, the additional directory is
	'%TEMP%\\comtypes_cache\\<imagebasename>-25'.
	"""

There is no easy way for users to change gen_dir to a different path and have imports work well.

If we try to add this kind of feature, it will be very complex.
This package already uses difficult ways to work with Python's import system.
So, I can't easily agree to add more features.

If you only need type hints, here's what you can do.
You can make type stubs from the files created in comtypes/gen/... by using a tool like mypy's stubgen. Then, put these stubs in a folder, and specify that folder using the stubPath setting.

• For mypy's stubgen: https://mypy.readthedocs.io/en/stable/stubgen.html
• For VS Code's Pylance stubPath setting: https://code.visualstudio.com/docs/python/settings-reference#_pylance-language-server

[gexgd0419]

Actually I came from this issue nvaccess/nvda#17608, where NVDA changes gen_dir before generating wrappers:

comtypes.client.gen_dir = Dir("comInterfaces").abspath

and adds that path to the search path of comtypes.gen, so that imports from comtypes.gen still work:

def appendComInterfacesToGenSearchPath() -> None:
	# Initialise comtypes.client.gen_dir and the comtypes.gen search path
	# and append our comInterfaces directory to the comtypes.gen search path.
	import comtypes.client
	import comtypes.gen
	import comInterfaces

	comtypes.gen.__path__.append(comInterfaces.__path__[0])

NVDA tried to fix the IDE importing problem by modifying the generated code and replacing:

from comtypes.gen import SomeLib

with this:

try:
    from comtypes.gen import SomeLib
except ModuleNotFoundError:
    import SomeLib  # import relatively
    from SomeLib import *

which might work in the past, but as comtypes changed the generated code, this fix was broken.

Now the generated code references comtypes.gen._SomeLibId_ everywhere, not just some import lines. And needless to say, a future update of comtypes might just break this once again.

So do you think that replacing comtypes generated code is an idea good enough?

---

By the way, I have one more question about type hinting.

As an interface pointer is a pointer to an interface, should I type hint a pointer with a pointer to the interface type, or the interface type itself?

voice: ISpeechVoice = comtypes.client.CreateObject("SAPI.SpVoice")
voice: POINTER(ISpeechVoice) = comtypes.client.CreateObject("SAPI.SpVoice")

Type hinting with the interface type itself enables some member listings, which is great. But pointer to the interface seems more accurate. And in fact, the type of voice seems to be the latter one:

print(type(voice))  # <class 'comtypes._post_coinit.unknwn.POINTER(ISpeechVoice)'>
print(isinstance(voice, ISpeechVoice))  # True
print(isinstance(voice, POINTER(ISpeechVoice)))  # True
print(type(voice) is ISpeechVoice)  # False
print(type(voice) is POINTER(ISpeechVoice))  # True

[junkmd]

So do you think that replacing comtypes generated code is an idea good enough?

If you could provide an example of what kind of code you'd like comtypes to generate instead of the current output, I can review it.

Please share your specific ideas before making a pull request.

---

Type hinting with the interface type itself enables some member listings, which is great. But pointer to the interface seems more accurate.

I've written down my thoughts on this topic in several issue comments and code-based comments.

You might be concerned about the runtime type (comtypes.POINTER(IZcadApplication)) differing from static type (IZcadApplication).
This discrepancy arises because the current Python static typing system cannot express the behavior of the _cominterface_meta and _compointer_meta.
At runtime, comtypes.POINTER(IZcadApplication) behaves as a subclass of IZcadApplication due to those metaclasses.
To represent this in static typing, I believe that Intersection would be necessary.

#516 (comment)

comtypes/comtypes/_post_coinit/unknwn.py

Lines 390 to 394 in 5c564ec

	# NOTE: Why not `QueryInterface(T) -> _Pointer[T]`?
	# Any static type checkers is not able to provide members of `T` from `_Pointer[T]`,
	# regardless of the pointer is able to access members of contents in runtime.
	# And if `isinstance(p, POINTER(T))` is `True`, then `isinstance(p, T)` is also `True`.
	# So returning `T` is not a lie, and good way to know what members the class has.

If the current code were to annotate return values as _Pointer[IUnknown], it would likely cause false positive type checker errors in most projects using comtypes, which would hurt usability.

To correctly represent something as both a pointer type and an IUnknown, we'll probably have to wait for the adoption of Intersection.

[junkmd]

I recommend annotating with the interface type rather than the pointer type for type hints, as shown in #516 (comment).

I think this approach will feel more natural if you consider it similar to annotating with an ABC, like this:

a: typing.Sized = [1]

[gexgd0419]

A problem is that if you want to pass a pointer to an interface pointer to another function, like this:

voice: ISpeechVoice = POINTER(ISpeechVoice)()  # type checker error here
# This can also be done by comtypes, but here's just an example
oledll.ole32.CoCreateInstance(
    byref(SpVoice._reg_clsid_),
    None,
    comtypes.CLSCTX_ALL,
    byref(ISpeechVoice._iid_),
    byref(voice)
)

then you have to use the actual pointer type, and if you type hint the variable with the interface type, the type checker will complain. Though this could be solved by specifying the parameter as an out parameter.

But I can understand that there are limitations from Python's current structure.

Thank you for your answers.

---

If you could provide an example of what kind of code you'd like comtypes to generate instead of the current output, I can review it.

I don't think I have a deep understanding of comtypes, so I don't plan to change comtypes currently. Instead, I plan to adjust the NVDA code responsible for modifying the generated module, so that it can still work at least for the current version.

But here's my thought: as all comtypes generated files are in the same directory, is it possible to change those imports to relative ones? For example:

import comtypes.gen._SomeLibId as __wrapper_module__
from comtypes.gen._SomeLibId import (
    aaa, bbb, ccc,
)

import comtypes.gen._SomeLibId
class SomeType(comtypes.gen._SomeLibId.SomeBaseType):
    ...

would become:

from . import _SomeLibId as __wrapper_module__
from ._SomeLibId import (
    aaa, bbb, ccc,
)

from . import _SomeLibId
class SomeType(_SomeLibId.SomeBaseType):
    ...

Those modules will, by default, all be in comtypes.gen anyway, so from . import _SomeLibId still gets comtypes.gen._SomeLibId. What's better, you can move them to anywhere without breaking the imports.

So is this approach feasible? If not, what is requiring the absolute path comtypes.gen._SomeLibId?

[junkmd]

The import statements use absolute paths simply because this is how the code has been generated since 1.2.x and earlier.

Currently, customizing gen_dir isn't a high priority.
It would require a significant overhaul of the entire code generation process.
Most people already use the default comtypes/gen path without problems.

I'm curious why you still feel the need to use a custom path.

Could you explain more about the specific issues or benefits you gain from it?