Stubgen recursive mode for module with submodule should generate directory and init for top-level module

[wenjuno]

The recursive stubgen is great but with one problem, it generates a pyi file named after the top-level module name even if there are submodules. For instance, if I have a module mylib and it has a submodule lib, it will generate two files (1) mylib.pyi (2) lib/__init__.pyi. The content of mylib.pyi will have the import for lib: from . import lib as lib. This only works when mylib.pyi is within a package (i.e. is under a directory). I could put these two files under a directory "mylib" to make it a package, but it leads to a new problem, this package does not have __init__.py, so importing mylib will not get access to the submodule lib and any top-level classes/functions. The mylib.pyi needs to be renamed to __init__.py.
I think stubgen recursive mode should either always generate the top-level module as a package or at least does that when there are submodules.

[nurpax]

Yes, I also hit this and fixed it by renaming the produced top-level stub file:

python -m nanobind.stubgen -i build/cp312-abi3-macosx_15_0_arm64 -q -m slimgui_ext -r -O src/slimgui/slimgui_ext
mv src/slimgui/slimgui_ext/slimgui_ext.pyi src/slimgui/slimgui_ext/__init__.pyi

(https://github.com/nurpax/slimgui/blob/main/full_build.sh)

I put build steps into a separate shell script so that I can apply workarounds like this. I thought I was doing something wrong when I had to add this but looks like this a bug/limitation in nanobind's stubgen.

[wjakob]

@nurpax @wenjuno This is fixed on master now, please let me know if you still encounter issues. The recursive stubgen can now also be directly used from CMake.

[wjakob]

@nurpax

Regarding the clashes with builtins (bool etc.). Can you re-try with the master version? I made this more relaxed.

[wjakob]

Regarding putting things into foo.pyi as opposed to foo/__init__.pyi. The algorithm to make this decision was wrong in the past, and it was fixed as part of issue #1082 and commits 620c1c1 and ad9d3fe.

If you think it is (still) wrong, then it would be important to understand why and get a reproducer.

[wjakob]

In particular, stubs are put into a nested directory containing __init__.pyi if the current module has submodules, or when the current module was generated by a Python __init__.py file (to match the directory structure). Otherwise, a module_name.pyi file is locally created.

[nurpax]

Re: folder/file name structure. I adapted my package for nanobind at commit 10cc510 and after I got it working and wheels built on CI, I tried to get it to break on macOS and Windows with VSCode/Pylance and I'm happy to inform that I was unable to reproduce the problem I had a few months ago. Seems like the new Nanobind pyi convention for submodules is working as advertised!

For posteriority, a few details on how I've wired this up.

Here's how the files get laid out on disk (as extracted from a wheel build):

slimgui/__init__.py
   imgui/
       __init__.py
   implot/
       __init__.py
   integrations/
       __init__.py
       base.py
       glfw.py
       null.py
       opengl.py
   slimgui_ext/
       __init__.pyi
       imgui.pyi
       implot.pyi
   slimgui_ext.pyd

I currently generate the stubs by invoking stubgen directly:

python -m nanobind.stubgen -i build/cp312-abi3-macosx_15_0_arm64 -q -m slimgui_ext -r -O src/slimgui

(I do this as I apply some docstring postprocessing on the outputs.)

I also tried the same with CMake and it seems to work too. I'm generating the outputs directly into the source directory as I keep the generated files in git:

nanobind_add_stub(
    slimgui_ext_stub
    MODULE slimgui_ext
    PYTHON_PATH $<TARGET_FILE_DIR:slimgui_ext>
    DEPENDS slimgui_ext
    RECURSIVE
    OUTPUT_PATH "${CMAKE_SOURCE_DIR}/src/slimgui"
    OUTPUT 
        "${CMAKE_SOURCE_DIR}/src/slimgui/slimgui_ext/__init__.pyi"
        "${CMAKE_SOURCE_DIR}/src/slimgui/slimgui_ext/imgui.pyi"
        "${CMAKE_SOURCE_DIR}/src/slimgui/slimgui_ext/implot.pyi"
)

Thanks a lot for fixing this and thank you also for taking the time to explain how the new version differs from before!

PS. Just throwing it out there that a good starter project for someone would be to add stub generation to nanobind_example. :)

[wjakob]

Phew, that's good to hear. All agreed about nanobind_example!