When running pdoc on an extension modules (aka, a native "C" extensions), the extension module's submodules don't show up in the documentation even though <TAB>-autocomplete in a Python REPL can find the submodules. This seems to be because pdoc searches for submodules by inspecting the source directory, which isn't available for extension modules.

I've proposed PR #318 to fix this issue. The proposed solution works but I'm not sure if it is safe enough to remove the old "source directory traversal" method. I'd appreciate guidance on completing the PR.

Expected Behavior

Running pdoc on a native extension module should generate documentation for the entire extension module, including its submodules.

Actual Behavior

• With current master: only items in the top-level module appear in the documentation. Submodules don't show up in the documentation.
• With the proposed fix in Recurse through submodules in extension modules #318: works as expected.

Steps to Reproduce

The following steps generate a minimalistic native extension module in Rust that exhibits the problem. The language shouldn't matter though.

1. Install a rust toolchain, see https://rustup.rs
2. Create the following directory structure:

pyext/
├── Cargo.toml
└── src/
    └── lib.rs

with the following file contents:

• Cargo.toml:

[package]
authors = ["Name <[email protected]>"]
edition = "2018"
name = "pyext"
version = "0.1.0"

[lib]
crate-type = ["cdylib"]

[dependencies]
pyo3 = {version = "0.13.2", features = ["extension-module"]}

• src/lib.rs:

use pyo3::{prelude::*, wrap_pymodule};

/// Docstring of main module.
#[pymodule(pyext)]
fn init_main_module(_py: Python<'_>, module: &PyModule) -> PyResult<()> {
    module.add_wrapped(wrap_pymodule!(submodule))?;
    Ok(())
}

/// Docstring of submodule
#[pymodule(submodule)]
fn init_submodule(_py: Python<'_>, submodule: &PyModule) -> PyResult<()> {
    submodule.add("variable", 42)?;
    Ok(())
}

3. Compile the extension module: cargo build
4. Create a properly named symlink to the object file:
   • on Linux: ln -s target/debug/libpyext.so pyext.so
   • on Mac: ln -s target/debug/libpyext.dylib pyext.so
   • on Windows: rename target\debug\libpyext.dll to pyext.pyd
5. Start a Python REPL from the directory containing the pyext.so file and verify that the submodule exists and can be found by tab completion:

$ python
Python 3.6.10 (default, May 22 2020, 17:59:48) 
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyext
>>> pyext.<TAB>  --> autocompletes to "pyext.submodule", proving that the submodule can be found
>>> pyext.submodule.variable
42

6. Run pdoc --html pyext from the same directory.
   • With the version at current master, the generated documentation leaves out the submodule.
   • With the version proposed in Recurse through submodules in extension modules #318, the generated documentation includes the submodule.

Additional info

• pdoc version: master and 0.9.2 don't work, the one from Recurse through submodules in extension modules #318 works.
• tested on linux