Doesn't work with Cython3

[labeneator]

Thank you for the tutorial and the repo. I've verified that the project does work with cython 0.x series but fails on cython3.

$  /tmp/example-cython-poetry-pypi 
 ❯ poetry build -vvv                                                                                                                                                                                                               
Using virtualenv: /Users/lmwangi/Library/Caches/pypoetry/virtualenvs/poetry-pypi-example-wSzIWKA2-py3.9
Building poetry-pypi-example (0.0.1)
  - Building sdist
  - Adding: /private/tmp/example-cython-poetry-pypi/poetry_pypi_example/main.cpython-39-darwin.so
  - Adding: README.rst
  - Adding: pyproject.toml
  - Built poetry-pypi-example-0.0.1.tar.gz
  - Building wheel
  - Executing build script: build.py
[1/1] Cythonizing poetry_pypi_example/main.py
Traceback (most recent call last):
  File "/private/tmp/example-cython-poetry-pypi/build.py", line 80, in <module>
    build()
  File "/private/tmp/example-cython-poetry-pypi/build.py", line 30, in build
    build_ext_cmd.copy_extensions_to_source()
  File "/Users/lmwangi/Library/Caches/pypoetry/virtualenvs/poetry-pypi-example-wSzIWKA2-py3.9/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 103, in __getattr__
    raise AttributeError(attr)
AttributeError: copy_extensions_to_source

On drilling down, it looks like the import for build_ext in cython3 does not have ** copy_extensions_to_source**. To be precise in line 14, build_ext is None when it is imported from setuptools. Please see below:

Python 3.9.13 (main, May 24 2022, 21:28:31)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.6.0 -- An enhanced Interactive Python. Type '?' for help.

# Injected an IPython shell just before the setuptools import in Cython/Distutils/build_ext.py
In [1]: _build_ext_module

# Should return the build_ext module. Hmm, we can't get the build_ext module. Let's check the parents.
In [2]:  sys.modules.get('setuptools.command.build_ext')

# We can get the setuptools module
In [3]:  sys.modules.get('setuptools')
Out[3]: <module 'setuptools' from '/Users/lmwangi/Library/Caches/pypoetry/virtualenvs/wallet-api-ZqjMZXwH-py3.9/lib/python3.9/site-packages/setuptools/__init__.py'>


# We can get the setuptools.command module
In [5]:  sys.modules.get('setuptools.command')
Out[5]: <module 'setuptools.command' from '/Users/lmwangi/Library/Caches/pypoetry/virtualenvs/wallet-api-ZqjMZXwH-py3.9/lib/python3.9/site-packages/setuptools/command/__init__.py'>

# On the filesystem, the module does exist.
In [6]: !ls /Users/lmwangi/Library/Caches/pypoetry/virtualenvs/wallet-api-ZqjMZXwH-py3.9/lib/python3.9/site-packages/setuptools/command
 __init__.py   alias.py       bdist_rpm.py    build_ext.py   develop.py     easy_install.py   install.py	    install_lib.py	'launcher manifest.xml'   register.py   saveopts.py   setopt.py   upload.py
 __pycache__   bdist_egg.py   build_clib.py   build_py.py    dist_info.py   egg_info.py       install_egg_info.py   install_scripts.py	 py36compat.py		  rotate.py     sdist.py      test.py	  upload_docs.py


# Hmm, import does work
In [8]: import setuptools.command.build_ext

# Now a get works....
In [9]:  sys.modules.get('setuptools.command.build_ext')
Out[9]: <module 'setuptools.command.build_ext' from '/Users/lmwangi/Library/Caches/pypoetry/virtualenvs/wallet-api-ZqjMZXwH-py3.9/lib/python3.9/site-packages/setuptools/command/build_ext.py'>

Digging further, it looks like this is the [breaking commit](https://github.com/cython/cython/commit/c6f5c5ddcc021099febae7cb1194fedea01fd056).

< # Always inherit from the "build_ext" in distutils since setuptools already imports
< # it from Cython if available, and does the proper distutils fallback otherwise.
< # https://github.com/pypa/setuptools/blob/9f1822ee910df3df930a98ab99f66d18bb70659b/setuptools/command/build_ext.py#L16
<
< # setuptools imports Cython's "build_ext", so make sure we go first.
< _build_ext_module = sys.modules.get('setuptools.command.build_ext')
< if _build_ext_module is None:
<     import distutils.command.build_ext as _build_ext_module
<
< # setuptools remembers the original distutils "build_ext" as "_du_build_ext"
< _build_ext = getattr(_build_ext_module, '_du_build_ext', None)
< if _build_ext is None:
<     _build_ext = getattr(_build_ext_module, 'build_ext', None)
< if _build_ext is None:
---
> if 'setuptools' in sys.modules:
>     try:
>         from setuptools.command.build_ext import build_ext as _build_ext
>     except ImportError:
>         # We may be in the process of importing setuptools, which tries
>         # to import this.
>         from distutils.command.build_ext import build_ext as _build_ext

If we revert the import in Cython/Distutils/build_ext.py with the one in cython 0.x, the build now works

 ❯  head -17 /Users/lmwangi/Library/Caches/pypoetry/virtualenvs/poetry-pypi-example-wSzIWKA2-py3.9/lib/python3.9/site-packages/Cython/Distutils/build_ext.py 
import sys
import os


import sys

if 'setuptools' in sys.modules:
    try:
        from setuptools.command.build_ext import build_ext as _build_ext
    except ImportError:
        # We may be in the process of importing setuptools, which tries
        # to import this.
        from distutils.command.build_ext import build_ext as _build_ext
else:
    from distutils.command.build_ext import build_ext as _build_ext


 ❯ poetry run Cython -V 
Cython version 3.0.0a10

  ❯ poetry build  
Building poetry-pypi-example (0.0.1)
  - Building sdist
  - Built poetry-pypi-example-0.0.1.tar.gz
  - Building wheel
[1/1] Cythonizing poetry_pypi_example/main.py
  - Built poetry_pypi_example-0.0.1-cp39-cp39-macosx_12_0_x86_64.whl

[its-monotype]

@labeneator Hello! Did you figure out how to fix this error and use that script with latest version of Cython and Poetry?

[labeneator]

@labeneator Hello! Did you figure out how to fix this error and use that script with latest version of Cython and Poetry?

With the small patch above, it worked. However, I had to abandon the cython route because pydantic <> cython does not work. Cython maintainers were unwilling to revert the patch as per: https://groups.google.com/g/cython-users/c/CQ2ieXACtzk/m/guDy3kFZBAAJ

[its-monotype]

@labeneator Thanks for your answer! I already fixed this tomorrow by editing build.py a bit:

...

def build(setup_kwargs: Dict[str, Any]) -> Dict[str, Any]:
    # Generate stub files for type hints
    generate_stub_files()

    # Collect and cythonize all files
    extension_modules = cythonize_helper(get_extension_modules())

    # Use Setuptools to collect files
    distribution = Distribution(
        {
            "ext_modules": extension_modules,
            "cmdclass": {
                "build_ext": cython_build_ext,
            },
        }
    )

    # Grab the build_ext command
    build_ext_cmd = distribution.get_command_obj("build_ext")
    build_ext_cmd.ensure_finalized()
    # Set the value to 1 for "inplace", with the goal to build extensions
    # in build directory, and then copy all files back to the source dir
    # (under the hood, "copy_extensions_to_source" will be called after
    # building the extensions). This is done so Poetry grabs the files
    # during the next step in its build.
    build_ext_cmd.inplace = 1
    build_ext_cmd.run()

    return setup_kwargs
    
...

if __name__ == '__main__':
    build({})

And here are my build settings in pyproject.toml:

[tool.poetry]
...
# Only include cythonized files in final build
include = ["mylib/**/*.so", "mylib/**/*.pyd"]  # ignored in VCS, so we need to be explicit
exclude = ["mylib/**/*.py"]
...
[tool.poetry.build]
generate-setup-file = false
script = "build.py"
...

Now I'm using the latest version of Poetry and the latest version of Cython (3.0.0a11).

And as a result, I get a protected package without source code, only with .pyd extensions, and I also added auto-generation of .pyi files so that hints and types remain when using the library.

I heard that pydantic is good, but simple typing and sometimes dataclasses are enough for me. The main thing is that the source code is protected.

[its-monotype]

And I also recommend not converting Path to the str(Path) string, but using the .as_posix method, because backslashes on Windows can cause problems. Because of this, the cythonization process did not start for me at all.

So you need to replace this

# Convert path to module name
module_path = str(module_path).replace("/", ".")

to this

# Convert path to module name
module_path = module_path.as_posix().replace("/", ".")

and in other places too