Expose compiler classes for customized use

[minrk]

Summary

The porting from distutils docs are a bit sparse, and could use some more specific examples. Specifically, for distutils.ccompiler and distutils.util.get_platform were the replacements I was looking for, and didn't find.

Maybe it would be good to start populating a FAQ section at or linked from the docs. Starting out with finishing the the table in the pep with actual recommended import substitutions that work (4 are there already), would be a huge help for migrators like myself.

OS / Environment

No response

Additional Information

In attempting to migrate https://github.com/zeromq/pyzmq away from distutils, I came up with some questions that don't appear to have clear answers in docs, and it seems like setuptools is the right place for such docs (if that's wrong, what is the right place?).

The pep lists setuptools as the thing to switch to for several APIs, such as ccompiler, etc., but at least ccompiler does not appear to be available from public setuptools APIs. Is importing form setuptools._distutils the 'right' way in general to get distutils functionality that hasn't been exposed at public level of setuptools yet? Or are there plans to expose these as top-level APIs (e.g. setutools.ccompiler)?

I can import new_compiler and customize_compiler from setuptools.command.build_ext, but that doesn't seem better than setuptools._distutils.ccompiler to me. If it were documented as the suggested alternative, I'd be happy, though.

Could we perhaps have a statement in the docs about when, if ever, importing from setuptools._distutils is appropriate? The text already there:

If a project relies on uses of distutils that do not have a suitable replacement above, please search the Setuptools issue tracker and file a request

suggests anything that requires importing from setuptools._distutils should result in an Issue (how I got here), but maybe could be more explicit about what to do while finding a resolution. A blessing of "import from setuptools._distutils" in the interim would ease some stress on migrators, but may not be what you all want to support.

The pep also lists the platform module as the replacement for distutils.util.get_platform, but in my experience this is not the right advice. Where I've used distutils.util.get_platform(), what I really wanted was the build platform string, which is available as dist.get_command_obj('build').plat_name. I don't always call it from within a command so this is a bit inconvenient, but I think I can manage. I don't think it's possible to construct this from the platform module.

Code of Conduct

• I agree to follow the PSF Code of Conduct

[althonos]

I would also be interested in the migration process for distutils.ccompiler. I have several libraries that are using the interal CCompiler API to detect build-support for SIMD code. In my opinion, the distutils.ccompiler module has a lot of legacy code that doesn't always work (for instance, using a UNIX compiler on Windows through MinGW barely works) so I'd think it would be great to have new top-level APIs to work with. I'd be happy to contribute to this.

[minrk]

FWIW, this is how I've created compilers without calling private APIs:

from setuptools import setup
from setuptools import Extension
from setuptools.command.build_ext import build_ext

class GetCompiler(build_ext):
    def finalize_options(self):
        super().finalize_options()
        # need to have an Extension for `.run()` to create self.compiler
        self.extensions = [Extension("unused", ["unused.c"])]

    def build_extensions(self):
        """Empty to skip actual compilation"""
        # because we need self.extensions to be defined and unused,
        # we _also_ need to skip actually building them
    
    def run(self):
        super().run()
        print(f"Got my compiler! {self.compiler}")
        

setup(
    cmdclass={"compiler": GetCompiler},
)

Relying on build_ext's logic to create the compiler. The tedious bit is that build_ext only creates the compiler if it has some extensions to compile. So I have to pass it an unused Extension and also override build_extensions to then ignore those unused Extensions.

Then I replaced all my calls to new_compiler/configure_compiler to passing around copies of self.compiler.

To always get a fresh compiler, you can do the same thing by patching a build_ext on an empty Distribution:

from setuptools import Distribution, Extension

def get_compiler():
    d = Distribution()
    build_ext = Distribution().get_command_obj("build_ext")
    build_ext.finalize_options()
    # register an extension to ensure a compiler is created
    build_ext.extensions = [Extension("ignored", ["ignored.c"])]
    # disable building fake extensions
    build_ext.build_extensions = lambda: None
    # run to populate self.compiler
    build_ext.run()
    return build_ext.compiler

[neutrinoceros]

Hi @abravalheri, do you think this could get some "official" answer ?

[abravalheri]

Hi @neutrinoceros, compilers are still outside of my area of expertise in setuptools. Recently I got more acquainted with build_ext but I still don't feel like I can contribute directly on this discussion. Sorry for that.

I noticed that in #2372, @jaraco seems to be open to implement some changes in order to support the migration, e.g.:

• provide setuptools extensions to implement additional functionality where appropriate; setuptools could add extendable hooks if that helps.
• creating suitably-compatible versions of public interfaces entirely in the setuptools namespace and weaning users and packages off of import distutils*.
• explore what interfaces the users need for the distutils compiler classes/modules and consolidate the behaviours.

I think that one thing that can help in this process is to collect what are the requirements for the developers (and potentially, if you have any suggestions, what would be the desired interfaces).

Would you guys be happy if setuptools simply exposes part of the setuptools._distutils.ccompiler interface under setuptools.ccompiler (or another public module)? What are the functions/classes that are needed from this module? Is there a use case for using this API outside of a setuptools command?

---

Where I've used distutils.util.get_platform(), what I really wanted was the build platform string, which is available as dist.get_command_obj('build').plat_name. I don't always call it from within a command so this is a bit inconvenient, but I think I can manage. I don't think it's possible to construct this from the platform module.

What would be the use case for running get_platform outside of a setuptools command?

[neutrinoceros]

Thanks for your reply ! I will answer on behalf of yt

Would you guys be happy if setuptools simply exposes part of the setuptools._distutils.ccompiler interface under setuptools.ccompiler (or another public module)?

Pretty much, though any official recommendation would satisfy us.

What are the functions/classes that are needed from this module?

CCompiler and new_compiler are the only components that we use.

Is there a use case for using this API outside of a setuptools command?

Not that I know of.

[neutrinoceros]

As a side note, we also rely on distutils.sysconfig.customize_compiler. Should I open a different ticket or is this relevant enough to this conversation ?

[abravalheri]

Thank you very much @neutrinoceros. If the other maintainers are ok with exposing this interface, I can go ahead and try something out (please also feel free to submit a PR - I am a bit busy for July so it may take time...).

As a side note, we also rely on distutils.sysconfig.customize_compiler. Should I open a different ticket or is this relevant enough to this conversation ?

I would not mind to track them together...

[neutrinoceros]

@abravalheri @minrk I've opened a PR to try to address this: #3445
Your feedback is very welcome

[abravalheri]

Thank you very much @neutrinoceros, that is very helpful.

I think we should ask Jason to review the PR to see if he agrees with this direction.

[FirefoxMetzger]

I'm also interested in getting the CCompiler, because I want to write a custom command to compile a shared(!) library that is then accessed by ctypes. Clarifications on how this is meant to be done would help a ton 🚀

We can do this for statically linked archives (.lib) using the build_clib command and the libraries kwarg of setup(), but I couldn't find anything for dynamically linked libraries (.dll / .so). This might be slightly off-topic here, but would it be useful to discuss adding a build_shared_clib command to build such shared libraries?