[FR] Allow plugins to pull in configuration from `pyproject.toml`

[abravalheri]

Originally this issue was discussed in #2671.

Plugin writers would like a way of pulling in configuration/metadata from pyproject.toml.
Currently the finalize_distribution_options hook runs before parse_config_files, which makes integration difficult.

@RonnyPfannschmidt, please correct me if I understood something wrong.

It would be nice if we can clarify what would be the requirements of this feature.

[abravalheri]

@RonnyPfannschmidt, I am thinking along the lines of:

class BuildHooks:
   """
   Build hooks can be added via the `setuptools.build_hooks` entry-point, and they should implement
   this interface.
   
   All public methods are reserved for future setuptools use (despite just a few hooks
   being currently defined). All public methods specified in the interface are optional,
   but when defined, they **MUST** comply with the given signature.
   """
   
   def __init__(self, dist: Distribution):
       """Plugins are initialized  inside ``finalize_options``
       (just like the ``finalize_distribution_option`` entry-point runs nowadays).
       
       Plugin writers are encouraged to save the ``dist`` argument in a private property to
       use it in successive hooks.
       """
       
   def before_config_hook(self, project_metadata: dict, plugin_config: Any):
       """
       Setuptools will pass ``plugin_config`` with the value of the first of the following
       configurations defined in ``pyproject.toml``:
       ``tool.setuptools.plugin.{{entry_point_name}}`` or ``tool.{{entry_point_name}}``.
       If the configuration is not present, the value of ``config`` will be ``None``.
       If desired, the hook implementation is responsible for validating the ``plugin_config``
       value.
       
       ``project_metadata`` corresponds to the raw metadata in the ``[project]`` table
       **if** provided in ``pyproject.toml``.
       
       Plugin writers can take advantage of ``dist`` (passed on the initialization method)
       to decide if they want to manipulate ``dist.cmdclass`` and/or add sub commands to 
       ``setuptools.command.build.build.sub_commands`` based on the value of ``plugin_config``.
       It is recommended that plugins refrain from altering the build behaviour if
       ``plugin_config`` is ``None`` (i.e. users did not opt-in).
       
       Plugin writers can also modify ``dist`` properties, but they must be aware that
       setuptools might modify these properties again after processing the ``pyproject.toml``.
       """
       
   def after_config_hook(self):
       """
       Plugin writers can use this method when they need to access attributes
       in ``dist`` after it being fully configured, but they **SHOULD** not
       modify any attribute related to project metadata
       (otherwise it could cause a direct violation of the ``dynamic``
       aspect of PEP 621 and PEP 643).
       """
       
   def before_commands_hook(self):
       """Setuptools will run this hook immediatelly before running the first command."""

[RonnyPfannschmidt]

Standards for Metadata providing hooks would help better

[abravalheri]

Do you have a suggestion @RonnyPfannschmidt (e.g. a signature)?

[RonnyPfannschmidt]

Not yet, but if both setuptools and hatch where able to use the same api fit that it would be great

I already have the hacks in place and I'm not keen on supporting many versions of setuptools with different api sets

[abravalheri]

I cannot speak for everyone in the setuptools project, but I am more than open to discuss any API/interoperability (as long as it does not involve an overhaul on the setuptools internal design -- ideally we are looking for something that can be integrated easily into the existing flow).

[RonnyPfannschmidt]

Ping @ofek wrt potentially shareable api for dynamic Metadata sources

[moshez]

@abravalheri -- would part of fixing this ticket involve moving finalize_distribution_options to after parse_config_files()?

Or would plugin authors need to hook into new entry points to run after parse_config_files()?

Would it make sense to split this ticket into two: moving finalize_distribution_options to after parse_config_files and as a separate effort, have an API for plugins to read configuration from the toml?

[abravalheri]

would part of fixing this ticket involve moving finalize_distribution_options to after parse_config_files()?

Hi @moshez, there is an "unfinished" discussion about this in the pypa/setuptools discord. I don't know how safe this operation would be because there might be code depending on the fact that finalize_distribution_options runs before parse_config_files (plugins or setuptools itself).
In my mind it would be safer to have a separated hook, but this opinion is very personal.

Would it make sense to split this ticket into two: moving finalize_distribution_options to after parse_config_files and as a separate effort, have an API for plugins to read configuration from the toml?

We can definitely do that. For the time being I am postponing this, because first I would like to listen from the plugin developers what are their list of requirements and what would be their idea of an useful API.

[moshez]

Thanks @abravalheri .

As someone who has written a plugin, I can tell you that my preference would be for "finalize..." to be called after the parsing of the toml file, because I assume that anything I modify in "finalize..." does not change any more. However, if there is a new hook, I can modify my plugin to use that. Either way, the lack of any hook that is after the parsing of the toml file is limiting the abilities of a plugin. What are the next steps? I can help circulating an informal poll, but we will need to decide how long we wait for responses, and how the decision is made, before starting to write the poll: in my experience, these decisions influence the design of the poll.

Let me know if I can help in any way, or if there is any further feedback you need.

[KOLANICH]

I think that there should be a way to specify mode of invocation of a plugin. There should be at least 2 modes:

a. a plugin is always invoked
b. a plugin is invoked only if pyproject.toml has a section for it. If pyproject.toml doesn't have a section, plugin's package must not be imported at all.

It is proposed to provide the possibility to specify this it via "entry_points metadata" - JSON (or maybe any other serialization format like JSON) appended to the end of entry point name after a delimiter (currently in my packages I use @).