target globbing/grouping

[cburroughs]

Is your feature request related to a problem? Please describe.

There are several different use cases that are tangentially related to grouping or globbing targets.

• Sphinx or other documentation tools would want to run on "all the python code".
• One off tricky non-inferable entry points (as an alternative to "write your own dep inference plugin")

A few more I can add:

• Including all the python files in "big ML project" into one docker image. So one docker image may be able to run hundreds of different but related "ML flows". (These could each have their own image, but that would be a lot of images to juggle.)
• Bundling libraries: For a vendor with a manged notebook environment there isn't an "executable" per se that needs to be deployed, but a bunch of python "libraries"that need to be statically identified (.py files, that need to show up on disk at the expected location)
• Dynamic "registration:" I'll try to write this up in more detail but the basic idea is a tool for CI pipelines where where any folder can have some pipeline like my_pipline.py that defines a pipline, and a runner tool. The runner tool then has a glob dep like **/my_pipeline.py These are not exactly "entry points" since everything is only run through Pants, but the idea is that anyone can drop in pipeline file and it "just works".

Describe the solution you'd like

It feels like there is some sort of missing primitive here, but I'm not sure exactly what a non-plugin should look lik.e

Additional context

Slack threads:

• https://chat.pantsbuild.org/t/16601375/topic
• https://chat.pantsbuild.org/t/18811751/is-there-a-way-to-specify-for-a-particular-file-target-that-
• https://chat.pantsbuild.org/t/18783369/i-have-a-python-executable-that-can-dynamically-import-a-num

Somewhat related:

• Confusing 'DifferingFamiliesError: Expected AddressMaps to share the same parent directory' exception for use of file glob in dependencies #17733
• Add a global option to handle having recursive globs in target generators sources field #20450
• tailor should support non-1:1-1 mode, i.e. not generated 1 BUILD file per directory #15429
• Supporting custom entry points other than `console_scripts` in pex envs #17308

[cburroughs]

@grihabor wrote a spiffy plugin that I have adapted below as an example of how I've been hacking around this. Besides the fundamental "is this actually the right core primitive?" there might be some noodling around AllTargets use/performance needed.)

# Adopted from Gregory Borodin's example in Pants Slack

import fnmatch
import logging
from collections.abc import Mapping, Sequence, Set
from dataclasses import dataclass
from textwrap import dedent

from pants.engine.rules import collect_rules, rule
from pants.engine.target import (
    AllTargets,
    BoolField,
    Dependencies,
    FieldSet,
    InferDependenciesRequest,
    InferredDependencies,
    NestedDictStringToStringField,
    StringSequenceField,
    Target,
)
from pants.engine.unions import UnionRule

logger = logging.getLogger(__name__)


class MyGlobTargetTypes(NestedDictStringToStringField):
    alias = "target_types"
    help = dedent("""\
    dict[str, dict[str, str]] where the keys are names of target types (eg "python_source", "resources").

    The values are usually empty dictionaries. If non-empty, it acts as a filter: a particular source will only be
    picked up as a dependency if each value is equal to the field value for that key. If the field value in the target
    type is actually tuple | list | Set, then the match condition is based on a contained `in` check, rather than
    equality.

    If the field key is prefixed with "!", then the equality check is changed to an inequality check, and the contained
    `in` check is changed to a `not in` check.
    """)


class MyGlobIncludes(StringSequenceField):
    alias = "includes"
    help = "Iterable[str].  When present, sources will only be included if and only if their address path matches at least one of these fnmatch style globs."


class MyGlobExcludes(StringSequenceField):
    alias = "excludes"
    help = "Iterable[str]. Sources will be excluded as a dependency if its path matches any of these fnmatch style globs."


class MyGlobFromBuildRoot(BoolField):
    alias = "from_build_root"
    default = False
    help = "Bool. Should the glob work from the globs own address, or globally from the root?"


class MyGlobDependencies(Dependencies):
    pass


class MyGlobTarget(Target):
    alias = "my_target_glob"
    help = dedent("""\
    Depends on all targets in the subtree recursively.

    This can be used as a simple substitute for inferring dynamic dependencies,
    which Pants is not capable of doing automatically, if the sources for those
    dynamic dependencies are within a directory.

    On target matching:
     * By default, everything is matched by the target type
     * When `includes` or `excludes` is set, those predicates must also be satisfied.
     * When both `inclues` and `excludes` is set, both predicates must be satisfied.
    """)
    core_fields = (
        MyGlobTargetTypes,
        MyGlobDependencies,
        MyGlobIncludes,
        MyGlobExcludes,
        MyGlobFromBuildRoot,
    )


@dataclass(frozen=True)
class MyGlobDependenciesInferenceFieldSet(FieldSet):
    required_fields = (
        MyGlobTargetTypes,
        MyGlobDependencies,
        MyGlobIncludes,
        MyGlobExcludes,
        MyGlobFromBuildRoot,
    )

    target_types: MyGlobTargetTypes
    dependencies: MyGlobDependencies
    includes: MyGlobIncludes
    excludes: MyGlobExcludes
    from_build_root: MyGlobFromBuildRoot


class MyInferGlobDependenciesRequest(
    InferDependenciesRequest[MyGlobDependenciesInferenceFieldSet]
):
    infer_from = MyGlobDependenciesInferenceFieldSet


@rule(desc="Recursively add all targets as dependencies for the glob target.")
async def infer_glob_dependencies(
    request: MyInferGlobDependenciesRequest,
    all_targets: AllTargets,
) -> InferredDependencies:
    spec_path = request.field_set.address.spec_path
    target_types: Mapping = request.field_set.target_types.value or {}
    include_globs: Sequence = request.field_set.includes.value or []
    exclude_globs: Sequence = request.field_set.excludes.value or []
    from_build_root = request.field_set.from_build_root.value

    dependencies = []
    for t in all_targets:
        if t.alias not in target_types:
            continue

        if not from_build_root and not (
            t.address.spec_path == spec_path
            or (
                t.address.spec_path.startswith(spec_path)
                and t.address.spec_path[len(spec_path)] == "/"
            )
            or (spec_path == "")
        ):
            continue

        logger.debug("glob: check target %s", t.alias)
        target_type_filter = dict(target_types[t.alias])
        if not all(
            _field_matches_target_type_filter(field_cls, field, target_type_filter)
            for field_cls, field in t.field_values.items()
        ):
            logger.debug("glob: skip target %s, field values %s", t.alias, t.field_values)
            continue

        if include_globs:
            if not any([fnmatch.fnmatch(str(t.address), pat) for pat in include_globs]):
                logger.debug(f"glob: skip target {t.address}; no matches in {include_globs}")
                continue

        skip_from_exclude = False
        for pat in exclude_globs:
            if fnmatch.fnmatch(str(t.address), pat):
                logger.debug("glob: skip target %s, matched exclude %s", t.address, pat)
                skip_from_exclude = True
                break
        if skip_from_exclude:
            continue

        logger.debug("glob: add target %s:%s", t.alias, t.address)
        dependencies.append(t.address)

    return InferredDependencies(dependencies)


def _field_matches_target_type_filter(
    field_cls, field, target_type_filter: dict[str, str]
) -> bool:
    """See MyGlobTargetTypes for an explanation of the filtering logic."""
    if (value := target_type_filter.get(field_cls.alias)) is not None:
        if isinstance(field.value, tuple | list | Set):
            # NOTE: We can't check for isinstance(Sequence) because that would include str, but we don't want to treat
            # str as a collection of characters in this context.
            if value not in field.value:
                return False
        else:
            if value != field.value:
                return False
    if (value := target_type_filter.get(f"!{field_cls.alias}")) is not None:
        if isinstance(field.value, tuple | list | Set):
            if value in field.value:
                return False
        else:
            if value == field.value:
                return False
    return True


def target_types():
    return [MyGlobTarget]


def rules():
    return (
        *collect_rules(),
        UnionRule(InferDependenciesRequest, MyInferGlobDependenciesRequest),
    )