Merge branch 'main' into ramon/add-build_file_path-var

Author: rbuckland

docs/docs/using-pants/advanced-target-selection.mdx

@@ -145,7 +145,7 @@ Whereas `tags` are useful for _decentralized_ allow/block lists, `--spec-files`
 To pipe a Pants run, use your shell's `|` pipe operator and `xargs`:
 
 ```bash
-pants dependents helloworld/util | xargs pants  list
+pants dependents helloworld/util | xargs pants list
 ```
 
 You can, of course, pipe multiple times:
@@ -167,6 +167,14 @@ $ pants dependencies helloworld/util > util_dependencies.txt
 $ pants --spec-files=util_dependencies.txt lint
 ```
 
+Using spec files is also more robust because when piping output of a Pants goal to `xargs`, the specified command
+may be invoked by `xargs` as many times as necessary to use up the list of input items.
+This may break the structured data output, for instance, when you want to `peek` the targets as JSON:
+
+```bash
+$ pants list --filter-target-type=resource :: | xargs pants peek
+```
+
 If you don't want to save the output to an actual file—such as to not pollute version control—you can use a variable and a named pipe:
 
 ```bash
@@ -185,3 +193,28 @@ For other goals, you can leverage shell piping to partition the input targets in
 ```bash
 pants list :: | awk 'NR % 5 == 0' | xargs pants package
 ```
+
+## Using CLI aliases
+
+If setting tags on individual targets is not feasible, there are a few other options available to refer to multiple targets.
+
+If you have an operation that you perform often on a certain group of targets, you can use the
+[cli](../../reference/subsystems/cli) subsystem options to create shortcuts. For instance, this alias
+would let you run `pants publish-libraries` to publish all Python distributions declared in the `src/libA` and `src/libB`
+directories.
+
+```toml title="pants.toml"
+[cli.alias]
+publish-libraries = "--filter-target-type=python_distribution --filter-address-regex=\"['^src/libA/,^src/libB/']\" publish src::"
+```
+
+You can use any argument or goal, and the alias doesn't need to be a "full" invocation of Pants.
+For instance, you could combine filtering arguments along with `--changed-since` flag and a tag to refer to long-running
+integration tests that have been recently modified:
+
+```toml title="pants.toml"
+[cli.alias]
+--integration-long = "--changed-since --filter-target-type=python_test --tag=long"
+```
+
+You can now invoke `pants --integration-long test tests::` to run the relevant tests.

docs/docs/using-pants/key-concepts/targets-and-build-files.mdx

@@ -166,6 +166,50 @@ You can use the prefix `!!` to transitively exclude a dependency, meaning that e
 Transitive excludes can only be used in target types that conventionally are not depended upon by other targets, such as `pex_binary`, `python_distribution`, and `python_test` / `python_tests`. This is meant to limit confusion, as using `!!` in something like a `python_source` / `python_sources` target could result in surprising behavior for everything that depends on it. (Pants will print a helpful error when using `!!` when it's not legal.)
 :::
 
+## Using the generic `target`
+
+[`target`](../../..reference/targets/target) is a generic target with no specific type.
+It can be used as a generic collection of targets to group related, but distinct targets into one single target.
+
+### Referring to a group of targets
+
+You could use the generic `target` when you need to group multiple targets to refer to them as a unit
+(a single dependency) to reduce repetition:
+
+```python title="BUILD"
+target(
+    name="python-libs",
+    dependencies=["src/python/libraries/libA", "src/python/libraries/libB"],
+)
+```
+
+If declared in the root of your workspace, you can now address the Python libraries by `//:python-libs`:
+
+```bash
+$ pants dependencies //:python-libs
+````
+
+### Creating aliases for targets
+
+If you have some targets declared in BUILD files that are stored deep within the directory structure of your workspace,
+you can make it easier to refer to that target when listing that target among dependencies of other targets.
+
+For example, you can simplify accessing a target by creating another target that will serve as an alias definition in
+a BUILD file stored in a more convenient location in the workspace, for instance, in the build root directory:
+
+```python title="BUILD"
+target(
+    name="deployment-bins",
+    dependencies=["src/golang/production/cloud/deployment/binaries:tools"]
+)
+```
+
+You can now refer to that target more concisely in BUILD files:
+
+```python title="BUILD"
+python_sources(dependencies=["//:deployment-bins"])
+```
+
 ## Field default values
 
 As mentioned above in [BUILD files](./targets-and-build-files.mdx#build-files), most target fields have sensible defaults. And it's easy to override those values on a specific target. But applying the same non-default value on many targets can get unwieldy, error-prone and hard to maintain. Enter `__defaults__`.

docs/docs/using-pants/project-introspection.mdx

@@ -250,6 +250,10 @@ pants dependents  helloworld/main.py:lib | xargs pants peek --exclude-defaults
 ]
 ```
 
+Keep in mind, however, that the `peek` goal may be invoked by `xargs` as many times as necessary to use up the list
+of input items. This may break the structured data output, so it may be safer to use the
+[`--spec-files`](../../reference/global-options#spec_files) option.
+
 :::
 
 ## `paths` - find dependency paths

docs/docs/writing-plugins/the-rules-api/rules-and-the-target-api.mdx

@@ -235,7 +235,7 @@ class MyTarget(Target):
 Then, to resolve the addresses, you can use `UnparsedAddressInputs`:
 
 ```python
-from pants.engine.addresses import Address, Addresses, UnparsedAddressInputs
+from pants.engine.addresses import Addresses, UnparsedAddressInputs
 from pants.engine.target import Targets
 from pants.engine.rules import Get, rule
 
@@ -291,6 +291,30 @@ async def demo(...) -> Foo:
 
 `SourceFilesRequest` expects an iterable of `SourcesField` objects. `SourceFiles` has a field `snapshot: Snapshot` with the merged snapshot of all resolved input sources fields.
 
+To convert a list of target addresses to existing source file names, you can request `HydratedSources` for every input target:
+
+```python
+from itertools import chain
+from pants.engine.addresses import Addresses
+from pants.engine.collection import DeduplicatedCollection
+from pants.engine.rules import Get, MultiGet, rule
+from pants.engine.target import (HydratedSources, HydrateSourcesRequest, SourcesField, UnexpandedTargets)
+
+
+class ProjectSources(DeduplicatedCollection[str]):
+    pass
+
+
+@rule
+async def addresses_to_source_files(addresses: Addresses) -> ProjectSources:
+    targets = await Get(UnexpandedTargets, Addresses, addresses)
+    all_sources = await MultiGet(Get(HydratedSources, HydrateSourcesRequest(tgt.get(SourcesField))) for tgt in targets)
+    return ProjectSources(chain.from_iterable(sources.snapshot.files for sources in all_sources))
+```
+
+This is often useful when you need to pass target addresses to commands that are not Pants goals and would not
+be able to interpret them properly.
+
 ### Enabling codegen
 
 If you want your plugin to work with code generation, you must set the argument `enable_codegen=True`, along with `for_sources_types` with the types of `SourcesField` you're expecting.

src/python/pants/backend/docker/goals/package_image.py

@@ -522,6 +522,8 @@ def parse_image_id_from_docker_build_output(docker: DockerBinary, *outputs: byte
                 (
                     # BuildKit output.
                     r"(writing image (?P<digest>sha256:\S+) done)",
+                    # BuildKit with containerd-snapshotter output.
+                    r"(exporting manifest list (?P<manifest_list>sha256:\S+) done)",
                     # Docker output.
                     r"(Successfully built (?P<short_id>\S+))",
                 ),
@@ -540,7 +542,11 @@ def parse_image_id_from_docker_build_output(docker: DockerBinary, *outputs: byte
                 None,
             )
             if image_id_match:
-                image_id = image_id_match.group("digest") or image_id_match.group("short_id")
+                image_id = (
+                    image_id_match.group("digest")
+                    or image_id_match.group("short_id")
+                    or image_id_match.group("manifest_list")
+                )
                 return image_id
 
     return "<unknown>"

src/python/pants/backend/docker/goals/package_image_test.py

@@ -1646,6 +1646,47 @@ def test_get_context_root(
             ),
             "",
         ),
+        # Buildkit with containerd-snapshotter
+        (
+            DockerBinary("/bin/docker", "1234", is_podman=False),
+            "sha256:b2b51838586286a9e544ddb31b3dbf7f6a99654d275b6e56b5f69f90138b4c0e",
+            dedent(
+                """\
+                #9 exporting to image
+                #9 exporting layers done
+                #9 exporting manifest sha256:7802087e8e0801f6451d862a00a6ce8af3e4829b09bc890dea0dd2659c11b25a done
+                #9 exporting config sha256:c83bed954709ba0c546d66d8f29afaac87c597f01b03fec158f3b21977c3e143 done
+                #9 exporting attestation manifest sha256:399891f9628cfafaba9e034599bdd55675ac0a3bad38151ed1ebf03993669545 done
+                #9 exporting manifest list sha256:b2b51838586286a9e544ddb31b3dbf7f6a99654d275b6e56b5f69f90138b4c0e done
+                #9 naming to myhost.com/my_app:latest done
+                #9 unpacking to myhost.com/my_app:latest done
+                #9 DONE 0.0s
+                """
+            ),
+            "",
+        ),
+        # Buildkit with containerd-snapshotter and cross platform
+        (
+            DockerBinary("/bin/docker", "1234", is_podman=False),
+            "sha256:3c72de0e05bb75247e68e124e6500700f6e0597425db2ee9f08fd59ef28cea0f",
+            dedent(
+                """\
+                #12 exporting to image
+                #12 exporting layers done
+                #12 exporting manifest sha256:452598369b55c27d752c45736cf26c0339612077f17df31fb0cdd79c5145d081 done
+                #12 exporting config sha256:6fbcebfde0ec24b487045516c3b5ffd3f0633e756a6d5808c2e5ad75809e0ca6 done
+                #12 exporting attestation manifest sha256:32fcf615e85bc9c2f606f863e8db3ca16dd77613a1e175e5972f39267e106dfb done
+                #12 exporting manifest sha256:bcb911a3efbec48e3c58c2acfd38fe92321eed731c53253f0b5c883918420187 done
+                #12 exporting config sha256:86e7fd0c4fa2356430d4ca188ed9e86497b8d03996ccba426d92c7e145e69990 done
+                #12 exporting attestation manifest sha256:66f9e7af29dd04e6264b8e113571f7b653f1681ba124a386530145fb39ff0102 done
+                #12 exporting manifest list sha256:3c72de0e05bb75247e68e124e6500700f6e0597425db2ee9f08fd59ef28cea0f done
+                #12 naming to myhost.com/my_app:latest done
+                #12 unpacking to myhost.com/my_app:latest done
+                #12 DONE 0.0s
+                """
+            ),
+            "",
+        ),
         # Podman
         (
             DockerBinary("/bin/podman", "abcd", is_podman=True),

src/python/pants/backend/project_info/dependencies.py

@@ -41,8 +41,7 @@ class DependenciesSubsystem(LineOriented, GoalSubsystem):
     )
     closed = BoolOption(
         default=False,
-        help="Include the input targets in the output, along with the dependencies. This option "
-        "only applies when using the `text` format.",
+        help="Include the input targets in the output, along with the dependencies.",
     )
     format = EnumOption(
         default=DependenciesOutputFormat.text,
@@ -58,11 +57,7 @@ class Dependencies(Goal):
 async def list_dependencies_as_json(
     addresses: Addresses, dependencies_subsystem: DependenciesSubsystem, console: Console
 ) -> None:
-    """Get dependencies for given addresses and list them in the console in JSON.
-
-    Note that `--closed` option is ignored as it doesn't make sense to duplicate source address in
-    the list of its dependencies.
-    """
+    """Get dependencies for given addresses and list them in the console in JSON."""
     # NB: We must preserve target generators for the roots, i.e. not replace with their
     # generated targets.
     target_roots = await Get(UnexpandedTargets, Addresses, addresses)
@@ -80,10 +75,16 @@ async def list_dependencies_as_json(
         )
 
         iterated_targets = []
-        for transitive_targets in transitive_targets_group:
-            iterated_targets.append(
-                sorted([str(tgt.address) for tgt in transitive_targets.dependencies])
-            )
+        for idx, transitive_targets in enumerate(transitive_targets_group):
+            targets_collection = {
+                str(tgt.address)
+                for tgt in (
+                    transitive_targets.closure
+                    if dependencies_subsystem.closed
+                    else transitive_targets.dependencies
+                )
+            }
+            iterated_targets.append(sorted(targets_collection))
 
     else:
         dependencies_per_target_root = await MultiGet(
@@ -98,8 +99,11 @@ async def list_dependencies_as_json(
         )
 
         iterated_targets = []
-        for targets in dependencies_per_target_root:
-            iterated_targets.append(sorted([str(tgt.address) for tgt in targets]))
+        for idx, targets in enumerate(dependencies_per_target_root):
+            targets_collection = {str(tgt.address) for tgt in targets}
+            if dependencies_subsystem.closed:
+                targets_collection.add(str(target_roots[idx].address))
+            iterated_targets.append(sorted(targets_collection))
 
     # The assumption is that when iterating the targets and sending dependency requests
     # for them, the lists of dependencies are returned in the very same order.

src/python/pants/backend/project_info/dependencies_test.py

@@ -282,6 +282,32 @@ def test_python_dependencies_output_format_json_direct_deps(rule_runner: PythonR
             ],
         },
     )
+    # input: directory, recursively, closed
+    assert_deps(
+        specs=["some::"],
+        transitive=False,
+        closed=True,
+        expected={
+            "some/target:target": [
+                "some/target/a.py",
+                "some/target:target",
+            ],
+            "some/target/a.py": [
+                "3rdparty/python:req1",
+                "dep/target/a.py",
+                "some/target/a.py",
+            ],
+            "some/other/target:target": [
+                "some/other/target/a.py",
+                "some/other/target:target",
+            ],
+            "some/other/target/a.py": [
+                "3rdparty/python:req2",
+                "some/other/target/a.py",
+                "some/target/a.py",
+            ],
+        },
+    )
     assert_deps(
         specs=["some/other/target:target"],
         transitive=True,
@@ -295,6 +321,21 @@ def test_python_dependencies_output_format_json_direct_deps(rule_runner: PythonR
             ]
         },
     )
+    assert_deps(
+        specs=["some/other/target:target"],
+        transitive=True,
+        closed=True,
+        expected={
+            "some/other/target:target": [
+                "3rdparty/python:req1",
+                "3rdparty/python:req2",
+                "dep/target/a.py",
+                "some/other/target/a.py",
+                "some/other/target:target",
+                "some/target/a.py",
+            ]
+        },
+    )
 
 
 def test_python_dependencies_output_format_json_transitive_deps(
@@ -366,3 +407,38 @@ def test_python_dependencies_output_format_json_transitive_deps(
             ],
         },
     )
+
+    # input: directory, recursively, closed
+    assert_deps(
+        specs=["some::"],
+        transitive=True,
+        closed=True,
+        expected={
+            "some/target:target": [
+                "3rdparty/python:req1",
+                "dep/target/a.py",
+                "some/target/a.py",
+                "some/target:target",
+            ],
+            "some/target/a.py": [
+                "3rdparty/python:req1",
+                "dep/target/a.py",
+                "some/target/a.py",
+            ],
+            "some/other/target:target": [
+                "3rdparty/python:req1",
+                "3rdparty/python:req2",
+                "dep/target/a.py",
+                "some/other/target/a.py",
+                "some/other/target:target",
+                "some/target/a.py",
+            ],
+            "some/other/target/a.py": [
+                "3rdparty/python:req1",
+                "3rdparty/python:req2",
+                "dep/target/a.py",
+                "some/other/target/a.py",
+                "some/target/a.py",
+            ],
+        },
+    )