MyPy example (#414)

* (almost) working MyPy example

* Need to fully declare this target because of how it's (not) resolved

* [NO TESTS] WIP

* Don't need to check the render in

* A more full example

* Get it all wired up and working

* Finish off the story

* Fix incorrect example depgraph

* [NO TESTS] WIP

* Add aspect configurations

* Upgrade Python to fix importpath issues with rules_mypy

* [NO TESTS] WIP

* Update requirements

* Another missing no-mypy tag; invert logic?

* Switch to opt-in typechecking

* [NO TESTS] WIP

* Alex: Can reference output_groups via filegroup

* Alex: Note the `opt_out_tags` attribute

* Alex: Note that this is an opt-in tag

* Tweak .bazelrc explainer

* Drop FIXME

* Alex: Link to config docs

* Lint from Marvin

Co-authored-by: aspect-workflows[bot] <143031405+aspect-workflows[bot]@users.noreply.github.com>

* A few more words here

---------

Co-authored-by: aspect-workflows[bot] <143031405+aspect-workflows[bot]@users.noreply.github.com>

Author: arrdem

.bazelrc

@@ -36,6 +36,14 @@ common --incompatible_enable_proto_toolchain_resolution
 # see https://github.com/bazelbuild/rules_jvm_external/issues/445
 build --repo_env=JAVA_HOME=../bazel_tools/jdk
 
+# In support of py_mypy/...
+
+# register mypy_aspect with Bazel
+build --aspects //tools/mypy:defs.bzl%mypy_aspect
+
+# optionally, default enable the mypy checks
+build --output_groups=+mypy
+
 # Load any settings & overrides specific to the current user from `.aspect/bazelrc/user.bazelrc`.
 # This file should appear in `.gitignore` so that settings are not shared with team members. This
 # should be last statement in this config so the user configuration is able to overwrite flags from

MODULE.bazel

@@ -29,6 +29,7 @@ bazel_dep(name = "rules_oci", version = "2.0.1")
 bazel_dep(name = "rules_pkg", version = "1.0.1")
 bazel_dep(name = "rules_proto", version = "7.1.0")
 bazel_dep(name = "rules_python", version = "0.40.0")
+bazel_dep(name = "rules_mypy", version = "0.29.0")
 bazel_dep(name = "rules_python_gazelle_plugin", version = "0.35.0")
 bazel_dep(name = "rules_swift", version = "1.12.0")
 bazel_dep(name = "rules_swift_package_manager", version = "0.12.0")
@@ -123,16 +124,28 @@ pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
 
 python = use_extension("@rules_python//python/extensions:python.bzl", "python")
 python.toolchain(
-    python_version = "3.9",
+    python_version = "3.11",
 )
 
 pip.parse(
     hub_name = "pip",
-    python_version = "3.9",
+    python_version = "3.11",
     requirements_lock = "//requirements:all.txt",
 )
 use_repo(pip, "pip")
 
+# rules_mypy co-initializes with rules_python in order to discover types and
+# stubs packages in your requirements lock and create a mapping from normal
+# requirements to stub requirements which can be used to add appropriate stub
+# dependencies automatically during typechecking.
+types = use_extension("@rules_mypy//mypy:types.bzl", "types")
+types.requirements(
+    name = "pip_types",
+    pip_requirements = "@pip//:requirements.bzl",
+    requirements_txt = "//requirements:all.txt",
+)
+use_repo(types, "pip_types")
+
 #########################
 # Java and other JVM languages:
 # https://github.com/bazelbuild/rules_jvm_external/blob/master/examples/bzlmod/MODULE.bazel

py_mypy/BUILD.bazel

@@ -0,0 +1 @@
+exports_files(["requirements.txt"])

py_mypy/README.md

@@ -0,0 +1,112 @@
+# Using MyPy with rules_python
+
+This is a walkthrough of using [rules_mypy](https://github.com/theoremlp/rules_mypy) together with `rules_python` to apply typechecks as part of "building" a Python application.
+
+## How MyPy will work
+
+Bazel's [aspects](https://bazel.build/extending/aspects) allow extensions to traverse the build graph and apply rewrites to it between the analysis pass and before `build` happens.
+A common application of aspects is to "bolt on" behavior to existing rules without having to modify them.
+Which is exactly what we're going to do here.
+
+One way that an aspect can extend an existing rule is by adding an [`OutputGroupInfo`](https://bazel.build/versions/7.4.0/rules/lib/providers/OutputGroupInfo) provider to the rule.
+Output groups are a slightly unusual feature which allows for rule names to be overloaded, and for a rule to provide multiple kinds of outputs.
+To take a slightly familiar example, the `py_binary` rule normally outputs a launcher script and a `.runfiles` tree, but it also provides a zipapp output which can be selectively enabled.
+Outputs may be selected during a build using the [`--output_groups`](https://bazel.build/reference/command-line-reference#flag--output_groups) flag, or by specifying the `output_group` attribute on a `filegroup` rule consuming targets.
+
+Notionally how this will all work is that:
+
+- We need to create an aspect configured to use whatever `mypy` tool we may want.
+- That aspect will extend the `py_*` rules in the build graph to add an output group capturing the MyPy typecheck cache.
+  These typecheck cache outputs will depend on the typecheck cache outputs of all dependencies.
+
+This all has the effect of creating a build sub-graph parallel to our normal build graph which instead of producing and consuming Python files as dependencies produces and consumes the MyPy analysis caches.
+
+To take a simple example, let's say that we have a small build graph
+
+```mermaid
+graph TD
+    A[data_models] --> B[data_persistence];
+    A --> C[inventory_management];
+    A --> D[order_processing];
+    B --> C;
+    B --> D;
+    C --> E[cli];
+    D --> E;
+    F["click (3rdparty)"] --> E;
+    G["pydantic (3rdparty)"] --> A;
+```
+
+Ordinarily the dependencies between these rules take the form of the Python source files underlying the rules.
+But when we activate our `mypy` aspect and select the `mypy` output group, the cache files from typechecking each* of these targets also become part of that dependency chain.
+This allows Bazel to drive typechecking these libraries in depgraph order while caching intermediate results.
+
+We can demonstrate this by looking at the results of `bazel aquery`, which will show MyPy invocations and that the resulting cache trees are dependencies between each of the invocations.
+But more on that in a minute.
+
+## Setup
+
+In this example we've set up `rules_python` in combination with `rules_uv`, which provides lockfile compilation.
+These two give us a Python dependency solution (including the MyPy we want to use), which we'll feed into `rules_mypy`.
+
+The main trick is in `//tools/mypy:BUILD.bazel`, where we provide a definition of the MyPy CLI binary which we can feed into the checking aspect.
+This is important because it allows us to use our locked requirement for MyPy, and to provide MyPy plugins.
+If we didn't do this, `rules_mypy` would "helpfully" provide an embedded default version and configuration of MyPy which may or may not be what we want.
+
+We've configured our `.bazelrc` to apply the aspect so that users don't have to think about separately enabling it.
+Since there's other Python code in this monorepo which doesn't typecheck and we don't want to have to address that to adopt typing, we're going to use the `opt_in_tags` parameter on the aspect configuration.
+This allows us to specify `tags=["mypy"]` on relevant Python targets to selectively apply typechecking rather than just getting mypy checks applied to everything.
+We could also use the `opt_out_tags` parameter on the aspect and annotate stuff we don't want to typecheck, but that has more impact for initial adoption.
+
+Otherwise users explicitly have to list `--aspects=...` when they're interested in leveraging typechecks.
+
+For the same reason we've also configured our `.bazelrc` to enable the `mypy` output group by default.
+This may or may not be desired behavior, since enabling the `mypy` output makes passing typechecks a blocker for build and test operations.
+
+## Demo
+
+If we use `bazel aquery //py_mypy/cli`, we will see among much other output
+
+```
+action 'mypy //py_mypy/cli:cli'
+  Mnemonic: mypy
+  Target: //py_mypy/cli:cli
+  Configuration: darwin_arm64-fastbuild
+  Execution platform: @@platforms//host:host
+  AspectDescriptors: [
+    //tools/mypy:defs.bzl%mypy_aspect(cache='true', color='true')
+]
+  ActionKey: ...
+  Inputs: [
+    bazel-out/.../bin/py_mypy/inventory_management/inventory_management.mypy_cache,
+    bazel-out/.../bin/py_mypy/order_processing/order_processing.mypy_cache,
+    bazel-out/.../bin/tools/mypy/mypy,
+    ...
+  ]
+```
+
+This is the actual typecheck action of the `//py_mypy/cli:cli` target, showing that as inputs it takes (among many other things) the `.mypy_cache` tree results from typechecking the two sub-libraries `inventory_management` and `order_processing`.
+
+If we dig around in the action plan a bit more, we'll also find the typecheck definitions for those products.
+For instance if we inspect the `inventory_management` build, we'll find the production action for those cache files.
+
+```
+action 'mypy //py_mypy/inventory_management:inventory_management'
+  Mnemonic: mypy
+  Target: //py_mypy/inventory_management:inventory_management
+  Configuration: darwin_arm64-fastbuild
+  Execution platform: @@platforms//host:host
+  AspectDescriptors: [
+    //tools/mypy:defs.bzl%mypy_aspect(cache='true', color='true')
+  ]
+  ActionKey: ...
+  Inputs: [
+    bazel-out/.../bin/py_mypy/data_models/data_models.mypy_cache,
+    bazel-out/.../bin/tools/mypy/mypy,
+    ...
+  ]
+```
+
+This demonstrates that the `rules_mypy` configuration will perform incremental typechecking (only targets which changed will be re-checked except in the case of a cascading failure), to the limit of 1stparty code.
+
+Per [rules_mypy#23](https://github.com/theoremlp/rules_mypy/issues/23), the aspect which creates typecheck rules short-circuits and stops to create annotations when it encounters 3rdparty code.
+This bypasses the problem of attempting to apply 1stparty typecheck rules to code which may not conform to them, but creates the problem that because there is no shared `pyspark.mypy_cache` output, 3rdparty libraries may be typechecked (or at least analyzed as part of typechecking) more than once.

py_mypy/cli/BUILD.bazel

@@ -0,0 +1,13 @@
+py_binary(
+    name = "cli",
+    main = "__main__.py",
+    srcs = ["__main__.py"],
+    deps = [
+        "//py_mypy/order_processing",
+        "//py_mypy/inventory_management",
+        "@pip//click",
+    ],
+    tags = [
+        "mypy",  # We've explicitly configured the aspect for opt-in
+    ],
+)

py_mypy/cli/__main__.py

@@ -0,0 +1,33 @@
+import click
+from order_processing import create_order, update_order_shipping
+from inventory_management import adjust_stock
+
+@click.group()
+def cli():
+    pass
+
+@cli.command("create-order")
+@click.option("--customer-id", type=int, required=True)
+@click.option("--items", type=str, required=True) # example: "1:2,2:1"
+@click.option("--shipping-address", type=str)
+def create_order_cmd(customer_id, items, shipping_address):
+    items_list = [(int(item.split(":")[0]), int(item.split(":")[1])) for item in items.split(",")]
+    order = create_order(customer_id, items_list, shipping_address)
+    click.echo(f"Order created: {order.order_id}")
+
+@cli.command("update-shipping")
+@click.option("--order-id", type=int, required=True)
+@click.option("--new-address", type=str, required=True)
+def update_shipping_cmd(order_id, new_address):
+    update_order_shipping(order_id, new_address)
+    click.echo(f"Shipping address updated for order {order_id}")
+
+@cli.command("adjust-stock")
+@click.option("--product-id", type=int, required=True)
+@click.option("--quantity-change", type=int, required=True)
+def adjust_stock_cmd(product_id, quantity_change):
+    adjust_stock(product_id, quantity_change)
+    click.echo(f"Stock adjusted for product {product_id}")
+
+if __name__ == "__main__":
+    cli()

py_mypy/data_models/BUILD.bazel

@@ -0,0 +1,17 @@
+package(default_visibility=["//py_mypy:__subpackages__",])
+
+py_library(
+    name = "data_models",
+    srcs = [
+        "data_models.py",
+    ],
+    deps = [
+        "@pip//pydantic",
+    ],
+    imports = [
+        ".",
+    ],
+    tags = [
+        "mypy",  # We've explicitly configured the aspect for opt-in
+    ],
+)

py_mypy/data_models/data_models.py

@@ -0,0 +1,22 @@
+from pydantic import BaseModel
+from typing import Optional
+
+class Customer(BaseModel):
+    customer_id: int
+    name: str
+    email: str
+
+class Product(BaseModel):
+    product_id: int
+    name: str
+    price: float
+
+class OrderItem(BaseModel):
+    product: Product
+    quantity: int
+
+class Order(BaseModel):
+    order_id: int
+    customer: Customer
+    items: list[OrderItem]
+    shipping_address: Optional[str] = None

py_mypy/data_persistence/BUILD.bazel

@@ -0,0 +1,21 @@
+package(default_visibility=[
+    "//py_mypy/order_processing:__pkg__",
+    "//py_mypy/inventory_management:__pkg__",
+])
+
+py_library(
+    name = "data_persistence",
+    srcs = [
+        "data_persistence.py",
+    ],
+    imports = [
+        ".",
+    ],
+    deps = [
+        "//py_mypy/data_models",
+    ],
+    tags = [
+        "mypy",  # We've explicitly configured the aspect for opt-in
+    ],
+
+)

py_mypy/data_persistence/data_persistence.py

@@ -0,0 +1,15 @@
+from data_models import Order, Customer
+
+def save_order(order: Order):
+    with open(f"order_{order.order_id}.json", "w") as f:
+        f.write(order.json()) # Pydantic's .json() method
+
+def get_customer(customer_id: int) -> Customer | None:
+    #Simulating customer retrieval. In real code this would be from a database.
+    if customer_id == 1:
+        return Customer(customer_id=1, name="John Doe", email="[email protected]")
+    return None
+
+def update_stock(product_id: int, new_stock: int):
+    #Simulating database update.
+    print(f"Updating database: Product {product_id} stock to {new_stock}")