Add type annotations #1761
Add type annotations #1761
Conversation
| for variant in self.iter_variants(): | ||
| if variant.index == index: | ||
| return variant | ||
| return None |
There was a problem hiding this comment.
mypy prefer explicit return None statements
There was a problem hiding this comment.
This is a safe change so I left it.
| raise ResolvedContextError( | ||
| "Cannot perform operation in a failed context") | ||
| return _check | ||
|
|
There was a problem hiding this comment.
mypy does not like these decorators defined at the class-level.
There was a problem hiding this comment.
This is a relatively safe change so I left it
|
We have a few options:
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1761 +/- ##
==========================================
- Coverage 60.11% 60.04% -0.08%
==========================================
Files 163 164 +1
Lines 20098 20514 +416
Branches 3494 3534 +40
==========================================
+ Hits 12082 12317 +235
- Misses 7205 7348 +143
- Partials 811 849 +38 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
I got bored and added lots more, particularly focused on the solver module. Once the solver module is complete, we can experiment with compiling it to a c-extension using mypyc, which could provide a big speed boost! |
|
I now have |
| self.dirty = True | ||
| return super().append(*args, **kwargs) | ||
| if not TYPE_CHECKING: | ||
| def append(self, *args, **kwargs): |
There was a problem hiding this comment.
Since this class inherits from list it's easier to rely on the type hints coming from that base class than to redefine them here, so we hide them by placing them behind not TYPE_CHECKING. In reality, the runtime value of TYPE_CHECKING is always False.
|
|
||
| def get_plugin_class(self, plugin_type, plugin_name, expected_type=None): | ||
| """Return the class registered under the given plugin name.""" | ||
| plugin = self._get_plugin_type(plugin_type) |
There was a problem hiding this comment.
Added a new argument here to validate the returned result. This provides both runtime and static validation.
There was a problem hiding this comment.
Could plugin_type be an enum or something like that? This would remove the need for expected_type right? Or maybe we could use overloads with Literals for expected_type?
There was a problem hiding this comment.
this uses type vars to describe a generic function, i.e. a function where the types of its arguments are related to each other. In our case, the type of the argument expected_type is related to the output type.
Using a literal would mean we would have to define a literal value and a function overload for every possible output type, like this:
@overload
def get_plugin_class(self, plugin_type: str, plugin_name: str, expected_type: Literal["Foo"]) -> type[Foo]:
pass
@overload
def get_plugin_class(self, plugin_type: str, plugin_name: str, expected_type: Literal["Bar"]) -> type[Bar]:
passIn a plugin environment where users can define their own types, we obviously cannot define a string constant for every possible type.
There was a problem hiding this comment.
Users can't define their own types. They can only create new plugin of some pre-defined types. Anybody adding new plugin types should do so in rez itself and not outside.
There was a problem hiding this comment.
I implemented the Literal suggestion.
| Args: | ||
| package_requests (list[typing.Union[str, PackageRequest]]): request | ||
| package_requests (list[typing.Union[str, Requirement]]): request |
There was a problem hiding this comment.
I noticed that everywhere that we've documented types as PackageRequest, they appear to actually be Requirement. I'm not sure if there any real-world exceptions to this.
There was a problem hiding this comment.
Here it's really supposed to be a PackageRequest if I'm not mistaken. But there is technically no differences between the two once the instantiated since PackageRequest inherits from Requirement and only overloads __init__ to check the inputs.
There was a problem hiding this comment.
The kind of haphazard use and documentation of PackageRequest and Requirement results in some very difficult situations to accurately add type annotations. If you want to see for yourself, check out the code, change this to PackageRequest and observe the new errors produced by mypy.
|
|
||
| _pr("resolved packages:", heading) | ||
| rows = [] | ||
| rows3: list[tuple[str, str, str]] = [] |
There was a problem hiding this comment.
you can't redefine types with mypy, so you need to use new variable names.
| if TYPE_CHECKING: | ||
| cached_property = property | ||
| else: | ||
| class cached_property(object): |
There was a problem hiding this comment.
it's much easier to pretend that cached_property is property than to type hint all the subtleties of a descriptor.
There was a problem hiding this comment.
But we loose the uncache method.
There was a problem hiding this comment.
typing.TYPE_CHECKING always resolve to False at runtime and True only during static analysis. So the code within the if TYPE_CHECKING block will never run. It's a way to simplify certain type analysis situations that arise.
There was a problem hiding this comment.
typing.TYPE_CHECKING always resolve to False
I know, but we are loosing stuff during typing. That's my whole point (the same apply to all my comments that are similar to this one).
There was a problem hiding this comment.
in what way would the static analysis be degraded by using property instead of cached_property? To my knowledge, they are functionally equivalent from a static analysis POV: the types returned are the same.
There was a problem hiding this comment.
In another thread you mentioned losing the uncache method. Unfortunately, it's not possible to type annotate this in a way both the return type and preserve the uncached method. It's something I've looked into pretty extensively.
IMO, the best path forward is to migrate to functools.cached_property. Note that to clear the cache with functools.cached_property you simply delete the attribute.
| """Reset the solver, removing any current solve.""" | ||
| if not self.request_list.conflict: | ||
| phase = _ResolvePhase(self.request_list.requirements, solver=self) | ||
| phase = _ResolvePhase(solver=self) |
There was a problem hiding this comment.
This appears to be a bug: _ResolvePhase only takes one argument. mypy to the rescue.
|
I found I probably won't have time to dig into this much more, but once this PR is merged I'll make a new PR with the changes necessary for people to test the compiled version of rez. |
|
Note: this PR likely invalidates #1745 |
|
@instinct-vfx Can you or someone from the Rez group have a look at this, please? |
| return self.build_system.working_dir | ||
|
|
||
| def build(self, install_path=None, clean=False, install=False, variants=None): | ||
| def build(self, install_path: str | None = None, clean: bool = False, |
There was a problem hiding this comment.
Using str | None means we need to drop support for python 3.7. I'm not sure we are ready for this yet.
There was a problem hiding this comment.
No, use of str | None is safe in python 3.7 as long as you use from __future__ import annotations. This backports behavior from python 3.9 that ensures that type annotations are recorded as strings within __annotations__ attributes, which means they are not evaluated at runtime unless inspect.get_annoations is called. The effect of from __future__ import annotations is that you can put basically anything you want into an annotation, it doesn't need to be valid at runtime.
The only thing breaking python 3.7 compatibility here is the use of TypedDict and Protocol, as mentioned in another comment. I presented 3 options for preserving the use of these classes in the other comment.
I noticed that the only python 3.7 tests that are currently run are for MacOS, which I took as an indicator that python 3.7 would be dropped soon. Is there a schedule for deprecation?
There was a problem hiding this comment.
By the way, I fixed the python 3.7 compatibility issue with TypedDict and Protocol, so that should not be a blocker anymore.
There was a problem hiding this comment.
It's been a year since this thread was started: has rez dropped python 3.7 support yet?
This PR will work for python 3.7, but I can remove some workarounds if we've dropped support.
There was a problem hiding this comment.
Not yet, but we could do this in the next release.
There was a problem hiding this comment.
I added a check to mypy to ensure it raises errors if we use any unsupported typing features.
There are a few cases of unsupported features:
dict[X],list[X]: usefrom __future__ import annotationsX | None: usefrom __future__ import annotationstyping.Selfor other type not intyping: useif TYPE_CHECKING`
Should I document this somewhere?
For now, I recommend that we keep using from __future__ import annotations so that we can use the modern form.
Here's a doc I compiled of typing changes by python version:
https://docs.google.com/document/d/1uOJUvgjPDwP-PRYkmlypOAPtJ9s4nWYm6WRfrAAsn0g/edit?usp=sharing
|
@chadrik You need to sign the CLA before we can even start to look at the PR. |
I work for Scanline, which is owned by Netflix, and I'm meeting with our CLA manager on Monday. I made this contribution on my own time: can choose individual vs corporate CLA on a per PR basis? |
I don't think you "can" but your account can be associated to an an ICLA and a CCLA. But I'm not a lawyer so I can't help more than that. If you and or your employer/CLA manager have questions, you can contact the LF support by following the link in the EasyCLA comment: #1761 (comment). |
chadrik
force-pushed
the
typing
branch
4 times, most recently
from
e73c6c1 to
961420b
Compare
|
CLA is signed! |
|
|
||
|
|
||
| class PackageOrderList(list): | ||
| class PackageOrderList(List[PackageOrder]): |
There was a problem hiding this comment.
Note that we use typing.List here instead of list because list does not become indexable at runtime until python 3.9. It's still safe to use list[X] inside annotations as long as we use from __future__ import annotations.
|
Any thoughts on this PR? |
|
Hey @chadrik, I made a first good read last week and I'll try to do another one soon. If you have the time, I would really love to see a GitHub Actions workflow that would run mypy on all pull requests. |
Me too! The challenge is that there are still a lot of errors. These are not the result of incorrect annotations, but rather due to code patterns which are difficult to annotate correctly without more invasion refactors. For example, there are quite a few objects which dynamically generate attributes, and that's a static typing anti-pattern. If you'd like an Action that runs mypy but allows failure for now, that's pretty easy, but if you want failures to block PRs, that'll take a lot more work. I'd prefer not to make that a blocker to merging this, though, because I've had to rebase and fix merge conflicts a few times already. I do have a plan for how we can get to zero failures in the mid-term: I wrote a tool which allows you to specify patterns for errors to ignore, but I need to update it. |
|
I think we can introduce a workflow that will fail for newly introduced errors and warnings. I'm sure someone already thought of that somewhere and we can probably re-use what they did? Basically, I'd like to verify that your changes work as expected and that we don't regress in the future and that new code is typed hint. Mypy can also be configured on a per module basis right? |
|
I'm motivated again to get this merged. I've started addressing some of the outstanding notes here, but I would love it if we could focus on blocker issues. |
chadrik
force-pushed
the
typing
branch
4 times, most recently
from
84c1e43 to
95e64a9
Compare
|
In the next week or two I'll do a pass to rebase this and remove anything that looks even remotely like a runtime change. |
Thanks @chadrik. This will definitely help us get this merged with more confidence (not that we don't trust you, but this PR is quite big). |
chadrik
force-pushed
the
typing
branch
2 times, most recently
from
4129ab0 to
1ca4a2f
Compare
|
@JeanChristopheMorinPerso I've removed everything that could be considered remotely risky. |
|
Any chance that this can get merged? I’ve removed everything but the addition of annotations. |
|
Hey @chadrik when you get a chance, could you rebase your PR please? Now that we dropped support for Python 3.7, your PR is next in line to be released. |
|
You should also look into the CLA. EasyCLA doesn't seem to be happy. |
3 participants
This is a first pass at adding type annotations throughout the code-base. Mypy is not fully passing yet, but it's getting close.
Fixes #1631