Cannot anotate type of generic functions.

[danmur97]

The current scoping rules for type variables seems to deny type annotations over generic functions.

The issue

define two functions as follows:

def fx_1(f: Callable[[bool], int]) -> int:
    return f(True)

def fx_2[T](f: Callable[[bool], T]) -> T:
    return f(True)

If a function alias is required, these functions can be assigned to variables. However, if type annotations are used, fx_2 cannot be annotated using compliant type checkers (i.e. pyright).

T = TypeVar("T")
fx_1_alias: Callable[[Callable[[bool], int]], int] = fx_1  # OK!
fx_2_alias: Callable[[Callable[[bool], T]], T] = fx_2  # complains that T has no meaning

Justification

• All variables should be able to be type annotated, even when those variables are functions or the annotation is not required.
• def is a syntactic sugar equivalent to a variable assignation over a lambda function. As a result, it should be possible to define generic functions without def statements e.g.

fx_2_alternative: Callable[[Callable[[bool], T]], T] =  lambda f: f(True)

• Generics are the first ladder of what is called a "dependent type". In the case of functions, this corresponds to a Π type where the input itself is the concrete type that will be replaced on the type var of a generic type. Here type-vars scope is over the function type itself rather than some outer entity (class, function, method).
• By these definitions of Π type, it cannot exist an unbound type variable when function/callable types are involved, since in python, the type argument is implicit and supplied when referring a TypeVar.
  e.g.

T = TypeVar("T")
fx_2_alias: Callable[[Callable[[bool], T]], T] = fx_2
fx_2_alias_2: Callable[[Callable[[bool], T]], T] = fx_2

Even when using the same type var fx_2_alias and fx_2_alias_2 each var is an independent one.
A more clear syntax (like the introduced on def statements on python312) could be

fx_2_alias: Callable[X][[Callable[[bool], X]], X] = fx_2
fx_2_alias_2: Callable[X][[Callable[[bool], X]], X] = fx_2

This marks explicitly that the scope of the var is over the type itself. However, this issue does not aim to propose a new syntax, but rather aims to adjust the scoping rules of type vars.

Motivation

This issue was derived from a pyright issue where generic functions as class instance variables are denied by the checker because of the same scope issues described before.

[erictraut]

The Callable special form has numerous limitations (no keyword parameters, no *args or **kwargs parameters, no default arguments, etc.). For more complex cases that cannot be defined using a simple Callable, the type system provides callback protocols. You can use a callback protocol to define a generic callable.

class GenericCallable(Protocol):
    def __call__[T](self, f: Callable[[bool], T]) -> T: ...

fx_2_alias: GenericCallable = fx_2

When the typing community was working on the design for PEP 695, we had some long and involved discussions about whether we wanted to extend the type system to support type variables scoped to a Callable special form. There are definitely use cases for this, as you've demonstrated, but these use cases are relatively uncommon. We explored a number of new syntax options for making this work in a way that is consistent with the rest of PEP 695. We struggled to find good syntax options, and it didn't seem at the time that this use case was sufficiently common to justify the added complexity. We therefore concluded that callback protocols, while more verbose to define, were an acceptable solution for this use case.

If this use case is sufficiently common to justify a less-verbose alternative to callback protocols, we could revisit this decision. If you want to drive that, I think you'd need to overcome the original obstacles that stopped us from adding this in the past:

1. The scoping rules for type variables within a type that includes a Callable can be ambiguous, so the rules would need to be very clear. For example, in the type Callable[[Callable[[T], T], None], is T scoped to the outer Callable or the inner Callable? In the type Callable[[T], T] | Callable[[list[T]], list[T]], are there two different type variables scoped to the two different Callable subtypes? If not, what scope does T have?

2. We wouldn't want to add a new feature that is available only with the outdated (pre-3.12) TypeVar declaration mechanism. Any new feature involving generics should have a "modern" (PEP 695 style) syntax. What is the modern syntax for declaring a type variable that is scoped to a Callable form? Will such syntax require a change to the Python grammar? (Answer: probably yes). If so, there is a very high bar for convincing the SC to adopt such a change because grammar changes have significant compatibility impacts and a high cost for tooling updates.

3. Such a change (especially if it requires new syntax) will require additional spec'ing, conformance tests, implementations in all major type checkers, implementation in the runtime, support in syntax highlighters and code formatters, etc. In other words, it will require hundreds of hours of investment across the ecosystem, and it will take a year or two for such a change to be broadly available. We'd need to collectively determine whether the feature is sufficiently compelling to justify this work, and you'd need to be able to make a strong argument for this.

[erictraut]

One more note that may be of historical interest... There was an effort a few years back to create a more modern syntax for Callable and eliminate some of its existing limitations. The result of this effort was PEP 677. This proposal was ultimately rejected by the Python steering council (SC) because they didn't feel that it met the bar for a grammar change.

[carljm]

I think this issue should be moved from the python/typing-council repo to the python/typing repo?

[erictraut]

Yes, good point. This should be moved to the python/typing repo. Even better, the Typing forum would be a good place to get more input on this topic.