Future of typing_extensions.Doc

[JelleZijlstra]

PEP-727 proposed a new Doc construct, but the PEP is unlikely to be accepted and will probably be withdrawn (https://discuss.python.org/t/pep-727-documentation-metadata-in-typing/32566/181).

That makes it so we have to figure out what to do with typing_extensions.Doc in the future. We don't have any previous cases where a feature was proposed in a PEP that didn't get accepted. We do have the precedent of typing_extensions.IntVar, which was added a long time ago without ever being proposed in a PEP. It is still in typing_extensions but undocumented.

I'm hesitant to remove importable objects from typing_extensions, because this library is so widely used and is a dependency of many other important libraries. If we remove something, we could end up breaking an important third-party library. Libraries could also end up pinning the version of typing-extensions they depend on, which is disruptive because it restricts users using that library from using new features of newer versions of typing-extensions.

Doc is also quite simple (a single class of a few dozen lines; no interaction with other features). Therefore, I don't see a big problem with keeping it around. However, we should mark it as deprecated in the documentation and probably move it into its own section.

[HexDecimal]

I've imagined a worst case scenario: After the proposed hint is abandoned the name is later reused in a new PEP with incompatible syntax than the initial proposal. This situation would force typing_extensions to make a breaking change no matter what.

Have you considered PEP 702, the @deprecated decorator? It's supposed to tell type checkers to warn on any usage of that type. When later looking at the code this will make the decorated hints deprecation obvious at a glance. Because this library also defines deprecated this might mean rearranging the source, putting deprecated hints near the bottom would be easiest. At that point it's on the type checkers to warn about deprecated usages of these types.

Because a hint might be reused for something different I would suggest using FutureWarning to indicate clearly that these should be removed immediately from current code and should never be touched. The deprecation message could account for the possibly that the hint was reintroduced and the user simply has an outdated version of typing_extensions.

@deprecated does not guarantee a warning will be raised at runtime, and I'm not sure which type checkers actually warn on the usage of deprecated types, but it's the best option which doesn't involve using hacks. It's also very little code in comparison to the other options.

The other option is mainly using PEP 562 which even shows examples of using module level __getattr__ to handle deprecations. This option will usually break type-hints without additional hacks to correct it, but runtime warnings will always be displayed. I personally don't think this is worth the effort if the goal is to not break anything.

[HexDecimal]

I realize after reading the linked discussion that this issue might've been more about the philosophy of keeping Doc around than the actual implementation of deprecating it.

Obviously you can't remove typing_extensions.Doc, but is it really a good idea to endorse it's usage? If it's not going to be part of the Python standard then it feels unstable to me.

[JelleZijlstra]

"Have you considered PEP 702" was pretty amusing to read :). Yes, I have considered it. In fact, I wrote it.

We could use it here, but it feels wrong to mark something as @deprecated if we're not going to actually remove it. That's why I suggest a docs-only deprecation.

[HexDecimal]

I apologize for being a dunce, I tend to gloss over the author field of PEP's. Obviously the maintainers of typing_extensions are going to be familiar with typing related PEP's, even if they didn't literally write it themselves.

We could use it here, but it feels wrong to mark something as @deprecated if we're not going to actually remove it.

It's simple, you mark it as @deprecated and then you never actually remove it, ever, a classic soft deprecation. Marked as do-not-use without ever having to even add a docstring. Perhaps even using PendingDepecationWarning rather than anything louder.

Of course, any deprecation warning for attempting to use the hint is the opposite of an endorsement. So less of a chance of having typing_extensions.Doc gain widespread usage and then using that as an example of PEP-727's usefulness. No chance at all if it's a DepecationWarning or FutureWarning.

Instead of deprecation, keeping typing_extensions.Doc around and documenting it as an official-unofficial type-hint is technically an option. Other third-party libraries could make similar extensions to Annotated but realistically it'd have to be in typing_extensions to gain any traction. It's just that typing_extensions is more well known for having back-ports of official types so including a semi-rejected type-hint feels unusual.

I personally don't like Doc aesthetically. I'm very anti-boilerplate. I just want plain docstrings in more places which most tools already support.

Maybe mark typing_extensions.IntVar as deprecated. It took me a while to even figure out where this came from and that tkinter.IntVar was completely unrelated to it. Is IntVar still being used?

[frezX]

What is IntVar for? I haven't found information about it anywhere

[JelleZijlstra]

It's for whatever you want it to do.

I think it was added as an early attempt to provide int-valued TypeVars for something like array types, but it never went anywhere. Now it's just an Easter egg, I suppose.

[HexDecimal]

That's consistent with what I found about IntVar. It's like Literal but only for int's. I think Numpy is still trying to figure out how to make the shape of arrays part of the array type, and IntVar was an old attempt at doing that.

[jakkdl]

We could use it here, but it feels wrong to mark something as @deprecated if we're not going to actually remove it. That's why I suggest a docs-only deprecation.

Well, there's no concrete plan to remove it - but the reason we're considering marking it is because it might be necessary to remove it in the future in case typing.Doc becomes something else.

I'm hesitant to remove importable objects from typing_extensions, because this library is so widely used and is a dependency of many other important libraries. If we remove something, we could end up breaking an important third-party library. Libraries could also end up pinning the version of typing-extensions they depend on, which is disruptive because it restricts users using that library from using new features of newer versions of typing-extensions.

For a third option we could introduce a new package, typing-extension-doc, that adds Doc to typing_extensions, with an error message in typing_extensions telling users to install it (after a deprecation period etc etc). That way they don't need to pin typing-extensions or similar.

Perhaps overkill for Doc in particular, but could be an option.