Added context manager mix-in classes #905

agronholm · 2025-04-02T12:34:42Z

Changes

This adds two new mix-in classes: ContextManagerMixin and AsyncContextManagerMixin. They're meant to enable users to write context manager classes in the same way as @contextmanager and @asynccontextmanager, respectively. The aim is to enable users to more easily embed cancel scopes and task groups into other context manager classes.

Checklist

If this is a user-facing code change, like a bugfix or a new feature, please ensure that
you've fulfilled the following conditions (where applicable):

You've added tests (in tests/) added which would fail without your patch
You've updated the documentation (in docs/, in case of behavior changes or new
features)
You've added a new changelog entry (in docs/versionhistory.rst).

If this is a trivial change, like a typo fix or a code reformatting, then you can ignore
these instructions.

Updating the changelog

If there are no entries after the last release, use **UNRELEASED** as the version.
If, say, your patch fixes issue #123, the entry should look like this:

- Fix big bad boo-boo in task groups
  (`#123 <https://github.com/agronholm/anyio/issues/123>`_; PR by @yourgithubaccount)

If there's no issue linked, just link to your pull request instead by updating the
changelog after you've created the PR.

agronholm · 2025-04-02T13:06:39Z

Open issues:

~~Can't type annotate to return Self~~
~~Should T be co/contravariant?~~ (typeshed uses a covariant typevar in AbstractAsyncContextManager)
~~This may require a new documentation section~~
~~We should probably check for corner cases in the @asynccontextmanager implementation~~
~~Reentrancy (must consider the case where the CM is entered twice before exiting)~~ (disallowed re-entrancy)

smurfix · 2025-04-02T13:32:20Z

src/anyio/_core/_contextmanagers.py

+
+    This class is designed to streamline the use of context management by
+    requiring the implementation of the `__contextmanager__` method, which
+    should yield instances of the class itself. It then wraps this generator


Well, there's no requirement for a context manager to return Self and the wrapper shouldn't enforce that. It should accept / forward whatever type __contextmanager__ is declared to return.

On the other hand, what this wrapper does not do is to ensure that the context is not entered twice, and that really is a bug.

Well, there's no requirement for a context manager to return Self and the wrapper shouldn't enforce that. It should accept / forward whatever type contextmanager is declared to return.

I've already updated the docstring before you posted this. This version was never intended to be pushed to GH.

On the other hand, what this wrapper does not do is to ensure that the context is not entered twice, and that really is a bug.

Hm, I see what you mean. But rather than raise an error here, perhaps reentrancy should be properly supported instead?

On second thought, I don't think it's feasible to support reentrancy, as there's no way to track which generator to use in __aexit__().

I've thus disallowed reentrancy.

…gain before exiting first

smurfix · 2025-04-15T08:22:07Z

I have been doing something like this in my own codebase for ages by now.

One problem, though: in a subclass, how do I "cleanly" call the superclass's __asynccontextmanager__? The best way I can come up with is

async with asynccontextmanager(super().__asynccontextmanager__)():
    ...

which is … meh. Anyway, whichever way we come up with needs documentation.

agronholm · 2025-04-15T09:25:55Z

I have been doing something like this in my own codebase for ages by now.

One problem, though: in a subclass, how do I "cleanly" call the superclass's __asynccontextmanager__? The best way I can come up with is
async with asynccontextmanager(super().__asynccontextmanager__)():
    ...
which is … meh. Anyway, whichever way we come up with needs documentation.

This is a very good point and we need to have an answer before going forward with this.

agronholm · 2025-04-15T09:27:08Z

I think we need to start by thinking what the ideal solution would look like, and then see if we can implement that.

agronholm · 2025-04-15T10:24:15Z

@Zac-HD mentioned elsewhere that he wanted to enforce the use of @asynccontextmanager on this special method, for the benefit of PEP 789 linter checks. If we do that, then this gets considerably cleaner: async with super().__asynccontextmanager__() as obj:

smurfix · 2025-04-15T14:50:13Z

Right.

Ideally we'd be able to simply do async with super() but that'd require some interesting core CPython changes AFAICT.

agronholm · 2025-04-16T09:58:52Z

I pushed those changes, but I encountered a different problem: How can we override the type parameter when subclassing? The pre-commit checks fail for this reason, and I don't know how to fix this.

smurfix · 2025-04-16T10:11:47Z

Huh. pyright doesn't understand this either, which frankly surprises me somewhat.
Maybe ask there?

agronholm · 2025-04-16T10:18:17Z

Huh. pyright doesn't understand this either, which frankly surprises me somewhat. Maybe ask there?

Doesn't understand what? The parent class pins the type variable to itself, and that class does not have the child_started and child_finished attributes, so it seems pretty clear to me why the mypy checks fail here. What I don't understand is how to improve the type annotations to get around this.

agronholm · 2025-04-16T10:19:23Z

There appears to be no way to pin the type variable to Self, which is exactly what we'd need here.

tapetersen · 2025-04-16T11:13:04Z

The problem here is that DummyContextManager binds the typevar concretely in its definition which sets the type for __enter__ unless we override that as well.

If it was allowed/defined and possible we would want to parametrize it directly with Self as:

class DummyContextManager(ContextManagerMixin[Self]):
    ...
    @contextmanager
    def __contextmanager__(self) -> Generator[Self]:
        ...

What does work here is the old workaround with a typevar with a bound as:

_TSelf = TypeVar("_TSelf", bound="DummyContextManager")
class DummyContextManager(ContextManagerMixin[_TSelf]):
    def __init__(self, handle_exc: bool = False) -> None:
        self.started = False
        self.finished = False
        self.handle_exc = handle_exc

    @contextmanager
    def __contextmanager__(self: _TSelf) -> Generator[_TSelf]:
        self.started = True
        try:
            yield self
        except RuntimeError:
            if not self.handle_exc:
                raise

        self.finished = True

That's of course not ideal, needing a declared typevar with correct bound for every class more or less, but the case of further extending the contextmanagers overriding the returned type maybe isn't that common either?

agronholm · 2025-04-16T11:21:38Z

The problem here is that DummyContextManager binds the typevar concretely in its definition which sets the type for __enter__ unless we override that as well.

If it was allowed/defined and possible we would want to parametrize it directly with Self as:
class DummyContextManager(ContextManagerMixin[Self]):
    ...
    @contextmanager
    def __contextmanager__(self) -> Generator[Self]:
        ...
What does work here is the old workaround with a typevar with a bound as:
_TSelf = TypeVar("_TSelf", bound="DummyContextManager")
class DummyContextManager(ContextManagerMixin[_TSelf]):
    def __init__(self, handle_exc: bool = False) -> None:
        self.started = False
        self.finished = False
        self.handle_exc = handle_exc

    @contextmanager
    def __contextmanager__(self: _TSelf) -> Generator[_TSelf]:
        self.started = True
        try:
            yield self
        except RuntimeError:
            if not self.handle_exc:
                raise

        self.finished = True
That's of course not ideal, needing a declared typevar with correct bound for every class more or less, but the case of further extending the contextmanagers overriding the returned type maybe isn't that common either?

By "does work" do you mean you get the correct type out of reveal_type() in an inherited CM? Can you show me?

agronholm · 2025-04-16T13:51:41Z

I don't think forcing context manager authors to use a typevar just to enable proper inheritance is a good solution to the problem. Let's see what the pyright folks have to say about this.

tapetersen · 2025-04-16T13:57:57Z

As in I checked out your branch but there are some issues with pyproject.toml parsing (the license checks but also some things about the tox config) so it was a bit hard to get the pre-commit to run.

Looking closer I can't get it to work without explicitly passing through the leaf-class as a typevar.

I suspect that getting this to work without explicit annotation when inheriting is not really possible with current typing system.

(The return type of __aenter__ in the mixin is basically dependent on a possibly non-self type-parameter in a future child-class)

Have added it as a pull request to your branch anyway but it's not really any better except maybe the default (and maybe demonstrating that it works if you specify the return type explicitly for every child)

agronholm · 2025-04-16T14:13:25Z

As in I checked out your branch but there are some issues with pyproject.toml parsing (the license checks but also some things about the tox config) so it was a bit hard to get the pre-commit to run.

What license checks? Pre-commit has no issues running over here, nor on CI.

I suspect that getting this to work without explicit annotation when inheriting is not really possible with current typing system.

That was my conclusion too.

agronholm · 2025-04-16T15:28:41Z

I've worked around this by adding specialized versions of the mix-in classes for yielding self. I'll post the code once I have 100% coverage again.

tapetersen · 2025-04-16T15:42:59Z

(this is very possibly older versions of tox and/or problems with this specific environment and I can sort it out and report as a real issue when/if i figure out if is some requirement that needs to be bumped)

Trying to run tox for me right now results in

$ tox 
ROOT: No tox.ini or setup.cfg or pyproject.toml found, assuming empty tox.ini at /home/tobias/Projects/anyio
.pkg: _optional_hooks> python /home/tobias/.local/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python /home/tobias/.local/lib/python3.10/site-packages/pyproject_api/_backend.py True setuptools.build_meta
py: packaging backend failed (code=1), with ValueError: invalid pyproject.toml config: `project.license`.
configuration error: `project.license` must be valid exactly by one definition (2 matches found):

    - keys:
        'file': {type: string}
      required: ['file']
    - keys:
        'text': {type: string}
      required: ['text']

agronholm · 2025-04-16T15:44:22Z

Yeah, tox doesn't seem to understand the SPDX license specifiers.

agronholm · 2025-04-16T15:47:31Z

Which would be weird because tox, too uses one: https://github.com/tox-dev/tox/blob/main/pyproject.toml#L19

tapetersen · 2025-04-16T15:49:59Z

Yeah but the error seems to come from setuptools and tox itself uses hatchling as the build-backend which may work fine.

agronholm · 2025-04-16T15:50:16Z

It seems like this is a setuptools issue. When I manually updated setuptools in the tox env, it resolved the problem.

tapetersen · 2025-04-16T15:51:57Z

Indeed. Updating the version in build-system requires solves it for tox here as well.

[build-system]
requires = [
    "setuptools >= 78.1.0",
    "setuptools_scm >= 6.4"
]
build-backend = "setuptools.build_meta"

tapetersen · 2025-04-16T15:53:40Z

tox still can't find it's settings from the pyproject.toml file for me though

ROOT: No tox.ini or setup.cfg or pyproject.toml found, assuming empty tox.ini at /home/tobias/Projects/anyio

agronholm · 2025-04-16T16:02:42Z

Which Python and tox version are you running?

tapetersen · 2025-04-16T16:06:01Z

It was just an old version of tox (4.5) that was picked up from outside the venv due to vscode's restoration of env-vars that messed up some things.

May still not hurt to add a requires in the tox section as specified here: https://tox.wiki/en/latest/config.html#pyproject-toml-native
Wouldn't have solved this as the old version couldn't parse it anyway but for other incompatibilites it may be a good idea.

[tool.tox]
requires = ["tox>=4.19"]
env_list = ["pre-commit", "py39", "py310", "py311", "py312", "py313", "py314", "pypy3"]
skip_missing_interpreters = true

agronholm · 2025-04-16T16:08:04Z

It was just an old version of tox (4.5) that was picked up from outside the venv due to vscode's restoration of env-vars that messed up some things.

May still not hurt to add a requires in the tox section as specified here: https://tox.wiki/en/latest/config.html#pyproject-toml-native Wouldn't have solved this as the old version couldn't parse it anyway but for other incompatibilites it may be a good idea.

I originally skipped specifying that for precisely that reason.

* Enforced context manager decorators on the dunder methods * Fix contextmanager typing in tests * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add exit type for contextmanager-mixins as well --------- Co-authored-by: Alex Grönholm <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

graingert · 2025-04-24T08:26:19Z

docs/cancellation.rst

-    class MyAsyncContextManager:
-        async def __aenter__(self):
-            self._exitstack = AsyncExitStack()
-            await self._exitstack.__aenter__()


oh this was slightly wrong it should have used pop_all()

I think it's worth including an (Async)ExitStack instruction on how to do this

graingert · 2025-04-24T08:33:47Z

src/anyio/_core/_contextmanagers.py

+
+    @final
+    def __enter__(self: _SupportsCtxMgr[_T_co, bool | None]) -> _T_co:
+        # Needed for mypy to assume self still has the __cm member


I think you can alternatively put _ContextManagerMixin__cm on the _SupportsCtxMgr protocol

graingert · 2025-04-24T08:34:23Z

tests/test_contextmanagers.py

+        self.finished = True
+
+
+class DummyAsyncContextManager(AsyncContextManagerMixin):


might be worth a test for a class that implements both AsyncContextManagerMixin and ContextManagerMixin

I'm not sure why? The mix-in classes have no overlap whatsoever.

Oh, I just realized: that's the problem. One could enter the CM synchronously AND asynchronously without first exiting.

agronholm added 3 commits April 2, 2025 15:31

Added context manager mix-in classes

dc683b0

Updated the documentation

5997105

Merge branch 'master' into contextmanagermixin

664a19f

agronholm added 2 commits April 2, 2025 16:19

Fixed updated documentation section

3765c28

Made the type variable covariant

7aaec76

smurfix reviewed Apr 2, 2025

View reviewed changes

agronholm added 2 commits April 2, 2025 17:21

Addressed reentrancy by raising RuntimeError if attempting to enter a…

84685d9

…gain before exiting first

Merge branch 'master' into contextmanagermixin

a8fbf36

agronholm mentioned this pull request Apr 14, 2025

TypeError: '_GeneratorContextManager' using anyio CM after yielding from custom async CM with TaskGroup #909

Closed

2 tasks

agronholm added 2 commits April 14, 2025 12:50

Merge branch 'master' into contextmanagermixin

68884e3

Merge branch 'master' into contextmanagermixin

5254a11

Merge branch 'master' into contextmanagermixin

915b172

agronholm added 2 commits April 16, 2025 12:01

Delete self.__cm instead of setting to None

7f5c4b5

Enforced context manager decorators on the dunder methods

921c880

tapetersen and others added 3 commits April 18, 2025 14:34

Merge branch 'master' into contextmanagermixin

dcba443

Added documentation section for context manager mix-ins

562d21a

agronholm added this to the 4.10 milestone Apr 18, 2025

Added documentation cross-references to the API docs

09a5a9e

agronholm marked this pull request as ready for review April 18, 2025 12:44

agronholm requested review from Zac-HD and graingert April 18, 2025 12:44

agronholm added 4 commits April 18, 2025 15:48

Added attribution

7334290

Merge branch 'master' into contextmanagermixin

b3c38db

Merge branch 'master' into contextmanagermixin

b67e36c

Improved the documentation

492120a

graingert reviewed Apr 24, 2025

View reviewed changes

graingert approved these changes Apr 24, 2025

View reviewed changes

Merge branch 'master' into contextmanagermixin

55a941a

agronholm merged commit 6977fc3 into master Apr 24, 2025
31 of 32 checks passed

agronholm deleted the contextmanagermixin branch April 24, 2025 09:24

		self.finished = True


		class DummyAsyncContextManager(AsyncContextManagerMixin):

Added context manager mix-in classes #905

Added context manager mix-in classes #905

Conversation

agronholm commented Apr 2, 2025

Changes

Checklist

Updating the changelog

agronholm commented Apr 2, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smurfix commented Apr 15, 2025

agronholm commented Apr 15, 2025

agronholm commented Apr 15, 2025

agronholm commented Apr 15, 2025

smurfix commented Apr 15, 2025

agronholm commented Apr 16, 2025

smurfix commented Apr 16, 2025

agronholm commented Apr 16, 2025

agronholm commented Apr 16, 2025

tapetersen commented Apr 16, 2025 • edited Loading

agronholm commented Apr 16, 2025

agronholm commented Apr 16, 2025 • edited Loading

tapetersen commented Apr 16, 2025

agronholm commented Apr 16, 2025

agronholm commented Apr 16, 2025

tapetersen commented Apr 16, 2025

agronholm commented Apr 16, 2025 • edited Loading

agronholm commented Apr 16, 2025

tapetersen commented Apr 16, 2025

agronholm commented Apr 16, 2025

tapetersen commented Apr 16, 2025

tapetersen commented Apr 16, 2025

agronholm commented Apr 16, 2025

tapetersen commented Apr 16, 2025 • edited Loading

agronholm commented Apr 16, 2025

graingert Apr 24, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

agronholm commented Apr 2, 2025 •

edited

Loading

tapetersen commented Apr 16, 2025 •

edited

Loading

agronholm commented Apr 16, 2025 •

edited

Loading

agronholm commented Apr 16, 2025 •

edited

Loading

tapetersen commented Apr 16, 2025 •

edited

Loading

graingert Apr 24, 2025 •

edited

Loading