Test support for free-threading

[SnoopJ]

Requested Feature

Follow-on work from #2608

It would be good to know if Sopel can run under the new (as of 3.13.0) CPython support for free-threading. This issue exists to represent that work.

At time of filing no specific feature work is anticipated for this, but it is likely that there are blockers in upstream dependencies that would require some adjustments to version to support free-threading, so I expect this to evolve as we learn more about how Sopel behaves in this mode.

Problems Solved

Although Sopel has excellent support for running each Trigger event on a separate Thread, the GIL means that only one of these threads will be doing something useful at any given moment. Free-threading could provide a nice boost in bot performance when handling multiple simultaneous events, e.g. across multiple active channels.

It is also helpful to know what potential issues are along this path of development, even if free-threading is not a priority feature. Think of it as 'feature reconnaissance'.

Alternatives

Ignore free-threading.

Notes

I am filing this issue because I've just become aware of the existence of the pytest-freethreaded plugin that aims to make this kind of testing for free-threading support more straightforward.

[SnoopJ]

The score on the starting line:

• The unadulterated test suite passes on 3.13.3t
• With pytest-freethreaded run in --threads 1 --iterations 1 mode, there are 183 test failures. See detailed log for a full listing.
  • The GIL is forced back on in this test mode, by the import of SQLAlchemy. I do not see any upstream issue about free-threading support in SQLAlchemy.

I would divide these 183 failures into a few categories given below. My counts don't add up to 183 and I may be double-counting the capability error class, so there's a bit more going on there, but this is hopefully a good start on categorizing the issues.

Message duplication (~111 failures)

Many of these seem to come from the Sopel core. Many of the failures exhibit as duplicate protocol messages (including but not limited to PRIVMSG) between the bot and 'server'. Perhaps this indicates some kind of data race? It seems to be consistent between runs.

Example failure

_______________________________ test_wrapper_say _______________________________                                                                                        
test/test_bot.py:138: in test_wrapper_say                                                                                                                               
    assert mockbot.backend.message_sent == rawlist(                                                                                                                     
E   AssertionError: assert [b'PRIVMSG #c...nel :Hi!\r\n'] == [b'PRIVMSG #channel :Hi!\r\n']                                                                             
E     Left contains one more item: b'PRIVMSG #channel :Hi!\r\n'                                                                                                         
E     Full diff:                                                                                                                                                        
E     - [b'PRIVMSG #channel :Hi!\r\n']                                                                                                                                  
E     + [b'PRIVMSG #channel :Hi!\r\n', b'PRIVMSG #channel :Hi!\r\n']

Database failures (~13 failures)

Many of these also look like problems with duplicated entries, and most involve SQLAlchemy directly. There are also issues with uniqueness constraints not being satisfied. Could be caused by lack of upstream support.

Example failure

_________________ test_get_nick_id_casemapping[[][]-{}{}-[}{]] _________________
test/test_db.py:136: in test_get_nick_id_casemapping
    assert 1 == session.scalar(
E   assert 1 == 2
E    +  where 2 = <bound method Session.scalar of <sqlalchemy.orm.session.Session object at 0x50a39354890>>(<sqlalchemy.sql.selectable.Select object at 0x50a390b2230>)
E    +    where <bound method Session.scalar of <sqlalchemy.orm.session.Session object at 0x50a39354890>> = <sqlalchemy.orm.session.Session object at 0x50a39354890>.scalar
E    +    and   <sqlalchemy.sql.selectable.Select object at 0x50a390b2230> = <bound method Select.select_from of <sqlalchemy.sql.selectable.Select object at 0x50a390b3ab0>>(NickIDs)
E    +      where <bound method Select.select_from of <sqlalchemy.sql.selectable.Select object at 0x50a390b3ab0>> = <sqlalchemy.sql.selectable.Select object at 0x50a390b3ab0>.select_from
E    +        where <sqlalchemy.sql.selectable.Select object at 0x50a390b3ab0> = select(<sqlalchemy.sql.functions.count at 0x50a39354b10; count>)
E    +          where <sqlalchemy.sql.functions.count at 0x50a39354b10; count> = <sqlalchemy.sql.functions._FunctionGenerator object at 0x50a39092010>()
E    +            where <sqlalchemy.sql.functions._FunctionGenerator object at 0x50a39092010> = func.count
_ test_get_nick_id_casemapping[Mr\xc9r\xefcPr\xe2lin\xe9-mr\xc9r\xefcpr\xe2lin\xe9-MR\xc9R\xefCPR\xe2LIN\xe9] _
test/test_db.py:136: in test_get_nick_id_casemapping
    assert 1 == session.scalar(
E   assert 1 == 2
E    +  where 2 = <bound method Session.scalar of <sqlalchemy.orm.session.Session object at 0x50a393553d0>>(<sqlalchemy.sql.selectable.Select object at 0x50a390b47d0>)
E    +    where <bound method Session.scalar of <sqlalchemy.orm.session.Session object at 0x50a393553d0>> = <sqlalchemy.orm.session.Session object at 0x50a393553d0>.scalar
E    +    and   <sqlalchemy.sql.selectable.Select object at 0x50a390b47d0> = <bound method Select.select_from of <sqlalchemy.sql.selectable.Select object at 0x50a390b09b0>>(NickIDs)
E    +      where <bound method Select.select_from of <sqlalchemy.sql.selectable.Select object at 0x50a390b09b0>> = <sqlalchemy.sql.selectable.Select object at 0x50a390b09b0>.select_from
E    +        where <sqlalchemy.sql.selectable.Select object at 0x50a390b09b0> = select(<sqlalchemy.sql.functions.count at 0x50a39090b10; count>)
E    +          where <sqlalchemy.sql.functions.count at 0x50a39090b10; count> = <sqlalchemy.sql.functions._FunctionGenerator object at 0x50a390b3650>()
E    +            where <sqlalchemy.sql.functions._FunctionGenerator object at 0x50a390b3650> = func.count

Filesystem errors (~14 failures)

These mostly look like failure to create files that already exist.

Example failure

____________________________ test_plugin_load_pymod ____________________________
E   FileExistsError: [Errno 17] File exists: '/tmp/pytest-of-jgerity/pytest-13/test_plugin_load_pymod0/loader_mods'
All traceback entries are hidden. Pass `--full-trace` to see hidden and internal frames.

During handling of the above exception, another exception occurred:
test/test_plugins.py:57: in test_plugin_load_pymod
    root = tmpdir.mkdir('loader_mods')
3.13t-venv/lib/python3.13t/site-packages/_pytest/_py/path.py:887: in mkdir
    error.checked_call(os.mkdir, os.fspath(p))
E   py.error.EEXIST: [File exists]: mkdir('/tmp/pytest-of-jgerity/pytest-13/test_plugin_load_pymod0/loader_mods',)

Capability errors (~16 failures)

There are a variety of failures in capability negotiation or handling.

Example failure

__________________________ test_cap_ack_config_error ___________________________
test/coretasks/test_coretasks_cap.py:243: in test_cap_ack_config_error
    assert mockbot.backend.message_sent == rawlist(
E   AssertionError: assert [b'CAP REQ :e...mple/cap\r\n'] == [b'CAP REQ :example/cap\r\n']
E     Left contains 3 more items, first extra item: b'CAP END\r\n'
E     Full diff:
E       [
E        b'CAP REQ :example/cap\r\n',
E     +  b'CAP END\r\n',
E     +  b'QUIT :Configuration error.\r\n',
E     +  b'CAP REQ :example/cap\r\n',
E       ]
------------------------------ Captured log call -------------------------------
ERROR    sopel.coretasks:coretasks.py:1064 Configuration error on ACK capability "example/cap": ConfigurationError: Improperly configured.

vcrpy errors (~18 failures)

These all appear to be caused by vcr.errors.CannotOverwriteExistingCassetteException

Example failure

_____________________________ test_example_duck_0 ______________________________
sopel/tests/pytest_plugin.py:119: in test
    tested_func(wrapper, test_trigger)
sopel/builtins/search.py:108: in duck
    result = duck_api(query)
sopel/builtins/search.py:72: in duck_api
    results = requests.get(base, parameters).json()
3.13t-venv/lib/python3.13t/site-packages/requests/api.py:73: in get
    return request("get", url, params=params, **kwargs)
3.13t-venv/lib/python3.13t/site-packages/requests/api.py:59: in request
    return session.request(method=method, url=url, **kwargs)
3.13t-venv/lib/python3.13t/site-packages/requests/sessions.py:589: in request
    resp = self.send(prep, **send_kwargs)
3.13t-venv/lib/python3.13t/site-packages/requests/sessions.py:703: in send
    r = adapter.send(request, **kwargs)
3.13t-venv/lib/python3.13t/site-packages/requests/adapters.py:667: in send
    resp = conn.urlopen(
3.13t-venv/lib/python3.13t/site-packages/urllib3/connectionpool.py:789: in urlopen
    response = self._make_request(
3.13t-venv/lib/python3.13t/site-packages/urllib3/connectionpool.py:536: in _make_request
    response = conn.getresponse()
3.13t-venv/lib/python3.13t/site-packages/vcr/stubs/__init__.py:263: in getresponse
    raise CannotOverwriteExistingCassetteException(
E   vcr.errors.CannotOverwriteExistingCassetteException: Can't overwrite existing cassette ('test/vcr/builtins/search/test_example_duck_0.yaml') in your current record mode (<RecordMode.ONCE: 'once'>).
E   No match for the request (<Request (GET) https://api.duckduckgo.com/?format=json&no_html=1&no_redirect=1&q=sopel.chat+irc+bot>) was found.
E   Found 1 similar requests with 0 different matcher(s) :
E   
E   1 - (<Request (GET) https://api.duckduckgo.com/?format=json&no_html=1&no_redirect=1&q=sopel.chat+irc+bot>).
E   Matchers succeeded : ['method', 'scheme', 'host', 'port', 'path', 'query']
E   Matchers failed :

[dgw]

• With pytest-freethreaded run in --threads 1 --iterations 1 mode, there are 183 test failures.

The newest issue on pytest-freethreaded (tonybaloney/pytest-freethreaded#20) indicates that these tests might actually run twice despite the single iteration requested. Observed behavior (message duplication, trying to create the same file multiple times) supports the idea that such a bug exists in the test plugin.

Overall I'm somewhat wary of this test plugin. It had a bunch of work done over just a few weeks in Sept-Oct last year; then in the remaining 6 months up to now, the author seems to have stepped away. (I looked at all 20 issues+PRs, since there aren't that many. No comment/event by the repo owner was dated after October 26, 2024.)

We will obviously need to add free-threaded tests in our CI eventually, but I hope there's a more actively-developed alternative (since free-threading itself is so new to CPython).

[SnoopJ]

Hmm, well-spotted and it does seem to be a tidy resolution of the apparent paradox here. I think I agree with your assessment.

I got the recommendation after joining a discord instance advertised on Mastodon, and there were also some other materials pointed to there, including a different pytest plugin, pytest-run-parallel. Worth a try, I suppose.