Clarify the Purpose of the PyGILState API and Identify Better Alternatives

[ericsnowcurrently]

This especially applies to the following sections:

• https://docs.python.org/3/c-api/init.html#c.PyGILState_Ensure
• https://docs.python.org/3/c-api/init.html#c.PyGILState_Release

but also:

• https://docs.python.org/3/c-api/init.html#c.PyGILState_GetThisThreadState
• https://docs.python.org/3/c-api/init.html#c.PyGILState_Check

The PyGILState_* C-API was introduced by PEP 311 as a convenience for extension maintainers for some very specific situations:

• code running in the current thread does not know if it holds the GIL
• this situation may involve reentrancy

The PyGILState_* API also (almost incidentally) supports use cases that are more appropriately satisfied by other API, like Py_BEGIN_ALLOW_THREADS()/Py_END_ALLOW_THREADS() for balanced acquire-release.

The documentation isn't so clear about the specific use cases, nor that an interested user may actually want a different API. For example, from a recent issue I saw: We were wrapping each evaluation in PyGILState_Ensure and PyGILState_Release, which I believe is a correct way to do this.

FTR, https://docs.python.org/3/c-api/init.html#non-python-created-threads does a decent job of explaining, as does the PEP, but the entries for PyGILState_Ensure() and PyGILState_Release() are less clear.

FWIW, I expect that it is easier now to deal with the motivating cases using the exiting C-API than it was in 2003. I'm not sure we would accept something like PEP 311 if it were proposed today. Then again, for a pair of operations that needs to balanced like this, having a specific pair of functions is helpful.

---

Related: a recent discussion about replacing the PyGILState_* API: https://discuss.python.org/t/a-new-api-for-ensuring-releasing-thread-states/83959.

Also see gh-131264 and gh-131265.

[sharktide]

@ericsnowcurrently I can make a pull request if you would like

[ZeroIntensity]

PTAL at #131087, which implements the "alternatives" part here.