PEP 743: Add Py_COMPAT_API_VERSION to the Python C API

Author: vstinner

.github/CODEOWNERS

@@ -621,6 +621,7 @@ peps/pep-0738.rst  @encukou
 peps/pep-0740.rst  @dstufft
 peps/pep-0741.rst  @vstinner
 peps/pep-0742.rst  @JelleZijlstra
+peps/pep-0743.rst  @vstinner
 # ...
 # peps/pep-0754.rst
 # ...

peps/pep-0743.rst

@@ -0,0 +1,214 @@
+PEP: 743
+Title: Add Py_COMPAT_API_VERSION to the Python C API
+Author: Victor Stinner <[email protected]>
+Discussions-To: https://github.com/capi-workgroup/api-evolution/issues/24
+Status: Draft
+Type: Standards Track
+Created: 11-Mar-2024
+Python-Version: 3.13
+Post-History: `11-Oct-2023 <https://github.com/capi-workgroup/api-evolution/issues/24>`__,
+              `23-Jun-2023  <https://github.com/capi-workgroup/problems/issues/54>`__,
+
+
+Abstract
+========
+
+Add ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX`` macros
+to opt-in for planned incompatible C API changes in a C extension.
+Maintainers can decide when they make their C extension compatible
+and also decide which future Python version they want to be compatible
+with.
+
+
+Rationale
+=========
+
+Python releases enforce C API changes
+-------------------------------------
+
+Every Python 3.x release has a long list of C API changes, including
+incompatible changes. C extensions have to be updated to work on the
+newly released Python.
+
+Some incompatible changes are driven by new features: they cannot be
+avoided, unless we decide to not add these features. Other reasons:
+
+* Remove deprecated API (see :pep:`387`).
+* Ease the implementation of another change.
+* Change or remove error-prone API.
+
+Currently, there is no middle ground between "not change the C API" and
+"incompatible C API changes impact everybody". Either a C extension is
+updated or the new Python version cannot be used. Such all-or-nothing
+deal does not satisfy C extension maintainers nor C extensions users.
+
+
+Limited C API
+-------------
+
+The limited C API is versioned: the ``Py_LIMITED_API`` macro can be set
+to a Python version to select which API is available. On the Python
+side, it allows introducing incompatible changes at a specific
+``Py_LIMITED_API`` version. For example, if ``Py_LIMITED_API`` is set to
+Python 3.11 or newer, the ``<stdio.h>`` is no longer included by
+``Python.h``, whereas C extensions targeting Python 3.10 are not
+affected.
+
+The difference here is that upgrading Python does not change if
+``<stdio.h>`` is included or not, but updating ``Py_LIMITED_API`` does.
+Updating ``Py_LIMITED_API`` is an deliberate action made by the C
+extension maintainer. It gives more freedom to decide **when** the
+maintainer is ready to deal with the latest batch of incompatible
+changes.
+
+A similiar version can be used with the regular (non-limited) C API.
+
+
+Deprecation and compiler warnings
+---------------------------------
+
+Some C API changes are scheduled over 3 years: deprecations. For
+example, a function is deprecated in Python 3.11 and only removed in
+Python 3.13. Deprecated functions are marked with ``Py_DEPRECATED()``
+which emits a compiler warning.
+
+The problem is that ``pip`` and ``build`` tools hide compiler logs by
+default, unless a build fails.  Moreover, it's easy to miss a single
+warning in the middle of hundred lines of logs.
+
+Schedule changes
+----------------
+
+Currently, there is no way to schedule a C API change: announce it but
+also provide a way to maintainers to test their C extensions with the
+change. Either a change is not made, or everybody must update their code
+if they want to update Python.
+
+
+Specification
+=============
+
+New macros
+----------
+
+Add new ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX``
+macros. They can be set to test if a C extension is prepared for future
+C API changes: compatible with future Python versions.
+
+The ``Py_COMPAT_API_VERSION`` macro can be set to a specific Python
+version. For example, ``Py_COMPAT_API_VERSION=0x030e0000`` tests C API
+changes scheduled in Python 3.14.
+
+If the ``Py_COMPAT_API_VERSION`` macro is set to
+``Py_COMPAT_API_VERSION_MAX``, all scheduled C API changes are tested at
+once.
+
+If the ``Py_COMPAT_API_VERSION`` macro is not set, it is to
+``PY_VERSION_HEX`` by default.
+
+The ``Py_COMPAT_API_VERSION`` macro can be set in a single C file or for
+a whole project in compiler flags. The macro does not affected other
+projects or Python itself.
+
+
+Example in Python
+-----------------
+
+For example, the ``PyImport_ImportModuleNoBlock()`` function is
+deprecated in Python 3.13 and scheduled for removal in Python 3.15. The
+function can be declared in the Python C API with the following
+declaration::
+
+    #if Py_COMPAT_API_VERSION < 0x030f0000
+    Py_DEPRECATED(3.13) PyAPI_FUNC(PyObject *) PyImport_ImportModuleNoBlock(
+        const char *name            /* UTF-8 encoded string */
+        );
+    #endif
+
+If ``if Py_COMPAT_API_VERSION`` is equal to or greate than Python 3.15
+(``0x030f0000``), the ``PyImport_ImportModuleNoBlock()`` function is not
+declared, and so using it fails with a build error.
+
+Goals
+-----
+
+* Reduce the number of C API changes affecting C extensions when
+  updating Python.
+* When testing C extensions (ex: optional CI test),
+  ``Py_COMPAT_API_VERSION`` can be set to ``Py_COMPAT_API_VERSION_MAX``
+  to detect future incompatibilities. For mandatory tests, it is
+  recommended to set ``Py_COMPAT_API_VERSION`` to a specific Python
+  version.
+* For core developers, make sure that the C API can still evolve
+  without being afraid of breaking an unknown number of C extensions.
+
+Non-goals
+---------
+
+* Freeze the API forever: this is not the stable ABI. For example,
+  deprecated functions will continue to be removed on a regular basis.
+* C extensions maintainers not using ``Py_COMPAT_API_VERSION`` will
+  still be affected by C API changes when updating Python.
+* Provide a stable ABI: the macro only impacts the regular (non-limited)
+  API.
+* Silver bullet solving all C API issues.
+
+
+Examples of ``Py_COMPAT_API_VERSION`` usages
+============================================
+
+* Remove deprecated functions.
+* Remove deprecated structure members, such as
+  ``PyBytesObject.ob_shash``.
+* Remove a standard ``#include``, such as ``#include <string.h>``,
+  from ``<Python.h>``.
+* Change the behavior of a function or a macro. For example, calling
+  ``PyObject_SetAttr(obj, name, NULL)`` can fail, to enforce the usage
+  of the ``PyObject_DelAttr()`` function instead to delete an attribute.
+
+
+Implementation
+==============
+
+* `Issue gh-116587 <https://github.com/python/cpython/issues/116587>`_
+* PR: `Add Py_COMPAT_API_VERSION and Py_COMPAT_API_VERSION_MAX macros
+  <https://github.com/python/cpython/pull/116588>`_
+
+
+Backwards Compatibility
+=======================
+
+There is no impact on backward compatibility.
+
+Adding ``Py_COMPAT_API_VERSION`` and ``Py_COMPAT_API_VERSION_MAX``
+macros has no effect on backward compatibility. Only developers setting
+the ``Py_COMPAT_API_VERSION`` macro in their project will be impacted by
+effects of this macro which is the expected behavior.
+
+
+Discussions
+===========
+
+* C API Evolutions: `Macro to hide deprecated functions
+  <https://github.com/capi-workgroup/api-evolution/issues/24>`_
+  (October 2023)
+* C API Problems: `Opt-in macro for a new clean API? Subset of functions
+  with no known issues
+  <https://github.com/capi-workgroup/problems/issues/54>`_
+  (June 2023)
+
+
+Prior Art
+=========
+
+* ``Py_LIMITED_API`` macro of `PEP 384 – Defining a Stable ABI
+  <https://peps.python.org/pep-0384/>`_.
+* Rejected `PEP 606 – Python Compatibility Version
+  <https://peps.python.org/pep-0606/>`_ which has a global scope.
+
+
+Copyright
+=========
+
+This document is placed in the public domain or under the
+CC0-1.0-Universal license, whichever is more permissive.