Merge remote-tracking branch 'upstream/master' into fix-segfault

Author: XuehaiPan

docs/changelog.md

@@ -13,6 +13,161 @@ Changes will be added here periodically from the "Suggested changelog
 entry" block in pull request descriptions.
 
 
+## Version 3.0.2 (release date TBD)
+
+Bug fixes:
+
+- MSVC 19.16 and earlier were blocked from using `std::launder` due to internal compiler errors.
+  [#5968](https://github.com/pybind/pybind11/pull/5968)
+
+- Internals destructors were updated to check the owning interpreter before clearing Python objects.
+  [#5965](https://github.com/pybind/pybind11/pull/5965)
+
+- pybind11 internals were updated to be deallocated during (sub-)interpreter shutdown to avoid memory leaks.
+  [#5958](https://github.com/pybind/pybind11/pull/5958)
+
+- Fixed ambiguous `str(handle)` construction for `object`-derived types like `kwargs` or `dict` by templatizing the constructor with SFINAE.
+  [#5949](https://github.com/pybind/pybind11/pull/5949)
+
+- Fixed concurrency consistency for `internals_pp_manager` under multiple-interpreters.
+  [#5947](https://github.com/pybind/pybind11/pull/5947)
+
+- Fixed MSVC LNK2001 in C++20 builds when /GL (whole program optimization) is enabled.
+  [#5939](https://github.com/pybind/pybind11/pull/5939)
+
+- Added per-interpreter storage for `gil_safe_call_once_and_store` to make it safe under multi-interpreters.
+  [#5933](https://github.com/pybind/pybind11/pull/5933)
+
+- A workaround for a GCC `-Warray-bounds` false positive in `argument_vector` was added.
+  [#5908](https://github.com/pybind/pybind11/pull/5908)
+
+- Corrected a mistake where support for `__index__` was added, but the type hints did not reflect acceptance of `SupportsIndex` objects. Also fixed a long-standing bug: the complex-caster did not accept `__index__` in `convert` mode.
+  [#5891](https://github.com/pybind/pybind11/pull/5891)
+
+- Fixed `*args/**kwargs` return types. Added type hinting to `py::make_tuple`.
+  [#5881](https://github.com/pybind/pybind11/pull/5881)
+
+- Fixed compiler error in `type_caster_generic` when casting a `T` implicitly convertible from `T*`.
+  [#5873](https://github.com/pybind/pybind11/pull/5873)
+
+- Updated `py::native_enum` bindings to unregister enum types on destruction, preventing a use-after-free when returning a destroyed enum instance.
+  [#5871](https://github.com/pybind/pybind11/pull/5871)
+
+- Fixed undefined behavior that occurred when importing pybind11 modules from non-main threads created by C API modules or embedded python interpreters.
+  [#5870](https://github.com/pybind/pybind11/pull/5870)
+
+- Fixed dangling pointer in `internals::registered_types_cpp_fast`.
+  [#5867](https://github.com/pybind/pybind11/pull/5867)
+
+- Added support for `std::shared_ptr<T>` when loading module-local or conduit types from other modules.
+  [#5862](https://github.com/pybind/pybind11/pull/5862)
+
+- Fixed thread-safety issues if types were concurrently registered while `get_local_type_info()` was called in free threaded Python.
+  [#5856](https://github.com/pybind/pybind11/pull/5856)
+
+- Fixed py::float_ casting and py::int_ and py::float_ type hints.
+  [#5839](https://github.com/pybind/pybind11/pull/5839)
+
+- Fixed two `smart_holder` bugs in `shared_ptr` and `unique_ptr` adoption with multiple/virtual inheritance:
+    - `shared_ptr` to-Python caster was updated to register the correct subobject pointer (fixes #5786).
+    - `unique_ptr` adoption was updated to own the proper object start while aliasing subobject pointers for registration, which fixed MSVC crashes during destruction.
+  [#5836](https://github.com/pybind/pybind11/pull/5836)
+
+- Constrained `accessor::operator=` templates to avoid obscuring special members.
+  [#5832](https://github.com/pybind/pybind11/pull/5832)
+
+- Fixed crash that can occur when finalizers acquire and release the GIL.
+  [#5828](https://github.com/pybind/pybind11/pull/5828)
+
+- Fixed compiler detection in `pybind11/detail/pybind11_namespace_macros.h` for clang-cl on Windows, to address warning suppression macros.
+  [#5816](https://github.com/pybind/pybind11/pull/5816)
+
+- Fixed compatibility with CMake policy CMP0190 by not always requiring a Python interpreter when cross-compiling.
+  [#5829](https://github.com/pybind/pybind11/pull/5829)
+
+- Added a static assertion to disallow `keep_alive` and `call_guard` on properties.
+  [#5533](https://github.com/pybind/pybind11/pull/5533)
+
+Internal:
+
+- CMake policy limit was set to 4.1.
+  [#5944](https://github.com/pybind/pybind11/pull/5944)
+
+- Improved performance of function calls between Python and C++ by switching to the "vectorcall" calling protocol.
+  [#5948](https://github.com/pybind/pybind11/pull/5948)
+
+- Many C-style casts were replaced with C++-style casts.
+  [#5930](https://github.com/pybind/pybind11/pull/5930)
+
+- Added `cast_sources` abstraction to `type_caster_generic`.
+  [#5866](https://github.com/pybind/pybind11/pull/5866)
+
+- Improved the performance of from-Python conversions of legacy pybind11 enum objects bound by `py::enum_`.
+  [#5860](https://github.com/pybind/pybind11/pull/5860)
+
+- Reduced size overhead by deduplicating functions' readable signatures and type information.
+  [#5857](https://github.com/pybind/pybind11/pull/5857)
+
+- Used new Python 3.14 C APIs when available.
+  [#5854](https://github.com/pybind/pybind11/pull/5854)
+
+- Improved performance of function dispatch and type casting by porting two-level type info lookup strategy from nanobind.
+  [#5842](https://github.com/pybind/pybind11/pull/5842)
+
+- Updated `.gitignore` to exclude `__pycache__/` directories.
+  [#5838](https://github.com/pybind/pybind11/pull/5838)
+
+- Changed internals to use `thread_local` instead of `thread_specific_storage` for increased performance.
+  [#5834](https://github.com/pybind/pybind11/pull/5834)
+
+- Reduced function call overhead by using thread_local for loader_life_support when possible.
+  [#5830](https://github.com/pybind/pybind11/pull/5830)
+
+- Removed heap allocation for the C++ argument array when dispatching functions with 6 or fewer arguments.
+  [#5824](https://github.com/pybind/pybind11/pull/5824)
+
+
+Documentation:
+
+- Fixed docstring for `long double` complex types to use `numpy.clongdouble` instead of the deprecated `numpy.longcomplex` (removed in NumPy 2.0).
+  [#5952](https://github.com/pybind/pybind11/pull/5952)
+
+- The "Supported compilers" and "Supported platforms" sections in the main `README.rst` were replaced with a new "Supported platforms & compilers" section that points to the CI test matrix as the living source of truth.
+  [#5910](https://github.com/pybind/pybind11/pull/5910)
+
+- Fixed documentation formatting.
+  [#5903](https://github.com/pybind/pybind11/pull/5903)
+
+- Updated upgrade notes for `py::native_enum`.
+  [#5885](https://github.com/pybind/pybind11/pull/5885)
+
+- Clarified in the docs to what extent bindings are global.
+  [#5859](https://github.com/pybind/pybind11/pull/5859)
+
+
+Tests:
+
+- Calls to `env.deprecated_call()` were replaced with direct calls to `pytest.deprecated_call()`.
+  [#5893](https://github.com/pybind/pybind11/pull/5893)
+
+- Updated pytest configuration to use `log_level` instead of `log_cli_level`.
+  [#5890](https://github.com/pybind/pybind11/pull/5890)
+
+
+CI:
+
+- Added CI tests for windows-11-arm with clang/MSVC (currently python 3.13), windows-11-arm with clang/mingw (currently python 3.12).
+  [#5932](https://github.com/pybind/pybind11/pull/5932)
+
+- These clang-tidy rules were added: `readability-redundant-casting`, `readability-redundant-inline-specifier`, `readability-redundant-member-init`
+  [#5924](https://github.com/pybind/pybind11/pull/5924)
+
+- Replaced deprecated macos-13 runners with macos-15-intel in CI.
+  [#5916](https://github.com/pybind/pybind11/pull/5916)
+
+- Restored `runs-on: windows-latest` in CI.
+  [#5835](https://github.com/pybind/pybind11/pull/5835)
+
 ## Version 3.0.1 (August 22, 2025)
 
 Bug fixes:

include/pybind11/detail/common.h

@@ -87,7 +87,7 @@
 #    endif
 #endif
 
-#if defined(__cpp_lib_launder) && !(defined(_MSC_VER) && (_MSC_VER < 1914))
+#if defined(__cpp_lib_launder) && !(defined(_MSC_VER) && (_MSC_VER < 1920)) // See PR #5968
 #    define PYBIND11_STD_LAUNDER std::launder
 #    define PYBIND11_HAS_STD_LAUNDER 1
 #else

include/pybind11/detail/internals.h

@@ -143,6 +143,38 @@ inline PyTypeObject *make_default_metaclass();
 inline PyObject *make_object_base_type(PyTypeObject *metaclass);
 inline void translate_exception(std::exception_ptr p);
 
+inline PyThreadState *get_thread_state_unchecked() {
+#if defined(PYPY_VERSION) || defined(GRAALVM_PYTHON)
+    return PyThreadState_GET();
+#elif PY_VERSION_HEX < 0x030D0000
+    return _PyThreadState_UncheckedGet();
+#else
+    return PyThreadState_GetUnchecked();
+#endif
+}
+
+inline PyInterpreterState *get_interpreter_state_unchecked() {
+    auto *tstate = get_thread_state_unchecked();
+    return tstate ? tstate->interp : nullptr;
+}
+
+inline object get_python_state_dict() {
+    object state_dict;
+#if defined(PYPY_VERSION) || defined(GRAALVM_PYTHON)
+    state_dict = reinterpret_borrow<object>(PyEval_GetBuiltins());
+#else
+    auto *istate = get_interpreter_state_unchecked();
+    if (istate) {
+        state_dict = reinterpret_borrow<object>(PyInterpreterState_GetDict(istate));
+    }
+#endif
+    if (!state_dict) {
+        raise_from(PyExc_SystemError, "pybind11::detail::get_python_state_dict() FAILED");
+        throw error_already_set();
+    }
+    return state_dict;
+}
+
 // Python loads modules by default with dlopen with the RTLD_LOCAL flag; under libc++ and possibly
 // other STLs, this means `typeid(A)` from one module won't equal `typeid(A)` from another module
 // even when `A` is the same, non-hidden-visibility type (e.g. from a common include).  Under
@@ -195,14 +227,6 @@ struct override_hash {
 
 using instance_map = std::unordered_multimap<const void *, instance *>;
 
-inline bool is_interpreter_alive() {
-#if PY_VERSION_HEX < 0x030D0000
-    return Py_IsInitialized() != 0 || _Py_IsFinalizing() != 0;
-#else
-    return Py_IsInitialized() != 0 || Py_IsFinalizing() != 0;
-#endif
-}
-
 #ifdef Py_GIL_DISABLED
 // Wrapper around PyMutex to provide BasicLockable semantics
 class pymutex {
@@ -293,11 +317,8 @@ struct internals {
 
     internals()
         : static_property_type(make_static_property_type()),
-          default_metaclass(make_default_metaclass()) {
+          default_metaclass(make_default_metaclass()), istate(get_interpreter_state_unchecked()) {
         tstate.set(nullptr); // See PR #5870
-        PyThreadState *cur_tstate = PyThreadState_Get();
-
-        istate = cur_tstate->interp;
         registered_exception_translators.push_front(&translate_exception);
 #ifdef Py_GIL_DISABLED
         // Scale proportional to the number of cores. 2x is a heuristic to reduce contention.
@@ -320,8 +341,10 @@ struct internals {
         // Normally this destructor runs during interpreter finalization and it may DECREF things.
         // In odd finalization scenarios it might end up running after the interpreter has
         // completely shut down, In that case, we should not decref these objects because pymalloc
-        // is gone.
-        if (is_interpreter_alive()) {
+        // is gone.  This also applies across sub-interpreters, we should only DECREF when the
+        // original owning interpreter is active.
+        auto *cur_istate = get_interpreter_state_unchecked();
+        if (cur_istate && cur_istate == istate) {
             Py_CLEAR(instance_base);
             Py_CLEAR(default_metaclass);
             Py_CLEAR(static_property_type);
@@ -336,20 +359,25 @@ struct internals {
 // impact any other modules, because the only things accessing the local internals is the
 // module that contains them.
 struct local_internals {
+    local_internals() : istate(get_interpreter_state_unchecked()) {}
+
     // It should be safe to use fast_type_map here because this entire
     // data structure is scoped to our single module, and thus a single
     // DSO and single instance of type_info for any particular type.
     fast_type_map<type_info *> registered_types_cpp;
 
     std::forward_list<ExceptionTranslator> registered_exception_translators;
     PyTypeObject *function_record_py_type = nullptr;
+    PyInterpreterState *istate = nullptr;
 
     ~local_internals() {
         // Normally this destructor runs during interpreter finalization and it may DECREF things.
         // In odd finalization scenarios it might end up running after the interpreter has
         // completely shut down, In that case, we should not decref these objects because pymalloc
-        // is gone.
-        if (is_interpreter_alive()) {
+        // is gone.  This also applies across sub-interpreters, we should only DECREF when the
+        // original owning interpreter is active.
+        auto *cur_istate = get_interpreter_state_unchecked();
+        if (cur_istate && cur_istate == istate) {
             Py_CLEAR(function_record_py_type);
         }
     }
@@ -436,16 +464,6 @@ struct native_enum_record {
     "__pybind11_module_local_v" PYBIND11_TOSTRING(PYBIND11_INTERNALS_VERSION)                     \
         PYBIND11_COMPILER_TYPE_LEADING_UNDERSCORE PYBIND11_PLATFORM_ABI_ID "__"
 
-inline PyThreadState *get_thread_state_unchecked() {
-#if defined(PYPY_VERSION) || defined(GRAALVM_PYTHON)
-    return PyThreadState_GET();
-#elif PY_VERSION_HEX < 0x030D0000
-    return _PyThreadState_UncheckedGet();
-#else
-    return PyThreadState_GetUnchecked();
-#endif
-}
-
 /// We use this to figure out if there are or have been multiple subinterpreters active at any
 /// point. This must never go from true to false while any interpreter may be running in any
 /// thread!
@@ -558,27 +576,6 @@ inline void translate_local_exception(std::exception_ptr p) {
 }
 #endif
 
-inline object get_python_state_dict() {
-    object state_dict;
-#if defined(PYPY_VERSION) || defined(GRAALVM_PYTHON)
-    state_dict = reinterpret_borrow<object>(PyEval_GetBuiltins());
-#else
-#    if PY_VERSION_HEX < 0x03090000
-    PyInterpreterState *istate = _PyInterpreterState_Get();
-#    else
-    PyInterpreterState *istate = PyInterpreterState_Get();
-#    endif
-    if (istate) {
-        state_dict = reinterpret_borrow<object>(PyInterpreterState_GetDict(istate));
-    }
-#endif
-    if (!state_dict) {
-        raise_from(PyExc_SystemError, "pybind11::detail::get_python_state_dict() FAILED");
-        throw error_already_set();
-    }
-    return state_dict;
-}
-
 // Get or create per-storage capsule in the current interpreter's state dict.
 //   - The storage is interpreter-dependent: different interpreters will have different storage.
 //     This is important when using multiple-interpreters, to avoid sharing unshareable objects

include/pybind11/subinterpreter.h

@@ -20,15 +20,6 @@
 #endif
 
 PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE)
-PYBIND11_NAMESPACE_BEGIN(detail)
-inline PyInterpreterState *get_interpreter_state_unchecked() {
-    auto *cur_tstate = get_thread_state_unchecked();
-    if (cur_tstate) {
-        return cur_tstate->interp;
-    }
-    return nullptr;
-}
-PYBIND11_NAMESPACE_END(detail)
 
 class subinterpreter;
 

tests/CMakeLists.txt

@@ -545,7 +545,9 @@ source_group(
   FILES ${PYBIND11_HEADERS})
 
 # Make sure pytest is found or produce a warning
-pybind11_find_import(pytest VERSION 3.1)
+if(NOT CMAKE_CROSSCOMPILING)
+  pybind11_find_import(pytest VERSION 3.1)
+endif()
 
 if(NOT CMAKE_CURRENT_SOURCE_DIR STREQUAL CMAKE_CURRENT_BINARY_DIR)
   # This is not used later in the build, so it's okay to regenerate each time.

tools/pybind11Common.cmake

@@ -41,7 +41,18 @@ set(pybind11_INCLUDE_DIRS
     "${pybind11_INCLUDE_DIR}"
     CACHE INTERNAL "Include directory for pybind11 (Python not requested)")
 
-if(CMAKE_CROSSCOMPILING AND PYBIND11_USE_CROSSCOMPILING)
+# CMP0190 prohibits calling FindPython with both Interpreter and Development components
+# when cross-compiling, unless the CMAKE_CROSSCOMPILING_EMULATOR variable is defined.
+if(CMAKE_VERSION VERSION_GREATER_EQUAL "4.1")
+  cmake_policy(GET CMP0190 _pybind11_cmp0190)
+  if(_pybind11_cmp0190 STREQUAL "NEW")
+    set(PYBIND11_USE_CROSSCOMPILING "ON")
+  endif()
+endif()
+
+if(CMAKE_CROSSCOMPILING
+   AND PYBIND11_USE_CROSSCOMPILING
+   AND NOT DEFINED CMAKE_CROSSCOMPILING_EMULATOR)
   set(_PYBIND11_CROSSCOMPILING
       ON
       CACHE INTERNAL "")