Remove libpython dependency on standalone libqiskit build

[mtreinish]

What should we add?

Since #14006 merged the output libqiskit standalone dylib that we build from the cext crate now requires libpython. This is only because the CircuitData Rust struct is built with Python objects internally so libpython is needed for the definition of those objects in the data model. They're not exposed through any public interface and the GIL is never used or required for anything that can be called from the standalone C API as it only exposes the part of the data model that is Python free so you can't ever create something that has a dependency on Python. However, because there are still various components in the data model that depend on Python we still need libpython so those have definitions. While we're working to remove these last components in the data model which should be moved to Rust (see #13264). There will be an additional step to conditionally compile the PyGate, PyInstruction, and PyOperation types as those by definition have a dependency on Python (they're Rust space containers for objects defined in Python) so that code isn't included when building in standalone mode.

There will be later work (potentially for Qiskit 3.0 as I expect there will be potential backwards compatibility implications) to move the python definition of custom operations to internally create rust custom operations (which don't exist yet) to remove the hard dependency on Python for these objects, but in the medium term conditional compilation after #13264 is complete will likely be simpler.

[cometta]

I improved the document Qiskit/documentation#3934

[DBFerPra]

Hi @mtreinish, I’m the author of issue 15164. I didn’t notice this one! It’s great to see that you’re already working on it. If you ever need an external user to test or validate a branch or new functionality, please don’t hesitate to ask me (specially if you need to validate in HPC environments). I’d also be happy to help with any specific tasks you’re working on. Thanks in advance for your time. Regards.

[jakelishman]

Jotting down some thoughts before I start splitting this up into smaller issues:

In an ideal world, we'd be able to completely isolate the Rust data model from Python. In practice, we're still some distance away from that, and we want to get libpython out of the standalone C library without having to completely solve all the remaining Python links, so we'll end up with some conditional compilation.

Now that Param is all in Rust, the big places left with things to solve are:

• control flow in Rust
• stretches in Rust
• the bit/var/stretch Registry objects linking Rust/Python objects
• the py_op cache and python-ish methods in PackedInstruction
• the Py* variants in PackedOperation

Control flow is already in the process of moving to Rust entirely, with some component of it being #14568, some being the follow-up #15123, and I think there's probably a couple of other bits too.

We don't have to solve all of these, though - several of them we'll be able to just feature gate (conditionally compile out). The trick will be finding the line where it's easier to conditionally compile now, and where it'll be faster/cleaner to remove the Python-space dependency entirely. We'll need a bit of investigation there. For the most part, I expect us to be able to conditionally compile out the entirety of all #[pymethods] blocks, but I think there are still places in DAGCircuit especially where we use a pymethod as a Rust-native one as well, so those will probably want separating, just for ease of construction.

The py_op caches are already hidden behind feature flags (just ones that are enabled by default), so they should be easy enough to get rid of. The Registry objects within CircuitData and DAGCircuit might need a bit more thought, though I think at this point it's probably just the internal caches that they keep that need to be compiled out, so that should be easy - the actual mappings between the two objects are between Rust-space objects.