[python] Support Python-defined QPUs

[lmondada]

Here I propose a solution to let packages downstream of cudaq define their own QPUs in Python. The scope is intentionally limited to leave breathing room for the ongoing runtime refactor:

• Do not expose any C++ APIs (in particular do not expose the C++ QPU class)
• Only expose a small subset of QPU capabilities, which correspond to the already refactored policies (sample and observe)

We could also consider marking this functionality as experimental, e.g. by nesting it within a new cudaq.experimental module.

How this makes everyone happy

• @anpaz The goal is to enable your work without requiring C++ dynamic plugins and tight coupling at the C++ level. Do you think this can work?
• @taalexander I am trying to steer us closer to the kind of dependency injection API you've been a proponent of with a new overload for cudaq.set_target
• @Renaud-K This keeps the current C++ QPU code hidden, meaning this should not get in the way of the refactor (removing QPU inheritance, policy-based dispatch, etc). I have ideas on what DynamicQPU could look like if we were to move to CRTP -- the whole concept is close to the any_qpu type that you had proposed, but with a more limited Python-only scope.
• @schweitzpgi No plugin infrastructure needed on the CUDAQ side, and no dynamic library loading. These Python QPU definitions can live as independent Python packages, fully downstream of CUDAQ.

Small example

Simplified from python/tests/backends/test_dynamic_qpu.py. Have a look at that file for a more comprehensive example.

import cudaq
from cudaq.qpu import CompileTarget, SampleResult, SupportsSampleQPU

class DummyQPU:
    """Default compilation, replace launch with MLIR dump. Only supports sample."""
    def get_compile_target_sample(self) -> CompileTarget:
        return CompileTarget.default_sample()

    def launch_sample(self, module, args):
        print(module.mlir_module)
        return SampleResult()

qpu = DummyQPU()

# types that implement SupportsSampleQPU are valid QPU targets
assert isinstance(qpu, SupportsSampleQPU)
cudaq.set_target(qpu)

# Now proceed as with any other target
@cudaq.kernel
def kernel(n_qubits: int):
    qbs = cudaq.qvector(n_qubits)
    h(qbs[0])

cudaq.sample(kernel, 1)

How this is achieved

1. Introduce the Python protocols cudaq.qpu.SupportsSampleQPU and cudaq.qpu.SupportsObserveQPU. These define the interfaces that Python classes must implement to be valid QPUs. They can choose to support sample, observe or both, depending on which protocols are implemented.
2. Within the Python extension, define a new QPU type in C++ called DynamicQPU. It wraps the Python object and exposes implementations for launchKernel and getCompileTarget by calling the python methods defined on the Python object. Note that the DynamicQPU class is NOT exposed publicly, so we are free to change how we represent Python QPUs as part of the runtime refactor (e.g. CRTPs)
3. Add an overload to cudaq.set_target in Python that accepts arbitrary Python objects: if none of the earlier overloads apply, the Python object is treated as a QPU instance, wrapped as DynamicQPU and registered within the DefaultQuantumPlatform

Note: this won't pass until #4859 gets merged in.

[github-actions]

CI Summary (push) — ✅ passed

Run #29253383561 · ✅ 6 · ⏩ 7 · ❌ 0 · ⛔ 0

Top-level jobs (13)

Job	Result
binaries	⏩ skipped
build_and_test	✅ success
config_devdeps	✅ success
config_source_build	⏩ skipped
config_wheeldeps	✅ success
devdeps	✅ success
docker_image	⏩ skipped
gen_code_coverage	⏩ skipped
metadata	✅ success
python_metapackages	⏩ skipped
python_wheels	⏩ skipped
source_build	⏩ skipped
wheeldeps	✅ success

⏩ Skipped jobs (7) — intentionally skipped on PR builds; run on merge_group / workflow_dispatch

Job
binaries
config_source_build
docker_image
gen_code_coverage
python_metapackages
python_wheels
source_build

All sub-jobs (42) — every matrix leg, with links

Job	Status	Link
Build and test (amd64, gcc12, openmpi) / Dev environment (Debug)	✅ success	view
Build and test (amd64, gcc12, openmpi) / Dev environment (Python)	✅ success	view
Build and test (amd64, llvm, openmpi) / Dev environment (Debug)	✅ success	view
Build and test (amd64, llvm, openmpi) / Dev environment (Python)	✅ success	view
Build and test (arm64, llvm, openmpi) / Dev environment (Debug)	✅ success	view
Build and test (arm64, llvm, openmpi) / Dev environment (Python)	✅ success	view
CI Summary	❔ in_progress	view
Configure build (devdeps)	✅ success	view
Configure build (source_build)	⏩ skipped	view
Configure build (wheeldeps)	✅ success	view
Create CUDA Quantum installer	⏩ skipped	view
Create Docker images	⏩ skipped	view
Create Python metapackages	⏩ skipped	view
Create Python wheels	⏩ skipped	view
Gen code coverage	⏩ skipped	view
Load dependencies (amd64, gcc12) / Caching	✅ success	view
Load dependencies (amd64, gcc12) / Finalize	✅ success	view
Load dependencies (amd64, gcc12) / Metadata	✅ success	view
Load dependencies (amd64, llvm) / Caching	✅ success	view
Load dependencies (amd64, llvm) / Finalize	✅ success	view
Load dependencies (amd64, llvm) / Metadata	✅ success	view
Load dependencies (arm64, gcc12) / Caching	✅ success	view
Load dependencies (arm64, gcc12) / Finalize	✅ success	view
Load dependencies (arm64, gcc12) / Metadata	✅ success	view
Load dependencies (arm64, llvm) / Caching	✅ success	view
Load dependencies (arm64, llvm) / Finalize	✅ success	view
Load dependencies (arm64, llvm) / Metadata	✅ success	view
Load source build cache	⏩ skipped	view
Load wheel dependencies (amd64, 12.6) / Caching	✅ success	view
Load wheel dependencies (amd64, 12.6) / Finalize	✅ success	view
Load wheel dependencies (amd64, 12.6) / Metadata	✅ success	view
Load wheel dependencies (amd64, 13.0) / Caching	✅ success	view
Load wheel dependencies (amd64, 13.0) / Finalize	✅ success	view
Load wheel dependencies (amd64, 13.0) / Metadata	✅ success	view
Load wheel dependencies (arm64, 12.6) / Caching	✅ success	view
Load wheel dependencies (arm64, 12.6) / Finalize	✅ success	view
Load wheel dependencies (arm64, 12.6) / Metadata	✅ success	view
Load wheel dependencies (arm64, 13.0) / Caching	✅ success	view
Load wheel dependencies (arm64, 13.0) / Finalize	✅ success	view
Load wheel dependencies (arm64, 13.0) / Metadata	✅ success	view
Prepare cache clean-up	✅ success	view
Retrieve PR info	✅ success	view

✅ Required checks (6/6) — declared in .github/required-checks.yml for push

Required check	Status	Link
Build and test (amd64, llvm, openmpi) / Dev environment (Debug)	✅ success	view
Build and test (amd64, llvm, openmpi) / Dev environment (Python)	✅ success	view
Build and test (arm64, llvm, openmpi) / Dev environment (Debug)	✅ success	view
Build and test (arm64, llvm, openmpi) / Dev environment (Python)	✅ success	view
Build and test (amd64, gcc12, openmpi) / Dev environment (Debug)	✅ success	view
Build and test (amd64, gcc12, openmpi) / Dev environment (Python)	✅ success	view

[taalexander]

I really like this, Luca. I think this is a good compromise moving us toward being a more flexible R&D integration partner.

[taalexander]

Is the intent here that we would expose Cuda-Q downstream simulators/runners for invocation (or would this require more of a "driver level" API)?

[lmondada]

You're hitting the nail on the head. With the current APIs, you can recover the MLIR module and further process that as you like (including execution). This might be enough for some things we have in mind, such as custom lowerings etc. However, you're limited in what you can do by

• the lack of APIs for existing targets (i.e. you can't forward your launch to an existing CUDAQ target)
• the lack of launch options passed as arguments (e.g. shot number, noise model, spin observable etc)
• the limited APIs to construct return types

Some are easy to add (additional arguments), others not so much (target APIs). We need a more in-depth discussion on which of the above APIs we want to expose. I'm hoping to use this PR as a starting point and see what's missing. The fact that external users will presumably quickly hit one of these limits is what is prodding me to flag this as 'experimental' for the time being.

[Renaud-K]

I am missing a lot of context here. But I am not seeing what this does. For instance, I am not seeing how the connection to ServerHelpers is done. Also, I see that you are calling cudaq.sample. This calls a C++ qpu. You are not proposing to change this right?

[sacpis]

Thanks @lmondada. Left initial review for these changes.

[lmondada]

I am not seeing how the connection to ServerHelpers is done

It does not support defining QPUs that require a ServerHelper. Only new QPU types that want to define their own compilation targets and their own kernel launch logic.

You are calling cudaq.sample. This calls a C++ qpu. You are not proposing to change this right?

Indeed. That's what the PyDynamicQPU type is for: it's a C++ QPU that calls back into Python. So cudaq.sample remains unchanged: it calls sample on the C++ QPU stored in the quantum_platform, which is set to an instance of PyDynamicQPU. That instance then forwards the calls to the Python implementation provided by the user.

It's true that the flow here is Python -> C++ -> Python (and then back to C++, for good). Unlike other parts of the codebase where we do this, I think in this case it's actually justified: we must call the QPU from within our runtime (which we obviously want to keep in C++).

[anpaz]

Can we change this? We currently have a nice mechanism for this. Maybe the Python QPU could somehow adversite the arguments it supports/expects.

[sacpis]

+1 to @anpaz comment.

[lmondada]

Of course, we can change whatever you want :) I'm not sure though what change and what 'nice mechanism' you mean. Can you expand?

[anpaz]

I think we need a Protocol for 'Run'?

[lmondada]

Of course, that is an easy thing to add.

[sacpis]

Overall LGTM. Thanks @lmondada.

[Renaud-K]

Thank you for the discussions.