External backends: support Fermioniq/QESEM-style server-side observe (not only ServerHelper + remote_rest sample)

[tomers-qedma]

Context

Qedma is integrating with CUDA-Q and was directed to the external-plugin packaging model on the external-backends work:

• Issue: Enable External Backend Integration #4619
• PR: External backends Plugin Framework #4754 (External backends Plugin Framework)
• Reference plugin: mock_rest on @anpaz's fork

We want that packaging story (pip install cudaq + vendor wheel via cudaq.backends). We cannot use the execution model that mock_rest demonstrates today (platform-qpu: remote_rest + custom ServerHelper only), because QESEM is a server-side observe API (same class of problem as in-tree Fermioniq).

cc @anpaz

What QESEM needs for cudaq.observe(U, H)

One job containing:

1. Preparation circuit U (QASM)
2. Full multi-term observable H (Pauli terms + coefficients)
3. QESEM options (precision, measurement bases, provider/backend, …)

Server returns mitigated { value, error_bar }. Job lifecycle is:

create → ESTIMATED → start → poll → SUCCEEDED

Not “create → poll sample counts → client reconstruct ⟨H⟩”.

The estimate→start handshake is handled in the vendor plugin via a custom Executor (same extension point as in-tree Braket-style backends). We are not asking CUDA-Q for first-class estimate→start hooks on stock remote_rest.

Why mock_rest / stock remote_rest is insufficient

Under remote_rest, observe typically splits H = Σ cᵢ Pᵢ into measurement circuits before ServerHelper::createJob. The helper sees post-split KernelExecutions — not the original (U, H) request. That destroys the observe contract before any external helper runs.

Coalescing the split /jobs array inside ServerHelper is not enough: wrong artifacts (measurement circuits), observable already gone, wrong product semantics (joint mitigation over full H), wrong result type (counts vs mitigated expectation).

What we do today

Fermioniq-style custom QPU in a CUDA-Q fork / in-tree backend:

• platform-qpu: qedma + custom QedmaQPU
• pauliTermSplitObservable = std::nullopt
• Attach full spin_op; custom executor for estimate→start→poll

That works, but blocks the recommended end-user story (stock pip CUDA-Q + cudaq-qedma wheel) until external plugins can express the same observe hooks (full H, no Pauli split, expectation results). Lifecycle remains a plugin Executor concern.

Proposed directions (smallest change preferred)

Proposal A (preferred): External custom QPU plugins

Allow a package to ship a QPU .so (not only ServerHelper), e.g.:

plugin_root/
  targets/qedma.yml          # platform-qpu: qedma
  lib/libcudaq-qedma-qpu.so  # CUDAQ_REGISTER_TYPE(..., QPU, qedma)
  lib/libcudaq-serverhelper-qedma.so

Requirements:

1. Stock CUDA-Q dlopens external QPU types the same way it loads external ServerHelpers
2. Target YAML platform-qpu: <name> resolves to that registered QPU
3. Documented reference (e.g. external Fermioniq-style observe), not only mock_rest

Proposal B: Observe mode on stock remote_rest

If external QPUs are out of scope, add hooks so remote_rest + ServerHelper can do server-side observe:

1. YAML / target flag — e.g. observe-mode: server-side (equiv. pauliTermSplitObservable = std::nullopt): one KernelExecution for bare U
2. Pass full spin_op into the helper — attach on execution / createObserveJob(circuit, spin_op, config)
3. Observe-shaped results — e.g. processObserveResults(...) so CUDA-Q does not re-estimate ⟨H⟩ from bitstrings

Non-trivial job lifecycles (QESEM estimate→start) stay in a vendor Executor, not as new stock remote_rest APIs.

Proposal C: Document in-tree-only for observe backends

If A/B are not near-term: state clearly that server-side-observe backends remain in-tree, and mock_rest is sample-style REST only. Workable, but a different distribution story than “pip install on stock CUDA-Q.”

Ask

1. Is Proposal A (external custom QPU) on the roadmap for external-backends / External backends Plugin Framework #4754?
2. If not, can Proposal B be scoped so mock_rest-style packages cover Fermioniq/QESEM-class observe APIs (items 1–3 above)?
3. Until then, should Qedma continue the in-tree / fork path?

We are happy to prototype against a draft API, contribute a reference plugin alongside mock_rest (e.g. mock_observe_rest), or share QESEM job payload examples on a working session.

Decision matrix

Choice	Qedma distribution	Effort on CUDA-Q
A External QPU plugins	Wheel on stock CUDA-Q	Medium — generalize Fermioniq load path
B Observe mode on stock remote_rest	Wheel; ServerHelper (+ optional Executor)	Medium — flags + observe path on existing REST QPU
C In-tree only for observe backends	Upstream PR / custom build	Low short-term; ongoing fork cost for us

Fermioniq precedent (in-tree today)

Mechanism	Location
Custom QPU	runtime/cudaq/platform/fermioniq/FermioniqQPU.*
Disable Pauli split	pauliTermSplitObservable = std::nullopt
Attach full spin_op	attachFermioniqObservable(...)
Target YAML	platform-qpu: fermioniq

Externalizing that observe pattern (A or B) is what Qedma needs to use the recommended pip packaging without changing QESEM’s API.

— Qedma (cudaq-qedma)

[tomers-qedma]

Update: dropped the former Proposal B item 4 (first-class estimate→start hooks on stock remote_rest). QESEM’s create → ESTIMATED → start → poll lifecycle is handled in the vendor plugin via a custom Executor (same extension point as in-tree Braket-style backends). We are not asking CUDA-Q to add those hooks; Proposal B is scoped to observe-mode items 1–3 only (no Pauli split, full H on the wire, expectation-shaped results).

[tomers-qedma]

Implementation of Proposal B (observe-mode items 1–3) is in anpaz#10:

anpaz#10

That PR adds observe-mode: server-side on stock remote_rest (no Pauli split, full H in user_data["observable"], expectation-shaped results). QESEM estimate→start remains a vendor Executor concern, as updated above.