Improve integration with QiskitIBMRuntime

[ElePT]

Summary

This PR proposes a series of changes to allow a better integration with the QiskitRuntimeService.

• From the function developer perspective, we expose a new helper function called get_runtime_service(). This helper function instantiates a runtime service pulling by default the authentication function from the environment variables. This simplifies the instantiation process from something like:

import os
from qiskit_serverless import get_arguments, save_result
from qiskit_ibm_runtime import QiskitRuntimeService

service = QiskitRuntimeService(
   channel=os.environ["QISKIT_IBM_CHANNEL"],
    instance=os.environ["QISKIT_IBM_INSTANCE"],
   token=os.environ["QISKIT_IBM_TOKEN"],
)

backend = service.backend("ibm_fez")
...

to doing:

from qiskit_serverless import get_arguments, save_result, get_runtime_service

service = get_runtime_service()
backend = service.backend("ibm_fez")

get_runtime_service() doesn't only simplify the instantiation service, it also allows the API to store the runtime job ids and session ids created within a serverless job.

• After running a function that applies get_runtime_service() (this will NOT work if the function developers use QiskitRuntimeService independently), users can retrieve the runtime job and session ids following the code below:

job = function.run(...)

runtime_ids = job.runtime_jobs()
runtime_sessions =  job.runtime_sessions()

# a specific session id can be passed to the runtime_jobs() method to filter by session:

session_id = job.runtime_sessions()[0]
session_job_ids =  job.runtime_jobs(session_id)

The outputs of these methods are list of id strings that can be directly plugged into a QiskitRuntimeService with access to these jobs for all the usual operations. They can also be used to search in the job dashboards from IQP.

Details and comments

This PR also updates the job.stop() method to attempt to authenticate a QiskitRuntimeService using the credentials provided to the ServerlessClient and stop the runtime jobs associated to the serverless job id. This mechanism will only work if the credentials used to authenticate to a QiskitRuntimeService in the function code MATCH those used to authenticate to the ServerlessClient, which is the general use-case when accessing a deployed function and the function code runs get_runtime_service() (or an independently instantiated QiskitRuntimeService using environment variables).

There are some cases where this mechanism won't work automatically. To cover these cases, job.stop() accepts a service input parameter where you can customize the runtime service used to stop the jobs:

• For local testing, where the ServerlessClient is instantiated with a dummy token/instance, these dummy credentials can't be used to instantiate a QiskitRuntimeService. In these cases, you can still use the job.stop() functionality providing an independently instantiated service. For example:

serverless_client = ServerlessClient(
    host="http://localhost:8000",
    token="awesome_token",
    instance="awesome_instance",
)

# use client to get function here
...

job = function.run()

# to stop runtime jobs, instantiate runtime service independently
service = QiskitRuntimeService(
    channel="ibm_quantum_platform",
    token="MY_TOKEN",
    instance="MY_CRN",
)

job.stop(service)

• If the jobs have been submitted to a different URL than the standard QiskitRuntimeService (for example, to access devices in staging), even if the token an instance match, the automatic instantiation in job.stop() won't be able to get the right service instance, as the runtime URL is not a parameter of the ServerlessClient. In these cases, you should also provide an independently instantiated service:

serverless_client = ServerlessClient(
    host="http://localhost:8000",
    token="MY_TOKEN",
    instance="MY_CRN",
)

# use client to get function here
...

job = function.run()

# to stop runtime jobs, instantiate runtime service independently
service = QiskitRuntimeService(
    channel="ibm_quantum_platform",
    token="MY_TOKEN",
    instance="MY_CRN",
    url="my.staging.url.com",
)

job.stop(service)

Code has changed substantially since review.

[ElePT]

I went multiple times back and forth with these endpoints. In the end decided to keep both the add and the list one for various reasons. I ran into limitations in result serialization, where the automatic pipeline would not return a dict for "runtime_jobs", instead it would return a list. I tried to figure this out but I think I need to improve my django knowledge. In any case:

• Making formatting changes is easier if we have a dedicated endpoint
• Keeping the add/list also adds transparency of what the endpoint is for in my opinion
• It creates separation of concerns. I don't see the need to have everything together in retrieve

[ElePT]

I tried adding a release note, and realized that I am only used to communicating interface changes. I think we should also have API release notes but I am not sure how these should be structured.

[Tansito]

Yes, let's discuss the better way to handle this!

[Tansito]

I couldn't have time to look at this in-depth sorry, Elena. Just a little comment from the warnings that I saw in the tests.

Also, I saw that you didn't apply the refactor for the end-points to the new architecture. Do you prefer to do that in another PR? (it's fine for me, this one has changes enough so no problem).

[Tansito]

If I remember correctly the interface job.stop was deprecated in favor of job.cancel to follow a similar UI as Runtime 👀

[ElePT]

Hmmm I did miss that, I am surprised I never saw the warnings running my tests...

[ElePT]

Aha, so only job.stop was deprecated, not service.stop(job_id). Should we add a new service.cancel() for consistency? Keeping track of these naming changes can be challenging.

[ElePT]

Anyways, addressed in afe300b.

[Tansito]

Yes, let's discuss the better way to handle this!

[Tansito]

I think the error from the integration test comes from this change. Why are we removing this hello-world? (a naive question, I don't remember in detail this configuration for the test, maybe @korgan00 has more context).

[ElePT]

yeah, that makes sense, I originally had 2 of these conftests but then decided to move the new tests to the docker folder and use the "smallest" conftest. Up until this point I hadn't understood what this part of the code was for. I will undo this change.