Performance regression found in "QJIT compilation with Qrack and Catalyst" PennyLane demo

[andrijapau]

Context 🧠

The "QJIT compilation with Qrack and Catalyst" demo is currently marked as non-executable with the latest CPL release and all graphs present in the demo are currently being hard-coded.

Since we should have compatibility with the latest version of pyqrack (2.1.2), I have a PR that updates the configuration to re-enable execution of the demo in our CI.

Concerns ⚠️

Some of the graphs have changed and we are seeing performance regressions from qrack in the graphs at the end of the demo. Demo renders can be found here ⤵️ ,

Old: https://pennylane.ai/qml/demos/qrack
New: https://pennylane.ai/qml/demos/qrack?pr=1614 (note there's a bug right now on our cloud platform, you need to manually type this URL in)

Any idea what's going wrong?

Environment 💻

pennylane-qrack==1.0.0
pyqrack==2.1.2
pennylane-catalyst==0.16.0.dev15
pennylane==0.46.0.dev12

Let me know if there's anything else you need! 🙏🏼

[WrathfulSpatula]

New: https://pennylane.ai/qml/demos/qrack?pr=1614 (note there's a bug right now on our cloud platform, you need to manually type this URL in)

Even if I type this into my URL bar, I still get directed to https://pennylane.ai/demos/qrack. Maybe server-side rendering is handling it, but I don't see any particular regression: I think it's the same as what's currently on production.

I'd love to help, but I'm not sure how to proceed. (Off the top of my head, relative performance can differ greatly depending upon many factors, like hardware acceleration and CPU instruction set.)

[andrijapau]

New: https://pennylane.ai/qml/demos/qrack?pr=1614 (note there's a bug right now on our cloud platform, you need to manually type this URL in)

Even if I type this into my URL bar, I still get directed to https://pennylane.ai/demos/qrack. Maybe server-side rendering is handling it, but I don't see any particular regression: I think it's the same as what's currently on production.

I'd love to help, but I'm not sure how to proceed. (Off the top of my head, relative performance can differ greatly depending upon many factors, like hardware acceleration and CPU instruction set.)

Huh, that's odd. Let me manually show you:

Here's the previous graph,

and here's what I'm getting with the new environment (slow down in Qrack),

I'd love to help, but I'm not sure how to proceed. (Off the top of my head, relative performance can differ greatly depending upon many factors, like hardware acceleration and CPU instruction set.)

Right, we do have a blurb about that in the demo,

These results were obtained using Ubuntu 22.04 LTS, Linux kernel version 5.19.0-35-generic, using one Intel(R) Core(TM) i9-10980HK CPU @ 2.40GHz, one NVIDIA GeForce RTX 3080 Laptop GPU, and 32 GB of SK hynix 3200 MT/s DDR4 in 2x16 GB row configuration. Results may differ if run on other system configurations.

but I'm wondering if this slow down is significant enough to warrant an investigation?

[WrathfulSpatula]

...but I'm wondering if this slow down is significant enough to warrant an investigation?

Chances are, on a CI instance, you have no hardware accelerator and a smaller CPU. On top of that, if the runner is ARM Mac-based, you no longer have SIMD instruction sets. These are just a couple of the factors that might affect relative performance.

If that's the case, this really isn't surprising: a CI instance isn't a representative benchmark for what you'd get on a Linux or Windows gaming laptop. Part of the strength of Qrack is that it'll happily run with limited or no dependencies on virtually any system. But its primary use case tends to be > 20 qb with GPU acceleration.

[andrijapau]

...but I'm wondering if this slow down is significant enough to warrant an investigation?

Chances are, on a CI instance, you have no hardware accelerator and a smaller CPU. On top of that, if the runner is ARM Mac-based, you no longer have SIMD instruction sets. These are just a couple of the factors that might affect relative performance.

If that's the case, this really isn't surprising: a CI instance isn't a representative benchmark for what you'd get on a Linux or Windows gaming laptop. Part of the strength of Qrack is that it'll happily run with limited or no dependencies on virtually any system. But its primary use case tends to be > 20 qb with GPU acceleration.

Great, thanks for the information. I'll check in with my team to see how we want to proceed with the demo. 🙏🏼