Quantum Reinforcement Learning Demo

[AVON257]

General information

Name
Thomas Nathaniel Toseland (My GitHub ID is AVON257).

Affiliation (optional)
University of St Andrews.

Twitter (optional)
I don't have Twitter but anyone interested can email me at tntavon27@gmail.com.

Image (optional)
How do you make a computer solve this problem fast?
https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html [online] (accessed 21/10/2023).

---

Demo information

Title
Quantum Reinforcement Learning using complex variables.

Abstract
A short abstract describing your demo. Try to keep it to 1-3 sentences that make clear the goal and outcome of the demo.

The purpose of this is to solve the cart-pole problem of reinforcement learning, faster than has been previously possible using a simulated, quantum computing demo. This has not yet been tested on real hardware but on my desktop (with an RTX3090), this only takes about 2 minutes or less to train.

Relevant links
[Add a link to your demo (as a GitHub repository, Jupyter Notebook, Python script, etc.) as well as links to any papers/resources used
https://github.com/AVON257/Complex-Valued-Quantum-Neural-Networks/blob/main/Reinforcement%20Learning(1).ipynb.

References:
http://bayesiandeeplearning.org/2021/papers/22.pdf [online] (28/02/2022). (1).
https://github.com/LauraGentini/QRL [online] (accessed 28/02/2022). (2).
https://ionq.com/best-practices [online] (accessed 05/07/2022). (3).
https://pytorch.org/docs/stable/generated/torch.complex.html [online] (accessed 09/07/2022). (4).
https://ionq.com/docs/get-started-with-qiskit [online] (accessed 09/07/2022). (5).
https://qiskit.org/documentation/partners/ionq/guides/usage.html [online] (accessed 09/07/2022). (6).
https://qiskit.org/documentation/stubs/qiskit.circuit.library.RZZGate.html [online] (accessed 14/07/2022). (7).
https://qiskit.org/documentation/stubs/qiskit.circuit.library.RXXGate.html [online] (accessed 14/07/2022). (8).
https://qiskit.org/documentation/stubs/qiskit.circuit.library.RZZGate.html [online] (accessed 14/07/2022). (9).
https://arxiv.org/abs/1611.02779 [online] (accessed 14/07/2022). (10).
https://ionq.com/docs/getting-started-with-native-gates [online] (accessed 16/07/2022). (11).
appropriate resources from https://qiskit.org/ [online] (accessed 16/07/2022). (12).
https://www.nature.com/articles/s41467-019-13534-2.pdf [online] (accessed 20/07/2022). (13).
S.Russell.P.Norvig. Artificial Intelligence: A Modern Approach. (14).
https://github.com/openqasm/openqasm/tree/OpenQASM2.x [online] (accessed 21/07/2022). (15).
https://arxiv.org/pdf/2104.14722.pdf [online] (accessed 21/07/2022). (16).
https://arxiv.org/pdf/2109.00506.pdf [online] (accessed 21/07/2022). (17).
https://strawberryfields.readthedocs.io/en/stable/ [online] (accessed 21/07/2022). (18).
https://pennylane.ai/qml/ [online] (accessed 21/07/2022). (19).
https://arxiv.org/pdf/2108.12926.pdf [online] (accessed 22/07/2022). (20).
https://strawberryfields.readthedocs.io/en/stable/code/sf_ops.html [online] (accessed 21/07/2022). (21).
https://www.nature.com/articles/s41586-021-04301-9 [online] (accessed 28/07/2022). (22).
https://link.springer.com/content/pdf/10.1007/s13244-018-0639-9.pdf [online] (accessed 28/07/2022). (23).
https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html [online] (accessed 21/10/2023). (24)

https://pennylane.ai/qml/demos/tutorial_qnn_module_torch [online] (accessed 21/10/2023). (25).

Lectures on Several Complex Variables P.M. Gauthier (kindle edition). (25).
Combinatorial and Toric Homotopy (Kindle edition, A.Darby, J.Grbic, Z.Lu, J.Wu) (26).
BiHarmonic Submanifolds and Biharmonic Maps in Riemannian Geometry (Ye-Lin Ou, Bang-Yen Chen, Kindle Edition), (27).
https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html [online] (accessed 21/10/2023). (28).

[CatalinaAlbornoz]

Thank you for submitting this demo @AVON257 ! We will review it and come back to you in a couple of days.

[CatalinaAlbornoz]

To improve readability it's useful to keep all text to the same size unless it's actually a heading. I think you added "#" before some lines of text in this issue and in your demo, which accidentally turned these lines into a heading (very large size).

[AVON257]

Sorry about that, I'm not very experienced at this.

…

On Mon, 23 Oct 2023, 22:19 Catalina Albornoz, ***@***.***> wrote: To improve readability it's useful to keep all text to the same size unless it's actually a heading. I think you added "#" before some lines of text in this issue and in your demo, which accidentally turned these lines into a heading (very large size). — Reply to this email directly, view it on GitHub <#962 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMWQGN4ATYFC2YYLJHVNLS3YA3NNPAVCNFSM6AAAAAA6KKKJ3GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZWGAZTQMJYGA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[CatalinaAlbornoz]

No problem @AVON257 ! You should be able to fix it in your notebook easily. 😄

[AVON257]

I think what I'd like to do, is wait until I've got all the feedback on your decision, and then fix it and resubmit a revised version of it to you.

…

On Tue, 24 Oct 2023, 01:00 Catalina Albornoz, ***@***.***> wrote: No problem @AVON257 <https://github.com/AVON257> ! You should be able to fix it in your notebook easily. 😄 — Reply to this email directly, view it on GitHub <#962 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMWQGN66UPHG5FSZ3EQYV23YA4AH7AVCNFSM6AAAAAA6KKKJ3GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZWGI2DCMBRGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[CatalinaAlbornoz]

Hi @AVON257 ,

Thank you again for submitting this demo. You will need to make some changes before we can publish it. Quality is key so that people who go to your demo to understand it and learn from it as much as possible.

First, take a look at the other examples here and see how they write their README, their notebook, what files they include, etc. Each file should be concise, including only the relevant information. For example, it would be good to explain what the cart pole problem is about at the beginning of your notebook.

It's great that you want to offer permission for anyone to use, distribute, or modify your source code as long as they cite you in their work. A good way to convey this information is by adding a License to your project. If you look for example at the PennyLane GitHub you see that we use the Apache 2.0 license. There are many kinds of licenses so please make sure you understand what the license means. You can add it as a separate file in your repository.

Did you make the entire code yourself or are there parts of the code that you took from somewhere else? At the moment you have so many references that someone coming to your project won't know what parts of the project came from which one. I'm sure that they all influenced your learning but which of these actually need to be referenced? For example, did you base your project mostly on a specific paper? After you have made a selection of the references you can add them at the end of the document that's most appropriate for each reference. Look for example at how we add references in PennyLane demos.

Regarding the abstract, make sure that it looks very professional. Imagine that you're submitting a paper for a journal, how would you write a short abstract for that?

Finally, take a look at the demos submission guidelines and make sure you're following them.

Let me know if you have any questions and please let me know here when you have a new version for review!

[AVON257]

Thank you for your feedback, do you have any information on how to keep a laboratory notebook sensibly? I think that is partly why my submission was a mess. I am hoping to try again with this if you don't mind once I have made appropriate changes, and maybe even make something that is a bit more original than just copying/pasting code from Penny lane and Pytorch. Thanks, Tom T. P.S. I tried to give credit to both authors of PyTorch and Pennylane, but obviously I have a few things to learn about how to do better with this.

…

On Thu, Oct 26, 2023 at 5:59 PM Catalina Albornoz ***@***.***> wrote: Hi @AVON257 <https://github.com/AVON257> , Thank you again for submitting this demo. You will need to make some changes before we can publish it. Quality is key so that people who go to your demo to understand it and learn from it as much as possible. First, take a look at the other examples here <https://pennylane.ai/qml/demos_community/> and see how they write their README, their notebook, what files they include, etc. Each file should be concise, including only the relevant information. For example, it would be good to explain what the cart pole problem is about at the beginning of your notebook. It's great that you want to offer permission for anyone to use, distribute, or modify your source code as long as they cite you in their work. A good way to convey this information is by adding a License to your project. If you look for example at the PennyLane GitHub <https://github.com/PennyLaneAI/pennylane> you see that we use the Apache 2.0 license. There are many kinds of licenses so please make sure you understand what the license means. You can add it as a separate file in your repository. Did you make the entire code yourself or are there parts of the code that you took from somewhere else? At the moment you have so many references that someone coming to your project won't know what parts of the project came from which one. I'm sure that they all influenced your learning but which of these actually need to be referenced? For example, did you base your project mostly on a specific paper? After you have made a selection of the references you can add them at the end of the document that's most appropriate for each reference. Look for example at how we add references in PennyLane demos <https://pennylane.ai/qml/demonstrations/>. Regarding the abstract, make sure that it looks very professional. Imagine that you're submitting a paper for a journal, how would you write a short abstract for that? Finally, take a look at the demos submission guidelines <https://pennylane.ai/qml/demos_submission/> and make sure you're following them. Let me know if you have any questions and please let me know here when you have a new version for review! — Reply to this email directly, view it on GitHub <#962 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMWQGN5LNSY6V7BYB6ANUDDYBKJIHAVCNFSM6AAAAAA6KKKJ3GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBRGUYDEMBQGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[CatalinaAlbornoz]

Hi @AVON257 , no problem at all. You can make changes and let us know when you would like for us to give it another look. You can use the demos in pennylane.ai/qml/demonstrations as an example. They include a short explanation at the beginning, then they include blocks of code and explanation where it's relevant, and a couple of references at the end. They're written with the reader always in mind.

I hope this helps you improve your demo!

Let me know if you have any more questions.