05-panel-local-llm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/panel-logo.png\" width=\"200px\" align=\"right\" vspace=\"20px\"/>\n",
    "\n",
    "# Local LLMs Chat App with Panel\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the [previous notebook](04-panel-intro.ipynb), `pn.chat.ChatInterface` was introduced with a callback that simply echoed the sent message.\n",
    "\n",
    "In this section, we will make it much more interesting by connecting a local LLM, specifically Llama3 from earlier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ragna & ExLlama2-quantized Llama3\n",
    "\n",
    "### Setup\n",
    "\n",
    "Let's first initialize the model and create a Ragna chat. We'll explore the Python Software Foundation's annual reports."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from local_llm import Llama38BInstruct\n",
    "from ragna import Rag, source_storages\n",
    "import panel as pn\n",
    "pn.extension()\n",
    "\n",
    "documents = [\n",
    "    \"files/psf-report-2021.pdf\",\n",
    "    \"files/psf-report-2022.pdf\",\n",
    "    \"files/psf-report-2023.pdf\",\n",
    "]\n",
    "\n",
    "chat = Rag().chat(\n",
    "    documents=documents,\n",
    "    source_storage=source_storages.Chroma,\n",
    "    assistant=Llama38BInstruct,\n",
    ")\n",
    "\n",
    "await chat.prepare();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test the chat\n",
    "\n",
    "We can first do a test run to see if it works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "message = await chat.answer(\"What does the PSF do?\", stream=True)\n",
    "\n",
    "async for chunk in message:\n",
    "    print(chunk, end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Migrate to `ChatInterface`\n",
    "\n",
    "Now, let's migrate this functionality into `pn.chat.ChatInterface` with a callback.\n",
    "\n",
    "To do this, we copy paste the prior cell's code into a function, and then:\n",
    "\n",
    "1. Prefix the `def` with `async` to make it async\n",
    "2. Replace the hard-coded string with `contents`\n",
    "3. Concatenate the chunks into a `response` string\n",
    "4. Yield the `response`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "async def reply(contents, user, instance):\n",
    "    message = await chat.answer(contents, stream=True)\n",
    "\n",
    "    response = \"\"\n",
    "    async for chunk in message:\n",
    "        response += chunk\n",
    "        yield response\n",
    "\n",
    "chat_interface = pn.chat.ChatInterface(callback=reply)\n",
    "chat_interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now try entering \"Who is the Python Developer in Residence?\" into the chat."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LlamaCpp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For posterity, we can use `llama-cpp-python` for quantized models too!\n",
    "\n",
    "`llama-cpp` can run on both CPU and GPU, and has an API that mimics OpenAI's API.\n",
    "\n",
    "Personally, I use it because I don't have any spare GPUs lying around and it runs extremely well on my local Mac M2 Pro! It also handles chat template formats internally so it's just a matter of specifying a the proper `chat_format` key.\n",
    "\n",
    "Here, we:\n",
    "1. download the quantized model (if it doesn't exist already) in GGUF format\n",
    "2. instantiate the model; first checking the cache\n",
    "3. serialize all messages into `transformers` format (new)\n",
    "4. calls the chat completion Openai-like API on the messages\n",
    "5. stream the chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "import llama_cpp\n",
    "import panel as pn\n",
    "from huggingface_hub import hf_hub_download\n",
    "pn.extension()\n",
    "\n",
    "model_path = hf_hub_download(\n",
    "    \"TheBloke/Mistral-7B-Instruct-v0.2-GGUF\",\n",
    "    \"mistral-7b-instruct-v0.2.Q5_K_M.gguf\",\n",
    "    local_dir=str(Path.home() / \"shared/scipy/rags-to-riches\")\n",
    ")  # 1.\n",
    "\n",
    "# 2.\n",
    "if model_path in pn.state.cache:\n",
    "    llama = pn.state.cache[model_path]\n",
    "else:\n",
    "    llama = llama_cpp.Llama(\n",
    "        model_path=model_path,\n",
    "        n_gpu_layers=-1,\n",
    "        chat_format=\"mistral-instruct\",\n",
    "        n_ctx=2048,\n",
    "        logits_all=True,\n",
    "        verbose=False,\n",
    "    )\n",
    "    pn.state.cache[model_path] = llama\n",
    "\n",
    "def reply(contents: str, user: str, instance: pn.chat.ChatInterface):\n",
    "    messages = instance.serialize()  # 3.\n",
    "    message = llama.create_chat_completion_openai_v1(messages=messages, stream=True)  # 4.\n",
    "\n",
    "    response = \"\"\n",
    "    for chunk in message:\n",
    "        part = chunk.choices[0].delta.content or \"\"\n",
    "        response += part\n",
    "        yield response  # 5.\n",
    "\n",
    "chat_interface = pn.chat.ChatInterface(callback=reply)\n",
    "chat_interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can even give the model a personality by setting a system message!\n",
    "\n",
    "Update the callback with the a system message.\n",
    "\n",
    "Note, Mistral Instruct does NOT support the `system` role so we use `user` instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "system_message = \"You are an excessively passionate Pythonista.\"\n",
    "\n",
    "def reply(contents: str, user: str, instance: pn.chat.ChatInterface):\n",
    "    messages = [\n",
    "        {\"role\": \"user\", \"content\": system_message}  # updated here\n",
    "    ] + instance.serialize()\n",
    "    message = llama.create_chat_completion_openai_v1(messages=messages, stream=True)\n",
    "\n",
    "    response = \"\"\n",
    "    for chunk in message:\n",
    "        part = chunk.choices[0].delta.content or \"\"\n",
    "        response += part\n",
    "        yield response\n",
    "\n",
    "chat_interface.callback = reply\n",
    "chat_interface"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your turn ✨\n",
    "\n",
    "Try aggregating all you've learned to customize the personality of the chatbot on the go!\n",
    "\n",
    "Again, replace the ellipses with the appropriate code snippets!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import llama_cpp\n",
    "import panel as pn\n",
    "from pydantic import BaseModel\n",
    "from huggingface_hub import hf_hub_download\n",
    "\n",
    "pn.extension()\n",
    "\n",
    "model_path = hf_hub_download(\n",
    "    \"TheBloke/Mistral-7B-Instruct-v0.2-GGUF\",\n",
    "    \"mistral-7b-instruct-v0.2.Q5_K_M.gguf\",\n",
    "    local_dir=str(Path.home() / \"shared/scipy/rags-to-riches\")\n",
    ")\n",
    "\n",
    "if model_path in pn.state.cache:\n",
    "    llama = pn.state.cache[model_path]\n",
    "else:\n",
    "    llama = llama_cpp.Llama(\n",
    "        model_path=model_path,\n",
    "        n_gpu_layers=-1,\n",
    "        chat_format=\"mistral-instruct\",\n",
    "        n_ctx=2048,\n",
    "        logits_all=True,\n",
    "        verbose=False,\n",
    "    )\n",
    "    pn.state.cache[model_path] = llama\n",
    "\n",
    "def reply(contents: str, user: str, instance: pn.chat.ChatInterface):\n",
    "    messages = [\n",
    "        {\"role\": \"user\", \"content\": ...}  # Fill this out\n",
    "    ] + instance.serialize()\n",
    "    message = llama.create_chat_completion_openai_v1(\n",
    "        messages=messages, stream=True\n",
    "    )\n",
    "\n",
    "    response = \"\"\n",
    "    for chunk in message:\n",
    "        part = chunk.choices[0].delta.content or \"\"\n",
    "        response += part\n",
    "        yield response\n",
    "\n",
    "\n",
    "system_input = ...  # Fill this out\n",
    "chat_interface = pn.chat.ChatInterface(callback=reply, min_height=350)\n",
    "layout = pn.Column(\n",
    "    system_input,\n",
    "    chat_interface,\n",
    ")\n",
    "layout"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's all for now. Check out [the examples](https://holoviz-topics.github.io/panel-chat-examples/) to see more on how you can integrate `pn.chat.ChatInterface` with other services!\n",
    "\n",
    "Again, there is also a [HoloViz Discourse](https://discourse.holoviz.org/) if you want to ask questions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "_❗️ **Warning:** Make sure to stop the Jupyter Kernel (in the JupyterLab Menu Bar, click on \"Kernel\" -> \"Shut down Kernel\") before proceeding to prevent the \"insufficient VRAM\" error._\n",
    "\n",
    "<br>\n",
    "\n",
    "**✨ Next: [Ragna's UI and Experiments](06-UI-and-experiments.ipynb) →**\n",
    "\n",
    "\n",
    "💬 _Wish to continue discussions after the tutorial? Contact the presenters: [@pavithraes](https://github.com/pavithraes), [@dharhas](https://github.com/dharhas), [@ahuang11](https://github.com/ahuang11)_\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "scipy-scipy-rags-to-riches",
   "language": "python",
   "name": "conda-env-scipy-scipy-rags-to-riches-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}