notebooks/langchain-pipeline-rosa.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question Answering with LangChain and a locally running model\n",
    "\n",
    "The goal of this notebook is only to illustrate how to run a [LangChain](https://github.com/hwchase17/langchain) question answering chain on top of a [Hugging Face](https://huggingface.co) language model that is running locally.\n",
    "\n",
    "LangChain currently offers [two wrappers around Hugging Face LLMs](https://langchain.readthedocs.io/en/latest/ecosystem/huggingface.html), one for a local pipeline and one to access a hosted model in Hugging Face Hub. This notebook uses the local pipeline version, i.e. running the LLM locally, implemented by the `HuggingFacePipeline` class. \n",
    "\n",
    "References: \n",
    "- https://blog.langchain.dev/tutorial-chatgpt-over-your-data/ explains the overall workflow that this notebook follows. The example in the blog post uses OpenAI LLMs and a FAISS vector store; here we will use a locally hosted LM and ChromaDB instead.\n",
    "- https://github.com/hwchase17/langchain/blob/master/langchain/llms/huggingface_pipeline.py"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import torch\n",
    "from langchain.chains import LLMChain\n",
    "from langchain.chains.question_answering import load_qa_chain\n",
    "from langchain.document_loaders import DirectoryLoader, TextLoader\n",
    "from langchain.embeddings import HuggingFaceEmbeddings\n",
    "from langchain.llms import HuggingFacePipeline\n",
    "from langchain.prompts import PromptTemplate\n",
    "from langchain.text_splitter import MarkdownTextSplitter\n",
    "from langchain.vectorstores import Chroma\n",
    "from transformers import AutoModelForCausalLM, AutoModelForSeq2SeqLM, AutoTokenizer, pipeline\n",
    "\n",
    "device = -1  # cpu\n",
    "if torch.cuda.is_available():\n",
    "    torch.set_default_tensor_type(torch.cuda.FloatTensor)\n",
    "    device = 0  # first GPU"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List of models\n",
    "\n",
    "We define a dictionary with various language models available on Hugging Face for text generation. In the dictionary we specify, for each model:\n",
    "- its Hugging Face path. Example: `bigscience/bloom-1b7`\n",
    "- the task they support. This is specific to each model. Example: text-generation. **NOTE**: the `HuggingFacePipeline` only supports *text-generation* (decoder-only) or *text2text-generation* (encoder-decoder) models for now.\n",
    "- an optional dictionary with additional parameters for that model\n",
    "\n",
    "The purpose of defining this list of models is to facilitate the process of switching models quickly while running the notebook multiple times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Model selection\n",
    "\n",
    "# A dictionary of models and their corresponding tasks\n",
    "models = {\n",
    "    \"bloom-1b7\": {\n",
    "        \"task\": \"text-generation\",\n",
    "        \"model\": \"bigscience/bloom-1b7\",\n",
    "        \"extra_args\": {\"max_new_tokens\": 100},\n",
    "    },\n",
    "    \"bloomz-1b7\": {\n",
    "        \"task\": \"text-generation\",\n",
    "        \"model\": \"bigscience/bloomz-1b7\",\n",
    "        \"extra_args\": {\"max_new_tokens\": 500, \"temperature\": 0.9},\n",
    "    },\n",
    "    \"bloom-3b\": {\n",
    "        \"task\": \"text-generation\",\n",
    "        \"model\": \"bigscience/bloom-3b\",\n",
    "        \"extra_args\": {\"max_new_tokens\": 100},\n",
    "    },\n",
    "    \"bloomz-3b\": {\n",
    "        \"task\": \"text-generation\",\n",
    "        \"model\": \"bigscience/bloomz-3b\",\n",
    "        \"extra_args\": {\"max_new_tokens\": 500, \"temperature\": 0.9},\n",
    "    },\n",
    "    \"gpt2\": {\n",
    "        \"task\": \"text-generation\",\n",
    "        \"model\": \"gpt2\",\n",
    "        \"extra_args\": {\"max_new_tokens\": 30},\n",
    "    },\n",
    "    \"gpt2-large\": {\n",
    "        \"task\": \"text-generation\",\n",
    "        \"model\": \"gpt2-large\",\n",
    "        \"extra_args\": {\"max_new_tokens\": 50},\n",
    "    },\n",
    "    \"mt0-large\": {\n",
    "        \"task\": \"text2text-generation\",\n",
    "        \"model\": \"bigscience/mt0-large\",\n",
    "        \"extra_args\": {},\n",
    "    },\n",
    "    \"opt-1b3\": {\n",
    "        \"task\": \"text-generation\",\n",
    "        \"model\": \"facebook/opt-1.3b\",\n",
    "        \"extra_args\": {\"max_new_tokens\": 100},\n",
    "    },\n",
    "}\n",
    "\n",
    "# A mapping between the task and the corresponding transformers class\n",
    "auto_classes = {\n",
    "    \"text-generation\": AutoModelForCausalLM,\n",
    "    \"text2text-generation\": AutoModelForSeq2SeqLM,\n",
    "}"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model selection\n",
    "\n",
    "When running the notebook, pick just one of the models from the options listed above to use in the rest of the notebook.\n",
    "\n",
    "A comparison between different models is beyond the scope of this notebook; the goal here is to show the workflow of running a question answering pipeline locally - allowing to pick among various models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "model_id = \"bloomz-3b\""
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pipeline creation\n",
    "\n",
    "Here we manually create a Hugging Face pipeline. LangChain's `HuggingFacePipeline` can create one automatically by calling `from_model_id`, but creating it manually and passing it to the constructor offers a bit more of control."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using a text-generation pipeline with bigscience/bloomz-3b\n"
     ]
    }
   ],
   "source": [
    "task = models[model_id][\"task\"]\n",
    "model_name = models[model_id][\"model\"]\n",
    "auto_class = auto_classes[task]\n",
    "print(f\"Using a {task} pipeline with {model_name}\")\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "model = auto_class.from_pretrained(model_name, low_cpu_mem_usage=True, torch_dtype=torch.float16)\n",
    "pipe = pipeline(\n",
    "    task, model=model, tokenizer=tokenizer, device=device, **models[model_id][\"extra_args\"]\n",
    ")\n",
    "llm = HuggingFacePipeline(pipeline=pipe)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing the pipeline\n",
    "\n",
    "Let's verify that the local language model / pipeline works with LangChain by creating a simple prompt template and completing a simple prompt:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
      "Prompt after formatting:\n",
      "\u001b[32;1m\u001b[1;3mThe day after monday is\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      " the day of the week\n"
     ]
    }
   ],
   "source": [
    "prompt = PromptTemplate(\n",
    "    input_variables=[\"day\"],\n",
    "    template=\"The day after {day} is\",\n",
    ")\n",
    "chain = LLMChain(llm=llm, prompt=prompt, verbose=True)\n",
    "\n",
    "print(chain.run(\"monday\"))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Document indexing\n",
    "\n",
    "Here we create and populate an embeddings database (AKA vectorstore) with embeddings from the data we have: a set of official documentation and two websites with resources about Red Hat OpenShift Service on AWS (ROSA). The data is in a set of Markdown files.\n",
    "\n",
    "Each language model has a maximum context lenght, and we must trim the documents to ensure that when they get passed as context to the language model, they fit into that limit.\n",
    "\n",
    "The `MarkdownTextSplitter` handles that document partition while maintaining the structure in the original Markdown files, aiming to keep relevant/related information together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "74 documents were loaded in 1484 chunks\n"
     ]
    }
   ],
   "source": [
    "# Note: using TextLoader here instead of UnstructuredMarkdownLoader. MarkdownTextSplitter will do the job of parsing markdown\n",
    "loader = DirectoryLoader('../data/external', glob=\"**/*.md\", loader_cls=TextLoader)\n",
    "documents = loader.load()  # FYI with the current dataset, documents[42] is the FAQ\n",
    "\n",
    "text_splitter = MarkdownTextSplitter(chunk_overlap=0)  # Consider adding chunk_size=1000\n",
    "texts = text_splitter.split_documents(documents)\n",
    "print(f\"{len(documents)} documents were loaded in {len(texts)} chunks\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a quick summary of the result of the split, let's take a look at how big is the largest chunk:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The longest text chunk is index 525, with lenght 3998\n"
     ]
    }
   ],
   "source": [
    "# Find the index of the longest text in texts\n",
    "max_len = max(len(text.page_content) for text in texts)\n",
    "max_len_idx = [i for i, text in enumerate(texts) if len(text.page_content) == max_len][0]\n",
    "print(f\"The longest text chunk is index {max_len_idx}, with lenght {max_len}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE**: depending on how big your \"chunks\" are you might face a situation where, when they are added as part of the prompt to provide context to the language model, they do not fit into the model's maximum input lenght.\n",
    "\n",
    "It is possible to split into smaller chunks by setting the `chunk_size` parameter, but this means that we _will_ start to miss on relevant related information.\n",
    "\n",
    "This compromise is one of the problems/challenges that this notebook attempts to highlight.\n",
    "\n",
    "Now, let's check what is the content of that largest chunk to understand why did we end up with such a long piece:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Delete objects\n",
      "\n",
      "This section describes the `delete` commands for clusters and resources.\n",
      "\n",
      "### delete admin\n",
      "\n",
      "Deletes a cluster administrator from a specified cluster.\n",
      "\n",
      "\n",
      "\n",
      "**Syntax**\n",
      "\n",
      "\n",
      "\n",
      "``` terminal\n",
      "$ rosa delete admin --cluster=<cluster_name> | <cluster_id>\n",
      "```\n",
      "\n",
      "| Option    | Definition                                                                              |\n",
      "|-----------|-----------------------------------------------------------------------------------------|\n",
      "| --cluster | Required: The name or ID (string) of the cluster to add to the identity provider (IDP). |\n",
      "\n",
      "Arguments\n",
      "\n",
      "| Option        | Definition                                                    |\n",
      "|---------------|---------------------------------------------------------------|\n",
      "| --help        | Shows help for this command.                                  |\n",
      "| --debug       | Enables debug mode.                                           |\n",
      "| --interactive | Enables interactive mode.                                     |\n",
      "| --profile     | Specifies an AWS profile (string) from your credentials file. |\n",
      "| --v \\<level>  | The log level for V logs.                                     |\n",
      "\n",
      "Optional arguments inherited from parent commands\n",
      "\n",
      "\n",
      "\n",
      "**Example**\n",
      "\n",
      "\n",
      "\n",
      "Delete a cluster administrator from a cluster named `mycluster`.\n",
      "\n",
      "``` terminal\n",
      "$ rosa delete admin --cluster=mycluster\n",
      "```\n",
      "\n",
      "### delete cluster\n",
      "\n",
      "Deletes a cluster.\n",
      "\n",
      "\n",
      "\n",
      "**Syntax**\n",
      "\n",
      "\n",
      "\n",
      "``` terminal\n",
      "$ rosa delete cluster --cluster=<cluster_name> | <cluster_id> [arguments]\n",
      "```\n",
      "\n",
      "| Option    | Definition                                                  |\n",
      "|-----------|-------------------------------------------------------------|\n",
      "| --cluster | Required: The name or ID (string) of the cluster to delete. |\n",
      "| --watch   | Watches the cluster uninstallation logs.                    |\n",
      "\n",
      "Arguments\n",
      "\n",
      "| Option        | Definition                                                    |\n",
      "|---------------|---------------------------------------------------------------|\n",
      "| --help        | Shows help for this command.                                  |\n",
      "| --debug       | Enables debug mode.                                           |\n",
      "| --interactive | Enables interactive mode.                                     |\n",
      "| --profile     | Specifies an AWS profile (string) from your credentials file. |\n",
      "| --v \\<level>  | The log level for V logs.                                     |\n",
      "| --yes         | Automatically answers `yes` to confirm the operation.         |\n",
      "\n",
      "Optional arguments inherited from parent commands\n",
      "\n",
      "\n",
      "\n",
      "**Examples**\n",
      "\n",
      "\n",
      "\n",
      "Delete a cluster named `mycluster`.\n",
      "\n",
      "``` terminal\n",
      "$ rosa delete cluster --cluster=mycluster\n",
      "```\n",
      "\n",
      "### delete idp\n",
      "\n",
      "Deletes a specific identity provider (IDP) from a cluster.\n",
      "\n",
      "\n",
      "\n",
      "**Syntax**\n",
      "\n",
      "\n",
      "\n",
      "``` terminal\n",
      "$ rosa delete idp --cluster=<cluster_name> | <cluster_id> [arguments]\n",
      "```\n",
      "\n",
      "| Option    | Definition                                                                           |\n",
      "|-----------|--------------------------------------------------------------------------------------|\n",
      "| --cluster | Required: The name or ID (string) of the cluster from which the IDP will be deleted. |\n",
      "\n",
      "Arguments\n",
      "\n",
      "| Option        | Definition                                                    |\n",
      "|---------------|---------------------------------------------------------------|\n",
      "| --help        | Shows help for this command.                                  |\n",
      "| --debug       | Enables debug mode.                                           |\n",
      "| --interactive | Enables interactive mode.                                     |\n",
      "| --profile     | Specifies an AWS profile (string) from your credentials file. |\n",
      "| --v \\<level>  | The log level for V logs.                                     |\n",
      "| --yes         | Automatically answers `yes` to confirm the operation.         |\n",
      "\n",
      "Optional arguments inherited from parent commands\n",
      "\n",
      "\n",
      "\n",
      "**Example**\n",
      "\n",
      "\n",
      "\n",
      "Delete an identity provider named `github` from a cluster named `mycluster`.\n",
      "\n",
      "``` terminal\n",
      "$ rosa delete idp github --cluster=mycluster\n",
      "```\n"
     ]
    }
   ],
   "source": [
    "# show the biggest chunk\n",
    "print(texts[max_len_idx].page_content)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we create the embeddings for the text documents (chunks) that we collected:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using embedded DuckDB with persistence: data will be stored in: ../data/interim\n"
     ]
    }
   ],
   "source": [
    "embeddings = HuggingFaceEmbeddings()\n",
    "\n",
    "# https://langchain.readthedocs.io/en/latest/modules/indexes/vectorstore_examples/chroma.html#persist-the-database\n",
    "db_dir = \"../data/interim\"\n",
    "docsearch = None\n",
    "if os.path.isdir(os.path.join(db_dir, \"index\")):\n",
    "    # Load the existing vector store\n",
    "    docsearch = Chroma(persist_directory=db_dir, embedding_function=embeddings)\n",
    "else:\n",
    "    # Create a new vector store\n",
    "    docsearch = Chroma.from_documents(texts, embeddings, persist_directory=db_dir)\n",
    "    docsearch.persist()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question Answering\n",
    "\n",
    "Now it is time to go through the actual question answering.\n",
    "\n",
    "Here we will be running a question answering chain from LangChain in *verbose* mode, so that we also get a peek into how LangChain builds a prompt for the model.\n",
    "\n",
    "References:\n",
    "- https://langchain.readthedocs.io/en/latest/modules/indexes/combine_docs.html for types of chains to combine documents\n",
    "- https://langchain.readthedocs.io/en/latest/modules/indexes/chain_examples/question_answering.html for the QA example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Types of chains: stuff, map_reduce, refine, map-rerank\n",
    "# https://langchain.readthedocs.io/en/latest/modules/indexes/chain_examples/question_answering.html\n",
    "# https://langchain.readthedocs.io/en/latest/modules/indexes/combine_docs.html\n",
    "chain = load_qa_chain(llm, chain_type=\"stuff\", verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "question = \"What is the roadmap for ROSA?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new StuffDocumentsChain chain...\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
      "Prompt after formatting:\n",
      "\u001b[32;1m\u001b[1;3mUse the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.\n",
      "\n",
      "Next steps\n",
      "\n",
      "-   [Create a ROSA cluster](#rosa-creating-cluster) or [Create an AWS PrivateLink cluster on ROSA](#rosa-aws-privatelink-creating-cluster).\n",
      "\n",
      "### Additional resources\n",
      "\n",
      "-   [AWS prerequisites](#prerequisites)\n",
      "\n",
      "-   [Required AWS service quotas and requesting increases](#rosa-required-aws-service-quotas)\n",
      "\n",
      "-   [Understanding the ROSA deployment workflow](#rosa-understanding-the-deployment-workflow)\n",
      "\n",
      "Next steps\n",
      "\n",
      "-   [Installing the ROSA CLI](#rosa-installing-cli)\n",
      "\n",
      "### Additional resources\n",
      "\n",
      "-   [AWS prerequisites](#prerequisites)\n",
      "\n",
      "-   [Required AWS service quotas and requesting increases](#rosa-required-aws-service-quotas)\n",
      "\n",
      "-   [Understanding the ROSA deployment workflow](#rosa-understanding-the-deployment-workflow)\n",
      "\n",
      "How does it work?\n",
      "Red Hat OpenShift Service on AWS (ROSA) has infrastructure components (virtual machines, storage disks, etc.) and a software component (OpenShift). When you provision ROSA clusters you will incur the infrastructure and OpenShift charges at the pay-as-you-go hourly rate. Refer to the Red Hat OpenShift Service on AWS pricing page for more information. You can also do 1 or 3 year commits for even deeper discounts.\n",
      "\n",
      "### Where can I see a roadmap or make feature requests for the service?\n",
      "You can visit our [ROSA roadmap](https://red.ht/rosa-roadmap) to stay up to date with the status of features currently in development. Please feel free to open a new issue if you have any suggestions for the product team.\n",
      "\n",
      "Question: What is the roadmap for ROSA?\n",
      "Helpful Answer:\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "docs = docsearch.similarity_search(question, k=3)\n",
    "answer = chain({\"input_documents\": docs, \"question\": question}, return_only_outputs=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The interesting part of the output below is to see the prompt that was created by LangChain. Notice how the prompt includes context collected using the embeddings database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Query: What is the roadmap for ROSA?\n",
      "Answer:  https://red.ht/rosa-roadmap\n"
     ]
    }
   ],
   "source": [
    "print(f'Query: {question}\\nAnswer:{answer[\"output_text\"]}')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a technically correct correct answer - even if maybe a bit too concise.\n",
    "\n",
    "As we saw above, the answer obtained was extracted from the context that was provided to the model via the generated prompt."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A longer context\n",
    "\n",
    "Now let's try another question. In order to test the limits of this approach, this time we will try to hit the big chunk we found above. This can cause the prompt to become too big for the model's context to handle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new StuffDocumentsChain chain...\u001b[0m\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
      "Prompt after formatting:\n",
      "\u001b[32;1m\u001b[1;3mUse the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.\n",
      "\n",
      "## Deleting a ROSA Cluster\n",
      "\n",
      "To delete a ROSA cluster follow the steps below.\n",
      "\n",
      "1. If you need to list your clusters in order to make sure you are deleting the correct one run:\n",
      "\n",
      "\t\trosa list clusters\n",
      "\n",
      "1. Once you know which one to delete run:\n",
      "\n",
      "\t\trosa delete cluster --cluster <cluster-name>\n",
      "\n",
      "\t!!! danger\n",
      "\t\t\t**THIS IS NON-RECOVERABLE.**\n",
      "\n",
      "1. It will prompt you to confirm that you want to delete it. Press “y”, then enter. The cluster and all its associated infrastructure will be deleted.\n",
      "\n",
      "\t!!! note\n",
      "\t\t\tAll AWS STS/IAM roles and policies will remain and must be deleted manually once the cluster deletion is complete by following the steps below.\n",
      "\n",
      "1. The command will output the next two commands to delete the other resources that were created.  You must wait until the cluster has finished deleting (from the previous command). You can use `rosa list clusters` to do a quick status check.\n",
      "\n",
      "1. Once complete, delete the:\n",
      "\n",
      "\t1. OpenID Connect (OIDC) provider that you created for Operator authentication. Run\n",
      "\n",
      "\t\t\trosa delete oidc-provider -c <clusterID> --mode auto --yes\n",
      "\n",
      "\t1. Cluster-specific Operator IAM roles.  Run\n",
      "\n",
      "\t\t\trosa delete operator-roles -c <clusterID> --mode auto --yes\n",
      "\n",
      "\t\t!!! note\n",
      "\t\t\t\tThe above require the cluster id to be used and not the cluster name.\n",
      "\n",
      "1. The remaining roles would be account-scoped and should only be removed if they are <u>no longer needed by other clusters in the same account</u>. In other words, if you would still like to create other ROSA clusters in this account, do not perform this step.\n",
      "\n",
      "\tTo delete these, you need to know the prefix used when creating them.  The default is \"ManagedOpenShift\" unless you specified otherwise.\n",
      "\n",
      "\t\trosa delete account-roles --prefix <prefix> --mode auto --yes\n",
      "\n",
      "\n",
      "*[ROSA]: Red Hat OpenShift Service on AWS\n",
      "*[IdP]: Identity Provider\n",
      "*[OCM]: OpenShift Cluster Manager\n",
      "*[STS]: AWS Security Token Service\n",
      "\n",
      "Delete objects\n",
      "\n",
      "This section describes the `delete` commands for clusters and resources.\n",
      "\n",
      "### delete admin\n",
      "\n",
      "Deletes a cluster administrator from a specified cluster.\n",
      "\n",
      "\n",
      "\n",
      "**Syntax**\n",
      "\n",
      "\n",
      "\n",
      "``` terminal\n",
      "$ rosa delete admin --cluster=<cluster_name> | <cluster_id>\n",
      "```\n",
      "\n",
      "| Option    | Definition                                                                              |\n",
      "|-----------|-----------------------------------------------------------------------------------------|\n",
      "| --cluster | Required: The name or ID (string) of the cluster to add to the identity provider (IDP). |\n",
      "\n",
      "Arguments\n",
      "\n",
      "| Option        | Definition                                                    |\n",
      "|---------------|---------------------------------------------------------------|\n",
      "| --help        | Shows help for this command.                                  |\n",
      "| --debug       | Enables debug mode.                                           |\n",
      "| --interactive | Enables interactive mode.                                     |\n",
      "| --profile     | Specifies an AWS profile (string) from your credentials file. |\n",
      "| --v \\<level>  | The log level for V logs.                                     |\n",
      "\n",
      "Optional arguments inherited from parent commands\n",
      "\n",
      "\n",
      "\n",
      "**Example**\n",
      "\n",
      "\n",
      "\n",
      "Delete a cluster administrator from a cluster named `mycluster`.\n",
      "\n",
      "``` terminal\n",
      "$ rosa delete admin --cluster=mycluster\n",
      "```\n",
      "\n",
      "### delete cluster\n",
      "\n",
      "Deletes a cluster.\n",
      "\n",
      "\n",
      "\n",
      "**Syntax**\n",
      "\n",
      "\n",
      "\n",
      "``` terminal\n",
      "$ rosa delete cluster --cluster=<cluster_name> | <cluster_id> [arguments]\n",
      "```\n",
      "\n",
      "| Option    | Definition                                                  |\n",
      "|-----------|-------------------------------------------------------------|\n",
      "| --cluster | Required: The name or ID (string) of the cluster to delete. |\n",
      "| --watch   | Watches the cluster uninstallation logs.                    |\n",
      "\n",
      "Arguments\n",
      "\n",
      "| Option        | Definition                                                    |\n",
      "|---------------|---------------------------------------------------------------|\n",
      "| --help        | Shows help for this command.                                  |\n",
      "| --debug       | Enables debug mode.                                           |\n",
      "| --interactive | Enables interactive mode.                                     |\n",
      "| --profile     | Specifies an AWS profile (string) from your credentials file. |\n",
      "| --v \\<level>  | The log level for V logs.                                     |\n",
      "| --yes         | Automatically answers `yes` to confirm the operation.         |\n",
      "\n",
      "Optional arguments inherited from parent commands\n",
      "\n",
      "\n",
      "\n",
      "**Examples**\n",
      "\n",
      "\n",
      "\n",
      "Delete a cluster named `mycluster`.\n",
      "\n",
      "``` terminal\n",
      "$ rosa delete cluster --cluster=mycluster\n",
      "```\n",
      "\n",
      "### delete idp\n",
      "\n",
      "Deletes a specific identity provider (IDP) from a cluster.\n",
      "\n",
      "\n",
      "\n",
      "**Syntax**\n",
      "\n",
      "\n",
      "\n",
      "``` terminal\n",
      "$ rosa delete idp --cluster=<cluster_name> | <cluster_id> [arguments]\n",
      "```\n",
      "\n",
      "| Option    | Definition                                                                           |\n",
      "|-----------|--------------------------------------------------------------------------------------|\n",
      "| --cluster | Required: The name or ID (string) of the cluster from which the IDP will be deleted. |\n",
      "\n",
      "Arguments\n",
      "\n",
      "| Option        | Definition                                                    |\n",
      "|---------------|---------------------------------------------------------------|\n",
      "| --help        | Shows help for this command.                                  |\n",
      "| --debug       | Enables debug mode.                                           |\n",
      "| --interactive | Enables interactive mode.                                     |\n",
      "| --profile     | Specifies an AWS profile (string) from your credentials file. |\n",
      "| --v \\<level>  | The log level for V logs.                                     |\n",
      "| --yes         | Automatically answers `yes` to confirm the operation.         |\n",
      "\n",
      "Optional arguments inherited from parent commands\n",
      "\n",
      "\n",
      "\n",
      "**Example**\n",
      "\n",
      "\n",
      "\n",
      "Delete an identity provider named `github` from a cluster named `mycluster`.\n",
      "\n",
      "``` terminal\n",
      "$ rosa delete idp github --cluster=mycluster\n",
      "```\n",
      "\n",
      "Deleting a ROSA cluster\n",
      "\n",
      "Delete a Red Hat OpenShift Service on AWS (ROSA) cluster using the `rosa` command-line.\n",
      "\n",
      "\n",
      "\n",
      "AWS Security Token Service (STS) is the recommended credential mode for installing and interacting with clusters on Red Hat OpenShift Service on AWS (ROSA) because it provides enhanced security.\n",
      "\n",
      "\n",
      "\n",
      "### Prerequisites\n",
      "\n",
      "-   If Red Hat OpenShift Service on AWS created a VPC, you must remove the following items from your cluster before you can successfully delete your cluster:\n",
      "\n",
      "    -   Network configurations, such as VPN configurations and VPC peering connections\n",
      "\n",
      "    -   Any additional services that were added to the VPC\n",
      "\n",
      "If these configurations and services remain, the cluster does not delete properly.\n",
      "\n",
      "Question: How can I delete my cluster?\n",
      "Helpful Answer:\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "Query: How can I delete my cluster?\n",
      "Answer:  rosa delete cluster\n"
     ]
    }
   ],
   "source": [
    "question = \"How can I delete my cluster?\"\n",
    "docs = docsearch.similarity_search(question, k=3)\n",
    "answer = chain({\"input_documents\": docs, \"question\": question}, return_only_outputs=True)\n",
    "print(f'Query: {question}\\nAnswer:{answer[\"output_text\"]}')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see that the generated prompt is indeed quite longer than the previous attempt (over 7000 characters that translate to more than 2500 tokens), to the point that it exceeds the model's sequence lenght (2048 for BLOOM). This can have different consequences depending on the model being used. BLOOM uses [ALiBi positional encodings](https://arxiv.org/abs/2108.12409) which is meant to enable extrapolation to longer sequences.\n",
    "\n",
    "Still, the obtained answer is very terse, and in this case also partially incomplete/incorrect as it is missing a mandatory option."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "This notebook provides an example of how to run a LangChain chain using an LLM from HuggingFace running locally. The main goal here is to provide an example of how to develop a question answering pipeline with this approach.\n",
    "\n",
    "Beyond the particular examples shown in the notebook, we also observe a few challenges:\n",
    "\n",
    "- Different models exhibit different behaviours and limitations (the notebook only shows one specific example, and a detailed comparison is beyond the goal of the notebook)\n",
    "- In the particular example shown above, we see how the answers could be more user friendly and complete.\n",
    "- For an approach that uses prompt engineering that includes information indexed with a vector database, building that database accurately deserves its own attention: quantity and quality of the data that is then made available to the model as context; how to keep the prompt size under control while still keeping all the necessary context.\n",
    "\n",
    "Exploring each of these challenges in more detail are topics for other notebooks."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "dfef540ac477f3be91e5308acd671ac2a81f2c8cc1125947821b3502f71b441e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}