13_1_LangChain document loader__M.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network\" target=\"_blank\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\"  />\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load Documents Using LangChain for Different Sources \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimated time needed: **20** minutes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Imagine you are working as a data scientist at a consulting firm, and you've been tasked with analyzing documents from multiple clients. Each client provides their data in different formats: some in PDFs, others in Word documents, CSV files, or even HTML webpages. Manually loading and parsing each document type is not only time-consuming but also prone to errors. Your goal is to streamline this process, making it efficient and error-free.\n",
    "\n",
    "To achieve this, you'll use LangChain’s powerful document loaders. These loaders allow you to read and convert various file formats into a unified document structure that can be easily processed. For example, you'll load client policy documents from text files, financial reports from PDFs, marketing strategies from Word documents, and product reviews from JSON files. By the end of this lab, you will have a robust pipeline that can handle any new file formats clients might send, saving you valuable time and effort.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/Hvf-jk8b5Fs-E_E4AJyEow/loader.png\" width=\"50%\" alt=\"indexing\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab, you will explore how to use various loaders provided by LangChain to load and process data from different file formats. These loaders simplify the task of reading and converting files into a document format that can be processed downstream. By the end of this lab, you will be able to efficiently load text, PDF, Markdown, JSON, CSV, DOCX, and other file formats into a unified format, allowing for seamless data analysis and manipulation for LLM applications.\n",
    "\n",
    "(Note: In this lab, we just introduced several commonly used file format loaders. LangChain provides more document loaders for various document formats [here](https://python.langchain.com/v0.2/docs/integrations/document_loaders/).)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## __Table of Contents__\n",
    "\n",
    "<ol>\n",
    "    <li><a href=\"#Objectives\">Objectives</a></li>\n",
    "    <li>\n",
    "        <a href=\"#Setup\">Setup</a>\n",
    "        <ol>\n",
    "            <li><a href=\"#Installing-required-libraries\">Installing required libraries</a></li>\n",
    "            <li><a href=\"#Importing-Required-Libraries\">Importing required libraries</a></li>\n",
    "        </ol>\n",
    "    </li>\n",
    "    <li><a href=\"#Load-from-TXT-files\">Load from TXT files</a></li>\n",
    "    <li><a href=\"#Load-from-PDF-files\">Load from PDF files</a></li>\n",
    "    <li><a href=\"#Load-from-Markdown-files\">Load from Markdown files</a></li>\n",
    "    <li><a href=\"#Load-from-JSON-files\">Load from JSON files</a></li>\n",
    "    <li><a href=\"#Load-from-CSV-files\">Load from CSV files</a></li>\n",
    "    <li><a href=\"#Load-from-URL/Website-files\">Load from URL/Webpage files</a></li>\n",
    "    <li><a href=\"#Load-from-WORD-files\">Load from WORD files</a></li>\n",
    "    <li><a href=\"#Load-from-Unstructured-Files\">Load from Unstructured Files</a></li>\n",
    "</ol>\n",
    "\n",
    "<a href=\"#Exercises\">Exercises</a>\n",
    "<ol>\n",
    "    <li><a href=\"#Exercise-1---Try-to-use-other-PDF-loaders\">Exercise 1 - Try to use other PDF loaders</a></li>\n",
    "    <li><a href=\"#Exercise-2---Load-from-Arxiv\">Exercise 2 - Load from Arxiv</a></li>\n",
    "</ol>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n",
    "\n",
    "After completing this lab you will be able to:\n",
    "\n",
    " - Understand how to use `TextLoader` to load text files.\n",
    " - Learn how to load PDFs using `PyPDFLoader` and `PyMuPDFLoader`.\n",
    " - Use `UnstructuredMarkdownLoader` to load Markdown files.\n",
    " - Load JSON files with `JSONLoader` using jq schemas.\n",
    " - Process CSV files with `CSVLoader` and `UnstructuredCSVLoader`.\n",
    " - Load Webpage content using `WebBaseLoader`.\n",
    " - Load Word documents using `Docx2txtLoader`.\n",
    " - Utilize `UnstructuredFileLoader` for various file types.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Installing required libraries\n",
    "\n",
    "The following required libraries are __not__ preinstalled in the Skills Network Labs environment. __You must run the following cell__ to install them:\n",
    "\n",
    "**Note:** We are pinning the version here to specify the version. It's recommended that you do this as well. Even if the library is updated in the future, the installed library could still support this lab work.\n",
    "\n",
    "This might take approximately 1 minute. \n",
    "\n",
    "As we use `%%capture` to capture the installation, you won't see the output process. But after the installation completes, you will see a number beside the cell.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "%%capture\n",
    "#After executing the cell,please RESTART the kernel and run all the cells.\n",
    "!pip install --user \"langchain-community==0.2.1\"\n",
    "!pip install --user \"pypdf==4.2.0\"\n",
    "!pip install --user \"PyMuPDF==1.24.5\"\n",
    "!pip install --user \"unstructured==0.14.8\"\n",
    "!pip install --user \"markdown==3.6\"\n",
    "!pip install --user  \"jq==1.7.0\"\n",
    "!pip install --user \"pandas==2.2.2\"\n",
    "!pip install --user \"docx2txt==0.8\"\n",
    "!pip install --user \"requests==2.32.3\"\n",
    "!pip install --user \"beautifulsoup4==4.12.3\"\n",
    "!pip install --user \"nltk==3.8.0\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After you install the libraries, restart your kernel. You can do that by clicking the **Restart the kernel** icon.\n",
    "\n",
    "<p style=\"text-align:left\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/1_Bd_EvpEzLH9BbxRXXUGQ/screenshot-to-replace.png\" width=\"50%\"/>\n",
    "    </a>\n",
    "</p>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing Required Libraries\n",
    "\n",
    "_We recommend you import all required libraries in one place (here):_\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt_tab to\n",
      "[nltk_data]     /home/jupyterlab/nltk_data...\n",
      "[nltk_data]   Unzipping tokenizers/punkt_tab.zip.\n",
      "[nltk_data] Downloading package averaged_perceptron_tagger_eng to\n",
      "[nltk_data]     /home/jupyterlab/nltk_data...\n",
      "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger_eng.zip.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You can also use this section to suppress warnings generated by your code:\n",
    "\n",
    "def warn(*args, **kwargs):\n",
    "    pass\n",
    "import warnings\n",
    "warnings.warn = warn\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "from pprint import pprint\n",
    "import json\n",
    "from pathlib import Path\n",
    "import nltk\n",
    "from langchain_community.document_loaders import TextLoader\n",
    "from langchain_community.document_loaders import PyPDFLoader\n",
    "from langchain_community.document_loaders import PyMuPDFLoader\n",
    "from langchain_community.document_loaders import UnstructuredMarkdownLoader\n",
    "from langchain_community.document_loaders import JSONLoader\n",
    "from langchain_community.document_loaders.csv_loader import CSVLoader\n",
    "from langchain_community.document_loaders.csv_loader import UnstructuredCSVLoader\n",
    "from langchain_community.document_loaders import WebBaseLoader\n",
    "from langchain_community.document_loaders import Docx2txtLoader\n",
    "from langchain_community.document_loaders import UnstructuredFileLoader\n",
    "\n",
    "nltk.download('punkt_tab')\n",
    "nltk.download('averaged_perceptron_tagger_eng')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load from TXT files\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `TextLoader` is a tool designed to load textual data from various sources.\n",
    "\n",
    "It is the simplest loader, reading a file as text and placing all the content into a single document.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have prepared a .txt file for you to load. First, we need to download it from a remote source.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have prepared a .txt file for you to load. First, we need to download it from a remote source.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-12-03 14:11:04--  https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/Ec5f3KYU1CpbKRp1whFLZw/new-Policies.txt\n",
      "Resolving cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)... 169.63.118.104, 169.63.118.104\n",
      "Connecting to cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)|169.63.118.104|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 6363 (6.2K) [text/plain]\n",
      "Saving to: ‘new-Policies.txt’\n",
      "\n",
      "new-Policies.txt    100%[===================>]   6.21K  --.-KB/s    in 0s      \n",
      "\n",
      "2024-12-03 14:11:05 (1.11 GB/s) - ‘new-Policies.txt’ saved [6363/6363]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/Ec5f3KYU1CpbKRp1whFLZw/new-Policies.txt\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will use the `TextLoader` class to load the file.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<langchain_community.document_loaders.text.TextLoader at 0x7fbd6d29a790>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loader = TextLoader(\"new-Policies.txt\")\n",
    "loader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we use the `load` method to load the data as documents.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = loader.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's present the entire data (document) here.\n",
    "\n",
    "This is a `document` object that includes `page_content` and `metadata` attributes.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': 'new-Policies.txt'}, page_content=\"1. Code of Conduct\\n\\nOur Code of Conduct establishes the core values and ethical standards that all members of our organization must adhere to. We are committed to fostering a workplace characterized by integrity, respect, and accountability.\\n\\nIntegrity: We commit to the highest ethical standards by being honest and transparent in all our dealings, whether with colleagues, clients, or the community. We protect sensitive information and avoid conflicts of interest.\\n\\nRespect: We value diversity and every individual's contribution. Discrimination, harassment, or any form of disrespect is not tolerated. We promote an inclusive environment where differences are respected, and everyone is treated with dignity.\\n\\nAccountability: We are responsible for our actions and decisions, complying with all relevant laws and regulations. We aim for continuous improvement and report any breaches of this code, supporting investigations into such matters.\\n\\nSafety: We prioritize the safety of our employees, clients, and the community. We encourage a culture of safety, including reporting any unsafe practices or conditions.\\n\\nEnvironmental Responsibility: We strive to reduce our environmental impact and promote sustainable practices.\\n\\nThis Code of Conduct is the cornerstone of our organizational culture. We expect every employee to uphold these principles and act as role models, ensuring our reputation for ethical conduct, integrity, and social responsibility.\\n\\n2. Recruitment Policy\\n\\nOur Recruitment Policy is dedicated to attracting, selecting, and integrating the most qualified and diverse candidates into our organization. The success of our company depends on the talent, skills, and commitment of our employees.\\n\\nEqual Opportunity: We are an equal opportunity employer and do not discriminate based on race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, or any other protected status. We actively support diversity and inclusion.\\n\\nTransparency: We maintain a transparent recruitment process. Job vacancies are advertised both internally and externally when appropriate. Job descriptions and requirements are clear and accurately reflect the role.\\n\\nSelection Criteria: We base our selection on qualifications, experience, and skills relevant to the role. Our interviews and assessments are objective, and decisions are made impartially.\\n\\nData Privacy: We are dedicated to protecting candidates' personal information and comply with all applicable data protection laws.\\n\\nFeedback: Candidates receive timely and constructive feedback on their applications and interview performance.\\n\\nOnboarding: New hires receive thorough onboarding to help them integrate effectively, including an overview of our culture, policies, and expectations.\\n\\nEmployee Referrals: We welcome employee referrals as they help build a strong and engaged team.\\n\\nThis policy lays the foundation for a diverse, inclusive, and talented workforce. It ensures that we hire candidates who align with our values and contribute to our success. We regularly review and update this policy to incorporate best practices in recruitment.\\n\\n\\n3. Internet and Email Policy\\n\\nOur Internet and Email Policy ensures the responsible and secure use of these tools within our organization, recognizing their importance in daily operations and the need for compliance with security, productivity, and legal standards.\\n\\nAcceptable Use: Company-provided internet and email are primarily for job-related tasks. Limited personal use is permitted during non-work hours as long as it does not interfere with work duties.\\n\\nSecurity: Protect your login credentials and avoid sharing passwords. Be cautious with email attachments and links from unknown sources, and promptly report any unusual online activity or potential security threats.\\n\\nConfidentiality: Use email for confidential information, trade secrets, and sensitive customer data only with encryption. Be careful when discussing company matters on public platforms or social media.\\n\\nHarassment and Inappropriate Content: Internet and email must not be used for harassment, discrimination, or the distribution of offensive content. Always communicate respectfully and sensitively online.\\n\\nCompliance: Adhere to all relevant laws and regulations concerning internet and email use, including copyright and data protection laws.\\n\\nMonitoring: The company reserves the right to monitor internet and email usage for security and compliance purposes.\\n\\nConsequences: Violations of this policy may lead to disciplinary action, including potential termination.\\n\\nThis policy promotes the safe and responsible use of digital communication tools in line with our values and legal obligations. Employees must understand and comply with this policy. Regular reviews will ensure it remains relevant with changing technology and security standards.\\n\\n4. Mobile Phone Policy\\n\\nOur Mobile Phone Policy defines standards for responsible use of mobile devices within the organization to ensure alignment with company values and legal requirements.\\n\\nAcceptable Use: Mobile devices are primarily for work-related tasks. Limited personal use is allowed if it does not disrupt work responsibilities.\\n\\nSecurity: Secure your mobile device and credentials. Be cautious with app downloads and links from unknown sources, and report any security issues promptly.\\n\\nConfidentiality: Avoid sharing sensitive company information via unsecured messaging apps or emails. Exercise caution when discussing company matters in public.\\n\\nCost Management: Personal use of mobile phones should be separate from company accounts, and any personal charges on company-issued phones must be reimbursed.\\n\\nCompliance: Comply with all relevant laws and regulations concerning mobile phone usage, including data protection and privacy laws.\\n\\nLost or Stolen Devices: Immediately report any lost or stolen mobile devices to the IT department or your supervisor.\\n\\nConsequences: Non-compliance with this policy may result in disciplinary actions, including potential loss of mobile phone privileges.\\n\\nThis policy encourages the responsible use of mobile devices in line with legal and ethical standards. Employees are expected to understand and follow these guidelines. The policy is regularly reviewed to stay current with evolving technology and security best practices.\\n\")]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the `pprint` function to print the first 1000 characters of the `page_content` here.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('1. Code of Conduct\\n'\n",
      " '\\n'\n",
      " 'Our Code of Conduct establishes the core values and ethical standards that '\n",
      " 'all members of our organization must adhere to. We are committed to '\n",
      " 'fostering a workplace characterized by integrity, respect, and '\n",
      " 'accountability.\\n'\n",
      " '\\n'\n",
      " 'Integrity: We commit to the highest ethical standards by being honest and '\n",
      " 'transparent in all our dealings, whether with colleagues, clients, or the '\n",
      " 'community. We protect sensitive information and avoid conflicts of '\n",
      " 'interest.\\n'\n",
      " '\\n'\n",
      " \"Respect: We value diversity and every individual's contribution. \"\n",
      " 'Discrimination, harassment, or any form of disrespect is not tolerated. We '\n",
      " 'promote an inclusive environment where differences are respected, and '\n",
      " 'everyone is treated with dignity.\\n'\n",
      " '\\n'\n",
      " 'Accountability: We are responsible for our actions and decisions, complying '\n",
      " 'with all relevant laws and regulations. We aim for continuous improvement '\n",
      " 'and report any breaches of this code, supporting investigations into such '\n",
      " 'matters.\\n'\n",
      " '\\n'\n",
      " 'Safety: We prioritize the safety of our employees, c')\n"
     ]
    }
   ],
   "source": [
    "pprint(data[0].page_content[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load from PDF files\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, we may have files in PDF format that we want to load for processing.\n",
    "\n",
    "LangChain provides several classes for loading PDFs. Here, we introduce two classes: `PyPDFLoader` and `PyMuPDFLoader`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### PyPDFLoader\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the PDF using `PyPDFLoader` into an array of documents, where each document contains the page content and metadata with the page number.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "pdf_url = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/Q81D33CdRLK6LswuQrANQQ/instructlab.pdf\"\n",
    "\n",
    "loader = PyPDFLoader(pdf_url)\n",
    "\n",
    "pages = loader.load_and_split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the first page of the PDF.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='LAB: L ARGE -SCALE ALIGNMENT FOR CHATBOTS\n",
      "MIT-IBM Watson AI Lab and IBM Research\n",
      "Shivchander Sudalairaj∗\n",
      "Abhishek Bhandwaldar∗\n",
      "Aldo Pareja∗\n",
      "Kai Xu\n",
      "David D. Cox\n",
      "Akash Srivastava∗,†\n",
      "*Equal Contribution, †Corresponding Author\n",
      "ABSTRACT\n",
      "This work introduces LAB (Large-scale Alignment for chatBots), a novel method-\n",
      "ology designed to overcome the scalability challenges in the instruction-tuning\n",
      "phase of large language model (LLM) training. Leveraging a taxonomy-guided\n",
      "synthetic data generation process and a multi-phase tuning framework, LAB sig-\n",
      "nificantly reduces reliance on expensive human annotations and proprietary mod-\n",
      "els like GPT-4. We demonstrate that LAB-trained models can achieve compet-\n",
      "itive performance across several benchmarks compared to models trained with\n",
      "traditional human-annotated or GPT-4 generated synthetic data. Thus offering a\n",
      "scalable, cost-effective solution for enhancing LLM capabilities and instruction-\n",
      "following behaviors without the drawbacks of catastrophic forgetting, marking a\n",
      "step forward in the efficient training of LLMs for a wide range of applications.\n",
      "1 I NTRODUCTION\n",
      "Large language models (LLMs) have achieved remarkable levels of success in various natural lan-\n",
      "guage processing (NLP) applications, including question-answering , entity extraction , and sum-\n",
      "marization . This has been made possible, in large part, by the introduction of the transformer\n",
      "architecture , which can leverage large amounts of unlabeled, unstructured data, enabling the scal-\n",
      "ing of LLMs to billions, or even trillions of parameters. LLMs are typically trained in phases: a\n",
      "self-supervised pre-training phase, followed by supervised alignment tuning phases.\n",
      "The majority of the cost of training an LLM comes from the pre-training phase. During this phase, a\n",
      "model is trained in an auto-regressive manner to predict the next token in the target language using\n",
      "trillions of tokens worth of unlabeled data, requiring thousands of GPUs training for months at a\n",
      "time. Alignment tuning, typically happens in two stages: instruction tuning, followed by prefer-\n",
      "ence tuning. Instruction tuning is more akin to the traditional model training approach in machine\n",
      "learning, where the model is trained directly on tasks of interest. In this stage, the model is given a\n",
      "task description in the form of an natural language instuction (e.g. Summarize the following news\n",
      "article in 2 lines: {News article }) and the model is trained to maximize the likelihood of the pro-\n",
      "vided ground truth summary. Preference tuning, on the other hand, is done using techniques such\n",
      "as RLHF (Stiennon et al., 2022; Ouyang et al., 2022) and DPO (Rafailov et al., 2023), where the\n",
      "response from an instruction-tuned model is rated as preferred or unpreferred using human feedback.\n",
      "In comparison to pre-training, the instruction tuning and preference tuning stages comprise a small\n",
      "fraction of the overall training procedure, both in terms of the data used as well as the compute\n",
      "infrastructure required to train models Touvron et al. (2023). For example, Meta’s LLaMA 2 models\n",
      "were trained with just tens of thousands of high quality human-generated instruction/response data\n",
      "pairs, followed by multiple rounds of RLHF with a comparatively limited number of examples as\n",
      "compared to pretraining data volumes Touvron et al. (2023). From a traditional machine learning\n",
      "training perspective, this imbalance in the scale across the phases is unconventional—typically one\n",
      "would expect a model to perform best when it has been trained directly on the desired tasks, using as\n",
      "much data as possible. The deviation from the tradtional LLM approach relies on the idea that pre-\n",
      "1arXiv:2403.01081v3  [cs.CL]  29 Apr 2024' metadata={'source': 'https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/Q81D33CdRLK6LswuQrANQQ/instructlab.pdf', 'page': 0}\n"
     ]
    }
   ],
   "source": [
    "print(pages[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the first three pages of the PDF.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page number 1\n",
      "page_content='LAB: L ARGE -SCALE ALIGNMENT FOR CHATBOTS\n",
      "MIT-IBM Watson AI Lab and IBM Research\n",
      "Shivchander Sudalairaj∗\n",
      "Abhishek Bhandwaldar∗\n",
      "Aldo Pareja∗\n",
      "Kai Xu\n",
      "David D. Cox\n",
      "Akash Srivastava∗,†\n",
      "*Equal Contribution, †Corresponding Author\n",
      "ABSTRACT\n",
      "This work introduces LAB (Large-scale Alignment for chatBots), a novel method-\n",
      "ology designed to overcome the scalability challenges in the instruction-tuning\n",
      "phase of large language model (LLM) training. Leveraging a taxonomy-guided\n",
      "synthetic data generation process and a multi-phase tuning framework, LAB sig-\n",
      "nificantly reduces reliance on expensive human annotations and proprietary mod-\n",
      "els like GPT-4. We demonstrate that LAB-trained models can achieve compet-\n",
      "itive performance across several benchmarks compared to models trained with\n",
      "traditional human-annotated or GPT-4 generated synthetic data. Thus offering a\n",
      "scalable, cost-effective solution for enhancing LLM capabilities and instruction-\n",
      "following behaviors without the drawbacks of catastrophic forgetting, marking a\n",
      "step forward in the efficient training of LLMs for a wide range of applications.\n",
      "1 I NTRODUCTION\n",
      "Large language models (LLMs) have achieved remarkable levels of success in various natural lan-\n",
      "guage processing (NLP) applications, including question-answering , entity extraction , and sum-\n",
      "marization . This has been made possible, in large part, by the introduction of the transformer\n",
      "architecture , which can leverage large amounts of unlabeled, unstructured data, enabling the scal-\n",
      "ing of LLMs to billions, or even trillions of parameters. LLMs are typically trained in phases: a\n",
      "self-supervised pre-training phase, followed by supervised alignment tuning phases.\n",
      "The majority of the cost of training an LLM comes from the pre-training phase. During this phase, a\n",
      "model is trained in an auto-regressive manner to predict the next token in the target language using\n",
      "trillions of tokens worth of unlabeled data, requiring thousands of GPUs training for months at a\n",
      "time. Alignment tuning, typically happens in two stages: instruction tuning, followed by prefer-\n",
      "ence tuning. Instruction tuning is more akin to the traditional model training approach in machine\n",
      "learning, where the model is trained directly on tasks of interest. In this stage, the model is given a\n",
      "task description in the form of an natural language instuction (e.g. Summarize the following news\n",
      "article in 2 lines: {News article }) and the model is trained to maximize the likelihood of the pro-\n",
      "vided ground truth summary. Preference tuning, on the other hand, is done using techniques such\n",
      "as RLHF (Stiennon et al., 2022; Ouyang et al., 2022) and DPO (Rafailov et al., 2023), where the\n",
      "response from an instruction-tuned model is rated as preferred or unpreferred using human feedback.\n",
      "In comparison to pre-training, the instruction tuning and preference tuning stages comprise a small\n",
      "fraction of the overall training procedure, both in terms of the data used as well as the compute\n",
      "infrastructure required to train models Touvron et al. (2023). For example, Meta’s LLaMA 2 models\n",
      "were trained with just tens of thousands of high quality human-generated instruction/response data\n",
      "pairs, followed by multiple rounds of RLHF with a comparatively limited number of examples as\n",
      "compared to pretraining data volumes Touvron et al. (2023). From a traditional machine learning\n",
      "training perspective, this imbalance in the scale across the phases is unconventional—typically one\n",
      "would expect a model to perform best when it has been trained directly on the desired tasks, using as\n",
      "much data as possible. The deviation from the tradtional LLM approach relies on the idea that pre-\n",
      "1arXiv:2403.01081v3  [cs.CL]  29 Apr 2024' metadata={'source': 'https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/Q81D33CdRLK6LswuQrANQQ/instructlab.pdf', 'page': 0}\n",
      "page number 2\n",
      "page_content='training captures enough of the distribution of language and knowledge, such that a small amount of\n",
      "supervised training can “unlock” or shape latent abilities related to the ultimate desired instruction-\n",
      "following behavior of the model. However, unlike the unstructured data that is abundantly available\n",
      "in the public domain, high-quality, human-generated task-specific instruction data is costly to pro-\n",
      "cure, even via crowd-sourcing, and human-generated instruction data is typically closely guarded\n",
      "by model builders, even for ostensibly “open” model-building efforts. In this work, we address\n",
      "the challenges associated with scaling of the alignment-tuning phase and propose a new method\n",
      "called LAB: Large-scale Alignment for chatBots. The LAB method consists of two components:\n",
      "(i) a taxonomy-guided synthetic data generation method and quality assurance process that yields a\n",
      "highly diverse and high-quality instruction dataset, without resorting to the use of proprietary LLMs\n",
      "like GPT-4 or substantial human curation, and (ii) a novel multi-phase training framework and un-\n",
      "conventional tuning regime that allows for adding new knowledge and instruction-following abilities\n",
      "into pre-trained LLMs without suffering from catastrophic forgetting. Our findings show that LAB-\n",
      "trained models can perform competitively with proprietary and open-source models that use human\n",
      "annotations and/or synthetic data generated using GPT-4 on a number of benchmarks.\n",
      "2 R ELATED WORK\n",
      "Existing methods for instruction tuning typically either rely on humans for generating high-quality\n",
      "datasets, or use synthetic data generation using a large teacher model. OpenAI (Ouyang et al.,\n",
      "2022) arguably set the standard for model alignment from human data, employing human annota-\n",
      "tors to gather data for supervised fine tuning (SFT) and reinforcement learning with human feed-\n",
      "back (RLHF) training. Collecting human-generated data for these steps is complex undertaking; the\n",
      "selection of annotators requires a rigorous multi-stage screening process aimed at achieving high\n",
      "inter-annotator agreement, and collecting even modest amounts data (by LLM standards) requires\n",
      "the coordination of large groups of annotators. The creators of the LLaMA 2 model series (Touvron\n",
      "et al., 2023) followed a similar recipe, collecting tens of thousands of human-generated instruction\n",
      "samples, and approximately 1 million human-annotated binary comparisons for reward modeling.\n",
      "Not only are such approaches expensive and time consuming, but they can also potentially limit\n",
      "agility in exploring the space of instructions and capabilities the model is trained to perform. Alter-\n",
      "natives to this approach, such as transforming existing human datasets into instructions via templat-\n",
      "ing (Wei et al.) can be more cost effective, but face limitations in the naturalness and length of the\n",
      "responses used for training.\n",
      "More recently, training with synthetic data generated from LLMs has emerged as an alternative to\n",
      "purely human-data-based approaches. Wang et al. (2023) introduced Self-Instruct, which leverages\n",
      "a small number of handwritten human seed instructions as input to bootstrapping process to generate\n",
      "a large number of samples using an LLM’s own generation abilities. Taori et al. (2023) built upon\n",
      "Self-Instruct, using a larger teacher model to generate synthetic data to train a smaller student model,\n",
      "and incorporating principles in the generation prompt to promote diversity in the generated instruc-\n",
      "tion data. Xu et al. (2023) introduces Evol-Instruct, another variant of Self-Instruct, that synthesizes\n",
      "iteratively more complex instruction to overcome shortcomings of previous methods. Mukherjee\n",
      "et al. (2023), Mitra et al. (2023) present a synthetic data generation approach to enhance task di-\n",
      "versity and scalability, alongside a progressive training framework aimed at improving the model’s\n",
      "reasoning ability and response style to match teacher models. This is achieved by generating rich' metadata={'source': 'https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/Q81D33CdRLK6LswuQrANQQ/instructlab.pdf', 'page': 1}\n",
      "page number 3\n",
      "page_content='versity and scalability, alongside a progressive training framework aimed at improving the model’s\n",
      "reasoning ability and response style to match teacher models. This is achieved by generating rich\n",
      "reasoning signals in the generated answer and progressively training on datasets of varying difficulty\n",
      "in incremental phases.\n",
      "Similar to LAB, concurrent work, GLAN (Li et al., 2024), employs a semi-automatic approach to\n",
      "synthetic data generation that uses a human-curated taxonomy to generate instruction tuning data\n",
      "from a teacher model. However, as explained in section 3.2.2, unlike LAB, GLAN cannot be used\n",
      "to generate synthetic data from domains that are not captured in the teacher model’s support. As\n",
      "such, while LAB uses the open-source Mixtral model as the teacher, like many other synthetic\n",
      "data generation approaches, GLAN has to rely on a large proprietary model (GPT-4). This poses\n",
      "complicated questions about the usability of generated data (especially for commercial purposes)\n",
      "since the terms of use of proprietary models typically forbid using the model to improve other\n",
      "models.\n",
      "2' metadata={'source': 'https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/Q81D33CdRLK6LswuQrANQQ/instructlab.pdf', 'page': 1}\n"
     ]
    }
   ],
   "source": [
    "for p,page in enumerate(pages[0:3]):\n",
    "    print(f\"page number {p+1}\")\n",
    "    print(page)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### PyMuPDFLoader\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`PyMuPDFLoader` is the fastest of the PDF parsing options. It provides detailed metadata about the PDF and its pages, and returns one document per page.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<langchain_community.document_loaders.pdf.PyMuPDFLoader at 0x7fbd88bbdc10>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loader = PyMuPDFLoader(pdf_url)\n",
    "loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='LAB: LARGE-SCALE ALIGNMENT FOR CHATBOTS\n",
      "MIT-IBM Watson AI Lab and IBM Research\n",
      "Shivchander Sudalairaj∗\n",
      "Abhishek Bhandwaldar∗\n",
      "Aldo Pareja∗\n",
      "Kai Xu\n",
      "David D. Cox\n",
      "Akash Srivastava∗,†\n",
      "*Equal Contribution, †Corresponding Author\n",
      "ABSTRACT\n",
      "This work introduces LAB (Large-scale Alignment for chatBots), a novel method-\n",
      "ology designed to overcome the scalability challenges in the instruction-tuning\n",
      "phase of large language model (LLM) training. Leveraging a taxonomy-guided\n",
      "synthetic data generation process and a multi-phase tuning framework, LAB sig-\n",
      "nificantly reduces reliance on expensive human annotations and proprietary mod-\n",
      "els like GPT-4. We demonstrate that LAB-trained models can achieve compet-\n",
      "itive performance across several benchmarks compared to models trained with\n",
      "traditional human-annotated or GPT-4 generated synthetic data. Thus offering a\n",
      "scalable, cost-effective solution for enhancing LLM capabilities and instruction-\n",
      "following behaviors without the drawbacks of catastrophic forgetting, marking a\n",
      "step forward in the efficient training of LLMs for a wide range of applications.\n",
      "1\n",
      "INTRODUCTION\n",
      "Large language models (LLMs) have achieved remarkable levels of success in various natural lan-\n",
      "guage processing (NLP) applications, including question-answering , entity extraction , and sum-\n",
      "marization . This has been made possible, in large part, by the introduction of the transformer\n",
      "architecture , which can leverage large amounts of unlabeled, unstructured data, enabling the scal-\n",
      "ing of LLMs to billions, or even trillions of parameters. LLMs are typically trained in phases: a\n",
      "self-supervised pre-training phase, followed by supervised alignment tuning phases.\n",
      "The majority of the cost of training an LLM comes from the pre-training phase. During this phase, a\n",
      "model is trained in an auto-regressive manner to predict the next token in the target language using\n",
      "trillions of tokens worth of unlabeled data, requiring thousands of GPUs training for months at a\n",
      "time. Alignment tuning, typically happens in two stages: instruction tuning, followed by prefer-\n",
      "ence tuning. Instruction tuning is more akin to the traditional model training approach in machine\n",
      "learning, where the model is trained directly on tasks of interest. In this stage, the model is given a\n",
      "task description in the form of an natural language instuction (e.g. Summarize the following news\n",
      "article in 2 lines: {News article}) and the model is trained to maximize the likelihood of the pro-\n",
      "vided ground truth summary. Preference tuning, on the other hand, is done using techniques such\n",
      "as RLHF (Stiennon et al., 2022; Ouyang et al., 2022) and DPO (Rafailov et al., 2023), where the\n",
      "response from an instruction-tuned model is rated as preferred or unpreferred using human feedback.\n",
      "In comparison to pre-training, the instruction tuning and preference tuning stages comprise a small\n",
      "fraction of the overall training procedure, both in terms of the data used as well as the compute\n",
      "infrastructure required to train models Touvron et al. (2023). For example, Meta’s LLaMA 2 models\n",
      "were trained with just tens of thousands of high quality human-generated instruction/response data\n",
      "pairs, followed by multiple rounds of RLHF with a comparatively limited number of examples as\n",
      "compared to pretraining data volumes Touvron et al. (2023). From a traditional machine learning\n",
      "training perspective, this imbalance in the scale across the phases is unconventional—typically one\n",
      "would expect a model to perform best when it has been trained directly on the desired tasks, using as\n",
      "much data as possible. The deviation from the tradtional LLM approach relies on the idea that pre-\n",
      "1\n",
      "arXiv:2403.01081v3  [cs.CL]  29 Apr 2024\n",
      "' metadata={'source': 'https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/Q81D33CdRLK6LswuQrANQQ/instructlab.pdf', 'file_path': 'https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/Q81D33CdRLK6LswuQrANQQ/instructlab.pdf', 'page': 0, 'total_pages': 10, 'format': 'PDF 1.5', 'title': '', 'author': '', 'subject': '', 'keywords': '', 'creator': 'LaTeX with hyperref', 'producer': 'pdfTeX-1.40.25', 'creationDate': 'D:20240501000524Z', 'modDate': 'D:20240501000524Z', 'trapped': ''}\n"
     ]
    }
   ],
   "source": [
    "print(data[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `metadata` attribute reveals that `PyMuPDFLoader` provides more detailed metadata information than `PyPDFLoader`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load from Markdown files\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, our file source might be in Markdown format.\n",
    "\n",
    "LangChain provides the `UnstructuredMarkdownLoader` to load content from Markdown files.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-12-03 14:11:06--  https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/eMSP5vJjj9yOfAacLZRWsg/markdown-sample.md\n",
      "Resolving cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)... 169.63.118.104, 169.63.118.104\n",
      "Connecting to cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)|169.63.118.104|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 3398 (3.3K) [text/markdown]\n",
      "Saving to: ‘markdown-sample.md’\n",
      "\n",
      "markdown-sample.md  100%[===================>]   3.32K  --.-KB/s    in 0s      \n",
      "\n",
      "2024-12-03 14:11:06 (724 MB/s) - ‘markdown-sample.md’ saved [3398/3398]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget 'https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/eMSP5vJjj9yOfAacLZRWsg/markdown-sample.md'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<langchain_community.document_loaders.markdown.UnstructuredMarkdownLoader at 0x7fbd7faa0cd0>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "markdown_path = \"markdown-sample.md\"\n",
    "loader = UnstructuredMarkdownLoader(markdown_path)\n",
    "loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /home/jupyterlab/nltk_data...\n",
      "[nltk_data]   Unzipping tokenizers/punkt.zip.\n",
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     /home/jupyterlab/nltk_data...\n",
      "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n"
     ]
    }
   ],
   "source": [
    "data = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': 'markdown-sample.md'}, page_content='An h1 header\\n\\nParagraphs are separated by a blank line.\\n\\n2nd paragraph. Italic, bold, and monospace. Itemized lists\\nlook like:\\n\\nthis one\\n\\nthat one\\n\\nthe other one\\n\\nNote that --- not considering the asterisk --- the actual text\\ncontent starts at 4-columns in.\\n\\nBlock quotes are\\nwritten like so.\\n\\nThey can span multiple paragraphs,\\nif you like.\\n\\nUse 3 dashes for an em-dash. Use 2 dashes for ranges (ex., \"it\\'s all\\nin chapters 12--14\"). Three dots ... will be converted to an ellipsis.\\nUnicode is supported. ☺\\n\\nAn h2 header\\n\\nHere\\'s a numbered list:\\n\\nfirst item\\n\\nsecond item\\n\\nthird item\\n\\nNote again how the actual text starts at 4 columns in (4 characters\\nfrom the left side). Here\\'s a code sample:\\n\\nAs you probably guessed, indented 4 spaces. By the way, instead of\\nindenting the block, you can use delimited blocks, if you like:\\n\\n~~~\\ndefine foobar() {\\n    print \"Welcome to flavor country!\";\\n}\\n~~~\\n\\n(which makes copying & pasting easier). You can optionally mark the\\ndelimited block for Pandoc to syntax highlight it:\\n\\n~~~python\\nimport time\\n\\nQuick, count to ten!\\n\\nfor i in range(10):\\n    # (but not too quick)\\n    time.sleep(0.5)\\n    print i\\n~~~\\n\\nAn h3 header\\n\\nNow a nested list:\\n\\nFirst, get these ingredients:\\n\\ncarrots\\ncelery\\nlentils\\n\\nBoil some water.\\n\\nDump everything in the pot and follow\\n    this algorithm:\\nfind wooden spoon\\nuncover pot\\nstir\\ncover pot\\nbalance wooden spoon precariously on pot handle\\nwait 10 minutes\\ngoto first step (or shut off burner when done)\\n\\nDo not bump wooden spoon or it will fall.\\n\\nNotice again how text always lines up on 4-space indents (including\\nthat last line which continues item 3 above).\\n\\nHere\\'s a link to a website, to a local\\ndoc, and to a section heading in the current\\ndoc. Here\\'s a footnote [^1].\\n\\n[^1]: Footnote text goes here.\\n\\nTables can look like this:\\n\\nsize  material      color\\n\\n9     leather       brown\\n10    hemp canvas   natural\\n11    glass         transparent\\n\\nTable: Shoes, their sizes, and what they\\'re made of\\n\\n(The above is the caption for the table.) Pandoc also supports\\nmulti-line tables:\\n\\nkeyword   text\\n\\nred       Sunsets, apples, and\\n          other red or reddish\\n          things.\\n\\ngreen     Leaves, grass, frogs\\n          and other things it\\'s\\n          not easy being.\\n\\nA horizontal rule follows.\\n\\nHere\\'s a definition list:\\n\\napples\\n  : Good for making applesauce.\\noranges\\n  : Citrus!\\ntomatoes\\n  : There\\'s no \"e\" in tomatoe.\\n\\nAgain, text is indented 4 spaces. (Put a blank line between each\\nterm/definition pair to spread things out more.)\\n\\nHere\\'s a \"line block\":\\n\\n| Line one\\n|   Line too\\n| Line tree\\n\\nand images can be specified like so:\\n\\nInline math equations go in like so: $\\\\omega = d\\\\phi / dt$. Display\\nmath should get its own line and be put in in double-dollarsigns:\\n\\n$$I = \\\\int \\\\rho R^{2} dV$$\\n\\nAnd note that you can backslash-escape any punctuation characters\\nwhich you wish to be displayed literally, ex.: `foo`, *bar*, etc.')]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load from JSON files\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The JSONLoader uses a specified [jq schema](https://en.wikipedia.org/wiki/Jq_(programming_language)) to parse the JSON files. It uses the jq python package, which we've installed before.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-12-03 14:11:08--  https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/hAmzVJeOUAMHzmhUHNdAUg/facebook-chat.json\n",
      "Resolving cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)... 169.63.118.104, 169.63.118.104\n",
      "Connecting to cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)|169.63.118.104|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2167 (2.1K) [application/json]\n",
      "Saving to: ‘facebook-chat.json’\n",
      "\n",
      "facebook-chat.json  100%[===================>]   2.12K  --.-KB/s    in 0s      \n",
      "\n",
      "2024-12-03 14:11:09 (765 MB/s) - ‘facebook-chat.json’ saved [2167/2167]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget 'https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/hAmzVJeOUAMHzmhUHNdAUg/facebook-chat.json'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's use `pprint` to take a look at the JSON file and its structure. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_path='facebook-chat.json'\n",
    "data = json.loads(Path(file_path).read_text())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'image': {'creation_timestamp': 1675549016, 'uri': 'image_of_the_chat.jpg'},\n",
      " 'is_still_participant': True,\n",
      " 'joinable_mode': {'link': '', 'mode': 1},\n",
      " 'magic_words': [],\n",
      " 'messages': [{'content': 'Bye!',\n",
      "               'sender_name': 'User 2',\n",
      "               'timestamp_ms': 1675597571851},\n",
      "              {'content': 'Oh no worries! Bye',\n",
      "               'sender_name': 'User 1',\n",
      "               'timestamp_ms': 1675597435669},\n",
      "              {'content': 'No Im sorry it was my mistake, the blue one is not '\n",
      "                          'for sale',\n",
      "               'sender_name': 'User 2',\n",
      "               'timestamp_ms': 1675596277579},\n",
      "              {'content': 'I thought you were selling the blue one!',\n",
      "               'sender_name': 'User 1',\n",
      "               'timestamp_ms': 1675595140251},\n",
      "              {'content': 'Im not interested in this bag. Im interested in the '\n",
      "                          'blue one!',\n",
      "               'sender_name': 'User 1',\n",
      "               'timestamp_ms': 1675595109305},\n",
      "              {'content': 'Here is $129',\n",
      "               'sender_name': 'User 2',\n",
      "               'timestamp_ms': 1675595068468},\n",
      "              {'photos': [{'creation_timestamp': 1675595059,\n",
      "                           'uri': 'url_of_some_picture.jpg'}],\n",
      "               'sender_name': 'User 2',\n",
      "               'timestamp_ms': 1675595060730},\n",
      "              {'content': 'Online is at least $100',\n",
      "               'sender_name': 'User 2',\n",
      "               'timestamp_ms': 1675595045152},\n",
      "              {'content': 'How much do you want?',\n",
      "               'sender_name': 'User 1',\n",
      "               'timestamp_ms': 1675594799696},\n",
      "              {'content': 'Goodmorning! $50 is too low.',\n",
      "               'sender_name': 'User 2',\n",
      "               'timestamp_ms': 1675577876645},\n",
      "              {'content': 'Hi! Im interested in your bag. Im offering $50. Let '\n",
      "                          'me know if you are interested. Thanks!',\n",
      "               'sender_name': 'User 1',\n",
      "               'timestamp_ms': 1675549022673}],\n",
      " 'participants': [{'name': 'User 1'}, {'name': 'User 2'}],\n",
      " 'thread_path': 'inbox/User 1 and User 2 chat',\n",
      " 'title': 'User 1 and User 2 chat'}\n"
     ]
    }
   ],
   "source": [
    "pprint(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use `JSONLoader` to load data from the JSON file. However, JSON files can have various attribute-value pairs. If we want to load a specific attribute and its value, we need to set an appropriate `jq schema`.\n",
    "\n",
    "So for example, if we want to load the `content` from the JSON file, we need to set `jq_schema='.messages[].content'`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = JSONLoader(\n",
    "    file_path=file_path,\n",
    "    jq_schema='.messages[].content',\n",
    "    text_content=False)\n",
    "\n",
    "data = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Document(metadata={'source': '/resources/AI0214EN/facebook-chat.json', 'seq_num': 1}, page_content='Bye!'),\n",
      " Document(metadata={'source': '/resources/AI0214EN/facebook-chat.json', 'seq_num': 2}, page_content='Oh no worries! Bye'),\n",
      " Document(metadata={'source': '/resources/AI0214EN/facebook-chat.json', 'seq_num': 3}, page_content='No Im sorry it was my mistake, the blue one is not for sale'),\n",
      " Document(metadata={'source': '/resources/AI0214EN/facebook-chat.json', 'seq_num': 4}, page_content='I thought you were selling the blue one!'),\n",
      " Document(metadata={'source': '/resources/AI0214EN/facebook-chat.json', 'seq_num': 5}, page_content='Im not interested in this bag. Im interested in the blue one!'),\n",
      " Document(metadata={'source': '/resources/AI0214EN/facebook-chat.json', 'seq_num': 6}, page_content='Here is $129'),\n",
      " Document(metadata={'source': '/resources/AI0214EN/facebook-chat.json', 'seq_num': 7}, page_content=''),\n",
      " Document(metadata={'source': '/resources/AI0214EN/facebook-chat.json', 'seq_num': 8}, page_content='Online is at least $100'),\n",
      " Document(metadata={'source': '/resources/AI0214EN/facebook-chat.json', 'seq_num': 9}, page_content='How much do you want?'),\n",
      " Document(metadata={'source': '/resources/AI0214EN/facebook-chat.json', 'seq_num': 10}, page_content='Goodmorning! $50 is too low.'),\n",
      " Document(metadata={'source': '/resources/AI0214EN/facebook-chat.json', 'seq_num': 11}, page_content='Hi! Im interested in your bag. Im offering $50. Let me know if you are interested. Thanks!')]\n"
     ]
    }
   ],
   "source": [
    "pprint(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load from CSV files\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "CSV files are a common format for storing tabular data. The `CSVLoader` provides a convenient way to read and process this data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-12-03 14:11:09--  https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IygVG_j0M87BM4Z0zFsBMA/mlb-teams-2012.csv\n",
      "Resolving cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)... 169.63.118.104, 169.63.118.104\n",
      "Connecting to cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)|169.63.118.104|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 848 [text/csv]\n",
      "Saving to: ‘mlb-teams-2012.csv’\n",
      "\n",
      "mlb-teams-2012.csv  100%[===================>]     848  --.-KB/s    in 0s      \n",
      "\n",
      "2024-12-03 14:11:09 (316 MB/s) - ‘mlb-teams-2012.csv’ saved [848/848]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget 'https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IygVG_j0M87BM4Z0zFsBMA/mlb-teams-2012.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = CSVLoader(file_path='mlb-teams-2012.csv')\n",
    "data = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 0}, page_content='Team: Nationals\\n\"Payroll (millions)\": 81.34\\n\"Wins\": 98'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 1}, page_content='Team: Reds\\n\"Payroll (millions)\": 82.20\\n\"Wins\": 97'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 2}, page_content='Team: Yankees\\n\"Payroll (millions)\": 197.96\\n\"Wins\": 95'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 3}, page_content='Team: Giants\\n\"Payroll (millions)\": 117.62\\n\"Wins\": 94'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 4}, page_content='Team: Braves\\n\"Payroll (millions)\": 83.31\\n\"Wins\": 94'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 5}, page_content='Team: Athletics\\n\"Payroll (millions)\": 55.37\\n\"Wins\": 94'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 6}, page_content='Team: Rangers\\n\"Payroll (millions)\": 120.51\\n\"Wins\": 93'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 7}, page_content='Team: Orioles\\n\"Payroll (millions)\": 81.43\\n\"Wins\": 93'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 8}, page_content='Team: Rays\\n\"Payroll (millions)\": 64.17\\n\"Wins\": 90'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 9}, page_content='Team: Angels\\n\"Payroll (millions)\": 154.49\\n\"Wins\": 89'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 10}, page_content='Team: Tigers\\n\"Payroll (millions)\": 132.30\\n\"Wins\": 88'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 11}, page_content='Team: Cardinals\\n\"Payroll (millions)\": 110.30\\n\"Wins\": 88'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 12}, page_content='Team: Dodgers\\n\"Payroll (millions)\": 95.14\\n\"Wins\": 86'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 13}, page_content='Team: White Sox\\n\"Payroll (millions)\": 96.92\\n\"Wins\": 85'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 14}, page_content='Team: Brewers\\n\"Payroll (millions)\": 97.65\\n\"Wins\": 83'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 15}, page_content='Team: Phillies\\n\"Payroll (millions)\": 174.54\\n\"Wins\": 81'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 16}, page_content='Team: Diamondbacks\\n\"Payroll (millions)\": 74.28\\n\"Wins\": 81'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 17}, page_content='Team: Pirates\\n\"Payroll (millions)\": 63.43\\n\"Wins\": 79'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 18}, page_content='Team: Padres\\n\"Payroll (millions)\": 55.24\\n\"Wins\": 76'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 19}, page_content='Team: Mariners\\n\"Payroll (millions)\": 81.97\\n\"Wins\": 75'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 20}, page_content='Team: Mets\\n\"Payroll (millions)\": 93.35\\n\"Wins\": 74'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 21}, page_content='Team: Blue Jays\\n\"Payroll (millions)\": 75.48\\n\"Wins\": 73'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 22}, page_content='Team: Royals\\n\"Payroll (millions)\": 60.91\\n\"Wins\": 72'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 23}, page_content='Team: Marlins\\n\"Payroll (millions)\": 118.07\\n\"Wins\": 69'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 24}, page_content='Team: Red Sox\\n\"Payroll (millions)\": 173.18\\n\"Wins\": 69'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 25}, page_content='Team: Indians\\n\"Payroll (millions)\": 78.43\\n\"Wins\": 68'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 26}, page_content='Team: Twins\\n\"Payroll (millions)\": 94.08\\n\"Wins\": 66'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 27}, page_content='Team: Rockies\\n\"Payroll (millions)\": 78.06\\n\"Wins\": 64'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 28}, page_content='Team: Cubs\\n\"Payroll (millions)\": 88.19\\n\"Wins\": 61'),\n",
       " Document(metadata={'source': 'mlb-teams-2012.csv', 'row': 29}, page_content='Team: Astros\\n\"Payroll (millions)\": 60.65\\n\"Wins\": 55')]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you load data from a CSV file, the loader typically creates a separate `Document` object for each row of data in the CSV.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### UnstructuredCSVLoader\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In contrast to `CSVLoader`, which treats each row as an individual document with headers defining the data, `UnstructuredCSVLoader` considers the entire CSV file as a single unstructured table element. This approach is beneficial when you want to analyze the data as a complete table rather than as separate entries.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = UnstructuredCSVLoader(\n",
    "    file_path=\"mlb-teams-2012.csv\", mode=\"elements\"\n",
    ")\n",
    "data = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n\\n\\nTeam\\n\"Payroll (millions)\"\\n\"Wins\"\\n\\n\\nNationals\\n81.34\\n98\\n\\n\\nReds\\n82.20\\n97\\n\\n\\nYankees\\n197.96\\n95\\n\\n\\nGiants\\n117.62\\n94\\n\\n\\nBraves\\n83.31\\n94\\n\\n\\nAthletics\\n55.37\\n94\\n\\n\\nRangers\\n120.51\\n93\\n\\n\\nOrioles\\n81.43\\n93\\n\\n\\nRays\\n64.17\\n90\\n\\n\\nAngels\\n154.49\\n89\\n\\n\\nTigers\\n132.30\\n88\\n\\n\\nCardinals\\n110.30\\n88\\n\\n\\nDodgers\\n95.14\\n86\\n\\n\\nWhite Sox\\n96.92\\n85\\n\\n\\nBrewers\\n97.65\\n83\\n\\n\\nPhillies\\n174.54\\n81\\n\\n\\nDiamondbacks\\n74.28\\n81\\n\\n\\nPirates\\n63.43\\n79\\n\\n\\nPadres\\n55.24\\n76\\n\\n\\nMariners\\n81.97\\n75\\n\\n\\nMets\\n93.35\\n74\\n\\n\\nBlue Jays\\n75.48\\n73\\n\\n\\nRoyals\\n60.91\\n72\\n\\n\\nMarlins\\n118.07\\n69\\n\\n\\nRed Sox\\n173.18\\n69\\n\\n\\nIndians\\n78.43\\n68\\n\\n\\nTwins\\n94.08\\n66\\n\\n\\nRockies\\n78.06\\n64\\n\\n\\nCubs\\n88.19\\n61\\n\\n\\nAstros\\n60.65\\n55\\n\\n\\n'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0].page_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<table border=\"1\" class=\"dataframe\">\n",
      "  <tbody>\n",
      "    <tr>\n",
      "      <td>Team</td>\n",
      "      <td>\"Payroll (millions)\"</td>\n",
      "      <td>\"Wins\"</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Nationals</td>\n",
      "      <td>81.34</td>\n",
      "      <td>98</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Reds</td>\n",
      "      <td>82.20</td>\n",
      "      <td>97</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Yankees</td>\n",
      "      <td>197.96</td>\n",
      "      <td>95</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Giants</td>\n",
      "      <td>117.62</td>\n",
      "      <td>94</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Braves</td>\n",
      "      <td>83.31</td>\n",
      "      <td>94</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Athletics</td>\n",
      "      <td>55.37</td>\n",
      "      <td>94</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Rangers</td>\n",
      "      <td>120.51</td>\n",
      "      <td>93</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Orioles</td>\n",
      "      <td>81.43</td>\n",
      "      <td>93</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Rays</td>\n",
      "      <td>64.17</td>\n",
      "      <td>90</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Angels</td>\n",
      "      <td>154.49</td>\n",
      "      <td>89</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Tigers</td>\n",
      "      <td>132.30</td>\n",
      "      <td>88</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Cardinals</td>\n",
      "      <td>110.30</td>\n",
      "      <td>88</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Dodgers</td>\n",
      "      <td>95.14</td>\n",
      "      <td>86</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>White Sox</td>\n",
      "      <td>96.92</td>\n",
      "      <td>85</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Brewers</td>\n",
      "      <td>97.65</td>\n",
      "      <td>83</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Phillies</td>\n",
      "      <td>174.54</td>\n",
      "      <td>81</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Diamondbacks</td>\n",
      "      <td>74.28</td>\n",
      "      <td>81</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Pirates</td>\n",
      "      <td>63.43</td>\n",
      "      <td>79</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Padres</td>\n",
      "      <td>55.24</td>\n",
      "      <td>76</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Mariners</td>\n",
      "      <td>81.97</td>\n",
      "      <td>75</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Mets</td>\n",
      "      <td>93.35</td>\n",
      "      <td>74</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Blue Jays</td>\n",
      "      <td>75.48</td>\n",
      "      <td>73</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Royals</td>\n",
      "      <td>60.91</td>\n",
      "      <td>72</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Marlins</td>\n",
      "      <td>118.07</td>\n",
      "      <td>69</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Red Sox</td>\n",
      "      <td>173.18</td>\n",
      "      <td>69</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Indians</td>\n",
      "      <td>78.43</td>\n",
      "      <td>68</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Twins</td>\n",
      "      <td>94.08</td>\n",
      "      <td>66</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Rockies</td>\n",
      "      <td>78.06</td>\n",
      "      <td>64</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Cubs</td>\n",
      "      <td>88.19</td>\n",
      "      <td>61</td>\n",
      "    </tr>\n",
      "    <tr>\n",
      "      <td>Astros</td>\n",
      "      <td>60.65</td>\n",
      "      <td>55</td>\n",
      "    </tr>\n",
      "  </tbody>\n",
      "</table>\n"
     ]
    }
   ],
   "source": [
    "print(data[0].metadata[\"text_as_html\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load from URL/Website files\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usually we use `BeautifulSoup` package to load and parse a HTML or XML file. But it has some limitations.\n",
    "\n",
    "The following code is using `BeautifulSoup` to parse a website. Let's see what limitation it has.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE HTML>\n",
      "<html lang=\"en\">\n",
      " <head>\n",
      "  <meta charset=\"utf-8\"/>\n",
      "  <meta content=\"en\" name=\"languageCode\"/>\n",
      "  <meta content=\"us\" name=\"countryCode\"/>\n",
      "  <meta content=\"What Is LangChain?\" name=\"searchTitle\"/>\n",
      "  <meta content=\"Data AI - All\" name=\"focusArea\"/>\n",
      "  <title>\n",
      "   What Is LangChain? | IBM\n",
      "  </title>\n",
      "  <script data-routing=\"program=131558,environment=1281329,tier=publish\" defer=\"defer\" src=\"https://rum.hlx.page/.rum/@adobe/helix-rum-js@%5E2/dist/rum-standalone.js\" type=\"text/javascript\">\n",
      "  </script>\n",
      "  <link href=\"/content/dam/adobe-cms/default-images/favicon.svg\" rel=\"icon\"/>\n",
      "  <meta content=\"LangChain,Large language models\" name=\"keywords\"/>\n",
      "  <meta content=\"LangChain is an open source orchestration framework for the development of applications using large language models (LLMs), like chatbots and virtual agents. \" name=\"description\"/>\n",
      "  <meta content=\"learn\" name=\"template\"/>\n",
      "  <meta content=\"width=device-width, initial-scale=1\" name=\"viewport\"/>\n",
      "  <meta content=\"index, follow, max-image-preview:large\" name=\"robots\"/>\n",
      "  <meta content=\"thinkhub\" name=\"ibm.com.search.appid\"/>\n",
      "  <meta content=\"thinkhub\" name=\"ibm.com.search.scopes\"/>\n",
      "  <meta content=\"2023-10-31T00:00:00.000\" name=\"dcterms.date\"/>\n",
      "  <meta content=\"taxonomy : Topics / AI and ML / Artificial intelligence / AI models / Large language models\" name=\"ibm.search.facet.field_hierarchy_01\"/>\n",
      "  <meta name=\"ibm.search.facet.field_hierarchy_02\"/>\n",
      "  <meta content=\"taxonomy : Content Format / Article\" name=\"ibm.search.facet.field_hierarchy_03\"/>\n",
      "  <meta content=\"taxonomy : Content Type / Explainer\" name=\"ibm.search.facet.field_hierarchy_04\"/>\n",
      "  <meta name=\"ibm.search.facet.field_hierarchy_05\"/>\n",
      "  <meta content=\"Large language models\" name=\"ibm.search.facet.field_keyword_01\"/>\n",
      "  <meta content=\"https://prod-cloud-publish.aem.ibm.net/content/dam/connectedassets-adobe-cms/worldwide-content/creative-assets/s-migr/ul/g/64/ea/content-hub-analytic-page-leadspace-short.component.crop-thumbnail-16-by-9-retina.ts=0.png/content/adobe-cms/us/en/topics/langchain/jcr:content/root/leadspace\" name=\"ibm.search.facet.field_keyword_09\"/>\n",
      "  <meta name=\"ibm.search.facet.field_keyword_14\"/>\n",
      "  <meta content=\"31 October 2023\" name=\"ibm.search.facet.field_text_01\"/>\n",
      "  <meta content=\"Explainer\" name=\"ibm.search.facet.field_text_03\"/>\n",
      "  <link href=\"https://www.ibm.com/topics/langchain\" rel=\"canonical\"/>\n",
      "  <style id=\"anti-flicker-style\">\n",
      "   :not(:defined) {\n",
      "          visibility: hidden;\n",
      "        }\n",
      "  </style>\n",
      "  <!--    <sly data-sly-test=\"false\">\n",
      "        <link href=\"//1.www.s81c.com/common/v18/css/www.css\" rel=\"preload\" as=\"style\"/>\n",
      "        <link rel=\"stylesheet\" href=\"//1.www.s81c.com/common/v18/css/www.css\" media=\"all\"/>\n",
      "        <link href=\"//1.www.s81c.com/common/v18/css/grid-fluid.css\" rel=\"preload\" as=\"style\"/>\n",
      "        <link rel=\"stylesheet\" href=\"//1.www.s81c.com/common/v18/css/grid-fluid.css\" media=\"all\"/>\n",
      "        <link href=\"//1.www.s81c.com/common/v19a/css/www.css\" rel=\"stylesheet\"/>\n",
      "    </sly>-->\n",
      "  <script async=\"async\" src=\"https://assets.adobedtm.com/0f2de1d5b89a/565c2aeb0d39/launch-560e54b3e83c.min.js\" type=\"text/javascript\">\n",
      "  </script>\n",
      "  <script async=\"async\" type=\"text/javascript\">\n",
      "   var adobeDataLayer = window.adobeDataLayer || [];\n",
      "  </script>\n",
      "  <script type=\"text/javascript\">\n",
      "   window.searchKey = {\n",
      "            templateName: 'learn',\n",
      "            taxonomylist: 'taxonomy : Brands \\/ LangChain,taxonomy : Topics \\/ AI and ML \\/ Artificial intelligence \\/ AI models \\/ Large language models',\n",
      "            L0Tag: 'Topics,Brands,Industries,Geography,Compliance Entities,Products,Events,Computer Languages,Content Format,Series,Content Type,CMaaS Focus Areas,brands,Deployment Types,Product Types,Search Index',\n",
      "        }\n",
      "  </script>\n",
      "  <link href=\"https://www.ibm.com/it-it/topics/langchain\" hreflang=\"it-it\" rel=\"alternate\"/>\n",
      "  <link href=\"https://www.ibm.com/de-de/topics/langchain\" hreflang=\"de-de\" rel=\"alternate\"/>\n",
      "  <link href=\"https://www.ibm.com/topics/langchain\" hreflang=\"en-us\" rel=\"alternate\"/>\n",
      "  <link href=\"https://www.ibm.com/kr-ko/topics/langchain\" hreflang=\"ko-kr\" rel=\"alternate\"/>\n",
      "  <link href=\"https://www.ibm.com/br-pt/topics/langchain\" hreflang=\"pt-br\" rel=\"alternate\"/>\n",
      "  <link href=\"https://www.ibm.com/id-id/topics/langchain\" hreflang=\"id-id\" rel=\"alternate\"/>\n",
      "  <link href=\"https://www.ibm.com/es-es/topics/langchain\" hreflang=\"es-es\" rel=\"alternate\"/>\n",
      "  <link href=\"https://www.ibm.com/fr-fr/topics/langchain\" hreflang=\"fr-fr\" rel=\"alternate\"/>\n",
      "  <link href=\"https://www.ibm.com/cn-zh/topics/langchain\" hreflang=\"zh-cn\" rel=\"alternate\"/>\n",
      "  <link href=\"https://www.ibm.com/mx-es/topics/langchain\" hreflang=\"es-mx\" rel=\"alternate\"/>\n",
      "  <link href=\"https://www.ibm.com/jp-ja/topics/langchain\" hreflang=\"ja-jp\" rel=\"alternate\"/>\n",
      "  <script type=\"text/javascript\">\n",
      "   window.antiFlicker = {\n",
      "            active: true,\n",
      "            timeout: 3000\n",
      "        }\n",
      "  </script>\n",
      "  <script type=\"text/javascript\">\n",
      "   var languageCode = document.getElementsByName('languageCode')[0].content;\n",
      "        var countryCode = document.getElementsByName('countryCode')[0].content;\n",
      "        var focusArea = document.getElementsByName('focusArea')[0].content;\n",
      "        /* Define digital data object based on _appInfo object */\n",
      "          window._ibmAnalytics = {\n",
      "              settings: {\n",
      "                 name: \"AEM Sites\",\n",
      "                 tealiumProfileName: \"aem-sites\"\n",
      "              },\n",
      "           };\n",
      "        window.digitalData = {\n",
      "            page: {\n",
      "                category: {\n",
      "                    primaryCategory: '',\n",
      "                },\n",
      "                pageInfo: {\n",
      "                    language: languageCode + '-' + countryCode,\n",
      "                    ibm: {\n",
      "                        siteID: 'MarketingAEM',\n",
      "                        country: countryCode,\n",
      "                        messaging: {\n",
      "                            routing: {\n",
      "                                focusArea: focusArea,\n",
      "                                languageCode: languageCode,\n",
      "                                regionCode: countryCode\n",
      "                            },\n",
      "                            translation: {\n",
      "                                languageCode: languageCode,\n",
      "                                regionCode: countryCode\n",
      "                            }\n",
      "                        },\n",
      "                        sections: 0,\n",
      "                        patterns: 0,\n",
      "                    },\n",
      "                    carbon: {\n",
      "                        '@carbon/web-components': 'v1.47.1',\n",
      "                        '@carbon/ibmdotcom-web-components': 'v1.51.1',\n",
      "                        'carbon-for-aem': 'v0.19.0',\n",
      "                    },\n",
      "                },\n",
      "            },\n",
      "        };\n",
      "  </script>\n",
      "  <script type=\"application/ld+json\">\n",
      "   {\n",
      "      \"@context\": \"https://schema.org\",\n",
      "      \"@type\": \"BlogPosting\",\n",
      "      \"mainEntityOfPage\": {\n",
      "        \"@type\": \"WebPage\",\n",
      "        \"@id\": \"https://www.ibm.com/topics/langchain\"\n",
      "        },\n",
      "      \"headline\": \"LangChain\",\n",
      "      \"image\": [\n",
      "        \"https://www.ibm.com/content/dam/connectedassets-adobe-cms/worldwide-content/creative-assets/s-migr/ul/g/64/ea/content-hub-analytic-page-leadspace-short.png\"\n",
      "       ],\n",
      "      \"datePublished\": \"\",\n",
      "      \"dateModified\": \"2024-10-17T18:25:11.563Z\",\n",
      "      \"author\": {\"@type\": \"Organization\",\"name\": \"IBM\",\"url\": \"https://www.ibm.com\"}\n",
      "    }\n",
      "  </script>\n",
      "  <!-- Instana script specific to Dev environment start -->\n",
      "  <!--      <script>\n",
      "          (function(s,t,a,n){s[t]||(s[t]=a,n=s[a]=function(){n.q.push(arguments)},\n",
      "          n.q=[],n.v=2,n.l=1*new Date)})(window,\"InstanaEumObject\",\"ineum\");\n",
      "\n",
      "          ineum('reportingUrl', 'https://eum-orange-saas.instana.io');\n",
      "          ineum('key', 'mFJnYtpBSiCKl9zenWHXtg');\n",
      "          ineum('trackSessions');\n",
      "      </script>\n",
      "      <script defer crossorigin=\"anonymous\" src=https://eum.instana.io/1.7.2/eum.min.js\n",
      "              integrity=\"sha384-cgeSlevgebehPauohUhsnAeBrpjXzaj94mSv3L2EXjCQH0RRb9xSQ2ErGOWkthIJ\"></script>-->\n",
      "  <!-- Instana script specific to Dev environment end -->\n",
      "  <meta content=\"https://www.ibm.com/topics/langchain\" property=\"og:url\"/>\n",
      "  <meta content=\"website\" property=\"og:type\"/>\n",
      "  <meta content=\"What Is LangChain? | IBM\" property=\"og:title\"/>\n",
      "  <meta content=\"LangChain is an open source orchestration framework for the development of applications using large language models (LLMs), like chatbots and virtual agents. \" property=\"og:description\"/>\n",
      "  <meta content=\"https://assets.ibm.com/is/image/ibm/ibm-8bar-logo-2560x2560?$original$\" property=\"og:image\"/>\n",
      "  <script async=\"async\" src=\"https://1.www.s81c.com/common/stats/ibm-common.js\" type=\"text/javascript\">\n",
      "  </script>\n",
      "  <script src=\"https://hybrid-cloud-widgets-production.s3.us.cloud-object-storage.appdomain.cloud/loader.js\" type=\"application/javascript\">\n",
      "  </script>\n",
      "  <link href=\"/etc.clientlibs/adobe-cms/clientlibs/clientlib-idlStylesCarbon.lc-92a431776e26f162dd2424f414af0c9a-lc.min.css\" rel=\"stylesheet\" type=\"text/css\"/>\n",
      "  <link href=\"/etc.clientlibs/adobe-cms/clientlibs/clientlib-idlStyles.lc-781111ab1b5d0bf5e1a3598eac05aba5-lc.min.css\" rel=\"stylesheet\" type=\"text/css\"/>\n",
      "  <link href=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/tag/v1/latest/plex.css\" rel=\"stylesheet\">\n",
      "   <link href=\"https://1.www.s81c.com/common/carbon/plex/sans.css\" rel=\"stylesheet\">\n",
      "    <link href=\"/etc.clientlibs/adobe-cms/clientlibs/clientlib-pdfviewer.lc-c79c6cf22a5b660f493131a4be3ded5f-lc.min.css\" rel=\"stylesheet\" type=\"text/css\"/>\n",
      "   </link>\n",
      "  </link>\n",
      " </head>\n",
      " <body class=\"content-page page basicpage publish\" data-fullwidthtemplate=\"false\" id=\"content-page-8d88ea7913\">\n",
      "  <dds-video-cta-container>\n",
      "   <div class=\"root container responsivegrid\">\n",
      "    <div class=\"cmp-container\" id=\"container-e9b80bd4f1\">\n",
      "     <div class=\"masthead\">\n",
      "      <c4d-masthead-container auth-method=\"profile-api\" data-endpoint=\"#\" has-cart=\"\" has-contact=\"true\" has-profile=\"\" has-search=\"\">\n",
      "      </c4d-masthead-container>\n",
      "      <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v2.15.0/masthead.min.js\" type=\"module\">\n",
      "      </script>\n",
      "      <script>\n",
      "       let l0Json = \"{\\x22profileMenu\\x22:{\\x22signedout\\x22:[{\\x22title\\x22:\\x22My IBM\\x22,\\x22url\\x22:\\x22https:\\/\\/myibm.ibm.com\\/?lnk=mmi\\x22},{\\x22title\\x22:\\x22Log in\\x22,\\x22url\\x22:\\x22https:\\/\\/login.ibm.com\\/oidc\\/endpoint\\/default\\/authorize?redirect_uri=https%3A%2F%2Fmyibm.ibm.com%2FOIDCHandler.html\\x26response_type=token\\x26client_id=v18LoginProdCI\\x26scope=openid\\x26state=https%3A%2F%2Fwww.ibm.com\\x26nonce=8675309\\x22}],\\x22signedin\\x22:[{\\x22title\\x22:\\x22My IBM\\x22,\\x22url\\x22:\\x22https:\\/\\/myibm.ibm.com\\/?lnk=mmi\\x22},{\\x22title\\x22:\\x22Log in\\x22,\\x22url\\x22:\\x22https:\\/\\/login.ibm.com\\/oidc\\/endpoint\\/default\\/authorize?redirect_uri=https%3A%2F%2Fmyibm.ibm.com%2FOIDCHandler.html\\x26response_type=token\\x26client_id=v18LoginProdCI\\x26scope=openid\\x26state=https%3A%2F%2Fwww.ibm.com\\x26nonce=8675309\\x22}]},\\x22mastheadNav\\x22:{\\x22links\\x22:[{\\x22title\\x22:\\x22Products\\x22,\\x22url\\x22:\\x22\\x22,\\x22submenu\\x22:{\\x22sections\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22Featured\\x22,\\x22description\\x22:\\x22\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22API Connect\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/api\\u002Dconnect?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software to rapidly create, protect, socialize and manage APIs\\x22},{\\x22title\\x22:\\x22Concert\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/concert?lnk=flatitem\\x22,\\x22description\\x22:\\x22AI to analyze and manage IT infrastructure using natural language\\x22},{\\x22title\\x22:\\x22Environmental Intelligence\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/environmental\\u002Dintelligence?lnk=flatitem\\x22,\\x22description\\x22:\\x22SaaS for predicting and responding to weather and climate events\\x22},{\\x22title\\x22:\\x22Envizi\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/envizi?lnk=flatitem\\x22,\\x22description\\x22:\\x22ESG data management, reporting and analysis SaaS\\x22},{\\x22title\\x22:\\x22FlashSystem\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/flashsystem?lnk=flatitem\\x22,\\x22description\\x22:\\x22Primary storage for performance and latency sensitive workloads\\x22},{\\x22title\\x22:\\x22Hybrid Cloud Mesh\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/hybrid\\u002Dcloud\\u002Dmesh?lnk=flatitem\\x22,\\x22description\\x22:\\x22SaaS for deploying and managing hybrid and multicloud networks\\x22},{\\x22title\\x22:\\x22IBM Cloud\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/cloud?lnk=flatitem\\x22,\\x22description\\x22:\\x22On\\u002Ddemand cloud computing platform and APIs\\x22},{\\x22title\\x22:\\x22IBM Z\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/z?lnk=flatitem\\x22,\\x22description\\x22:\\x22Flagship mainframe with on\\u002Dchip AI and quantum\\u002Dsafe cryptography\\x22},{\\x22title\\x22:\\x22Instana\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/instana?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software for application performance monitoring and automation\\x22},{\\x22title\\x22:\\x22MaaS360\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/maas360?lnk=flatitem\\x22,\\x22description\\x22:\\x22Unified endpoint management software for many device types\\x22},{\\x22title\\x22:\\x22Maximo\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/maximo?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software for asset management and related workflows\\x22},{\\x22title\\x22:\\x22Planning Analytics\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/planning\\u002Danalytics?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software to automate financial and operational planning\\x22},{\\x22title\\x22:\\x22Robotic Process Automation (RPA)\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/robotic\\u002Dprocess\\u002Dautomation?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software to automate workflows and business processes\\x22},{\\x22title\\x22:\\x22Storage Defender\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/storage\\u002Ddefender?lnk=flatitem\\x22,\\x22description\\x22:\\x22Data resiliency software for threat detection and data recovery\\x22},{\\x22title\\x22:\\x22Turbonomic\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/turbonomic?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software to manage and optimize IT resource usage\\x22},{\\x22title\\x22:\\x22watsonx\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/watsonx?lnk=flatitem\\x22,\\x22description\\x22:\\x22AI and data platform\\x22},{\\x22title\\x22:\\x22watsonx Assistant\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/watsonx\\u002Dassistant?lnk=flatitem\\x22,\\x22description\\x22:\\x22Virtual agents customizable to any domain\\x22},{\\x22title\\x22:\\x22watsonx Orchestrate\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/watsonx\\u002Dorchestrate?lnk=flatitem\\x22,\\x22description\\x22:\\x22Personal\\u002Dassistant software that automates repetitive tasks\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22AI \\x26 machine learning\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/artificial\\u002Dintelligence?lnk=flathl\\x22,\\x22description\\x22:\\x22Use IBM Watson’s AI or build your own machine learning models\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Cloud Pak for AIOps\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/cloud\\u002Dpak\\u002Dfor\\u002Daiops?lnk=flatitem\\x22,\\x22description\\x22:\\x22DevOps management tool with AI analysis and recommendations\\x22},{\\x22title\\x22:\\x22Cloud Pak for Data\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/cloud\\u002Dpak\\u002Dfor\\u002Ddata?lnk=flatitem\\x22,\\x22description\\x22:\\x22Tools for data analysis, organization and management\\x22},{\\x22title\\x22:\\x22Knowledge Catalog\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/knowledge\\u002Dcatalog?lnk=flatitem\\x22,\\x22description\\x22:\\x22SaaS to catalog data, AI models, metadata, policies and more\\x22},{\\x22title\\x22:\\x22Watson Discovery\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/watson\\u002Ddiscovery?lnk=flatitem\\x22,\\x22description\\x22:\\x22AI to search in and answer questions about business documents\\x22},{\\x22title\\x22:\\x22Watson Natural Language Understanding\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/natural\\u002Dlanguage\\u002Dunderstanding?lnk=flatitem\\x22,\\x22description\\x22:\\x22API for text analysis and metadata extraction\\x22},{\\x22title\\x22:\\x22Watson Speech to Text\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/speech\\u002Dto\\u002Dtext?lnk=flatitem\\x22,\\x22description\\x22:\\x22API for real\\u002Dtime speech recognition and transcription\\x22},{\\x22title\\x22:\\x22Watson Studio\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/watson\\u002Dstudio?lnk=flatitem\\x22,\\x22description\\x22:\\x22IDE to build, run and manage AI models\\x22},{\\x22title\\x22:\\x22Watson Text to Speech\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/text\\u002Dto\\u002Dspeech?lnk=flatitem\\x22,\\x22description\\x22:\\x22API for real\\u002Dtime text to speech conversion\\x22},{\\x22title\\x22:\\x22Z Anomaly Analytics\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/z\\u002Danomaly\\u002Danalytics?lnk=flatitem\\x22,\\x22description\\x22:\\x22Operational anomaly detection software for mainframes\\x22},{\\x22title\\x22:\\x22watsonx\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/watsonx?lnk=flatitem\\x22,\\x22description\\x22:\\x22AI and data platform\\x22},{\\x22title\\x22:\\x22watsonx Assistant\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/watsonx\\u002Dassistant?lnk=flatitem\\x22,\\x22description\\x22:\\x22Virtual agents customizable to any domain\\x22},{\\x22title\\x22:\\x22watsonx Code Assistant\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/watsonx\\u002Dcode\\u002Dassistant?lnk=flatitem\\x22,\\x22description\\x22:\\x22AI tool to generate code\\x22},{\\x22title\\x22:\\x22watsonx Orders\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/watsonx\\u002Dorders?lnk=flatitem\\x22,\\x22description\\x22:\\x22AI voice agent for taking restaurant orders\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Analytics\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/analytics?lnk=flathl\\x22,\\x22description\\x22:\\x22Aggregate and analyze large datasets\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Business Analytics Enterprise\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/business\\u002Danalytics\\u002Denterprise?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software for business planning and analysis\\x22},{\\x22title\\x22:\\x22CPLEX\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/ilog\\u002Dcplex\\u002Doptimization\\u002Dstudio?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software to build and solve complex optimization models\\x22},{\\x22title\\x22:\\x22Cloud Pak for Data\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/cloud\\u002Dpak\\u002Dfor\\u002Ddata?lnk=flatitem\\x22,\\x22description\\x22:\\x22Tools for data analysis, organization and management\\x22},{\\x22title\\x22:\\x22Cognos Analytics\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/cognos\\u002Danalytics?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software for business intelligence and performance management\\x22},{\\x22title\\x22:\\x22Databand\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/databand?lnk=flatitem\\x22,\\x22description\\x22:\\x22Data observability software for data engineers and DataOps teams\\x22},{\\x22title\\x22:\\x22InfoSphere Information Server\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/information\\u002Dserver?lnk=flatitem\\x22,\\x22description\\x22:\\x22Data integration suite for ETL, governance and analysis\\x22},{\\x22title\\x22:\\x22Manta Data Lineage\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/manta\\u002Ddata\\u002Dlineage?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software to visualize the flow of data from origin to consumption\\x22},{\\x22title\\x22:\\x22Netezza\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/netezza?lnk=flatitem\\x22,\\x22description\\x22:\\x22Data warehousing and analytics system on custom hardware\\x22},{\\x22title\\x22:\\x22Optim\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/infosphere\\u002Doptim?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software to manage test, production, and archived data\\x22},{\\x22title\\x22:\\x22Planning Analytics\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/planning\\u002Danalytics?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software to automate financial and operational planning\\x22},{\\x22title\\x22:\\x22SPSS\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/spss?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software for statistical analysis and business intelligence\\x22},{\\x22title\\x22:\\x22Spectrum Computing\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/high\\u002Dperformance\\u002Dcomputing?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software to optimize resources in complex computing clusters\\x22},{\\x22title\\x22:\\x22StreamSets\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/streamsets?lnk=flatitem\\x22,\\x22description\\x22:\\x22Graphical interface to build and manage streaming data pipelines\\x22},{\\x22title\\x22:\\x22Watson Discovery\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/watson\\u002Ddiscovery?lnk=flatitem\\x22,\\x22description\\x22:\\x22AI to search in and answer questions about business documents\\x22},{\\x22title\\x22:\\x22Watson Studio\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/watson\\u002Dstudio?lnk=flatitem\\x22,\\x22description\\x22:\\x22IDE to build, run and manage AI models\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Asset lifecycle management\\x22,\\x22description\\x22:\\x22Manage and maintain physical assets\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Environmental Intelligence\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/environmental\\u002Dintelligence?lnk=flatitem\\x22,\\x22description\\x22:\\x22SaaS for predicting and responding to weather and climate events\\x22},{\\x22title\\x22:\\x22Envizi\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/envizi?lnk=flatitem\\x22,\\x22description\\x22:\\x22ESG data management, reporting and analysis SaaS\\x22},{\\x22title\\x22:\\x22Maximo\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/maximo?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software for asset management and related workflows\\x22},{\\x22title\\x22:\\x22TRIRIGA\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/tririga?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software for real estate and workplace management\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Business automation\\x22,\\x22description\\x22:\\x22Automate workflows and business processes\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Blueworks Live\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/blueworkslive?lnk=flatitem\\x22,\\x22description\\x22:\\x22SaaS for collaboratively modeling business processes\\x22},{\\x22title\\x22:\\x22Business Automation Manager Open Editions\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/business\\u002Dautomation\\u002Dmanager\\u002Dopen\\u002Deditions?lnk=flatitem\\x22,\\x22description\\x22:\\x22Workflow automation and decision management software\\x22},{\\x22title\\x22:\\x22Business Automation Workflow\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/business\\u002Dautomation\\u002Dworkflow?lnk=flatitem\\x22,\\x22description\\x22:\\x22Cloud Pak for Business Automation add\\u002Don to automate workflows\\x22},{\\x22title\\x22:\\x22Cloud Pak for Business Automation\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/cloud\\u002Dpak\\u002Dfor\\u002Dbusiness\\u002Dautomation?lnk=flatitem\\x22,\\x22description\\x22:\\x22Operations management software with AI insights\\x22},{\\x22title\\x22:\\x22Datacap\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/data\\u002Dcapture\\u002Dand\\u002Dimaging?lnk=flatitem\\x22,\\x22description\\x22:\\x22Document management for IBM Cloud Pak for Business Automation\\x22},{\\x22title\\x22:\\x22Event Automation\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/event\\u002Dautomation?lnk=flatitem\\x22,\\x22description\\x22:\\x22No\\u002Dcode tool for building automated event driven workflows\\x22},{\\x22title\\x22:\\x22FileNet Content Manager\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/filenet\\u002Dcontent\\u002Dmanager?lnk=flatitem\\x22,\\x22description\\x22:\\x22Content management solution for Cloud Pak for Business Automation\\x22},{\\x22title\\x22:\\x22Operational Decision Manager\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/operational\\u002Ddecision\\u002Dmanager?lnk=flatitem\\x22,\\x22description\\x22:\\x22Cloud Pak for Business Automation addon for rule\\u002Dbased decisions\\x22},{\\x22title\\x22:\\x22Process Mining\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/process\\u002Dmining?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software to extract process data from business applications\\x22},{\\x22title\\x22:\\x22Robotic Process Automation (RPA)\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/robotic\\u002Dprocess\\u002Dautomation?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software to automate workflows and business processes\\x22},{\\x22title\\x22:\\x22watsonx Orchestrate\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/watsonx\\u002Dorchestrate?lnk=flatitem\\x22,\\x22description\\x22:\\x22Personal\\u002Dassistant software that automates repetitive tasks\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Containers\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/containers?lnk=flathl\\x22,\\x22description\\x22:\\x22Allocate compute infrastructure on demand\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Cloud Satellite\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/satellite?lnk=flatitem\\x22,\\x22description\\x22:\\x22Platform for running managed cloud services on any infrastructure\\x22},{\\x22title\\x22:\\x22Code Engine\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/code\\u002Dengine?lnk=flatitem\\x22,\\x22description\\x22:\\x22Managed serverless runtime for containers, batch jobs, or code\\x22},{\\x22title\\x22:\\x22Container Registry\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/container\\u002Dregistry?lnk=flatitem\\x22,\\x22description\\x22:\\x22Private SaaS registry for container storage and management\\x22},{\\x22title\\x22:\\x22Kubernetes Service\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/kubernetes\\u002Dservice?lnk=flatitem\\x22,\\x22description\\x22:\\x22Managed Kubernetes for high availability container deployments\\x22},{\\x22title\\x22:\\x22Red Hat OpenShift on IBM Cloud\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/openshift?lnk=flatitem\\x22,\\x22description\\x22:\\x22Managed service with tools for security, management and monitoring\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Databases\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/database?lnk=flathl\\x22,\\x22description\\x22:\\x22Store, query and analyze structured data\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Cloudant\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/cloudant?lnk=flatitem\\x22,\\x22description\\x22:\\x22Managed PCI\\u002Dcompliant JSON document store on Apache CouchDB\\x22},{\\x22title\\x22:\\x22Db2\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/db2\\u002Ddatabase?lnk=flatitem\\x22,\\x22description\\x22:\\x22Data management software including IBM Db2 Database\\x22},{\\x22title\\x22:\\x22Db2 for z\\/OS\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/db2\\u002Dfor\\u002Dzos?lnk=flatitem\\x22,\\x22description\\x22:\\x22IBM Db2 Database for IBM Z mainframes\\x22},{\\x22title\\x22:\\x22IBM Cloud Databases for Elastic Search\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/databases\\u002Dfor\\u002Delasticsearch?lnk=flatitem\\x22,\\x22description\\x22:\\x22JSON document store for full\\u002Dtext search\\x22},{\\x22title\\x22:\\x22IBM Cloud Databases for EnterpriseDB\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/databases\\u002Dfor\\u002Denterprisedb?lnk=flatitem\\x22,\\x22description\\x22:\\x22PostgreSQL with added performance, security and management features\\x22},{\\x22title\\x22:\\x22IBM Cloud Databases for MongoDB\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/databases\\u002Dfor\\u002Dmongodb?lnk=flatitem\\x22,\\x22description\\x22:\\x22Managed NoSQL JSON document store\\x22},{\\x22title\\x22:\\x22IBM Cloud Databases for MySQL\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/databases\\u002Dfor\\u002Dmysql?lnk=flatitem\\x22,\\x22description\\x22:\\x22Managed SQL database\\x22},{\\x22title\\x22:\\x22IBM Cloud Databases for PostgreSQL\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/databases\\u002Dfor\\u002Dpostgresql?lnk=flatitem\\x22,\\x22description\\x22:\\x22Managed SQL database with advanced features\\x22},{\\x22title\\x22:\\x22IBM Cloud Databases for etcd\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/databases\\u002Dfor\\u002Detcd?lnk=flatitem\\x22,\\x22description\\x22:\\x22Distributed key\\u002Dvalue store\\x22},{\\x22title\\x22:\\x22Information Management System (IMS)\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/ims?lnk=flatitem\\x22,\\x22description\\x22:\\x22Database and management system for high\\u002Dvolume transactions\\x22},{\\x22title\\x22:\\x22Informix\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/informix?lnk=flatitem\\x22,\\x22description\\x22:\\x22Embeddable database for SQL, NoSQL, time\\u002Dseries and spatial data\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22DevOps\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/devops?lnk=flathl\\x22,\\x22description\\x22:\\x22Manage infrastructure, environments and deployments\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Application Delivery Foundation for z\\/OS\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/app\\u002Ddelivery\\u002Dfoundation\\u002Dfor\\u002Dzos?lnk=flatitem\\x22,\\x22description\\x22:\\x22Application development and DevOps tools for z\\/OS\\x22},{\\x22title\\x22:\\x22Application Discovery and Delivery Intelligence\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/app\\u002Ddiscovery\\u002Dand\\u002Ddelivery\\u002Dintelligence?lnk=flatitem\\x22,\\x22description\\x22:\\x22Analysis tool for visualizing applications, data and jobs on z\\/OS\\x22},{\\x22title\\x22:\\x22Cloud Continuous Delivery\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/continuous\\u002Ddelivery?lnk=flatitem\\x22,\\x22description\\x22:\\x22Toolchains to automate building and deploying applications\\x22},{\\x22title\\x22:\\x22Cloud Pak for AIOps\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/cloud\\u002Dpak\\u002Dfor\\u002Daiops?lnk=flatitem\\x22,\\x22description\\x22:\\x22DevOps management tool with AI analysis and recommendations\\x22},{\\x22title\\x22:\\x22DevOps Build\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/devops\\u002Dbuild?lnk=flatitem\\x22,\\x22description\\x22:\\x22Management tool for configuring and running software builds\\x22},{\\x22title\\x22:\\x22DevOps Code ClearCase\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/devops\\u002Dcode\\u002Dclearcase?lnk=flatitem\\x22,\\x22description\\x22:\\x22Configuration management system for software asset access control\\x22},{\\x22title\\x22:\\x22DevOps Deploy\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/devops\\u002Ddeploy?lnk=flatitem\\x22,\\x22description\\x22:\\x22Automation software for continuous delivery and deployment\\x22},{\\x22title\\x22:\\x22DevOps Test\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/devops\\u002Dtest?lnk=flatitem\\x22,\\x22description\\x22:\\x22Continuous testing and virtualization platform\\x22},{\\x22title\\x22:\\x22DevOps Velocity\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/devops\\u002Dvelocity?lnk=flatitem\\x22,\\x22description\\x22:\\x22Release management software for pipeline orchestration and analytics\\x22},{\\x22title\\x22:\\x22Engineering Lifecycle Management\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/engineering\\u002Dlifecycle\\u002Dmanagement?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software for product and application lifecycle management\\x22},{\\x22title\\x22:\\x22IBM i Modernization Engine\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/ibm\\u002Di\\u002Dmerlin?lnk=flatitem\\x22,\\x22description\\x22:\\x22Development and modernization tools for IBM i applications\\x22},{\\x22title\\x22:\\x22Rational ClearQuest\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/rational\\u002Dclearquest?lnk=flatitem\\x22,\\x22description\\x22:\\x22Change management software\\x22},{\\x22title\\x22:\\x22Rational Software Architect Designer\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/rational\\u002Dsoftware\\u002Darchitect\\u002Ddesigner?lnk=flatitem\\x22,\\x22description\\x22:\\x22Tools to create, evaluate, and communicate software architecture\\x22},{\\x22title\\x22:\\x22Test Accelerator for Z\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/test\\u002Daccelerator\\u002Dz?lnk=flatitem\\x22,\\x22description\\x22:\\x22Test automation and test generation framework for z\\/OS\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22IT automation\\x22,\\x22description\\x22:\\x22Automate IT infrastructure management\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Apptio\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/apptio?lnk=flatitem\\x22,\\x22description\\x22:\\x22Hybrid cloud financial management and planning software\\x22},{\\x22title\\x22:\\x22Cloud App Configuration\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/app\\u002Dconfiguration?lnk=flatitem\\x22,\\x22description\\x22:\\x22Feature management and configuration service for web and mobile apps\\x22},{\\x22title\\x22:\\x22Cloud App ID\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/app\\u002Did?lnk=flatitem\\x22,\\x22description\\x22:\\x22Authentication and user management service for web and mobile apps\\x22},{\\x22title\\x22:\\x22Cloud Monitoring\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/cloud\\u002Dmonitoring?lnk=flatitem\\x22,\\x22description\\x22:\\x22Hosted monitoring tool for organization\\u002Dwide system performance\\x22},{\\x22title\\x22:\\x22Cloud Pak for AIOps\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/cloud\\u002Dpak\\u002Dfor\\u002Daiops?lnk=flatitem\\x22,\\x22description\\x22:\\x22DevOps management tool with AI analysis and recommendations\\x22},{\\x22title\\x22:\\x22Cloud Schematics\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/schematics?lnk=flatitem\\x22,\\x22description\\x22:\\x22Management and provisionining of cloud infrastructure with code\\x22},{\\x22title\\x22:\\x22Concert\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/concert?lnk=flatitem\\x22,\\x22description\\x22:\\x22AI to analyze and manage IT infrastructure using natural language\\x22},{\\x22title\\x22:\\x22Event Notifications\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/event\\u002Dnotifications?lnk=flatitem\\x22,\\x22description\\x22:\\x22Notification and workflow automation service for IBM Cloud events\\x22},{\\x22title\\x22:\\x22Flexera One\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/flexera\\u002Done?lnk=flatitem\\x22,\\x22description\\x22:\\x22IT asset management software\\x22},{\\x22title\\x22:\\x22IBM Cloud Logs\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/cloud\\u002Dlogs?lnk=flatitem\\x22,\\x22description\\x22:\\x22Logging and observability service for applications and infrastructure\\x22},{\\x22title\\x22:\\x22Instana\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/instana?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software for application performance monitoring and automation\\x22},{\\x22title\\x22:\\x22Turbonomic\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/turbonomic?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software to manage and optimize IT resource usage\\x22},{\\x22title\\x22:\\x22Z IntelliMagic Vision for z\\/OS\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/z\\u002Dintellimagic\\u002Dvision\\u002Dfor\\u002Dzos?lnk=flatitem\\x22,\\x22description\\x22:\\x22Infrastructure performance management and monitoring software\\x22},{\\x22title\\x22:\\x22Z Service Management Suite\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/z\\u002Dservice\\u002Dmanagement\\u002Dsuite?lnk=flatitem\\x22,\\x22description\\x22:\\x22Monitoring, control, and automation suite for Z\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Middleware\\x22,\\x22description\\x22:\\x22Connect your applications, data and events\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22API Connect\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/api\\u002Dconnect?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software to rapidly create, protect, socialize and manage APIs\\x22},{\\x22title\\x22:\\x22App Connect\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/app\\u002Dconnect?lnk=flatitem\\x22,\\x22description\\x22:\\x22Codeless connectors for your data, apps and APIs\\x22},{\\x22title\\x22:\\x22Aspera\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/aspera?lnk=flatitem\\x22,\\x22description\\x22:\\x22System to transfer, sync or stream huge data globally\\x22},{\\x22title\\x22:\\x22CICS Transaction Server for z\\/OS\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/cics\\u002Dtransaction\\u002Dserver?lnk=flatitem\\x22,\\x22description\\x22:\\x22Application server for online transaction processing\\x22},{\\x22title\\x22:\\x22Cloud Pak for Applications\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/cloud\\u002Dpak\\u002Dfor\\u002Dapplications?lnk=flatitem\\x22,\\x22description\\x22:\\x22Kubernetes\\u002Dbased platform for hybrid cloud applications\\x22},{\\x22title\\x22:\\x22Cloud Pak for Integration\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/cloud\\u002Dpak\\u002Dfor\\u002Dintegration?lnk=flatitem\\x22,\\x22description\\x22:\\x22Tools to connect all of your apps, data and events\\x22},{\\x22title\\x22:\\x22DataPower\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/datapower\\u002Dgateway?lnk=flatitem\\x22,\\x22description\\x22:\\x22Physical or virtual multi\\u002Dprotocol network gateway\\x22},{\\x22title\\x22:\\x22Event Streams\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/event\\u002Dstreams?lnk=flatitem\\x22,\\x22description\\x22:\\x22PaaS stream processing based on Apache Kafka\\x22},{\\x22title\\x22:\\x22IBM Cloud Messages for RabbitMQ\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/messages\\u002Dfor\\u002Drabbitmq?lnk=flatitem\\x22,\\x22description\\x22:\\x22Managed open\\u002Dsource message broker\\x22},{\\x22title\\x22:\\x22IBM webMethods\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/webmethods?lnk=flatitem\\x22,\\x22description\\x22:\\x22iPaaS capabilities including app integration, API management, B2B integration and managed file transfer\\x22},{\\x22title\\x22:\\x22Information Management System\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/ims?lnk=flatitem\\x22,\\x22description\\x22:\\x22Hierarchical database and transaction processor for Z\\x22},{\\x22title\\x22:\\x22MQ\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/mq?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software and hardware for messaging between distributed systems\\x22},{\\x22title\\x22:\\x22z\\/OS Connect\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/zos\\u002Dconnect?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software to build and serve RESTful APIs to z\\/OS applications\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Network\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/networking?lnk=flathl\\x22,\\x22description\\x22:\\x22Run and manage public, private and virtual networks\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Cloud Pak for Network Automation\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/cloud\\u002Dpak\\u002Dfor\\u002Dnetwork\\u002Dautomation?lnk=flatitem\\x22,\\x22description\\x22:\\x22Management software for telco network operations\\x22},{\\x22title\\x22:\\x22Hybrid Cloud Mesh\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/hybrid\\u002Dcloud\\u002Dmesh?lnk=flatitem\\x22,\\x22description\\x22:\\x22SaaS for deploying and managing hybrid and multicloud networks\\x22},{\\x22title\\x22:\\x22NS1 Connect\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/ns1\\u002Dconnect?lnk=flatitem\\x22,\\x22description\\x22:\\x22Authoritative DNS and traffic steering as a service\\x22},{\\x22title\\x22:\\x22Rapid Network Automation\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/rapid\\u002Dnetwork\\u002Dautomation?lnk=flatitem\\x22,\\x22description\\x22:\\x22No\\u002Dcode tool to build automation workflows across apps and APIs\\x22},{\\x22title\\x22:\\x22SevOne Network Performance Management\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/sevone\\u002Dnetwork\\u002Dperformance\\u002Dmanagement?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software to collect and analyze network performance data\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Operating systems\\x22,\\x22description\\x22:\\x22Run critical workloads on specialized operating systems\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22AIX\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/aix?lnk=flatitem\\x22,\\x22description\\x22:\\x22UNIX operating system for Power servers\\x22},{\\x22title\\x22:\\x22IBM i\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/ibm\\u002Di?lnk=flatitem\\x22,\\x22description\\x22:\\x22Integrated operating system for Power servers\\x22},{\\x22title\\x22:\\x22Linux\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/linux?lnk=flatitem\\x22,\\x22description\\x22:\\x22Run Linux workloads on IBM servers\\x22},{\\x22title\\x22:\\x22z\\/OS\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/zos?lnk=flatitem\\x22,\\x22description\\x22:\\x22Flagship OS for continuous, high\\u002Dvolume mainframe operation\\x22},{\\x22title\\x22:\\x22z\\/TPF\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/z\\u002Dtransaction\\u002Dprocessing\\u002Dfacility?lnk=flatitem\\x22,\\x22description\\x22:\\x22Real\\u002Dtime operating system for mainframes\\x22},{\\x22title\\x22:\\x22z\\/VM\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/zvm?lnk=flatitem\\x22,\\x22description\\x22:\\x22Hypervisor and virtualization software for IBM Z and LinuxONE\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Quantum\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/quantum?lnk=flathl\\x22,\\x22description\\x22:\\x22Run code on real quantum systems using a full\\u002Dstack SDK\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Qiskit Runtime\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/quantum\\/qiskit?lnk=flatitem\\x22,\\x22description\\x22:\\x22Cloud service for executing quantum workloads at scale\\x22},{\\x22title\\x22:\\x22Quantum Safe\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/quantum\\/quantum\\u002Dsafe?lnk=flatitem\\x22,\\x22description\\x22:\\x22Technology and consulting services for quantum\\u002Dsafe cryptography\\x22},{\\x22title\\x22:\\x22Quantum Systems\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/quantum\\/technology?lnk=flatitem\\x22,\\x22description\\x22:\\x22Quantum computing systems accessible on the cloud\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Security \\x26 identity\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/security?lnk=flathl\\x22,\\x22description\\x22:\\x22Cloud\\u002Dnative software to secure resources and simplify compliance\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Cloud AppID\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/app\\u002Did?lnk=flatitem\\x22,\\x22description\\x22:\\x22Authentication and user profiles as a service for mobile and web apps\\x22},{\\x22title\\x22:\\x22Cloud Security and Compliance Center\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/security\\u002Dand\\u002Dcompliance\\u002Dcenter?lnk=flatitem\\x22,\\x22description\\x22:\\x22SaaS to define and audit the compliance posture of your cloud\\x22},{\\x22title\\x22:\\x22Data Security Broker\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/data\\u002Dsecurity\\u002Dbroker?lnk=flatitem\\x22,\\x22description\\x22:\\x22Administrative console to enable data encryption in the cloud\\x22},{\\x22title\\x22:\\x22Guardium\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/guardium?lnk=flatitem\\x22,\\x22description\\x22:\\x22Suite for data encryption, management and simplified compliance\\x22},{\\x22title\\x22:\\x22Key Protect for IBM Cloud\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/key\\u002Dprotect?lnk=flatitem\\x22,\\x22description\\x22:\\x22Tool to provision and store keys for apps across IBM Cloud services\\x22},{\\x22title\\x22:\\x22MaaS360\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/maas360?lnk=flatitem\\x22,\\x22description\\x22:\\x22Unified endpoint management for mobile workforces\\x22},{\\x22title\\x22:\\x22QRadar EDR\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/qradar\\u002Dedr?lnk=flatitem\\x22,\\x22description\\x22:\\x22Advanced AI threat detection and response for endpoints\\x22},{\\x22title\\x22:\\x22QRadar SIEM\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/qradar\\u002Dsiem?lnk=flatitem\\x22,\\x22description\\x22:\\x22Threat detection and prioritization for real\\u002Dtime visibility\\x22},{\\x22title\\x22:\\x22QRadar SOAR\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/qradar\\u002Dsoar?lnk=flatitem\\x22,\\x22description\\x22:\\x22Automation and workflow management for security operations\\x22},{\\x22title\\x22:\\x22Trusteer\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/trusteer?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software to authenticate customers, detect fraud and assess risk\\x22},{\\x22title\\x22:\\x22Verify\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/verify?lnk=flatitem\\x22,\\x22description\\x22:\\x22Identity, authentication, and access control software\\x22},{\\x22title\\x22:\\x22zSecure\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/zsecure?lnk=flatitem\\x22,\\x22description\\x22:\\x22Security, risk management and compliance tools for Z hardware\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Servers\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/servers?lnk=flathl\\x22,\\x22description\\x22:\\x22Run workloads on hybrid cloud infrastructure\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22IBM Cloud Bare Metal Servers\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/bare\\u002Dmetal\\u002Dservers?lnk=flatitem\\x22,\\x22description\\x22:\\x22Dedicated hardware for maximum performance\\x22},{\\x22title\\x22:\\x22IBM Cloud Virtual Servers for VPC\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/virtual\\u002Dservers?lnk=flatitem\\x22,\\x22description\\x22:\\x22Virtual servers with Intel Xeon or IBM Z CPUs\\x22},{\\x22title\\x22:\\x22LinuxONE 4\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/linuxone\\u002D4?lnk=flatitem\\x22,\\x22description\\x22:\\x22Physical and virtual servers for running Linux\\x22},{\\x22title\\x22:\\x22Power\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/power?lnk=flatitem\\x22,\\x22description\\x22:\\x22Physical and virtual servers with IBM Power CPUs\\x22},{\\x22title\\x22:\\x22z16\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/z16?lnk=flatitem\\x22,\\x22description\\x22:\\x22Flagship mainframe with on\\u002Dchip AI and quantum\\u002Dsafe cryptography\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Storage\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/storage?lnk=flathl\\x22,\\x22description\\x22:\\x22Software, hardware and services for critical and operational workloads and data\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Ceph\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/storage\\u002Dceph?lnk=flatitem\\x22,\\x22description\\x22:\\x22Multi\\u002Dprotocol software\\u002Ddefined enterprise storage platform\\x22},{\\x22title\\x22:\\x22Cloud Block Storage\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/block\\u002Dstorage?lnk=flatitem\\x22,\\x22description\\x22:\\x22Managed service optimized for low latency and high transfer rates\\x22},{\\x22title\\x22:\\x22Cloud File Storage\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/file\\u002Dstorage?lnk=flatitem\\x22,\\x22description\\x22:\\x22Managed service for flash\\u002Dbacked, durable, NFS\\u002Dbased file storage\\x22},{\\x22title\\x22:\\x22Cloud Object Storage\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/cloud\\u002Dobject\\u002Dstorage?lnk=flatitem\\x22,\\x22description\\x22:\\x22Unstructured data storage accessible via cloud APIs\\x22},{\\x22title\\x22:\\x22DS8000\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/ds8000?lnk=flatitem\\x22,\\x22description\\x22:\\x22High\\u002Dcapacity, high\\u002Davailability mainframe storage system\\x22},{\\x22title\\x22:\\x22Defender\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/storage\\u002Ddefender?lnk=flatitem\\x22,\\x22description\\x22:\\x22Data resiliency software for threat detection and data recovery\\x22},{\\x22title\\x22:\\x22FlashSystem\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/flashsystem?lnk=flatitem\\x22,\\x22description\\x22:\\x22All\\u002Dflash array with performance and capacity for any workload\\x22},{\\x22title\\x22:\\x22Fusion\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/storage\\u002Dfusion?lnk=flatitem\\x22,\\x22description\\x22:\\x22Container\\u002Dnative storage and data orchestration for OpenShift\\x22},{\\x22title\\x22:\\x22Insights\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/storage\\u002Dinsights?lnk=flatitem\\x22,\\x22description\\x22:\\x22Storage AIOps for metrics\\u002Dpowered analysis and decisions\\x22},{\\x22title\\x22:\\x22Protect\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/storage\\u002Dprotect?lnk=flatitem\\x22,\\x22description\\x22:\\x22Data resiliency, backup and recovery delivered via SaaS\\x22},{\\x22title\\x22:\\x22Protect for Cloud\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/storage\\u002Dprotect\\u002Dfor\\u002Dcloud?lnk=flatitem\\x22,\\x22description\\x22:\\x22Backup and recovery SaaS for data in public cloud services\\x22},{\\x22title\\x22:\\x22SAN Switches\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/storage\\u002Darea\\u002Dnetwork?lnk=flatitem\\x22,\\x22description\\x22:\\x22Scalable and highly available storage area network solutions\\x22},{\\x22title\\x22:\\x22Scale\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/storage\\u002Dscale?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software defined storage for AI and high performance workloads\\x22},{\\x22title\\x22:\\x22Scale System\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/storage\\u002Dscale\\u002Dsystem?lnk=flatitem\\x22,\\x22description\\x22:\\x22All\\u002Dflash or hybrid performant storage for cloud scale use\\u002Dcases\\x22},{\\x22title\\x22:\\x22Tape Systems\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/tape\\u002Dstorage?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software and hardware for encrypted and air\\u002Dgapped archives\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Supply chain\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/supply\\u002Dchain?lnk=flathl\\x22,\\x22description\\x22:\\x22Manage the flow of goods and services\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Sterling Order and Fulfillment Suite\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/order\\u002Dfulfillment\\u002Dsuite?lnk=flatitem\\x22,\\x22description\\x22:\\x22Complete order management system with AI and machine learning\\x22},{\\x22title\\x22:\\x22Sterling Supply Chain Intelligence Suite\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/supply\\u002Dchain\\u002Dintelligence\\u002Dsuite?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software for supply chain optimization and automation\\x22}]}]}],\\x22viewAll\\x22:{\\x22title\\x22:\\x22View all products\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\x22}}},{\\x22title\\x22:\\x22Solutions\\x22,\\x22url\\x22:\\x22\\x22,\\x22submenu\\x22:{\\x22sections\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22Automation\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/automation?lnk=flathl\\x22,\\x22description\\x22:\\x22Intelligent automation for business and IT operations\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22AIOps\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/aiops?lnk=flatitem\\x22,\\x22description\\x22:\\x22Enable proactive, continuous application performance management\\x22},{\\x22title\\x22:\\x22Application delivery\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/application\\u002Ddelivery?lnk=flatitem\\x22,\\x22description\\x22:\\x22Modernize applications and deliver new cloud\\u002Dnative applications\\x22},{\\x22title\\x22:\\x22Application performance management\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/application\\u002Dperformance\\u002Dmanagement?lnk=flatitem\\x22,\\x22description\\x22:\\x22Bridge observability and automated resource management\\x22},{\\x22title\\x22:\\x22Business automation\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/business\\u002Dautomation?lnk=flatitem\\x22,\\x22description\\x22:\\x22Deliver intelligent automation solutions quickly with low\\u002Dcode\\x22},{\\x22title\\x22:\\x22Cloud cost management\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/cloud\\u002Dcost\\u002Dmanagement?lnk=flatitem\\x22,\\x22description\\x22:\\x22Optimize performance at the lowest cost and align business value\\x22},{\\x22title\\x22:\\x22Decision management\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/decision\\u002Dmanagement?lnk=flatitem\\x22,\\x22description\\x22:\\x22Model, manage and automate repeatable business processes\\x22},{\\x22title\\x22:\\x22DevOps automation\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/devops\\u002Dautomation?lnk=flatitem\\x22,\\x22description\\x22:\\x22Automate your software delivery process\\x22},{\\x22title\\x22:\\x22Enterprise content management\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/enterprise\\u002Dcontent\\u002Dmanagement?lnk=flatitem\\x22,\\x22description\\x22:\\x22Accelerate information management and governance processes\\x22},{\\x22title\\x22:\\x22FinOps\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/finops?lnk=flatitem\\x22,\\x22description\\x22:\\x22Manage the variable economics of cloud infrastructure\\x22},{\\x22title\\x22:\\x22Integration\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/integration?lnk=flatitem\\x22,\\x22description\\x22:\\x22Connect applications and systems to unlock critical data\\x22},{\\x22title\\x22:\\x22License compliance\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/license\\u002Dcompliance?lnk=flatitem\\x22,\\x22description\\x22:\\x22Reduce compliance risks with application resource management\\x22},{\\x22title\\x22:\\x22Networking\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/networking?lnk=flatitem\\x22,\\x22description\\x22:\\x22Automate network configuration, provisioning and troubleshooting\\x22},{\\x22title\\x22:\\x22Workflow\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/workflow?lnk=flatitem\\x22,\\x22description\\x22:\\x22Increase productivity by streamlining processes and tasks\\x22},{\\x22title\\x22:\\x22iPaaS\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/ipaas?lnk=flatitem\\x22,\\x22description\\x22:\\x22Deliver connectivity across all your apps and data\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Data \\x26 AI\\x22,\\x22description\\x22:\\x22Optimize your data strategy to support data\\u002Ddriven decisions\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22AI\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/artificial\\u002Dintelligence?lnk=flatitem\\x22,\\x22description\\x22:\\x22Artificial intelligence solutions built with industry expertise\\x22},{\\x22title\\x22:\\x22AI ethics\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/impact\\/ai\\u002Dethics?lnk=flatitem\\x22,\\x22description\\x22:\\x22Trustworthy, transparent AI with a multidimensional approach\\x22},{\\x22title\\x22:\\x22Customer experience\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/ai\\u002Dcustomer\\u002Dservice?lnk=flatitem\\x22,\\x22description\\x22:\\x22Build a full\\u002Dservice AI chatbot that integrates with your CRM\\x22},{\\x22title\\x22:\\x22Data fabric\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/data\\u002Dfabric?lnk=flatitem\\x22,\\x22description\\x22:\\x22Automate data discovery, governance and consumption\\x22},{\\x22title\\x22:\\x22Data lake\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/data\\u002Dlake?lnk=flatitem\\x22,\\x22description\\x22:\\x22Analyze structured, semi\\u002Dstructured and unstructured data\\x22},{\\x22title\\x22:\\x22Data management\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/data\\u002Dmanagement?lnk=flatitem\\x22,\\x22description\\x22:\\x22Consistent data access and delivery across apps and domains\\x22},{\\x22title\\x22:\\x22Data quality\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/data\\u002Dquality?lnk=flatitem\\x22,\\x22description\\x22:\\x22Robust tools help identify, understand and correct data flaws\\x22},{\\x22title\\x22:\\x22Data science\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/data\\u002Dscience?lnk=flatitem\\x22,\\x22description\\x22:\\x22Operationalize data science models on any cloud\\x22},{\\x22title\\x22:\\x22Data warehouse\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/data\\u002Dwarehouse?lnk=flatitem\\x22,\\x22description\\x22:\\x22A flexible foundation optimized for data from disparate sources\\x22},{\\x22title\\x22:\\x22DataOps\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/dataops?lnk=flatitem\\x22,\\x22description\\x22:\\x22Develop a coordinated overview of the data acquisition journey\\x22},{\\x22title\\x22:\\x22Master data management\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/master\\u002Ddata\\u002Dmanagement?lnk=flatitem\\x22,\\x22description\\x22:\\x22A trusted, 360\\u002Ddegree view of master data across the enterprise\\x22},{\\x22title\\x22:\\x22Prescriptive analytics\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/prescriptive\\u002Danalytics?lnk=flatitem\\x22,\\x22description\\x22:\\x22Use optimization technology to solve complex decisions\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Industry\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/industries?lnk=flathl\\x22,\\x22description\\x22:\\x22Customized, end\\u002Dto\\u002Dend, AI\\u002Dpowered solutions for your industry\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Automotive\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/industries\\/automotive?lnk=flatitem\\x22,\\x22description\\x22:\\x22Shift focus to connected vehicles and digital services\\x22},{\\x22title\\x22:\\x22Banking\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/industries\\/banking\\u002Dfinancial\\u002Dmarkets?lnk=flatitem\\x22,\\x22description\\x22:\\x22Create compliant, customer\\u002Dcentric banking  experiences\\x22},{\\x22title\\x22:\\x22Consumer goods\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/industries\\/consumer\\u002Dgoods?lnk=flatitem\\x22,\\x22description\\x22:\\x22Infuse agility and automation into operations and supply chains\\x22},{\\x22title\\x22:\\x22Energy \\x26 utilities\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/industries\\/energy?lnk=flatitem\\x22,\\x22description\\x22:\\x22Get your utility ready for the new, sustainable energy ecosystem\\x22},{\\x22title\\x22:\\x22Government\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/industries\\/government?lnk=flatitem\\x22,\\x22description\\x22:\\x22Support citizens with greater speed and confidence\\x22},{\\x22title\\x22:\\x22Healthcare\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/industries\\/healthcare?lnk=flatitem\\x22,\\x22description\\x22:\\x22Navigate a more interconnected, data\\u002Drich healthcare ecosystem\\x22},{\\x22title\\x22:\\x22Insurance\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/industries\\/insurance?lnk=flatitem\\x22,\\x22description\\x22:\\x22Improve operations and customer experiences at speed and scale\\x22},{\\x22title\\x22:\\x22Life Sciences\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/industries\\/life\\u002Dsciences?lnk=flatitem\\x22,\\x22description\\x22:\\x22Accelerate product innovation and drive patient centricity\\x22},{\\x22title\\x22:\\x22Manufacturing\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/industries\\/manufacturing?lnk=flatitem\\x22,\\x22description\\x22:\\x22Digitally transform traditional manufacturing processes\\x22},{\\x22title\\x22:\\x22Retail\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/industries\\/retail?lnk=flatitem\\x22,\\x22description\\x22:\\x22Meet changing consumer needs with new retail business models\\x22},{\\x22title\\x22:\\x22Telecommunications\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/industries\\/telecommunications?lnk=flatitem\\x22,\\x22description\\x22:\\x22Achieve a more flexible, seamless network architecture\\x22},{\\x22title\\x22:\\x22Travel\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/industries\\/travel\\u002Dtransportation?lnk=flatitem\\x22,\\x22description\\x22:\\x22Restore traveler trust and increase operational efficiency\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Infrastructure\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/it\\u002Dinfrastructure?lnk=flathl\\x22,\\x22description\\x22:\\x22\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Backup \\x26 recovery\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/backup\\u002Drecovery?lnk=flatitem\\x22,\\x22description\\x22:\\x22Comprehensive data resilience for physical and virtual servers\\x22},{\\x22title\\x22:\\x22Edge computing\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/edge\\u002Dcomputing?lnk=flatitem\\x22,\\x22description\\x22:\\x22Manage workloads from edge to core across any hybrid multicloud\\x22},{\\x22title\\x22:\\x22High performance computing (HPC)\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/high\\u002Dperformance\\u002Dcomputing?lnk=flatitem\\x22,\\x22description\\x22:\\x22HPC infrastructure for advanced and data\\u002Dintensive workloads\\x22},{\\x22title\\x22:\\x22Hybrid cloud\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/hybrid\\u002Dcloud?lnk=flatitem\\x22,\\x22description\\x22:\\x22Use a common platform across cloud, on\\u002Dprem and edge environments\\x22},{\\x22title\\x22:\\x22IT modernization\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/infrastructure\\u002Dmodernization?lnk=flatitem\\x22,\\x22description\\x22:\\x22Modernize legacy IT in place with containerized microservices\\x22},{\\x22title\\x22:\\x22Mainframe application modernization\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/mainframe\\u002Dapplication\\u002Dmodernization?lnk=flatitem\\x22,\\x22description\\x22:\\x22Apply modern development and DevOps tools to mainframe applications\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Security\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/security?lnk=flathl\\x22,\\x22description\\x22:\\x22Enterprise cybersecurity for today’s hybrid cloud environments\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Cloud security\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/cloud\\u002Dsecurity?lnk=flatitem\\x22,\\x22description\\x22:\\x22Trust IBM Cloud security for data protection and compliance\\x22},{\\x22title\\x22:\\x22Data security\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/data\\u002Dsecurity?lnk=flatitem\\x22,\\x22description\\x22:\\x22Meet data privacy regulations and simplify operational complexity\\x22},{\\x22title\\x22:\\x22Insider threat\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/insider\\u002Dthreat?lnk=flatitem\\x22,\\x22description\\x22:\\x22Protection from malicious or unintentional threats from insiders\\x22},{\\x22title\\x22:\\x22Ransomware\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/ransomware?lnk=flatitem\\x22,\\x22description\\x22:\\x22Proactively manage cyber risks to reduce the impact of attacks\\x22},{\\x22title\\x22:\\x22Threat detection and response\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/threat\\u002Ddetection\\u002Dresponse?lnk=flatitem\\x22,\\x22description\\x22:\\x22Accelerate and streamline responses across the attack lifecycle\\x22}]}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Sustainability\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/sustainability?lnk=flathl\\x22,\\x22description\\x22:\\x22Embed sustainability into your business transformation\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Asset lifecycle management\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/maximo\\/sustainability?lnk=flatitem\\x22,\\x22description\\x22:\\x22Optimize performance, extend asset lifecycles and reduce downtime\\x22},{\\x22title\\x22:\\x22ESG reporting\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/envizi?lnk=flatitem\\x22,\\x22description\\x22:\\x22Data foundation to streamline meeting ESG reporting requirements\\x22},{\\x22title\\x22:\\x22Facilities management\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/tririga\\/sustainability?lnk=flatitem\\x22,\\x22description\\x22:\\x22A single source of truth to manage your corporate real estate\\x22},{\\x22title\\x22:\\x22IT sustainability\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/turbonomic\\/sustainability?lnk=flatitem\\x22,\\x22description\\x22:\\x22Software to estimate infrastructure resource consumption\\x22},{\\x22title\\x22:\\x22Supply chain\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/supply\\u002Dchain?lnk=flatitem\\x22,\\x22description\\x22:\\x22Build supply chain resiliency with intelligent automation\\x22},{\\x22title\\x22:\\x22Weather Risk Management\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/environmental\\u002Dintelligence\\u002Dsuite\\/sustainability?lnk=flatitem\\x22,\\x22description\\x22:\\x22Manage the economic impact of weather and climate on your business\\x22}]}]}],\\x22viewAll\\x22:{\\x22title\\x22:\\x22\\x22,\\x22url\\x22:\\x22\\x22}}},{\\x22title\\x22:\\x22Consulting\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/consulting?lnk=L0G\\x22},{\\x22title\\x22:\\x22Support\\x22,\\x22url\\x22:\\x22\\x22,\\x22submenu\\x22:{\\x22sections\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22\\x22,\\x22description\\x22:\\x22\\x22},\\x22groups\\x22:[{\\x22heading\\x22:{\\x22title\\x22:\\x22What\\x27s New\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/new?lnk=flathl\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Community\\x22,\\x22url\\x22:\\x22https:\\/\\/community.ibm.com\\/community\\/user\\/community?lnk=flathl\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Developer\\x22,\\x22url\\x22:\\x22https:\\/\\/developer.ibm.com\\/?lnk=flathl\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Call for Code\\x22,\\x22url\\x22:\\x22https:\\/\\/developer.ibm.com\\/callforcode\\/?lnk=flatitem\\x22},{\\x22title\\x22:\\x22Generative AI\\x22,\\x22url\\x22:\\x22https:\\/\\/developer.ibm.com\\/generative\\u002Dai\\u002Dfor\\u002Ddevelopers\\/?lnk=flatitem\\x22},{\\x22title\\x22:\\x22Open Source @ IBM\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/opensource\\/?lnk=flatitem\\x22},{\\x22title\\x22:\\x22Products\\x22,\\x22url\\x22:\\x22https:\\/\\/developer.ibm.com\\/components\\/?lnk=flatitem\\x22},{\\x22title\\x22:\\x22Technologies\\x22,\\x22url\\x22:\\x22https:\\/\\/developer.ibm.com\\/technologies\\/?lnk=flatitem\\x22}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Documentation\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/docs\\/en?lnk=flathl\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22All product documentation\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/docs\\/en?lnk=flatitem\\x22},{\\x22title\\x22:\\x22IBM Cloud documentation\\x22,\\x22url\\x22:\\x22https:\\/\\/cloud.ibm.com\\/docs?lnk=flatitem\\x22},{\\x22title\\x22:\\x22IBM Redbooks\\x22,\\x22url\\x22:\\x22https:\\/\\/www.redbooks.ibm.com\\/?lnk=flatitem\\x22}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22IBM Cloud platform support\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/cloud\\/support?lnk=flathl\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Implementation\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Expert Labs\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/products\\/expertlabs?lnk=flatitem\\x22}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Support\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/mysupport\\/s\\/?language=en_US\\x26lnk=flathl\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Download fixes, updates \\x26 drivers\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/support\\/fixcentral?lnk=flatitem\\x22},{\\x22title\\x22:\\x22Download licensed software \\u002D Passport Advantage\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/software\\/passportadvantage\\/pao_customer.html?lnk=flatitem\\x22},{\\x22title\\x22:\\x22IBM Software Licensing\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/about\\/software\\u002Dlicensing\\/?lnk=flatitem\\x22},{\\x22title\\x22:\\x22Open a case\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/mysupport\\/s\\/redirecttoopencasepage?lnk=flatitem\\x22},{\\x22title\\x22:\\x22View more\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/mysupport\\/s\\/?language=en_US\\x26lnk=flatitem\\x22},{\\x22title\\x22:\\x22View support plans\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/support\\/pages\\/ibm\\u002Dsupport\\u002Dofferings?lnk=flatitem\\x22},{\\x22title\\x22:\\x22View your cases\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/mysupport\\/s\\/my\\u002Dcases?lnk=flatitem\\x22}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Technology Lifecycle Services\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/services\\/technology\\u002Dlifecycle\\u002Dservices?lnk=flathl\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Enterprise networking and security\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/services\\/networking\\u002Dsupport?lnk=flatitem\\x22},{\\x22title\\x22:\\x22Servers and storage\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/services\\/systems\\u002Dsupport?lnk=flatitem\\x22},{\\x22title\\x22:\\x22Software\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/services\\/software\\u002Dsupport?lnk=flatitem\\x22}]},{\\x22heading\\x22:{\\x22title\\x22:\\x22Training\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/training\\/?lnk=flathl\\x22,\\x22description\\x22:\\x22\\x22},\\x22links\\x22:[{\\x22title\\x22:\\x22Courses\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/training\\/search?q=course\\x26lnk=flatitem\\x22},{\\x22title\\x22:\\x22Digital learning subscriptions\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/training\\/subscriptions?lnk=flatitem\\x22},{\\x22title\\x22:\\x22Learning paths \\x26 collections\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/training\\/learning\\u002Dpaths\\u002Dand\\u002Dcollections?lnk=flatitem\\x22},{\\x22title\\x22:\\x22Professional certifications\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/training\\/credentials?lnk=flatitem\\x22}]}]}],\\x22viewAll\\x22:{\\x22title\\x22:\\x22\\x22,\\x22url\\x22:\\x22\\x22}}},{\\x22title\\x22:\\x22Think\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think?lnk=L0G\\x22}]}}\";\n",
      "    const searchPlaceholderText = \"Search all of IBM\";\n",
      "    const links = JSON.parse(l0Json).mastheadNav.links;\n",
      "    const unAuthenticated = JSON.parse(l0Json).profileMenu.signedout;\n",
      "    const authenticated = JSON.parse(l0Json).profileMenu.signedin;\n",
      "\n",
      "    const mastheadContainerEl = document.querySelector('c4d-masthead-container');\n",
      "\n",
      "    mastheadContainerEl.navLinks = links;\n",
      "    mastheadContainerEl.unauthenticatedProfileItems = unAuthenticated;\n",
      "    mastheadContainerEl.authenticatedProfileItems = authenticated;\n",
      "\n",
      "    const appendSearchPlaceholderText = () => {\n",
      "        const typeheadEl = mastheadContainerEl.querySelector('c4d-search-with-typeahead');\n",
      "        if (typeheadEl) {\n",
      "            typeheadEl.searchPlaceholder = searchPlaceholderText;\n",
      "            observer.disconnect();\n",
      "        }\n",
      "    }\n",
      "    let observer = new MutationObserver(appendSearchPlaceholderText);\n",
      "    observer.observe(mastheadContainerEl, {subtree: true, childList: true});\n",
      "      </script>\n",
      "      <script>\n",
      "       var json = \"{\\x22title\\x22:\\x22Think\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\x22,\\x22menuItems\\x22:[{\\x22title\\x22:\\x22Artificial intelligence\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/artificial\\u002Dintelligence\\x22},{\\x22title\\x22:\\x22Cloud\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/cloud\\x22},{\\x22title\\x22:\\x22Security\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/security\\x22},{\\x22title\\x22:\\x22Sustainability\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/sustainability\\x22},{\\x22title\\x22:\\x22Blog\\x22,\\x22submenu\\x22:{\\x22columns\\x22:2,\\x22footer\\x22:{\\x22title\\x22:\\x22View all blog posts\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/\\x22},\\x22menuSections\\x22:[{\\x22span\\x22:1,\\x22heading\\x22:{\\x22headingLevel\\x22:2,\\x22title\\x22:\\x22Categories\\x22},\\x22items\\x22:[{\\x22title\\x22:\\x22Analytics\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/category\\/analytics\\/\\x22},{\\x22title\\x22:\\x22Artificial intelligence\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/category\\/artificial\\u002Dintelligence\\/\\x22},{\\x22title\\x22:\\x22Automation\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/category\\/automation\\/\\x22},{\\x22title\\x22:\\x22Business transformation\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/category\\/business\\u002Dtransformation\\/\\x22},{\\x22title\\x22:\\x22Cloud\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/category\\/cloud\\/\\x22},{\\x22title\\x22:\\x22Security\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/category\\/security\\/\\x22},{\\x22title\\x22:\\x22Social impact\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/category\\/social\\u002Dimpact\\/\\x22},{\\x22title\\x22:\\x22Supply chain\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/category\\/supply\\u002Dchain\\/\\x22},{\\x22title\\x22:\\x22Sustainability\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/category\\/sustainability\\/\\x22}]},{\\x22span\\x22:1,\\x22heading\\x22:{\\x22headingLevel\\x22:2,\\x22title\\x22:\\x22Industries\\x22},\\x22items\\x22:[{\\x22title\\x22:\\x22Energy and utilities\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/category\\/energy\\u002Dand\\u002Dutilities\\/\\x22},{\\x22title\\x22:\\x22Financial services\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/category\\/financial\\u002Dservices\\/\\x22},{\\x22title\\x22:\\x22Government\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/category\\/government\\/\\x22},{\\x22title\\x22:\\x22Healthcare\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/category\\/healthcare\\/\\x22},{\\x22title\\x22:\\x22Manufacturing\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/category\\/manufacturing\\/\\x22},{\\x22title\\x22:\\x22Media and entertainment\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/category\\/media\\u002Dand\\u002Dentertainment\\/\\x22},{\\x22title\\x22:\\x22Mining and extraction\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/category\\/mining\\u002Dand\\u002Dextraction\\/\\x22},{\\x22title\\x22:\\x22Technology\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/category\\/technology\\/\\x22},{\\x22title\\x22:\\x22Telecommunications\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/category\\/telecommunications\\/\\x22},{\\x22title\\x22:\\x22Transportation\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/blog\\/category\\/transportation\\/\\x22}]}]}},{\\x22title\\x22:\\x22Videos\\x22,\\x22submenu\\x22:{\\x22columns\\x22:1,\\x22menuSections\\x22:[{\\x22span\\x22:1,\\x22items\\x22:[{\\x22title\\x22:\\x22Overview\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/videos\\x22},{\\x22title\\x22:\\x22AI Academy\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/videos\\/ai\\u002Dacademy\\x22},{\\x22title\\x22:\\x22Rethink use cases\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/videos\\/series\\/rethink\\x22},{\\x22title\\x22:\\x22Think 2024 on demand\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/videos\\/series\\/think\\u002Dkeynotes\\x22}]}]}},{\\x22title\\x22:\\x22Reports\\x22,\\x22submenu\\x22:{\\x22columns\\x22:1,\\x22menuSections\\x22:[{\\x22span\\x22:1,\\x22items\\x22:[{\\x22title\\x22:\\x22AI in Action 2024\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22http:\\/\\/www.ibm.com\\/think\\/reports\\/ai\\u002Din\\u002Daction\\x22},{\\x22title\\x22:\\x22State of Sustainability Readiness Report 2024\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/reports\\/sustainability\\u002Dreadiness\\x22},{\\x22title\\x22:\\x22Cost of a Data Breach 2024\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/reports\\/data\\u002Dbreach\\x22},{\\x22title\\x22:\\x22IBM X\\u002DForce Threat Intelligence Index 2024\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/reports\\/threat\\u002Dintelligence\\x22},{\\x22title\\x22:\\x22CEO’s Guide to Generative AI\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/thought\\u002Dleadership\\/institute\\u002Dbusiness\\u002Dvalue\\/en\\u002Dus\\/report\\/ceo\\u002Dgenerative\\u002Dai\\x22},{\\x22title\\x22:\\x22Hybrid by Design: The great tech reset\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/thought\\u002Dleadership\\/institute\\u002Dbusiness\\u002Dvalue\\/en\\u002Dus\\/report\\/hybrid\\u002Dby\\u002Ddesign\\x22},{\\x22title\\x22:\\x222024 CEO Study\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/thought\\u002Dleadership\\/institute\\u002Dbusiness\\u002Dvalue\\/en\\u002Dus\\/c\\u002Dsuite\\u002Dstudy\\/ceo\\x22},{\\x22title\\x22:\\x22View all IBV reports\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/thought\\u002Dleadership\\/institute\\u002Dbusiness\\u002Dvalue\\x22}]}]}},{\\x22title\\x22:\\x22Podcasts\\x22,\\x22submenu\\x22:{\\x22columns\\x22:1,\\x22menuSections\\x22:[{\\x22span\\x22:1,\\x22items\\x22:[{\\x22title\\x22:\\x22Overview\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/podcasts\\x22},{\\x22title\\x22:\\x22Mixture of Experts\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/podcasts\\/mixture\\u002Dof\\u002Dexperts\\x22},{\\x22title\\x22:\\x22AI in Action\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/podcasts\\/ai\\u002Din\\u002Daction\\x22},{\\x22title\\x22:\\x22Smart Talks with IBM\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/podcasts\\/smart\\u002Dtalks\\x22}]}]}},{\\x22title\\x22:\\x22Events\\x22,\\x22submenu\\x22:{\\x22columns\\x22:1,\\x22menuSections\\x22:[{\\x22span\\x22:1,\\x22items\\x22:[{\\x22title\\x22:\\x22Think 2024\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/events\\/think\\x22},{\\x22title\\x22:\\x22Think on Tour\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/events\\/think\\/on\\u002Dtour\\x22},{\\x22title\\x22:\\x22TechXchange\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/community\\/ibm\\u002Dtechxchange\\u002Dconference\\/\\x22}]}]}},{\\x22title\\x22:\\x22More\\x22,\\x22submenu\\x22:{\\x22columns\\x22:2,\\x22footer\\x22:{\\x22title\\x22:\\x22View all\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/search\\x22},\\x22menuSections\\x22:[{\\x22span\\x22:1,\\x22heading\\x22:{\\x22headingLevel\\x22:2,\\x22title\\x22:\\x22Topics\\x22},\\x22items\\x22:[{\\x22title\\x22:\\x22Analytics\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/analytics\\x22},{\\x22title\\x22:\\x22Asset management\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/asset\\u002Dmanagement\\x22},{\\x22title\\x22:\\x22Business automation\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/business\\u002Dautomation\\x22},{\\x22title\\x22:\\x22Business operations\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/business\\u002Doperations\\x22},{\\x22title\\x22:\\x22Compute and servers\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/compute\\x22},{\\x22title\\x22:\\x22DevOps\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/devops\\x22},{\\x22title\\x22:\\x22IT automation\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/it\\u002Dautomation\\x22},{\\x22title\\x22:\\x22IT infrastructure\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/it\\u002Dinfrastructure\\x22},{\\x22title\\x22:\\x22Middleware\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/middleware\\x22},{\\x22title\\x22:\\x22Network\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/network\\x22},{\\x22title\\x22:\\x22Quantum\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/quantum\\x22},{\\x22title\\x22:\\x22Storage\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/storage\\x22}]},{\\x22span\\x22:1,\\x22heading\\x22:{\\x22headingLevel\\x22:2,\\x22title\\x22:\\x22Content types\\x22},\\x22items\\x22:[{\\x22title\\x22:\\x22Explainers\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/topics\\x22},{\\x22title\\x22:\\x22News\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/think\\/news\\x22},{\\x22title\\x22:\\x22Newsletters\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/subscribe\\x22},{\\x22title\\x22:\\x22Reference architectures\\x22,\\x22target\\x22:\\x22_self\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/architectures\\x22}]}]}}],\\x22actions\\x22:{\\x22login\\x22:null,\\x22cta\\x22:{\\x22title\\x22:\\x22Subscribe\\x22,\\x22url\\x22:\\x22https:\\/\\/www.ibm.com\\/account\\/reg\\/signup?formid=news\\u002Durx\\u002D52954\\x22}},\\x22linkOrContactModule\\x22:false,\\x22optinalCta\\x22:true}\";\n",
      "    document.querySelector('c4d-masthead-container').l1Data = JSON.parse(json);\n",
      "    document.querySelector('.masthead').classList.add(\"hasNavigation\");\n",
      "    document.documentElement.style.setProperty('--anker-navigation-spacing', '-6rem');\n",
      "    document.addEventListener('DOMContentLoaded', () => {\n",
      "        document.querySelector('c4d-masthead-l1').overviewText = 'Overview';\n",
      "    });\n",
      "      </script>\n",
      "      <script type=\"text/javascript\">\n",
      "       if(window.location.href.includes(\"/careers\")){\n",
      "       document.querySelector('c4d-masthead-container').setAttribute(\"has-search\",\"false\");\n",
      "       document.querySelector('c4d-masthead-container').setAttribute(\"has-profile\",\"false\");\n",
      "    }\n",
      "      </script>\n",
      "     </div>\n",
      "     <div class=\"leadspace\">\n",
      "      <div class=\"bx--grid bx--no-gutter\">\n",
      "       <dds-leadspace gradient-style-scheme=\"\" loadinganimation=\"\" size=\"medium\" type=\"left\">\n",
      "        <bx-breadcrumb slot=\"navigation\">\n",
      "         <bx-breadcrumb-item>\n",
      "          <bx-breadcrumb-link href=\"/\">\n",
      "           <caem-paragraph size=\"normal\" type-style=\"body-short-01\">\n",
      "            Home\n",
      "           </caem-paragraph>\n",
      "          </bx-breadcrumb-link>\n",
      "         </bx-breadcrumb-item>\n",
      "         <bx-breadcrumb-item>\n",
      "          <bx-breadcrumb-link href=\"/topics\">\n",
      "           <caem-paragraph size=\"normal\" type-style=\"body-short-01\">\n",
      "            Topics\n",
      "           </caem-paragraph>\n",
      "          </bx-breadcrumb-link>\n",
      "         </bx-breadcrumb-item>\n",
      "         <bx-breadcrumb-item class=\"no-trailing-slash\">\n",
      "          <caem-paragraph size=\"normal\" type-style=\"body-short-01\">\n",
      "           LangChain\n",
      "          </caem-paragraph>\n",
      "         </bx-breadcrumb-item>\n",
      "        </bx-breadcrumb>\n",
      "        <dds-leadspace-heading id=\"leadspaceSimpleTitle\">\n",
      "         What is LangChain?\n",
      "        </dds-leadspace-heading>\n",
      "        <div class=\"leadspace__copy\">\n",
      "        </div>\n",
      "        <dds-button-group slot=\"action\">\n",
      "         <dds-button-cta aria-label=\"Optional Primary Option\" cta-type=\"local\" data-aem-autoid=\"aem--leadspace_cta-0\" href=\"https://www.ibm.com/products/watsonx-ai/foundation-models\" kind=\"primary\" target=\"_self\">\n",
      "          <span>\n",
      "           Use LangChain with watsonx.ai\n",
      "          </span>\n",
      "         </dds-button-cta>\n",
      "         <dds-button-cta aria-label=\"Optional Secondary Option\" cta-type=\"local\" data-aem-autoid=\"aem--leadspace_cta-1\" href=\"https://www.ibm.com/account/reg/signup?formid=news-urx-52954\" kind=\"tertiary\" target=\"_blank\">\n",
      "          <span>\n",
      "           Sign up for AI updates\n",
      "          </span>\n",
      "         </dds-button-cta>\n",
      "        </dds-button-group>\n",
      "        <dds-background-media gradient-hidden=\"\" mobile-position=\"bottom\" opacity=\"100\">\n",
      "         <div class=\"bx--image\">\n",
      "          <picture>\n",
      "           <source media=\"(min-width: 1312px) and (min-resolution: 192dpi)\" srcset=\"/content/dam/connectedassets-adobe-cms/worldwide-content/creative-assets/s-migr/ul/g/64/ea/content-hub-analytic-page-leadspace-short.component.xl-retina.ts=1729189504656.png/content/adobe-cms/us/en/topics/langchain/jcr:content/root/leadspace\"/>\n",
      "           <source media=\"(min-width: 1312px)\" srcset=\"/content/dam/connectedassets-adobe-cms/worldwide-content/creative-assets/s-migr/ul/g/64/ea/content-hub-analytic-page-leadspace-short.component.xl.ts=1729189504656.png/content/adobe-cms/us/en/topics/langchain/jcr:content/root/leadspace\"/>\n",
      "           <source media=\"(min-width: 1056px) and (min-resolution: 192dpi)\" srcset=\"/content/dam/connectedassets-adobe-cms/worldwide-content/creative-assets/s-migr/ul/g/64/ea/content-hub-analytic-page-leadspace-short.component.l-retina.ts=1729189504656.png/content/adobe-cms/us/en/topics/langchain/jcr:content/root/leadspace\"/>\n",
      "           <source media=\"(min-width: 1056px)\" srcset=\"/content/dam/connectedassets-adobe-cms/worldwide-content/creative-assets/s-migr/ul/g/64/ea/content-hub-analytic-page-leadspace-short.component.l.ts=1729189504656.png/content/adobe-cms/us/en/topics/langchain/jcr:content/root/leadspace\"/>\n",
      "           <source media=\"(min-width: 672px) and (min-resolution: 192dpi)\" srcset=\"/content/dam/connectedassets-adobe-cms/worldwide-content/creative-assets/s-migr/ul/g/64/ea/content-hub-analytic-page-leadspace-short.component.m-retina.ts=1729189504656.png/content/adobe-cms/us/en/topics/langchain/jcr:content/root/leadspace\"/>\n",
      "           <source media=\"(min-width: 672px)\" srcset=\"/content/dam/connectedassets-adobe-cms/worldwide-content/creative-assets/s-migr/ul/g/64/ea/content-hub-analytic-page-leadspace-short.component.m.ts=1729189504656.png/content/adobe-cms/us/en/topics/langchain/jcr:content/root/leadspace\"/>\n",
      "           <source media=\"(min-width: 481px) and (min-resolution: 192dpi)\" srcset=\"/content/dam/connectedassets-adobe-cms/worldwide-content/creative-assets/s-migr/ul/g/64/ea/content-hub-analytic-page-leadspace-short.component.s-retina.ts=1729189504656.png/content/adobe-cms/us/en/topics/langchain/jcr:content/root/leadspace\"/>\n",
      "           <source media=\"(min-width: 481px)\" srcset=\"/content/dam/connectedassets-adobe-cms/worldwide-content/creative-assets/s-migr/ul/g/64/ea/content-hub-analytic-page-leadspace-short.component.s.ts=1729189504656.png/content/adobe-cms/us/en/topics/langchain/jcr:content/root/leadspace\"/>\n",
      "           <source media=\"(max-width: 480px) and (min-resolution: 192dpi)\" srcset=\"/content/dam/connectedassets-adobe-cms/worldwide-content/creative-assets/s-migr/ul/g/64/ea/content-hub-analytic-page-leadspace-short.component.xs-retina.ts=1729189504656.png/content/adobe-cms/us/en/topics/langchain/jcr:content/root/leadspace\"/>\n",
      "           <source media=\"(max-width: 480px)\" srcset=\"/content/dam/connectedassets-adobe-cms/worldwide-content/creative-assets/s-migr/ul/g/64/ea/content-hub-analytic-page-leadspace-short.component.xs.ts=1729189504656.png/content/adobe-cms/us/en/topics/langchain/jcr:content/root/leadspace\"/>\n",
      "           <img alt=\"Illustration with collage of pictograms of clouds, pie chart, graph pictograms \" class=\"bx--image__img\" id=\"image--2015408711\" src=\"/content/dam/connectedassets-adobe-cms/worldwide-content/creative-assets/s-migr/ul/g/64/ea/content-hub-analytic-page-leadspace-short.component.xl.ts=1729189504656.png/content/adobe-cms/us/en/topics/langchain/_jcr_content/root/leadspace\"/>\n",
      "          </picture>\n",
      "         </div>\n",
      "        </dds-background-media>\n",
      "       </dds-leadspace>\n",
      "      </div>\n",
      "     </div>\n",
      "     <div class=\"table-of-contents container responsivegrid\">\n",
      "      <dds-table-of-contents class=\"cmp-table-of-contents\" data-autoid=\"dds--table-of-contents\" data-cmp-is=\"table-of-contents\" stickyoffset=\"96\">\n",
      "       <div class=\"cmp-table-of-contents\" id=\"table-of-contents-db314d2ad4\">\n",
      "        <div class=\"body container responsivegrid\">\n",
      "         <div class=\"cmp-container\" id=\"body-faaeaf7bd1\">\n",
      "          <div class=\"content-section-styled content-section container responsivegrid\">\n",
      "           <div class=\"bx--grid\">\n",
      "            <div class=\"bx--row\">\n",
      "             <div class=\"bx--col-lg-16 bx--no-gutter\">\n",
      "              <div class=\"bx--content-section\">\n",
      "               <div class=\"cmp-container\" id=\"container-c7507ad5b4\">\n",
      "                <div class=\"complex-narrative\">\n",
      "                 <dds-content-block-segmented content-section-body=\"\" data-autoid=\"dds-content-block-segmented\" no-image=\"\">\n",
      "                  <div slot=\"heading\">\n",
      "                   <a data-title=\"Overview\" name=\"Overview\">\n",
      "                   </a>\n",
      "                   <dds-content-block-heading data-autoid=\"dds--content-block__heading\" role=\"heading\">\n",
      "                    <a data-title=\"Overview\" name=\"Overview\">\n",
      "                    </a>\n",
      "                    What is LangChain?\n",
      "                   </dds-content-block-heading>\n",
      "                  </div>\n",
      "                  <dds-content-block-copy data-autoid=\"dds--content-block__copy\" slot=\"copy\">\n",
      "                   <dds-content-block-paragraph>\n",
      "                    <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-695fb6a9d5\">\n",
      "                     <p>\n",
      "                      LangChain is an open source orchestration framework for the development of applications using\n",
      "                      <a href=\"https://www.ibm.com/think/topics/open-source-llms\" rel=\"noopener noreferrer\" target=\"_self\">\n",
      "                       large language models (LLMs)\n",
      "                      </a>\n",
      "                      . Available in both Python- and Javascript-based libraries, LangChain’s tools and APIs simplify the process of building LLM-driven applications like\n",
      "                      <a href=\"https://www.ibm.com/topics/chatbots\" rel=\"noopener noreferrer\" target=\"_self\">\n",
      "                       chatbots\n",
      "                      </a>\n",
      "                      and\n",
      "                      <a href=\"https://www.ibm.com/topics/virtual-agent\" rel=\"noopener noreferrer\" target=\"_self\">\n",
      "                       virtual agents\n",
      "                      </a>\n",
      "                      .\n",
      "                     </p>\n",
      "                    </div>\n",
      "                   </dds-content-block-paragraph>\n",
      "                  </dds-content-block-copy>\n",
      "                  <dds-content-block-segmented-item>\n",
      "                   <dds-content-item-copy>\n",
      "                    <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-d47cf7023b\">\n",
      "                     <p>\n",
      "                      LangChain serves as a generic interface for nearly any LLM, providing a centralized development environment to build LLM applications and integrate them with external data sources and software workflows. LangChain’s module-based approach allows developers and data scientists to dynamically compare different prompts and even different\n",
      "                      <a href=\"https://research.ibm.com/blog/what-are-foundation-models\" rel=\"noopener noreferrer\" target=\"_self\">\n",
      "                       foundation models\n",
      "                      </a>\n",
      "                      with minimal need to rewrite code. This modular environment also allows for programs that use multiple LLMs: for example, an application that uses one LLM to interpret user queries and another LLM to author a response.\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      Launched by Harrison Chase in October 2022, LangChain enjoyed a meteoric rise to prominence: as of June 2023, it was the single fastest-growing open source project on Github.\n",
      "                      <sup>\n",
      "                       1\n",
      "                      </sup>\n",
      "                      Coinciding with the momentous launch of OpenAI’s ChatGPT the following month, LangChain has played a significant role in making\n",
      "                      <a href=\"https://www.ibm.com/think/insights/generative-ai-benefits\" rel=\"noopener noreferrer\" target=\"_self\">\n",
      "                       generative AI\n",
      "                      </a>\n",
      "                      more accessible to enthusiasts in the wake of its widespread popularity.\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      LangChain can facilitate most use cases for LLMs and\n",
      "                      <a href=\"https://www.ibm.com/natural-language-processing\" rel=\"noopener noreferrer\" target=\"_self\">\n",
      "                       natural language processing (NLP)\n",
      "                      </a>\n",
      "                      , like chatbots,\n",
      "                      <a href=\"https://www.ibm.com/topics/intelligent-search\" rel=\"noopener noreferrer\" target=\"_self\">\n",
      "                       intelligent search\n",
      "                      </a>\n",
      "                      , question-answering, summarization services or even virtual agents capable of\n",
      "                      <a href=\"https://www.ibm.com/topics/rpa\" rel=\"noopener noreferrer\" target=\"_self\">\n",
      "                       robotic process automation\n",
      "                      </a>\n",
      "                      .\n",
      "                     </p>\n",
      "                    </div>\n",
      "                   </dds-content-item-copy>\n",
      "                  </dds-content-block-segmented-item>\n",
      "                  <dds-content-block-segmented-item>\n",
      "                   <dds-content-group-heading>\n",
      "                    Integrations with LLMs\n",
      "                   </dds-content-group-heading>\n",
      "                   <dds-content-item-copy>\n",
      "                    <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-cb5219336b\">\n",
      "                     <p>\n",
      "                      LLMs are not standalone applications: they are pre-trained statistical models that must be paired with an application (and, in some cases, specific data sources) in order to meet their purpose.\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      For example, Chat-GPT is not an LLM: it is a chatbot application that, depending on the version you’ve chosen, uses the GPT-3.5 or GPT-4 language model. While it’s the GPT model that interprets the user’s input and composes a natural language response, it’s the application that (among other things) provides an interface for the user to type and read and a UX\n",
      "                      <a href=\"https://www.ibm.com/topics/chatbot-design\" rel=\"noopener noreferrer\" target=\"_self\">\n",
      "                       design that governs the chatbot experience\n",
      "                      </a>\n",
      "                      . Even at the enterprise level, Chat-GPT is not the only application using the GPT model: Microsoft uses GPT-4 to power Bing Chat.\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      Furthermore, though foundation models (like those powering LLMs) are pre-trained on massive datasets, they are not omniscient. If a particular task requires access to specific contextual information, like internal documentation or domain expertise, LLMs must be connected to those external data sources. Even if you simply want your model to reflect real-time awareness of current events, it requires external information: a model’s internal data is only up-to-date through the time period during which it was pre-trained.\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      Likewise, if a given generative AI task requires access to external software workflows—for example, if you wanted your virtual agent to\n",
      "                      <a href=\"https://www.ibm.com/docs/en/watsonx/watson-orchestrate/current?topic=collaborating-slack\" rel=\"noopener noreferrer\" target=\"_self\">\n",
      "                       integrate with Slack\n",
      "                      </a>\n",
      "                      —then you will need a way to integrate the LLM with the\n",
      "                      <a href=\"https://www.ibm.com/topics/api\" rel=\"noopener noreferrer\" target=\"_self\">\n",
      "                       API\n",
      "                      </a>\n",
      "                      for that software.\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      While these integrations can generally be achieved with fully manual code, orchestration frameworks like LangChain and the\n",
      "                      <a href=\"https://www.ibm.com/watsonx\" rel=\"noopener noreferrer\" target=\"_self\">\n",
      "                       IBM watsonx\n",
      "                      </a>\n",
      "                      platform greatly simplify the process. They also make it much easier to experiment with different LLMs to compare results, as different models can be swapped in and out with minimal changes to code.\n",
      "                     </p>\n",
      "                    </div>\n",
      "                   </dds-content-item-copy>\n",
      "                  </dds-content-block-segmented-item>\n",
      "                  <div slot=\"complementary\">\n",
      "                   <div class=\"aside-card\">\n",
      "                    <dds-card-cta cta-type=\"local\" href=\"https://www.ibm.com/account/reg/us-en/signup?formid=urx-52620\" pictogram-placement=\"top\" target=\"_blank\">\n",
      "                     <dds-card-eyebrow aria-hidden=\"true\" slot=\"eyebrow\">\n",
      "                      Ebook\n",
      "                     </dds-card-eyebrow>\n",
      "                     <dds-card-heading aria-level=\"3\" role=\"heading\" slot=\"heading\">\n",
      "                      How to choose the right foundation model\n",
      "                     </dds-card-heading>\n",
      "                     <p>\n",
      "                      Learn how to choose the right approach in preparing data sets and employing foundation models.\n",
      "                     </p>\n",
      "                     <dds-card-cta-footer aria-hidden=\"true\" href=\"https://www.ibm.com/account/reg/us-en/signup?formid=urx-52620\" icon-placement=\"right\" slot=\"footer\" target=\"_blank\">\n",
      "                      <span>\n",
      "                       Read the ebook\n",
      "                      </span>\n",
      "                     </dds-card-cta-footer>\n",
      "                    </dds-card-cta>\n",
      "                   </div>\n",
      "                   <dds-link-list type=\"default\">\n",
      "                    <dds-link-list-heading aria-level=\"4\" role=\"heading\" slot=\"heading\">\n",
      "                     Related content\n",
      "                    </dds-link-list-heading>\n",
      "                    <dds-link-list-item-card cta-type=\"local\" href=\"https://www.ibm.com/account/reg/signup?formid=urx-53231\" target=\"_blank\">\n",
      "                     <p>\n",
      "                      Read the 2024 AI in Action Report\n",
      "                     </p>\n",
      "                     <dds-card-cta-footer cta-type=\"local\" href=\"https://www.ibm.com/account/reg/signup?formid=urx-53231\" target=\"_blank\">\n",
      "                     </dds-card-cta-footer>\n",
      "                    </dds-link-list-item-card>\n",
      "                   </dds-link-list>\n",
      "                  </div>\n",
      "                 </dds-content-block-segmented>\n",
      "                 <div horizontal-ruler=\"\">\n",
      "                  <dds-hr>\n",
      "                  </dds-hr>\n",
      "                 </div>\n",
      "                </div>\n",
      "                <div class=\"complex-narrative\">\n",
      "                 <dds-content-block-segmented class=\"complex-narrative-container\" content-section-body=\"\" data-autoid=\"dds-content-block-segmented\" no-aside=\"\" no-image=\"\">\n",
      "                  <div slot=\"heading\">\n",
      "                   <a data-title=\"How LangChain works\" name=\"How+LangChain+works\">\n",
      "                   </a>\n",
      "                   <dds-content-block-heading data-autoid=\"dds--content-block__heading\" role=\"heading\">\n",
      "                    <a data-title=\"How LangChain works\" name=\"How+LangChain+works\">\n",
      "                    </a>\n",
      "                    How does LangChain work?\n",
      "                   </dds-content-block-heading>\n",
      "                  </div>\n",
      "                  <dds-content-block-copy data-autoid=\"dds--content-block__copy\" slot=\"copy\">\n",
      "                   <dds-content-block-paragraph>\n",
      "                    <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-b5ca4db876\">\n",
      "                     <p>\n",
      "                      At LangChain’s core is a development environment that streamlines the programming of LLM applications through the use of\n",
      "                      <em>\n",
      "                       abstraction\n",
      "                      </em>\n",
      "                      : the simplification of code by representing one or more complex processes as a named component that encapsulates all of its constituent steps.\n",
      "                     </p>\n",
      "                    </div>\n",
      "                   </dds-content-block-paragraph>\n",
      "                  </dds-content-block-copy>\n",
      "                  <dds-content-block-segmented-item>\n",
      "                   <dds-content-item-copy>\n",
      "                    <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-87560288b6\">\n",
      "                     <p>\n",
      "                      Abstractions are a common element of everyday life and language. For example, “\n",
      "                      <em>\n",
      "                       π\n",
      "                      </em>\n",
      "                      ” allows us to represent the ratio of the length of a circle’s circumference to that of its diameter without having to write out its infinite digits. Similarly, a thermostat allows us to control the temperature in our home without needing to understand the complex circuitry this entails—we only need to know how different thermostat settings translate to different temperatures.\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      LangChain is essentially a library of abstractions for Python and Javascript, representing common steps and concepts necessary to work with language models. These modular components—like functions and object classes—serve as the building blocks of generative AI programs. They can be “\n",
      "                      <em>\n",
      "                       chained\n",
      "                      </em>\n",
      "                      ” together to create applications, minimizing the amount of code and fine understanding required to execute complex NLP tasks. Though LangChain’s abstracted approach may limit the extent to which an expert programmer can finely customize an application, it empowers specialists and newcomers alike to quickly experiment and prototype.\n",
      "                     </p>\n",
      "                    </div>\n",
      "                   </dds-content-item-copy>\n",
      "                  </dds-content-block-segmented-item>\n",
      "                  <dds-content-block-segmented-item>\n",
      "                   <dds-content-group-heading>\n",
      "                    Importing language models\n",
      "                   </dds-content-group-heading>\n",
      "                   <dds-content-item-copy>\n",
      "                    <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-09ffe75cad\">\n",
      "                     <p>\n",
      "                      Nearly any LLM can be used in LangChain. Importing language models into LangChain is easy, provided you have an API key. The\n",
      "                      <em>\n",
      "                       LLM\n",
      "                      </em>\n",
      "                      class is designed to provide a standard interface for all models.\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      Most LLM providers will require you to create an account in order to receive an API key. Some of these APIs—particularly those for proprietary closed-source models, like those offered by OpenAI or Anthropic—may have associated costs.\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      Many open source models, like BigScience’s BLOOM, Meta AI’s LLaMa and Google’s Flan-T5, can be accessed through\n",
      "                      <a href=\"https://huggingface.co/models\" rel=\"noopener noreferrer\" target=\"_blank\">\n",
      "                       Hugging Face\n",
      "                      </a>\n",
      "                      (link resides outside ibm.com).\n",
      "                      <a href=\"https://www.ibm.com/watsonx\" rel=\"noopener noreferrer\" target=\"_self\">\n",
      "                       IBM watsonx\n",
      "                      </a>\n",
      "                      , through its\n",
      "                      <a href=\"https://developer.ibm.com/blogs/awb-hugging-face-and-ibm-working-together-in-open-source/\" rel=\"noopener noreferrer\" target=\"_self\">\n",
      "                       partnership with Hugging Face\n",
      "                      </a>\n",
      "                      , also offers a curated suite of open source models. Creating an account with either service will allow you to generate an API key for any of the models offered by that provider.\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      LangChain is not limited to out-of-the-box foundation models: the\n",
      "                      <a href=\"https://python.langchain.com/v0.1/docs/modules/model_io/llms/custom_llm/\" rel=\"noopener noreferrer\" target=\"_blank\">\n",
      "                       <em>\n",
      "                        CustomLLM\n",
      "                       </em>\n",
      "                       class\n",
      "                      </a>\n",
      "                      (link resides outside ibm.com) allows for custom LLM wrappers. Likewise, you can use the\n",
      "                      <a href=\"https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/wdp-apis.html?context=wx\" rel=\"noopener noreferrer\" target=\"_self\">\n",
      "                       IBM watsonx APIs and Python SDK\n",
      "                      </a>\n",
      "                      , which includes a LangChain integration, to build applications in LangChain with models that you’ve already trained or fine-tuned for your specific needs using the\n",
      "                      <em>\n",
      "                       WatsonxLLM\n",
      "                      </em>\n",
      "                      class (and that model’s specific\n",
      "                      <a href=\"https://www.ibm.com/docs/en/watsonx/saas\" rel=\"noopener noreferrer\" target=\"_self\">\n",
      "                       project ID\n",
      "                      </a>\n",
      "                      ).\n",
      "                     </p>\n",
      "                    </div>\n",
      "                   </dds-content-item-copy>\n",
      "                   <dds-text-cta cta-type=\"local\" data-dynamic-inner-content=\"ctaLabel\" data-dynamic-properties='{\"ctaUrl\":\"href\"}' href=\"https://dataplatform.cloud.ibm.com/exchange/public/entry/view/c3dbf23a-9a56-4c4b-8ce5-5707828fc981?context=wx\" icon-placement=\"right\" slot=\"footer\" target=\"_self\">\n",
      "                    Explore the demo: using watsonx and LangChain to make a series of calls to a language model\n",
      "                   </dds-text-cta>\n",
      "                  </dds-content-block-segmented-item>\n",
      "                  <dds-content-block-segmented-item>\n",
      "                   <dds-content-group-heading>\n",
      "                    Prompt templates\n",
      "                   </dds-content-group-heading>\n",
      "                   <dds-content-item-copy>\n",
      "                    <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-79c2e38e78\">\n",
      "                     <p>\n",
      "                      Prompts are the instructions given to an LLM. The “art” of composing prompts that effectively provide the context necessary for the LLM to interpret input and structure output in the way most useful to you is often called prompt engineering.\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      The\n",
      "                      <em>\n",
      "                       PromptTemplate\n",
      "                      </em>\n",
      "                      class in LangChain formalizes the composition of prompts without the need to manually hard code context and queries. Important elements of a prompt are likewise entered as formal classes, like\n",
      "                      <em>\n",
      "                       input_variables\n",
      "                      </em>\n",
      "                      . A prompt template can thus contain and reproduce context, instructions (like “do not use technical terms”), a set of examples to guide its responses (in what is called “few-shot prompting”), a specified output format or a standardized question to be answered.  You can save and name an effectively structured prompt template and easily reuse it as needed.\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      Though these elements can all be manually coded,\n",
      "                      <em>\n",
      "                       PromptTemplate\n",
      "                      </em>\n",
      "                      modules empower smooth integration with other LangChain features, like the eponymous\n",
      "                      <em>\n",
      "                       chains.\n",
      "                      </em>\n",
      "                     </p>\n",
      "                    </div>\n",
      "                   </dds-content-item-copy>\n",
      "                   <dds-text-cta cta-type=\"external\" data-dynamic-inner-content=\"ctaLabel\" data-dynamic-properties='{\"ctaUrl\":\"href\"}' href=\"https://www.youtube.com/watch?v=yu27PWzJI_Y\" icon-placement=\"right\" slot=\"footer\" target=\"_blank\">\n",
      "                    Watch the video: prompt engineering and prompt tuning\n",
      "                   </dds-text-cta>\n",
      "                  </dds-content-block-segmented-item>\n",
      "                  <dds-content-block-segmented-item>\n",
      "                   <dds-content-group-heading>\n",
      "                    Chains\n",
      "                   </dds-content-group-heading>\n",
      "                   <dds-content-item-copy>\n",
      "                    <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-e9215f6c95\">\n",
      "                     <p>\n",
      "                      As its name implies,\n",
      "                      <em>\n",
      "                       chains\n",
      "                      </em>\n",
      "                      are the core of LangChain’s workflows. They combine LLMs with other components, creating applications by executing a sequence of functions.\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      The most basic chain is\n",
      "                      <em>\n",
      "                       LLMChain\n",
      "                      </em>\n",
      "                      . It simply calls a model and prompt template for that model. For example, imagine you saved a prompt as “ExamplePrompt” and wanted to run it against Flan-T5. You can import LLMChain from langchain.chains, then define\n",
      "                      <em>\n",
      "                       chain_example = LLMChain(llm = flan-t5, prompt = ExamplePrompt)\n",
      "                      </em>\n",
      "                      . To run the chain for a given input, you simply call\n",
      "                      <em>\n",
      "                       chain_example.run(“input”)\n",
      "                      </em>\n",
      "                      .\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      To use the output of one function as the input for the next function, you can use\n",
      "                      <em>\n",
      "                       SimpleSequentialChain\n",
      "                      </em>\n",
      "                      . Each function could utilize different prompts, different tools, different parameters or even different models, depending on your specific needs.\n",
      "                     </p>\n",
      "                    </div>\n",
      "                   </dds-content-item-copy>\n",
      "                  </dds-content-block-segmented-item>\n",
      "                  <dds-content-block-segmented-item>\n",
      "                   <dds-content-group-heading>\n",
      "                    Indexes\n",
      "                   </dds-content-group-heading>\n",
      "                   <dds-content-item-copy>\n",
      "                    <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-ef9ca4e5fb\">\n",
      "                     <p>\n",
      "                      To achieve certain tasks, LLMs will need access to specific external data sources not included in its training dataset, such as internal documents, emails or datasets. LangChain collectively refers to such external documentation as “indexes”.\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      <strong>\n",
      "                       Document loaders\n",
      "                      </strong>\n",
      "                      <br/>\n",
      "                      LangChain offers\n",
      "                      <a href=\"https://python.langchain.com/v0.2/docs/integrations/document_loaders/\" rel=\"noopener noreferrer\" target=\"_blank\">\n",
      "                       a wide variety of document loaders for third party applications\n",
      "                      </a>\n",
      "                      (link resides outside ibm.com). This allows for easy importation of data from sources like file storage services (like Dropbox, Google Drive and Microsoft OneDrive), web content (like YouTube, PubMed or specific URLs), collaboration tools (like Airtable, Trello, Figma and Notion), databases (like Pandas, MongoDB and Microsoft), among many others.\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      <strong>\n",
      "                       Vector databases\n",
      "                      </strong>\n",
      "                      <br/>\n",
      "                      Unlike “traditional” structured databases,\n",
      "                      <a href=\"https://ibm.com/topics/vector-database\" rel=\"noopener noreferrer\" target=\"_self\">\n",
      "                       vector databases\n",
      "                      </a>\n",
      "                      represent data points by converting them into\n",
      "                      <em>\n",
      "                       vector embeddings\n",
      "                      </em>\n",
      "                      : numerical representations in the form of vectors with a fixed number of dimensions, often clustering related data points using\n",
      "                      <a href=\"https://www.ibm.com/topics/unsupervised-learning\" rel=\"noopener noreferrer\" target=\"_self\">\n",
      "                       unsupervised learning methods\n",
      "                      </a>\n",
      "                      . This enables low latency queries, even for massive datasets, which greatly increases efficiency. Vector embeddings also store each vector’s metadata, further enhancing search possibilities.\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      LangChain provides integrations for over 25 different embedding methods, as well as for over 50 different vector stores (both cloud-hosted and local).\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      <strong>\n",
      "                       Text splitters\n",
      "                      </strong>\n",
      "                      <br/>\n",
      "                      To increase speed and reduce computational demands, it’s often wise to split large text documents into smaller pieces. LangChain’s\n",
      "                      <em>\n",
      "                       TextSplitters\n",
      "                      </em>\n",
      "                      split text up into small, semantically meaningful chunks that can then be combined using methods and parameters of your choosing.\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      <strong>\n",
      "                       Retrieval\n",
      "                      </strong>\n",
      "                      <br/>\n",
      "                      Once external sources of knowledge have been connected, the model must be able to quickly retrieve and integrate relevant information as needed. Like watsonx, LangChain offers\n",
      "                      <a href=\"https://research.ibm.com/blog/retrieval-augmented-generation-RAG\" rel=\"noopener noreferrer\" target=\"_self\">\n",
      "                       retrieval augmented generation (RAG)\n",
      "                      </a>\n",
      "                      : its\n",
      "                      <em>\n",
      "                       retriever\n",
      "                      </em>\n",
      "                      modules accept a string query as an input and return a list of\n",
      "                      <em>\n",
      "                       Document\n",
      "                      </em>\n",
      "                      ’s as output.\n",
      "                     </p>\n",
      "                    </div>\n",
      "                   </dds-content-item-copy>\n",
      "                  </dds-content-block-segmented-item>\n",
      "                  <dds-content-block-segmented-item>\n",
      "                   <dds-content-group-heading>\n",
      "                    Memory\n",
      "                   </dds-content-group-heading>\n",
      "                   <dds-content-item-copy>\n",
      "                    <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-58adb7bdf4\">\n",
      "                     <p>\n",
      "                      LLMs, by default, do not have any long-term memory of prior conversations (unless that chat history is used as input for a query). LangChain solves this problem with simple utilities for adding memory to a system, with options ranging from retaining the entirety of all conversations to retaining a summarization of the conversation thus far to retaining the\n",
      "                      <em>\n",
      "                       n\n",
      "                      </em>\n",
      "                      most recent exchanges.\n",
      "                     </p>\n",
      "                    </div>\n",
      "                   </dds-content-item-copy>\n",
      "                  </dds-content-block-segmented-item>\n",
      "                  <dds-content-block-segmented-item>\n",
      "                   <dds-content-group-heading>\n",
      "                    Agents\n",
      "                   </dds-content-group-heading>\n",
      "                   <dds-content-item-copy>\n",
      "                    <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-b73eb7b9f4\">\n",
      "                     <p>\n",
      "                      LangChain agents can use a given language model as a “reasoning engine” to determine which actions to take. When building a chain for an agent, inputs include:\n",
      "                     </p>\n",
      "                     <ul>\n",
      "                      <li>\n",
      "                       a list of available\n",
      "                       <em>\n",
      "                        tools\n",
      "                       </em>\n",
      "                       to be leveraged.\n",
      "                      </li>\n",
      "                      <li>\n",
      "                       user input (like prompts and queries).\n",
      "                      </li>\n",
      "                      <li>\n",
      "                       any relevant previously executed steps.\n",
      "                      </li>\n",
      "                     </ul>\n",
      "                    </div>\n",
      "                   </dds-content-item-copy>\n",
      "                   <dds-text-cta cta-type=\"external\" data-dynamic-inner-content=\"ctaLabel\" data-dynamic-properties='{\"ctaUrl\":\"href\"}' href=\"https://python.langchain.com/v0.1/docs/modules/agents/\" icon-placement=\"right\" slot=\"footer\" target=\"_blank\">\n",
      "                    Learn more about agents in LangChain\n",
      "                   </dds-text-cta>\n",
      "                  </dds-content-block-segmented-item>\n",
      "                  <dds-content-block-segmented-item>\n",
      "                   <dds-content-group-heading>\n",
      "                    Tools\n",
      "                   </dds-content-group-heading>\n",
      "                   <dds-content-item-copy>\n",
      "                    <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-5d13f3798c\">\n",
      "                     <p>\n",
      "                      Despite their heralded power and versatility, LLMs have important limitations: namely, a lack of up-to-date information, a lack of domain-specific expertise and a general difficulty with math.\n",
      "                     </p>\n",
      "                     <p>\n",
      "                      <a href=\"https://python.langchain.com/v0.2/docs/integrations/tools/\" rel=\"noopener noreferrer\" target=\"_blank\">\n",
      "                       LangChain\n",
      "                       <em>\n",
      "                        tools\n",
      "                       </em>\n",
      "                      </a>\n",
      "                      (link resides outside ibm.com) are a set of functions that empower LangChain agents to interact with real-world information in order to expand or improve the services it can provide. Examples of prominent LangChain tools include:\n",
      "                      <br/>\n",
      "                     </p>\n",
      "                     <ul>\n",
      "                      <li>\n",
      "                       <p>\n",
      "                        <strong>\n",
      "                         Wolfram Alpha:\n",
      "                        </strong>\n",
      "                        provides access to powerful computational and data visualization functions, enabling sophisticated mathematical capabilities.\n",
      "                       </p>\n",
      "                      </li>\n",
      "                      <li>\n",
      "                       <p>\n",
      "                        <strong>\n",
      "                         Google Search:\n",
      "                        </strong>\n",
      "                        provides access to Google Search, equipping applications and agents with real-time information.\n",
      "                       </p>\n",
      "                      </li>\n",
      "                      <li>\n",
      "                       <p>\n",
      "                        <strong>\n",
      "                         OpenWeatherMap:\n",
      "                        </strong>\n",
      "                        fetches weather information.\n",
      "                       </p>\n",
      "                      </li>\n",
      "                      <li>\n",
      "                       <p>\n",
      "                        <strong>\n",
      "                         Wikipedia\n",
      "                        </strong>\n",
      "                        : provides efficient access to information from Wikipedia articles.\n",
      "                       </p>\n",
      "                      </li>\n",
      "                     </ul>\n",
      "                    </div>\n",
      "                   </dds-content-item-copy>\n",
      "                  </dds-content-block-segmented-item>\n",
      "                 </dds-content-block-segmented>\n",
      "                 <div horizontal-ruler=\"\">\n",
      "                  <dds-hr>\n",
      "                  </dds-hr>\n",
      "                 </div>\n",
      "                </div>\n",
      "                <div class=\"simple-narrative\">\n",
      "                 <dds-content-block-simple class=\"\" content-section-body=\"\" data-autoid=\"dds--content-block-simple\" no-aside=\"\">\n",
      "                  <div slot=\"heading\">\n",
      "                   <a data-title=\"LangSmith\" name=\"LangSmith\">\n",
      "                   </a>\n",
      "                   <dds-content-block-heading data-autoid=\"dds--content-block__heading\" role=\"heading\">\n",
      "                    <a data-title=\"LangSmith\" name=\"LangSmith\">\n",
      "                    </a>\n",
      "                    LangSmith\n",
      "                   </dds-content-block-heading>\n",
      "                  </div>\n",
      "                  <dds-content-block-copy data-autoid=\"dds--content-block__copy\" size=\"sm\" slot=\"copy\">\n",
      "                   <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-110287c943\">\n",
      "                    <p>\n",
      "                     Released in the fall of 2023, LangSmith aims to bridge the gap between the accessible prototyping capabilities that brought LangChain to prominence and building production-quality LLM applications.\n",
      "                    </p>\n",
      "                    <p>\n",
      "                     LangSmith provides tools to monitor, evaluate and debug applications, including the ability to automatically trace all model calls to spot errors and test performance under different model configurations. This visibility aims to empower more robust, cost-efficient applications.\n",
      "                     <br/>\n",
      "                    </p>\n",
      "                   </div>\n",
      "                  </dds-content-block-copy>\n",
      "                 </dds-content-block-simple>\n",
      "                 <div horizontal-ruler=\"\">\n",
      "                  <dds-hr>\n",
      "                  </dds-hr>\n",
      "                 </div>\n",
      "                </div>\n",
      "                <div class=\"simple-narrative\">\n",
      "                 <dds-content-block-simple class=\"\" content-section-body=\"\" data-autoid=\"dds--content-block-simple\" no-aside=\"\">\n",
      "                  <div slot=\"heading\">\n",
      "                   <a data-title=\"Getting started\" name=\"Getting+started\">\n",
      "                   </a>\n",
      "                   <dds-content-block-heading data-autoid=\"dds--content-block__heading\" role=\"heading\">\n",
      "                    <a data-title=\"Getting started\" name=\"Getting+started\">\n",
      "                    </a>\n",
      "                    Getting started with LangChain\n",
      "                   </dds-content-block-heading>\n",
      "                  </div>\n",
      "                  <dds-content-block-copy data-autoid=\"dds--content-block__copy\" size=\"sm\" slot=\"copy\">\n",
      "                   <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-6659548675\">\n",
      "                    <p>\n",
      "                     LangChain is open source and free to use: source code is\n",
      "                     <a href=\"https://github.com/langchain-ai/langchain\" rel=\"noopener noreferrer\" target=\"_blank\">\n",
      "                      available for download on Github\n",
      "                     </a>\n",
      "                     (link resides outside ibm.com).\n",
      "                    </p>\n",
      "                    <p>\n",
      "                     LangChain can also be installed on Python with a simple pip command:\n",
      "                     <em>\n",
      "                      pip install langchain\n",
      "                     </em>\n",
      "                     .  To install all LangChain dependencies (rather than only those you find necessary), you can run the command\n",
      "                     <em>\n",
      "                      pip install langchain[all]\n",
      "                     </em>\n",
      "                     .\n",
      "                    </p>\n",
      "                    <p>\n",
      "                     Many step-by-step tutorials are available from both the greater LangChain community ecosystem and the official documentation at\n",
      "                     <a href=\"https://python.langchain.com/v0.2/docs/introduction/\" rel=\"noopener noreferrer\" target=\"_blank\">\n",
      "                      docs.langchain.com\n",
      "                     </a>\n",
      "                     (link resides outside ibm.com).\n",
      "                    </p>\n",
      "                   </div>\n",
      "                  </dds-content-block-copy>\n",
      "                 </dds-content-block-simple>\n",
      "                 <div horizontal-ruler=\"\">\n",
      "                  <dds-hr>\n",
      "                  </dds-hr>\n",
      "                 </div>\n",
      "                </div>\n",
      "                <div class=\"mixed-series\">\n",
      "                 <div>\n",
      "                  <div class=\"bx--col-lg-12 bx--no-gutter\">\n",
      "                   <dds-content-block-mixed content-section-body=\"\">\n",
      "                    <div slot=\"heading\">\n",
      "                     <a data-title=\"LangChain use cases\" name=\"LangChain+use+cases\">\n",
      "                     </a>\n",
      "                     <dds-content-block-heading>\n",
      "                      <a data-title=\"LangChain use cases\" name=\"LangChain+use+cases\">\n",
      "                      </a>\n",
      "                      LangChain use cases\n",
      "                     </dds-content-block-heading>\n",
      "                    </div>\n",
      "                    <dds-content-block-copy>\n",
      "                     <dds-content-block-paragraph>\n",
      "                      <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-afd4809df7\">\n",
      "                       <p>\n",
      "                        Applications made with LangChain provide great utility for a variety of use cases, from straightforward question-answering and text generation tasks to more complex solutions that use an LLM as a “reasoning engine.”\n",
      "                       </p>\n",
      "                      </div>\n",
      "                     </dds-content-block-paragraph>\n",
      "                    </dds-content-block-copy>\n",
      "                    <div class=\"cmp-container\" id=\"mixed-series-container-87ffb86e6f\">\n",
      "                     <div class=\"simple-series\">\n",
      "                      <dds-content-group-simple>\n",
      "                       <dds-content-group-copy>\n",
      "                        <dds-content-group-paragraph>\n",
      "                         <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-530afa27ce\">\n",
      "                          <ul>\n",
      "                           <li>\n",
      "                            <strong>\n",
      "                             Chatbots:\n",
      "                            </strong>\n",
      "                            Chatbots are among the most intuitive uses of LLMs. LangChain can be used to provide proper context for the specific use of a chatbot, and to integrate chatbots into existing communication channels and workflows with their own APIs.\n",
      "                           </li>\n",
      "                           <li>\n",
      "                            <strong>\n",
      "                             Summarization:\n",
      "                            </strong>\n",
      "                            Language models can be tasked with summarizing many types of text, from breaking down complex academic articles and transcripts to providing a digest of incoming emails.\n",
      "                           </li>\n",
      "                           <li>\n",
      "                            <strong>\n",
      "                             Question answering\n",
      "                            </strong>\n",
      "                            : Using specific documents or specialized knowledge bases (like Wolfram, arXiv or PubMed), LLMs can retrieve relevant information from storage and articulate helpful answers). If fine-tuned or properly prompted, some LLMs can answer many questions even without external information.\n",
      "                           </li>\n",
      "                           <li>\n",
      "                            <strong>\n",
      "                             Data augmentation:\n",
      "                            </strong>\n",
      "                            LLMs can be used to generate\n",
      "                            <a href=\"https://www.ibm.com/topics/synthetic-data\" rel=\"noopener noreferrer\" style=\"\tfont-family: inherit;\n",
      "\tbackground-color: rgb(255,255,255);\n",
      "\" target=\"_self\">\n",
      "                             synthetic data\n",
      "                            </a>\n",
      "                            for use in machine learning. For example, an LLM can be trained to generate additional data samples that closely resemble the data points in a training dataset.\n",
      "                           </li>\n",
      "                           <li>\n",
      "                            <a href=\"https://www.ibm.com/topics/virtual-agent\" rel=\"noopener noreferrer\" style=\"\tfont-family: inherit;\n",
      "\tbackground-color: rgb(255,255,255);\n",
      "\" target=\"_self\">\n",
      "                             <strong>\n",
      "                              Virtual agents\n",
      "                             </strong>\n",
      "                            </a>\n",
      "                            : Integrated with the right workflows, LangChain’s Agent modules can use an LLM to autonomously determine next steps and take action using robotic process automation (RPA).\n",
      "                           </li>\n",
      "                          </ul>\n",
      "                         </div>\n",
      "                        </dds-content-group-paragraph>\n",
      "                       </dds-content-group-copy>\n",
      "                       <div class=\"cmp-container\" id=\"simple-series-items-container-2607349d6f\">\n",
      "                       </div>\n",
      "                      </dds-content-group-simple>\n",
      "                     </div>\n",
      "                    </div>\n",
      "                   </dds-content-block-mixed>\n",
      "                  </div>\n",
      "                 </div>\n",
      "                 <div horizontal-ruler=\"\">\n",
      "                  <dds-hr>\n",
      "                  </dds-hr>\n",
      "                 </div>\n",
      "                </div>\n",
      "                <div class=\"item-horizontal-group\">\n",
      "                 <div class=\"bx--content-block\">\n",
      "                  <a data-title=\"Products\" name=\"Products\">\n",
      "                  </a>\n",
      "                  <dds-content-block-horizontal border=\"\" content-section-body=\"\">\n",
      "                   <dds-content-block-heading aria-level=\"2\" data-autoid=\"dds--content-block__heading\" role=\"heading\" slot=\"heading\">\n",
      "                    Related solutions\n",
      "                   </dds-content-block-heading>\n",
      "                   <div class=\"content-item-horizontal\">\n",
      "                    <dds-content-item-horizontal>\n",
      "                     <dds-content-item-heading>\n",
      "                      watsonx.ai\n",
      "                     </dds-content-item-heading>\n",
      "                     <dds-content-item-horizontal-copy>\n",
      "                      <caem-paragraph size=\"normal\">\n",
      "                       <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-e65979b8b8\">\n",
      "                        <p>\n",
      "                         Train, validate, tune and deploy generative AI, foundation models and machine learning capabilities with ease and build AI applications in a fraction of the time with a fraction of the data.\n",
      "                        </p>\n",
      "                       </div>\n",
      "                      </caem-paragraph>\n",
      "                     </dds-content-item-horizontal-copy>\n",
      "                     <dds-link-list slot=\"footer\" type=\"vertical\">\n",
      "                      <dds-link-list-item-cta cta-type=\"local\" href=\"https://www.ibm.com/products/watsonx-ai\" icon-placement=\"right\" role=\"listitem\" target=\"_self\">\n",
      "                       Explore watsonx.ai\n",
      "                      </dds-link-list-item-cta>\n",
      "                     </dds-link-list>\n",
      "                    </dds-content-item-horizontal>\n",
      "                    <div>\n",
      "                     <hr class=\"bx--hr\"/>\n",
      "                    </div>\n",
      "                   </div>\n",
      "                   <div class=\"content-item-horizontal\">\n",
      "                    <dds-content-item-horizontal>\n",
      "                     <dds-content-item-heading>\n",
      "                      AI consulting services\n",
      "                     </dds-content-item-heading>\n",
      "                     <dds-content-item-horizontal-copy>\n",
      "                      <caem-paragraph size=\"normal\">\n",
      "                       <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-73b476e8ac\">\n",
      "                        <p>\n",
      "                         Reimagine how you work with AI: our diverse, global team of more than 20,000 AI experts can help you quickly and confidently design and scale AI and automation across your business, working across our own IBM watsonx technology and an open ecosystem of partners to deliver any AI model, on any cloud, guided by ethics and trust.\n",
      "                        </p>\n",
      "                       </div>\n",
      "                      </caem-paragraph>\n",
      "                     </dds-content-item-horizontal-copy>\n",
      "                     <dds-link-list slot=\"footer\" type=\"vertical\">\n",
      "                      <dds-link-list-item-cta cta-type=\"local\" href=\"https://www.ibm.com/consulting/artificial-intelligence\" icon-placement=\"right\" role=\"listitem\" target=\"_self\">\n",
      "                       Explore IBM AI consulting services\n",
      "                      </dds-link-list-item-cta>\n",
      "                     </dds-link-list>\n",
      "                    </dds-content-item-horizontal>\n",
      "                    <div>\n",
      "                     <hr class=\"bx--hr\"/>\n",
      "                    </div>\n",
      "                   </div>\n",
      "                   <div class=\"content-item-horizontal\">\n",
      "                    <dds-content-item-horizontal>\n",
      "                     <dds-content-item-heading>\n",
      "                      watsonx.data\n",
      "                     </dds-content-item-heading>\n",
      "                     <dds-content-item-horizontal-copy>\n",
      "                      <caem-paragraph size=\"normal\">\n",
      "                       <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-266e8ed93f\">\n",
      "                        <p>\n",
      "                         Scale analytics and AI workloads for all your data, anywhere with watsonx.data, the industry’s only data store that is open, hybrid and governed.\n",
      "                        </p>\n",
      "                       </div>\n",
      "                      </caem-paragraph>\n",
      "                     </dds-content-item-horizontal-copy>\n",
      "                     <dds-link-list slot=\"footer\" type=\"vertical\">\n",
      "                      <dds-link-list-item-cta cta-type=\"local\" href=\"https://www.ibm.com/products/watsonx-data\" icon-placement=\"right\" role=\"listitem\" target=\"_self\">\n",
      "                       Explore watsonx.data\n",
      "                      </dds-link-list-item-cta>\n",
      "                     </dds-link-list>\n",
      "                    </dds-content-item-horizontal>\n",
      "                   </div>\n",
      "                  </dds-content-block-horizontal>\n",
      "                 </div>\n",
      "                </div>\n",
      "                <div class=\"block-card-container container responsivegrid\">\n",
      "                 <dds-content-block-cards content-section-body=\"\" data-autoid=\"dds--content-block-cards\">\n",
      "                  <a data-title=\"Resources\" name=\"Resources\" slot=\"heading\">\n",
      "                  </a>\n",
      "                  <dds-content-block-heading data-autoid=\"dds--content-block__heading\" role=\"heading\" slot=\"heading\">\n",
      "                   <a data-title=\"Resources\" name=\"Resources\">\n",
      "                   </a>\n",
      "                   LangChain resources\n",
      "                  </dds-content-block-heading>\n",
      "                  <dds-content-block-copy>\n",
      "                   <dds-content-block-paragraph>\n",
      "                    <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-5ef784422c\">\n",
      "                     <p>\n",
      "                      Tools, tips and sample code to begin building applications with LangChain and watsonx.\n",
      "                     </p>\n",
      "                    </div>\n",
      "                   </dds-content-block-paragraph>\n",
      "                  </dds-content-block-copy>\n",
      "                  <dds-card-group cards-per-row=\"3\" grid-mode=\"border\">\n",
      "                   <dds-card-group-item cta-type=\"local\" href=\"https://www.ibm.com/granite\" no-poster=\"true\" pictogram-placement=\"top\" target=\"_self\">\n",
      "                    <dds-card-eyebrow aria-hidden=\"true\" class=\"bx--card__eyebrow\" slot=\"eyebrow\">\n",
      "                     AI Models\n",
      "                    </dds-card-eyebrow>\n",
      "                    <dds-card-heading aria-level=\"3\" class=\"bx--card__heading\" role=\"heading\" slot=\"heading\">\n",
      "                     Discover IBM's Granite LLM\n",
      "                    </dds-card-heading>\n",
      "                    <p>\n",
      "                     Granite is IBM's flagship series of LLM foundation models based on decoder-only transformer architecture. Granite language models are trained on trusted enterprise data spanning internet, academic, code, legal and finance.\n",
      "                    </p>\n",
      "                    <dds-card-cta-footer aria-hidden=\"true\" class=\"bx--card__footer-icon-size\" cta-type=\"local\" href=\"https://www.ibm.com/granite\" icon-placement=\"right\" slot=\"footer\" target=\"_self\">\n",
      "                    </dds-card-cta-footer>\n",
      "                   </dds-card-group-item>\n",
      "                   <dds-card-group-item cta-type=\"local\" data-video-modal-type=\"media-center\" href=\"https://www.ibm.com/account/reg/signup?formid=urx-53231\" no-poster=\"true\" pictogram-placement=\"top\" target=\"_blank\">\n",
      "                    <dds-card-eyebrow aria-hidden=\"true\" class=\"bx--card__eyebrow\" slot=\"eyebrow\">\n",
      "                     Report\n",
      "                    </dds-card-eyebrow>\n",
      "                    <dds-card-heading aria-level=\"3\" class=\"bx--card__heading\" role=\"heading\" slot=\"heading\">\n",
      "                     AI in Action 2024\n",
      "                    </dds-card-heading>\n",
      "                    <p>\n",
      "                     We surveyed 2,000 organizations about their AI initiatives to discover what’s working, what’s not and how you can get ahead.\n",
      "                    </p>\n",
      "                    <dds-card-cta-footer aria-hidden=\"true\" class=\"bx--card__footer-icon-size\" cta-type=\"local\" data-video-modal-type=\"media-center\" href=\"https://www.ibm.com/account/reg/signup?formid=urx-53231\" icon-placement=\"right\" slot=\"footer\" target=\"_blank\">\n",
      "                    </dds-card-cta-footer>\n",
      "                   </dds-card-group-item>\n",
      "                   <dds-card-group-item cta-type=\"local\" data-video-modal-type=\"media-center\" href=\"https://www.ibm.com/account/reg/signup?formid=urx-52356\" no-poster=\"true\" pictogram-placement=\"top\" target=\"_self\">\n",
      "                    <dds-card-eyebrow aria-hidden=\"true\" class=\"bx--card__eyebrow\" slot=\"eyebrow\">\n",
      "                     Ebook\n",
      "                    </dds-card-eyebrow>\n",
      "                    <dds-card-heading aria-level=\"3\" class=\"bx--card__heading\" role=\"heading\" slot=\"heading\">\n",
      "                     Unlock competitive advantage with generative AI + ML\n",
      "                    </dds-card-heading>\n",
      "                    <p>\n",
      "                     Explore the benefits of generative AI and ML and learn how to confidently incorporate these technologies into your business.\n",
      "                    </p>\n",
      "                    <dds-card-cta-footer aria-hidden=\"true\" class=\"bx--card__footer-icon-size\" cta-type=\"local\" data-video-modal-type=\"media-center\" href=\"https://www.ibm.com/account/reg/signup?formid=urx-52356\" icon-placement=\"right\" slot=\"footer\" target=\"_self\">\n",
      "                    </dds-card-cta-footer>\n",
      "                   </dds-card-group-item>\n",
      "                   <dds-card-group-item cta-type=\"local\" href=\"https://www.ibm.com/docs/en/watsonx/saas?topic=models-prompt-tips\" no-poster=\"true\" pictogram-placement=\"top\" target=\"_self\">\n",
      "                    <dds-card-eyebrow aria-hidden=\"true\" class=\"bx--card__eyebrow\" slot=\"eyebrow\">\n",
      "                     Documentation\n",
      "                    </dds-card-eyebrow>\n",
      "                    <dds-card-heading aria-level=\"3\" class=\"bx--card__heading\" role=\"heading\" slot=\"heading\">\n",
      "                     Tips for writing foundation model prompts\n",
      "                    </dds-card-heading>\n",
      "                    <p>\n",
      "                     Part art, part science, prompt engineering is the process of crafting prompt text to best effect for a given model and parameters. These tips will help you successfully prompt most text-generating foundation models.\n",
      "                    </p>\n",
      "                    <dds-card-cta-footer aria-hidden=\"true\" class=\"bx--card__footer-icon-size\" cta-type=\"local\" href=\"https://www.ibm.com/docs/en/watsonx/saas?topic=models-prompt-tips\" icon-placement=\"right\" slot=\"footer\" target=\"_self\">\n",
      "                    </dds-card-cta-footer>\n",
      "                   </dds-card-group-item>\n",
      "                   <dds-card-group-item cta-type=\"local\" href=\"https://dataplatform.cloud.ibm.com/exchange/public/entry/view/c3dbf23a-9a56-4c4b-8ce5-5707828fc981?context=wx\" no-poster=\"true\" pictogram-placement=\"top\" target=\"_self\">\n",
      "                    <dds-card-eyebrow aria-hidden=\"true\" class=\"bx--card__eyebrow\" slot=\"eyebrow\">\n",
      "                     Sample code\n",
      "                    </dds-card-eyebrow>\n",
      "                    <dds-card-heading aria-level=\"3\" class=\"bx--card__heading\" role=\"heading\" slot=\"heading\">\n",
      "                     Use watsonx and LangChain to call a language model\n",
      "                    </dds-card-heading>\n",
      "                    <p>\n",
      "                     This notebook contains the steps and code to demonstrate Simple Sequential Chain using langchain integration with watsonx models. Some familiarity with Python is helpful.\n",
      "                    </p>\n",
      "                    <dds-card-cta-footer aria-hidden=\"true\" class=\"bx--card__footer-icon-size\" cta-type=\"local\" href=\"https://dataplatform.cloud.ibm.com/exchange/public/entry/view/c3dbf23a-9a56-4c4b-8ce5-5707828fc981?context=wx\" icon-placement=\"right\" slot=\"footer\" target=\"_self\">\n",
      "                    </dds-card-cta-footer>\n",
      "                   </dds-card-group-item>\n",
      "                  </dds-card-group>\n",
      "                 </dds-content-block-cards>\n",
      "                </div>\n",
      "               </div>\n",
      "              </div>\n",
      "             </div>\n",
      "            </div>\n",
      "           </div>\n",
      "          </div>\n",
      "          <div class=\"experiencefragment\">\n",
      "           <div class=\"cmp-experiencefragment cmp-experiencefragment--watsonx-ai\" id=\"experiencefragment-dafcd502c7\">\n",
      "            <div class=\"xf-content-height\">\n",
      "             <div class=\"root container responsivegrid\">\n",
      "              <div class=\"cmp-container\" id=\"container-3c41aa7cd2\">\n",
      "               <div class=\"next-steps container responsivegrid\">\n",
      "                <div class=\"cmp-container\" id=\"next-steps-a5040a33cd\">\n",
      "                 <div class=\"next-steps-container\">\n",
      "                  <div class=\"bx--next-steps-container\" data-cmp-is=\"next-steps-container\">\n",
      "                   <div class=\"bx--cta-section--g10 bx--cta-section--theme\" data-component-identifier=\"next-steps\" data-component-name=\"next-steps\" data-dynamic-content-type=\"ADOBE_TARGET\">\n",
      "                    <dds-cta-block class=\"cta-block-section__container\" no-border=\"\">\n",
      "                     <div class=\"bx--content-block__heading\" data-autoid=\"dds--content-block__heading\" role=\"heading\" slot=\"heading\">\n",
      "                      <a data-title=\"Take the next step\" name=\"Take+the+next+step\">\n",
      "                      </a>\n",
      "                      <span class=\"enhanced-title\">\n",
      "                       Take the next step\n",
      "                      </span>\n",
      "                     </div>\n",
      "                     <dds-content-block-copy size=\"sm\">\n",
      "                      <dds-content-block-paragraph>\n",
      "                       <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-817ea9ba07\">\n",
      "                        <p>\n",
      "                         Train, validate, tune and deploy generative AI, foundation models and machine learning capabilities with IBM watsonx.ai, a next-generation enterprise studio for AI builders. Build AI applications in a fraction of the time with a fraction of the data.\n",
      "                        </p>\n",
      "                       </div>\n",
      "                      </dds-content-block-paragraph>\n",
      "                     </dds-content-block-copy>\n",
      "                     <dds-button-group slot=\"action\">\n",
      "                      <dds-button-cta cta-type=\"local\" href=\"https://www.ibm.com/products/watsonx-ai\" kind=\"primary\" target=\"_self\">\n",
      "                       <span>\n",
      "                        Explore watsonx.ai\n",
      "                       </span>\n",
      "                      </dds-button-cta>\n",
      "                      <dds-button-cta cta-type=\"local\" href=\"https://www.ibm.com/account/reg/signup?formid=DEMO-dataaiwatsonxai\" kind=\"tertiary\" target=\"_blank\">\n",
      "                       <span>\n",
      "                        Book a live demo\n",
      "                       </span>\n",
      "                      </dds-button-cta>\n",
      "                     </dds-button-group>\n",
      "                    </dds-cta-block>\n",
      "                   </div>\n",
      "                  </div>\n",
      "                 </div>\n",
      "                </div>\n",
      "               </div>\n",
      "              </div>\n",
      "             </div>\n",
      "            </div>\n",
      "           </div>\n",
      "          </div>\n",
      "          <div class=\"footnotes\">\n",
      "           <div driverlocation=\"1-2\">\n",
      "            <div class=\"bx--content-block\" data-autoid=\"dds--content-block\">\n",
      "             <div class=\"bx--grid\">\n",
      "              <div class=\"bx--row\">\n",
      "               <div class=\"bx--col-lg-4\">\n",
      "                <div class=\"bx--content-block-segmented\" data-autoid=\"dds--content-block-segmented\">\n",
      "                 <h5>\n",
      "                  Footnotes\n",
      "                 </h5>\n",
      "                </div>\n",
      "               </div>\n",
      "               <div class=\"bx--col-lg-12\">\n",
      "                <div class=\"bx--content-block-segmented\" data-autoid=\"dds--content-block-segmented\">\n",
      "                 <div class=\"cms-richtext\" data-dynamic-inner-content=\"description\" id=\"rich-text-5137c18379\">\n",
      "                  <p>\n",
      "                   <sup>\n",
      "                    1\n",
      "                   </sup>\n",
      "                   <a href=\"https://runacap.com/ross-index/q2_2023/\" rel=\"noopener noreferrer\" target=\"_blank\">\n",
      "                    The fastest-growing open-source startups in Q2 2023\n",
      "                   </a>\n",
      "                   (link resides outside ibm.com), Runa Capital, 2023\n",
      "                  </p>\n",
      "                 </div>\n",
      "                </div>\n",
      "               </div>\n",
      "              </div>\n",
      "             </div>\n",
      "            </div>\n",
      "           </div>\n",
      "          </div>\n",
      "         </div>\n",
      "        </div>\n",
      "       </div>\n",
      "      </dds-table-of-contents>\n",
      "     </div>\n",
      "     <div class=\"container responsivegrid\">\n",
      "      <div class=\"cmp-container\" id=\"container-a2cf03bed8\">\n",
      "      </div>\n",
      "     </div>\n",
      "     <div class=\"footer\">\n",
      "      <dds-footer-container class=\"cmp-footer\" data-autoid=\"dds--footer\" data-children-count=\"0\" data-cmp-is=\"footer\" size=\"tall\">\n",
      "      </dds-footer-container>\n",
      "     </div>\n",
      "    </div>\n",
      "   </div>\n",
      "  </dds-video-cta-container>\n",
      "  <div aria-hidden=\"true\" class=\"video-modal-overlay\">\n",
      "   <div class=\"video-modal-overlay__dialog\">\n",
      "    <div class=\"video-modal-overlay__dialog--body\">\n",
      "     <div class=\"video-modal-overlay__dialog--body__close-btn\">\n",
      "      <svg fill=\"none\" height=\"24\" viewbox=\"0 0 24 24\" width=\"24\" xmlns=\"http://www.w3.org/2000/svg\">\n",
      "       <rect fill=\"white\" fill-opacity=\"0.01\" height=\"24\" style=\"mix-blend-mode:multiply\" width=\"24\">\n",
      "       </rect>\n",
      "       <path d=\"M13.0606 12L19.5 5.56058L18.4394 4.5L12 10.9394L5.56072 4.5L4.5 5.56058L10.9394 12L4.5 18.4394L5.56072 19.5L12 13.0606L18.4394 19.5L19.5 18.4394L13.0606 12Z\" fill=\"#161616\" id=\"Vector\">\n",
      "       </path>\n",
      "      </svg>\n",
      "     </div>\n",
      "     <div class=\"video-modal-overlay__dialog--body-content\">\n",
      "      <div class=\"video-modal-overlay__loading-container\">\n",
      "       <bx-loading>\n",
      "       </bx-loading>\n",
      "      </div>\n",
      "      <div class=\"video-wms hide\">\n",
      "       <iframe allowfullscreen=\"\" class=\"video-modal-content\" data-original-src=\"https://video.ibm.com/embed/$[wmsChanneId]?volume=0&amp;autoplay=true&amp;controls=true\" frameborder=\"no\" referrerpolicy=\"no-referrer-when-downgrade\" src=\"\" webkitallowfullscreen=\"\">\n",
      "       </iframe>\n",
      "      </div>\n",
      "      <div class=\"video-media-center hide\">\n",
      "       <dds-video-player-container aspect-ratio=\"16x9\" auto-play=\"\" muted=\"\" playing-mode=\"inline\" video-id=\"\">\n",
      "       </dds-video-player-container>\n",
      "      </div>\n",
      "     </div>\n",
      "    </div>\n",
      "   </div>\n",
      "  </div>\n",
      "  <dds-lightbox-video-player-container>\n",
      "  </dds-lightbox-video-player-container>\n",
      "  <script src=\"/etc.clientlibs/adobe-cms/clientlibs/clientlib-idlBundle.lc-184f1369edfafb914817c747b0b7778e-lc.min.js\">\n",
      "  </script>\n",
      "  <script src=\"/etc.clientlibs/adobe-cms/clientlibs/clientlib-target-antiflicker.lc-36a1efb28e947b5873f4a28c2cf9d5eb-lc.min.js\">\n",
      "  </script>\n",
      "  <script src=\"/etc.clientlibs/adobe-cms/clientlibs/clientlib-idlStyles.lc-f5a7744980f951b41a897229a7802b6f-lc.min.js\">\n",
      "  </script>\n",
      "  <script async=\"\" defer=\"\" src=\"https://www.ibm.com/common/digitaladvisor/cm-app/latest/cm-app.min.js\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon/web-components/version/v1.47.1/tag.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon/web-components/version/v1.47.1/code-snippet.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon/web-components/version/v1.47.1/loading.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/leadspace.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/content-block-horizontal.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/content-item-horizontal.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/table-of-contents.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/tag-group.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/content-section.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/content-group-heading.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/content-block-simple.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/content-block-cards.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/cta-block.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/content-group.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/background-media.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/content-block-mixed.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/footer.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/content-group-simple.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/card-group.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/text-cta.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/content-block.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/link-list.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/link-list-item-cta.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/video-player.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/button-group.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/card-cta-footer.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/content-item.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/button-cta.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/content-block-segmented.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/horizontal-rule.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/lightbox-video-player.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon-for-ibm-dotcom/version/v1.51.1/video-cta-container.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon/carbon-for-aem/version/v0.19.0/paragraph.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon/carbon-for-aem/version/v0.19.0/breadcrumb.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script src=\"https://1.www.s81c.com/common/carbon/carbon-for-aem/version/v0.19.0/star-rating.min.js\" type=\"module\">\n",
      "  </script>\n",
      "  <script id=\"anti-flicker-script\" type=\"module\">\n",
      "   document.getElementById('anti-flicker-style').remove();\n",
      "        document.getElementById('anti-flicker-script').remove();\n",
      "  </script>\n",
      "  <script defer=\"\" src=\"/etc.clientlibs/adobe-cms/clientlibs/clientlib-pdfviewer.lc-e4412705ab74ab8b0e2e21371e0762c9-lc.min.js\">\n",
      "  </script>\n",
      "  <!-- Added for Adobe analytics implementation ADCMS-5834\n",
      "<script type=\"text/javascript\">\n",
      "\n",
      "    adobeDataLayer.push({\n",
      "        \"event\": \"linkClick\",\n",
      "        \"web\": {\n",
      "            \"webPageDetails\": {\n",
      "                \"URL\": document.URL,\n",
      "                \"name\": \"home\"\n",
      "            },\n",
      "            \"webInteraction\": {\n",
      "                \"linkClick\":\"event\",\n",
      "                \"value\":\"1\",\n",
      "                \"type\": \"other\",\n",
      "                \"URL\": document.URL,\n",
      "                \"name\": \"linkClick: \" +  document.URL\n",
      "            }\n",
      "        },\n",
      "    })\n",
      "</script>  !-->\n",
      "  <!-- Added for Adobe analytics implementation ADCMS-5834 & ADCMS-6152 !-->\n",
      "  <script type=\"text/javascript\">\n",
      "   document.addEventListener(\"DOMContentLoaded\", function() {\n",
      "        var templatePath = \"\\/conf\\/adobe\\u002Dcms\\u002Deditable\\/settings\\/wcm\\/templates\\/learn\";\n",
      "        var templateName = templatePath.split('/').pop();\n",
      "        const currentUrl = document.URL;\n",
      "        const parts = currentUrl.split(\"/\");\n",
      "        const sectionOne = parts.length > 3 ? parts[3].split(\"?\")[0] : \"\";\n",
      "        const sectionTwo = parts.length > 4 ? parts[4].split(\"?\")[0] : \"\";\n",
      "        const sectionThree = parts.length > 5 ? parts[5].split(\"?\")[0] : \"\";\n",
      "\n",
      "        function getCookieByName(name) {\n",
      "            const cookieArr = document.cookie.split(';');\n",
      "            for (let i = 0; i < cookieArr.length; i++) {\n",
      "                const cookie = cookieArr[i].trim();\n",
      "                if (cookie.startsWith(name + \"=\")) {\n",
      "                    return cookie.substring(name.length + 1);\n",
      "                }\n",
      "            }\n",
      "            return null;\n",
      "        }\n",
      "        function cleanUrl(url) {\n",
      "            let modifiedUrl = url;\n",
      "            if (modifiedUrl.startsWith(\"www\")) {\n",
      "                modifiedUrl = modifiedUrl.replace(/^www\\.?/, \"\");\n",
      "            }\n",
      "            return modifiedUrl;\n",
      "        }\n",
      "\n",
      "        const cisSessionId = getCookieByName(\"CISSESSIONIDP07A\");\n",
      "        let loginStatus = cisSessionId ? \"logged in\" : \"logged out\";\n",
      "\n",
      "        adobeDataLayer.push({\n",
      "            \"event\": \"pageLoad\",\n",
      "            \"_ibm\": {\n",
      "                \"page\": {\n",
      "                    \"siteSection2\": sectionTwo,\n",
      "                    \"siteSection3\": sectionThree,\n",
      "                    \"pageType\": templateName,\n",
      "                    \"domain\": cleanUrl(window.location.host),\n",
      "                    \"siteLanguage\": \"en\",\n",
      "                    \"pageName\": cleanUrl(window.location.host) + \" | \" + templateName + \" | LangChain\",\n",
      "                    \"siteCountry\": \"us\"\n",
      "                },\n",
      "                \"user\": {\n",
      "                    \"userAgent\": navigator.userAgent,\n",
      "                    \"loginStatus\": loginStatus\n",
      "                }\n",
      "            },\n",
      "            \"web\": {\n",
      "                \"webPageDetails\": {\n",
      "                    \"pageViews\": { \"value\": 1 },\n",
      "                    \"URL\": document.URL,\n",
      "                    \"name\": \"LangChain\",\n",
      "                    \"siteSection\": sectionOne\n",
      "                },\n",
      "                \"webReferrer\": {\n",
      "                    \"URL\": document.referrer\n",
      "                }\n",
      "            }\n",
      "        });\n",
      "\n",
      "        document.addEventListener('click', function(event) {\n",
      "            if (event.target.closest('.WACLauncher__ButtonContainer')) {\n",
      "                handleClick(event);\n",
      "            }\n",
      "        });\n",
      "\n",
      "        function handleClick(event) {\n",
      "            adobeDataLayer.push({\n",
      "                \"event\": \"chatInitialized\",\n",
      "                \"_ibm\": {\n",
      "                    \"chat\": {\n",
      "                        \"chatBotClick\": { \"value\": 1 },\n",
      "                        \"liveChatType\": \"Support or Sales\",\n",
      "                        \"chatLanguage\": \"en\",\n",
      "                        \"chatCountry\": \"us\"\n",
      "                    },\n",
      "                    \"click\": {\n",
      "                        \"linkTileNumber\": 1\n",
      "                    }\n",
      "                },\n",
      "                \"web\": {\n",
      "                    \"webPageDetails\": {\n",
      "                        \"URL\": document.URL,\n",
      "                        \"name\": \"LangChain\"\n",
      "                    },\n",
      "                    \"webInteraction\": {\n",
      "                        \"name\": \"chatInitialized\",\n",
      "                        \"URL\": document.URL,\n",
      "                        \"type\": \"other\"\n",
      "                    }\n",
      "                }\n",
      "            });\n",
      "        }\n",
      "    });\n",
      "  </script>\n",
      "  <script src=\"/0grriJA0WInNYxZHPg9B_tMXkf0/it7hk4rbpzhXmOa7/WChw/Ak5X/U212R3AB\" type=\"text/javascript\">\n",
      "  </script>\n",
      "  <link href=\"/0grriJA0WInNYxZHPg9B_tMXkf0/6k7hJi/LiNi/M345/KBJGA29X\" rel=\"stylesheet\" type=\"text/css\"/>\n",
      "  <script async=\"\" defer=\"\" src=\"/0grriJA0WInNYxZHPg9B_tMXkf0/6k7hJi/LiNi/Zjk2/fGtLLmAo\">\n",
      "  </script>\n",
      "  <div id=\"sec-overlay\" style=\"display:none;\">\n",
      "   <div id=\"sec-container\">\n",
      "   </div>\n",
      "  </div>\n",
      " </body>\n",
      "</html>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "url = 'https://www.ibm.com/topics/langchain'\n",
    "response = requests.get(url)\n",
    "\n",
    "soup = BeautifulSoup(response.content, 'html.parser')\n",
    "print(soup.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the print output, we can see that `BeautifulSoup` not only load the web content, but also a lot of HTML tags and external links, which are not necessary if we just want to load the text content of the web.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So LangChain's `WebBaseLoader` can effectively address this limitation.\n",
    "\n",
    "`WebBaseLoader` is designed to extract all text from HTML webpages and convert it into a document format suitable for further processing.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load from single web page\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = WebBaseLoader(\"https://www.ibm.com/topics/langchain\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': 'https://www.ibm.com/topics/langchain', 'title': 'What Is LangChain? | IBM', 'description': 'LangChain is an open source orchestration framework for the development of applications using large language models (LLMs), like chatbots and virtual agents.\\u202f', 'language': 'en'}, page_content=\"\\n\\n\\n\\n\\n\\n\\n\\n\\nWhat Is LangChain? | IBM\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nHome\\n\\n\\n\\n\\nTopics\\n\\n\\n\\nLangChain\\n\\n\\n\\n                    \\n\\n\\n\\n  \\n    What is LangChain?\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n                \\n\\n\\n\\n\\nUse LangChain with watsonx.ai\\n\\n\\nSign up for AI updates\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n  \\n\\n\\n\\n    What is LangChain?\\n\\n\\n\\n\\n\\n    \\n\\n\\n            \\n        \\n\\n\\n\\n\\nLangChain is an open source orchestration framework for the development of applications using large language models (LLMs). Available in both Python- and Javascript-based libraries, LangChain’s tools and APIs simplify the process of building LLM-driven applications like chatbots and virtual agents.\\u202f\\n\\n\\n\\n\\n\\n\\nLangChain serves as a generic interface for nearly any LLM, providing a centralized development environment to build LLM applications and integrate them with external data sources and software workflows. LangChain’s module-based approach allows developers and data scientists to dynamically compare different prompts and even different foundation models with minimal need to rewrite code. This modular environment also allows for programs that use multiple LLMs: for example, an application that uses one LLM to interpret user queries and another LLM to author a response.\\u202f\\nLaunched by Harrison Chase in October 2022, LangChain enjoyed a meteoric rise to prominence: as of June 2023, it was the single fastest-growing open source project on Github.1 Coinciding with the momentous launch of OpenAI’s ChatGPT the following month, LangChain has played a significant role in making generative AI more accessible to enthusiasts in the wake of its widespread popularity.\\u202f\\nLangChain can facilitate most use cases for LLMs and natural language processing (NLP), like chatbots, intelligent search, question-answering, summarization services or even virtual agents capable of robotic process automation.\\n\\n\\n\\n\\n\\n      \\n\\n\\n\\n  \\n    Integrations with LLMs\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n    \\n\\n\\nLLMs are not standalone applications: they are pre-trained statistical models that must be paired with an application (and, in some cases, specific data sources) in order to meet their purpose.\\u202f\\nFor example, Chat-GPT is not an LLM: it is a chatbot application that, depending on the version you’ve chosen, uses the GPT-3.5 or GPT-4 language model. While it’s the GPT model that interprets the user’s input and composes a natural language response, it’s the application that (among other things) provides an interface for the user to type and read and a UX design that governs the chatbot experience. Even at the enterprise level, Chat-GPT is not the only application using the GPT model: Microsoft uses GPT-4 to power Bing Chat.\\u202f\\nFurthermore, though foundation models (like those powering LLMs) are pre-trained on massive datasets, they are not omniscient. If a particular task requires access to specific contextual information, like internal documentation or domain expertise, LLMs must be connected to those external data sources. Even if you simply want your model to reflect real-time awareness of current events, it requires external information: a model’s internal data is only up-to-date through the time period during which it was pre-trained.\\u202f\\nLikewise, if a given generative AI task requires access to external software workflows—for example, if you wanted your virtual agent to integrate with Slack—then you will need a way to integrate the LLM with the API for that software.\\u202f\\nWhile these integrations can generally be achieved with fully manual code, orchestration frameworks like LangChain and the IBM watsonx platform greatly simplify the process. They also make it much easier to experiment with different LLMs to compare results, as different models can be swapped in and out with minimal changes to code. \\u202f\\n\\n\\n\\n\\n\\n\\n\\n            Ebook\\n        \\n\\n            How to choose the right foundation model\\n        \\nLearn how to choose the right approach in preparing data sets and employing foundation models.\\n\\nRead the ebook\\n\\n\\n\\n\\n\\n            \\n\\n\\n  \\n  \\n      Related content\\n  \\n\\n\\n\\n\\n    \\n\\n\\n        \\n\\nRead the 2024 AI in Action Report\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n  \\n\\n\\n\\n    How does LangChain work?\\n\\n\\n\\n\\n\\n    \\n\\n\\n            \\n        \\n\\n\\n\\n\\nAt LangChain’s core is a development environment that streamlines the programming of LLM applications through the use of\\xa0abstraction: the simplification of code by representing one or more complex processes as a named component that encapsulates all of its constituent steps.\\n\\n\\n\\n\\n\\n\\nAbstractions are a common element of everyday life and language. For example, “π” allows us to represent the ratio of the length of a circle’s circumference to that of its diameter without having to write out its infinite digits. Similarly, a thermostat allows us to control the temperature in our home without needing to understand the complex circuitry this entails—we only need to know how different thermostat settings translate to different temperatures.\\nLangChain is essentially a library of abstractions for Python and Javascript, representing common steps and concepts necessary to work with language models. These modular components—like functions and object classes—serve as the building blocks of generative AI programs. They can be “chained” together to create applications, minimizing the amount of code and fine understanding required to execute complex NLP tasks. Though LangChain’s abstracted approach may limit the extent to which an expert programmer can finely customize an application, it empowers specialists and newcomers alike to quickly experiment and prototype.\\n\\n\\n\\n\\n\\n      \\n\\n\\n\\n  \\n    Importing language models\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n    \\n\\n\\nNearly any LLM can be used in LangChain. Importing language models into LangChain is easy, provided you have an API key. The LLM class is designed to provide a standard interface for all models.\\nMost LLM providers will require you to create an account in order to receive an API key. Some of these APIs—particularly those for proprietary closed-source models, like those offered by OpenAI or Anthropic—may have associated costs.\\nMany open source models, like BigScience’s BLOOM, Meta AI’s LLaMa and Google’s Flan-T5, can be accessed through Hugging Face (link resides outside ibm.com). IBM watsonx, through its partnership with Hugging Face, also offers a curated suite of open source models. Creating an account with either service will allow you to generate an API key for any of the models offered by that provider.\\nLangChain is not limited to out-of-the-box foundation models: the CustomLLM class\\xa0(link resides outside ibm.com) allows for custom LLM wrappers. Likewise, you can use the IBM watsonx APIs and Python SDK, which includes a LangChain integration, to build applications in LangChain with models that you’ve already trained or fine-tuned for your specific needs using the WatsonxLLM class (and that model’s specific project ID).\\n\\n\\nExplore the demo: using watsonx and LangChain to make a series of calls to a language model \\n            \\n        \\n\\n\\n\\n      \\n\\n\\n\\n  \\n    Prompt templates\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n    \\n\\n\\nPrompts are the instructions given to an LLM. The “art” of composing prompts that effectively provide the context necessary for the LLM to interpret input and structure output in the way most useful to you is often called prompt engineering.\\u202f\\nThe PromptTemplate class in LangChain formalizes the composition of prompts without the need to manually hard code context and queries. Important elements of a prompt are likewise entered as formal classes, like input_variables. A prompt template can thus contain and reproduce context, instructions (like “do not use technical terms”), a set of examples to guide its responses (in what is called “few-shot prompting”), a specified output format or a standardized question to be answered.\\u202f You can save and name an effectively structured prompt template and easily reuse it as needed.\\nThough these elements can all be manually coded, PromptTemplate modules empower smooth integration with other LangChain features, like the eponymous chains.\\n\\n\\nWatch the video: prompt engineering and prompt tuning\\n            \\n        \\n\\n\\n\\n      \\n\\n\\n\\n  \\n    Chains\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n    \\n\\n\\nAs its name implies, chains are the core of LangChain’s workflows. They combine LLMs with other components, creating applications by executing a sequence of functions.\\u202f\\nThe most basic chain is LLMChain. It simply calls a model and prompt template for that model. For example, imagine you saved a prompt as “ExamplePrompt” and wanted to run it against Flan-T5. You can import LLMChain from langchain.chains, then define chain_example = LLMChain(llm = flan-t5, prompt = ExamplePrompt). To run the chain for a given input, you simply call chain_example.run(“input”).\\nTo use the output of one function as the input for the next function, you can use SimpleSequentialChain. Each function could utilize different prompts, different tools, different parameters or even different models, depending on your specific needs.\\u202f\\n\\n\\n\\n\\n\\n      \\n\\n\\n\\n  \\n    Indexes\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n    \\n\\n\\nTo achieve certain tasks, LLMs will need access to specific external data sources not included in its training dataset, such as internal documents, emails or datasets. LangChain collectively refers to such external documentation as “indexes”.\\nDocument loaders\\xa0\\r\\nLangChain offers a wide variety of document loaders for third party applications (link resides outside ibm.com). This allows for easy importation of data from sources like file storage services (like Dropbox, Google Drive and Microsoft OneDrive), web content (like YouTube, PubMed or specific URLs), collaboration tools (like Airtable, Trello, Figma and Notion), databases (like Pandas, MongoDB and Microsoft), among many others.\\u202f\\nVector databases\\xa0\\r\\nUnlike “traditional” structured databases, vector databases represent data points by converting them into vector embeddings: numerical representations in the form of vectors with a fixed number of dimensions, often clustering related data points using unsupervised learning methods. This enables low latency queries, even for massive datasets, which greatly increases efficiency. Vector embeddings also store each vector’s metadata, further enhancing search possibilities.\\nLangChain provides integrations for over 25 different embedding methods, as well as for over 50 different vector stores (both cloud-hosted and local).\\u202f\\nText splitters\\xa0\\r\\nTo increase speed and reduce computational demands, it’s often wise to split large text documents into smaller pieces. LangChain’s TextSplitters split text up into small, semantically meaningful chunks that can then be combined using methods and parameters of your choosing.\\nRetrieval\\xa0\\r\\nOnce external sources of knowledge have been connected, the model must be able to quickly retrieve and integrate relevant information as needed. Like watsonx, LangChain offers retrieval augmented generation (RAG):\\xa0its retriever modules accept a string query as an input and return a list of Document’s as output.\\n\\n\\n\\n\\n\\n      \\n\\n\\n\\n  \\n    Memory\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n    \\n\\n\\nLLMs, by default, do not have any long-term memory of prior conversations (unless that chat history is used as input for a query). LangChain solves this problem with simple utilities for adding memory to a system, with options ranging from retaining the entirety of all conversations to retaining a summarization of the conversation thus far to retaining the n\\xa0most recent exchanges.\\n\\n\\n\\n\\n\\n      \\n\\n\\n\\n  \\n    Agents\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n    \\n\\n\\nLangChain agents can use a given language model as a “reasoning engine” to determine which actions to take. When building a chain for an agent, inputs include:\\na list of available tools to be leveraged.user input (like prompts and queries).any relevant previously executed steps.\\n\\n\\nLearn more about agents in LangChain\\n            \\n        \\n\\n\\n\\n      \\n\\n\\n\\n  \\n    Tools\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n    \\n\\n\\nDespite their heralded power and versatility, LLMs have important limitations: namely, a lack of up-to-date information, a lack of domain-specific expertise and a general difficulty with math.\\u202f\\nLangChain tools\\xa0(link resides outside ibm.com) are a set of functions that empower LangChain agents to interact with real-world information in order to expand or improve the services it can provide. Examples of prominent LangChain tools include:\\n\\nWolfram Alpha: provides access to powerful computational and data visualization functions, enabling sophisticated mathematical capabilities.\\nGoogle Search: provides access to Google Search, equipping applications and agents with real-time information.\\nOpenWeatherMap: fetches weather information.\\nWikipedia: provides efficient access to information from Wikipedia articles.\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n  \\n\\n\\n\\n    LangSmith\\n\\n\\n\\n\\n\\n    \\n\\n\\n            \\n        \\n\\n\\n\\nReleased in the fall of 2023, LangSmith aims to bridge the gap between the accessible prototyping capabilities that brought LangChain to prominence and building production-quality LLM applications.\\nLangSmith provides tools to monitor, evaluate and debug applications, including the ability to automatically trace all model calls to spot errors and test performance under different model configurations. This visibility aims to empower more robust, cost-efficient applications.\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n  \\n\\n\\n\\n    Getting started with LangChain\\n\\n\\n\\n\\n\\n    \\n\\n\\n            \\n        \\n\\n\\n\\nLangChain is open source and free to use: source code is\\xa0available for download on Github\\xa0(link resides outside ibm.com).\\u202f\\nLangChain can also be installed on Python with a simple pip command:\\xa0pip install langchain.\\u202f\\xa0To install all LangChain dependencies (rather than only those you find necessary), you can run the command\\xa0pip install langchain[all].\\nMany step-by-step tutorials are available from both the greater LangChain community ecosystem and the official documentation at\\xa0docs.langchain.com\\xa0(link resides outside ibm.com).\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n  \\n\\n\\n\\n    LangChain use cases\\n\\n\\n\\n\\n\\n    \\n\\n\\n                        \\n                    \\n\\n\\n\\n\\nApplications made with LangChain provide great utility for a variety of use cases, from straightforward question-answering and text generation tasks to more complex solutions that use an LLM as a “reasoning engine.”\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nChatbots: Chatbots are among the most intuitive uses of LLMs. LangChain can be used to provide proper context for the specific use of a chatbot, and to integrate chatbots into existing communication channels and workflows with their own APIs.Summarization: Language models can be tasked with summarizing many types of text, from breaking down complex academic articles and transcripts to providing a digest of incoming emails.Question answering: Using specific documents or specialized knowledge bases (like Wolfram, arXiv or PubMed), LLMs can retrieve relevant information from storage and articulate helpful answers). If fine-tuned or properly prompted, some LLMs can answer many questions even without external information.Data augmentation: LLMs can be used to generate synthetic data for use in machine learning. For example, an LLM can be trained to generate additional data samples that closely resemble the data points in a training dataset.Virtual agents: Integrated with the right workflows, LangChain’s Agent modules can use an LLM to autonomously determine next steps and take action using robotic process automation (RPA).\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n                Related solutions\\n            \\n\\n\\n\\n            \\n\\n\\n\\n  \\n    watsonx.ai\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n        \\n\\n\\n\\nTrain, validate, tune and deploy generative AI, foundation models and machine learning capabilities with ease and build AI applications in a fraction of the time with a fraction of the data.\\n\\n\\n\\n\\n\\n            Explore watsonx.ai\\n            \\n            \\n        \\n\\n\\n\\n\\n\\n\\n\\n\\n\\n            \\n\\n\\n\\n  \\n     AI consulting services \\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n        \\n\\n\\n\\nReimagine how you work with\\xa0AI: our diverse, global team of more than 20,000 AI\\xa0experts can help you quickly and confidently\\xa0design and scale AI and automation across your\\xa0business, working across our own IBM\\xa0watsonx\\xa0technology\\xa0and an open ecosystem of partners to deliver any\\xa0AI model, on any cloud, guided by ethics and trust.\\n\\n\\n\\n\\n\\n            Explore IBM AI consulting services\\n            \\n            \\n        \\n\\n\\n\\n\\n\\n\\n\\n\\n\\n            \\n\\n\\n\\n  \\n    watsonx.data\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n        \\n\\n\\n\\nScale analytics and AI workloads for all your data, anywhere with watsonx.data, the industry’s\\xa0only data store that is open, hybrid and governed.\\n\\n\\n\\n\\n\\n            Explore watsonx.data\\n            \\n            \\n        \\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n  \\n\\n\\n\\n    LangChain resources\\n\\n\\n\\n\\n\\n    \\n\\n\\n                \\n            \\n\\n\\n\\nTools, tips and sample code to begin building applications with LangChain and watsonx.\\n\\n\\n\\n\\n\\n\\n            AI Models\\n        \\n\\n            Discover IBM's Granite LLM\\n        \\nGranite is IBM's flagship series of LLM foundation models based on decoder-only transformer architecture. Granite language models are trained on trusted enterprise data spanning internet, academic, code, legal and finance.\\n\\n\\n\\n\\n\\n            Report\\n        \\n\\n            AI in Action 2024\\n        \\nWe surveyed 2,000 organizations about their AI initiatives to discover what’s working, what’s not and how you can get ahead.\\n\\n\\n\\n\\n\\n            Ebook\\n        \\n\\n            Unlock competitive advantage with generative AI + ML\\n        \\nExplore the benefits of generative AI and ML and learn how to confidently incorporate these technologies into your business.\\n\\n\\n\\n\\n\\n            Documentation\\n        \\n\\n            Tips for writing foundation model prompts\\n        \\nPart art, part science, prompt engineering is the process of crafting prompt text to best effect for a given model and parameters. These tips will help you successfully prompt most text-generating foundation models.\\n\\n\\n\\n\\n\\n            Sample code\\n        \\n\\n            Use watsonx and LangChain to call a language model\\n        \\nThis notebook contains the steps and code to demonstrate Simple Sequential Chain using langchain integration with watsonx models. Some familiarity with Python is helpful. \\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nTake the next step\\n\\n\\n\\n\\nTrain, validate, tune and deploy generative AI, foundation models and machine learning capabilities with\\xa0IBM watsonx.ai, a next-generation enterprise studio for AI builders. Build AI applications in a fraction of the time with a fraction of the data.\\n\\n\\n\\n\\n\\nExplore watsonx.ai\\n\\n\\nBook a live demo\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n                            \\n\\n\\n\\n  \\n    Footnotes\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n                        \\n\\n\\n\\n\\n\\n1 The fastest-growing open-source startups in Q2 2023\\xa0(link resides outside ibm.com), Runa Capital, 2023\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\")]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load from multiple web pages\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can load multiple webpages simultaneously by passing a list of URLs to the loader. This will return a list of documents corresponding to the order of the URLs provided.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': 'https://www.ibm.com/topics/langchain', 'title': 'What Is LangChain? | IBM', 'description': 'LangChain is an open source orchestration framework for the development of applications using large language models (LLMs), like chatbots and virtual agents.\\u202f', 'language': 'en'}, page_content=\"\\n\\n\\n\\n\\n\\n\\n\\n\\nWhat Is LangChain? | IBM\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nHome\\n\\n\\n\\n\\nTopics\\n\\n\\n\\nLangChain\\n\\n\\n\\n                    \\n\\n\\n\\n  \\n    What is LangChain?\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n                \\n\\n\\n\\n\\nUse LangChain with watsonx.ai\\n\\n\\nSign up for AI updates\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n  \\n\\n\\n\\n    What is LangChain?\\n\\n\\n\\n\\n\\n    \\n\\n\\n            \\n        \\n\\n\\n\\n\\nLangChain is an open source orchestration framework for the development of applications using large language models (LLMs). Available in both Python- and Javascript-based libraries, LangChain’s tools and APIs simplify the process of building LLM-driven applications like chatbots and virtual agents.\\u202f\\n\\n\\n\\n\\n\\n\\nLangChain serves as a generic interface for nearly any LLM, providing a centralized development environment to build LLM applications and integrate them with external data sources and software workflows. LangChain’s module-based approach allows developers and data scientists to dynamically compare different prompts and even different foundation models with minimal need to rewrite code. This modular environment also allows for programs that use multiple LLMs: for example, an application that uses one LLM to interpret user queries and another LLM to author a response.\\u202f\\nLaunched by Harrison Chase in October 2022, LangChain enjoyed a meteoric rise to prominence: as of June 2023, it was the single fastest-growing open source project on Github.1 Coinciding with the momentous launch of OpenAI’s ChatGPT the following month, LangChain has played a significant role in making generative AI more accessible to enthusiasts in the wake of its widespread popularity.\\u202f\\nLangChain can facilitate most use cases for LLMs and natural language processing (NLP), like chatbots, intelligent search, question-answering, summarization services or even virtual agents capable of robotic process automation.\\n\\n\\n\\n\\n\\n      \\n\\n\\n\\n  \\n    Integrations with LLMs\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n    \\n\\n\\nLLMs are not standalone applications: they are pre-trained statistical models that must be paired with an application (and, in some cases, specific data sources) in order to meet their purpose.\\u202f\\nFor example, Chat-GPT is not an LLM: it is a chatbot application that, depending on the version you’ve chosen, uses the GPT-3.5 or GPT-4 language model. While it’s the GPT model that interprets the user’s input and composes a natural language response, it’s the application that (among other things) provides an interface for the user to type and read and a UX design that governs the chatbot experience. Even at the enterprise level, Chat-GPT is not the only application using the GPT model: Microsoft uses GPT-4 to power Bing Chat.\\u202f\\nFurthermore, though foundation models (like those powering LLMs) are pre-trained on massive datasets, they are not omniscient. If a particular task requires access to specific contextual information, like internal documentation or domain expertise, LLMs must be connected to those external data sources. Even if you simply want your model to reflect real-time awareness of current events, it requires external information: a model’s internal data is only up-to-date through the time period during which it was pre-trained.\\u202f\\nLikewise, if a given generative AI task requires access to external software workflows—for example, if you wanted your virtual agent to integrate with Slack—then you will need a way to integrate the LLM with the API for that software.\\u202f\\nWhile these integrations can generally be achieved with fully manual code, orchestration frameworks like LangChain and the IBM watsonx platform greatly simplify the process. They also make it much easier to experiment with different LLMs to compare results, as different models can be swapped in and out with minimal changes to code. \\u202f\\n\\n\\n\\n\\n\\n\\n\\n            Ebook\\n        \\n\\n            How to choose the right foundation model\\n        \\nLearn how to choose the right approach in preparing data sets and employing foundation models.\\n\\nRead the ebook\\n\\n\\n\\n\\n\\n            \\n\\n\\n  \\n  \\n      Related content\\n  \\n\\n\\n\\n\\n    \\n\\n\\n        \\n\\nRead the 2024 AI in Action Report\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n  \\n\\n\\n\\n    How does LangChain work?\\n\\n\\n\\n\\n\\n    \\n\\n\\n            \\n        \\n\\n\\n\\n\\nAt LangChain’s core is a development environment that streamlines the programming of LLM applications through the use of\\xa0abstraction: the simplification of code by representing one or more complex processes as a named component that encapsulates all of its constituent steps.\\n\\n\\n\\n\\n\\n\\nAbstractions are a common element of everyday life and language. For example, “π” allows us to represent the ratio of the length of a circle’s circumference to that of its diameter without having to write out its infinite digits. Similarly, a thermostat allows us to control the temperature in our home without needing to understand the complex circuitry this entails—we only need to know how different thermostat settings translate to different temperatures.\\nLangChain is essentially a library of abstractions for Python and Javascript, representing common steps and concepts necessary to work with language models. These modular components—like functions and object classes—serve as the building blocks of generative AI programs. They can be “chained” together to create applications, minimizing the amount of code and fine understanding required to execute complex NLP tasks. Though LangChain’s abstracted approach may limit the extent to which an expert programmer can finely customize an application, it empowers specialists and newcomers alike to quickly experiment and prototype.\\n\\n\\n\\n\\n\\n      \\n\\n\\n\\n  \\n    Importing language models\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n    \\n\\n\\nNearly any LLM can be used in LangChain. Importing language models into LangChain is easy, provided you have an API key. The LLM class is designed to provide a standard interface for all models.\\nMost LLM providers will require you to create an account in order to receive an API key. Some of these APIs—particularly those for proprietary closed-source models, like those offered by OpenAI or Anthropic—may have associated costs.\\nMany open source models, like BigScience’s BLOOM, Meta AI’s LLaMa and Google’s Flan-T5, can be accessed through Hugging Face (link resides outside ibm.com). IBM watsonx, through its partnership with Hugging Face, also offers a curated suite of open source models. Creating an account with either service will allow you to generate an API key for any of the models offered by that provider.\\nLangChain is not limited to out-of-the-box foundation models: the CustomLLM class\\xa0(link resides outside ibm.com) allows for custom LLM wrappers. Likewise, you can use the IBM watsonx APIs and Python SDK, which includes a LangChain integration, to build applications in LangChain with models that you’ve already trained or fine-tuned for your specific needs using the WatsonxLLM class (and that model’s specific project ID).\\n\\n\\nExplore the demo: using watsonx and LangChain to make a series of calls to a language model \\n            \\n        \\n\\n\\n\\n      \\n\\n\\n\\n  \\n    Prompt templates\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n    \\n\\n\\nPrompts are the instructions given to an LLM. The “art” of composing prompts that effectively provide the context necessary for the LLM to interpret input and structure output in the way most useful to you is often called prompt engineering.\\u202f\\nThe PromptTemplate class in LangChain formalizes the composition of prompts without the need to manually hard code context and queries. Important elements of a prompt are likewise entered as formal classes, like input_variables. A prompt template can thus contain and reproduce context, instructions (like “do not use technical terms”), a set of examples to guide its responses (in what is called “few-shot prompting”), a specified output format or a standardized question to be answered.\\u202f You can save and name an effectively structured prompt template and easily reuse it as needed.\\nThough these elements can all be manually coded, PromptTemplate modules empower smooth integration with other LangChain features, like the eponymous chains.\\n\\n\\nWatch the video: prompt engineering and prompt tuning\\n            \\n        \\n\\n\\n\\n      \\n\\n\\n\\n  \\n    Chains\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n    \\n\\n\\nAs its name implies, chains are the core of LangChain’s workflows. They combine LLMs with other components, creating applications by executing a sequence of functions.\\u202f\\nThe most basic chain is LLMChain. It simply calls a model and prompt template for that model. For example, imagine you saved a prompt as “ExamplePrompt” and wanted to run it against Flan-T5. You can import LLMChain from langchain.chains, then define chain_example = LLMChain(llm = flan-t5, prompt = ExamplePrompt). To run the chain for a given input, you simply call chain_example.run(“input”).\\nTo use the output of one function as the input for the next function, you can use SimpleSequentialChain. Each function could utilize different prompts, different tools, different parameters or even different models, depending on your specific needs.\\u202f\\n\\n\\n\\n\\n\\n      \\n\\n\\n\\n  \\n    Indexes\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n    \\n\\n\\nTo achieve certain tasks, LLMs will need access to specific external data sources not included in its training dataset, such as internal documents, emails or datasets. LangChain collectively refers to such external documentation as “indexes”.\\nDocument loaders\\xa0\\r\\nLangChain offers a wide variety of document loaders for third party applications (link resides outside ibm.com). This allows for easy importation of data from sources like file storage services (like Dropbox, Google Drive and Microsoft OneDrive), web content (like YouTube, PubMed or specific URLs), collaboration tools (like Airtable, Trello, Figma and Notion), databases (like Pandas, MongoDB and Microsoft), among many others.\\u202f\\nVector databases\\xa0\\r\\nUnlike “traditional” structured databases, vector databases represent data points by converting them into vector embeddings: numerical representations in the form of vectors with a fixed number of dimensions, often clustering related data points using unsupervised learning methods. This enables low latency queries, even for massive datasets, which greatly increases efficiency. Vector embeddings also store each vector’s metadata, further enhancing search possibilities.\\nLangChain provides integrations for over 25 different embedding methods, as well as for over 50 different vector stores (both cloud-hosted and local).\\u202f\\nText splitters\\xa0\\r\\nTo increase speed and reduce computational demands, it’s often wise to split large text documents into smaller pieces. LangChain’s TextSplitters split text up into small, semantically meaningful chunks that can then be combined using methods and parameters of your choosing.\\nRetrieval\\xa0\\r\\nOnce external sources of knowledge have been connected, the model must be able to quickly retrieve and integrate relevant information as needed. Like watsonx, LangChain offers retrieval augmented generation (RAG):\\xa0its retriever modules accept a string query as an input and return a list of Document’s as output.\\n\\n\\n\\n\\n\\n      \\n\\n\\n\\n  \\n    Memory\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n    \\n\\n\\nLLMs, by default, do not have any long-term memory of prior conversations (unless that chat history is used as input for a query). LangChain solves this problem with simple utilities for adding memory to a system, with options ranging from retaining the entirety of all conversations to retaining a summarization of the conversation thus far to retaining the n\\xa0most recent exchanges.\\n\\n\\n\\n\\n\\n      \\n\\n\\n\\n  \\n    Agents\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n    \\n\\n\\nLangChain agents can use a given language model as a “reasoning engine” to determine which actions to take. When building a chain for an agent, inputs include:\\na list of available tools to be leveraged.user input (like prompts and queries).any relevant previously executed steps.\\n\\n\\nLearn more about agents in LangChain\\n            \\n        \\n\\n\\n\\n      \\n\\n\\n\\n  \\n    Tools\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n    \\n\\n\\nDespite their heralded power and versatility, LLMs have important limitations: namely, a lack of up-to-date information, a lack of domain-specific expertise and a general difficulty with math.\\u202f\\nLangChain tools\\xa0(link resides outside ibm.com) are a set of functions that empower LangChain agents to interact with real-world information in order to expand or improve the services it can provide. Examples of prominent LangChain tools include:\\n\\nWolfram Alpha: provides access to powerful computational and data visualization functions, enabling sophisticated mathematical capabilities.\\nGoogle Search: provides access to Google Search, equipping applications and agents with real-time information.\\nOpenWeatherMap: fetches weather information.\\nWikipedia: provides efficient access to information from Wikipedia articles.\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n  \\n\\n\\n\\n    LangSmith\\n\\n\\n\\n\\n\\n    \\n\\n\\n            \\n        \\n\\n\\n\\nReleased in the fall of 2023, LangSmith aims to bridge the gap between the accessible prototyping capabilities that brought LangChain to prominence and building production-quality LLM applications.\\nLangSmith provides tools to monitor, evaluate and debug applications, including the ability to automatically trace all model calls to spot errors and test performance under different model configurations. This visibility aims to empower more robust, cost-efficient applications.\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n  \\n\\n\\n\\n    Getting started with LangChain\\n\\n\\n\\n\\n\\n    \\n\\n\\n            \\n        \\n\\n\\n\\nLangChain is open source and free to use: source code is\\xa0available for download on Github\\xa0(link resides outside ibm.com).\\u202f\\nLangChain can also be installed on Python with a simple pip command:\\xa0pip install langchain.\\u202f\\xa0To install all LangChain dependencies (rather than only those you find necessary), you can run the command\\xa0pip install langchain[all].\\nMany step-by-step tutorials are available from both the greater LangChain community ecosystem and the official documentation at\\xa0docs.langchain.com\\xa0(link resides outside ibm.com).\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n  \\n\\n\\n\\n    LangChain use cases\\n\\n\\n\\n\\n\\n    \\n\\n\\n                        \\n                    \\n\\n\\n\\n\\nApplications made with LangChain provide great utility for a variety of use cases, from straightforward question-answering and text generation tasks to more complex solutions that use an LLM as a “reasoning engine.”\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nChatbots: Chatbots are among the most intuitive uses of LLMs. LangChain can be used to provide proper context for the specific use of a chatbot, and to integrate chatbots into existing communication channels and workflows with their own APIs.Summarization: Language models can be tasked with summarizing many types of text, from breaking down complex academic articles and transcripts to providing a digest of incoming emails.Question answering: Using specific documents or specialized knowledge bases (like Wolfram, arXiv or PubMed), LLMs can retrieve relevant information from storage and articulate helpful answers). If fine-tuned or properly prompted, some LLMs can answer many questions even without external information.Data augmentation: LLMs can be used to generate synthetic data for use in machine learning. For example, an LLM can be trained to generate additional data samples that closely resemble the data points in a training dataset.Virtual agents: Integrated with the right workflows, LangChain’s Agent modules can use an LLM to autonomously determine next steps and take action using robotic process automation (RPA).\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n                Related solutions\\n            \\n\\n\\n\\n            \\n\\n\\n\\n  \\n    watsonx.ai\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n        \\n\\n\\n\\nTrain, validate, tune and deploy generative AI, foundation models and machine learning capabilities with ease and build AI applications in a fraction of the time with a fraction of the data.\\n\\n\\n\\n\\n\\n            Explore watsonx.ai\\n            \\n            \\n        \\n\\n\\n\\n\\n\\n\\n\\n\\n\\n            \\n\\n\\n\\n  \\n     AI consulting services \\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n        \\n\\n\\n\\nReimagine how you work with\\xa0AI: our diverse, global team of more than 20,000 AI\\xa0experts can help you quickly and confidently\\xa0design and scale AI and automation across your\\xa0business, working across our own IBM\\xa0watsonx\\xa0technology\\xa0and an open ecosystem of partners to deliver any\\xa0AI model, on any cloud, guided by ethics and trust.\\n\\n\\n\\n\\n\\n            Explore IBM AI consulting services\\n            \\n            \\n        \\n\\n\\n\\n\\n\\n\\n\\n\\n\\n            \\n\\n\\n\\n  \\n    watsonx.data\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n        \\n\\n\\n\\nScale analytics and AI workloads for all your data, anywhere with watsonx.data, the industry’s\\xa0only data store that is open, hybrid and governed.\\n\\n\\n\\n\\n\\n            Explore watsonx.data\\n            \\n            \\n        \\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n  \\n\\n\\n\\n    LangChain resources\\n\\n\\n\\n\\n\\n    \\n\\n\\n                \\n            \\n\\n\\n\\nTools, tips and sample code to begin building applications with LangChain and watsonx.\\n\\n\\n\\n\\n\\n\\n            AI Models\\n        \\n\\n            Discover IBM's Granite LLM\\n        \\nGranite is IBM's flagship series of LLM foundation models based on decoder-only transformer architecture. Granite language models are trained on trusted enterprise data spanning internet, academic, code, legal and finance.\\n\\n\\n\\n\\n\\n            Report\\n        \\n\\n            AI in Action 2024\\n        \\nWe surveyed 2,000 organizations about their AI initiatives to discover what’s working, what’s not and how you can get ahead.\\n\\n\\n\\n\\n\\n            Ebook\\n        \\n\\n            Unlock competitive advantage with generative AI + ML\\n        \\nExplore the benefits of generative AI and ML and learn how to confidently incorporate these technologies into your business.\\n\\n\\n\\n\\n\\n            Documentation\\n        \\n\\n            Tips for writing foundation model prompts\\n        \\nPart art, part science, prompt engineering is the process of crafting prompt text to best effect for a given model and parameters. These tips will help you successfully prompt most text-generating foundation models.\\n\\n\\n\\n\\n\\n            Sample code\\n        \\n\\n            Use watsonx and LangChain to call a language model\\n        \\nThis notebook contains the steps and code to demonstrate Simple Sequential Chain using langchain integration with watsonx models. Some familiarity with Python is helpful. \\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nTake the next step\\n\\n\\n\\n\\nTrain, validate, tune and deploy generative AI, foundation models and machine learning capabilities with\\xa0IBM watsonx.ai, a next-generation enterprise studio for AI builders. Build AI applications in a fraction of the time with a fraction of the data.\\n\\n\\n\\n\\n\\nExplore watsonx.ai\\n\\n\\nBook a live demo\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n                            \\n\\n\\n\\n  \\n    Footnotes\\n\\n\\n\\n\\n\\n\\n    \\n\\n\\n                        \\n\\n\\n\\n\\n\\n1 The fastest-growing open-source startups in Q2 2023\\xa0(link resides outside ibm.com), Runa Capital, 2023\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\"),\n",
       " Document(metadata={'source': 'https://www.redhat.com/en/topics/ai/what-is-instructlab', 'title': 'What is InstructLab?', 'description': 'InstructLab is an open source project for enhancing large language models (LLMs).', 'language': 'en'}, page_content='\\n\\n\\n\\n\\n\\nWhat is InstructLab?\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nSkip to contentFeatured linksSupportDocumentationConsoleDevelopersStart a trial\\n                All Red HatFor customersCustomer supportSubscription managementSupport casesRed Hat Ecosystem CatalogFind a partnerFor partnersPartner portalPartner supportBecome a partner Try, buy, & sellRed Hat MarketplaceRed Hat StoreContact salesStart a trialLearning resourcesDocumentationTraining and certification Hybrid cloud learning hubInteractive labsLearning communityRed Hat TVOpen source communitiesAnsibleGlobal advocacyHow we contributeRed HatProductsSolutionsTraining & servicesResourcesPartnersAboutExplore morePlatform productsRed Hat Enterprise LinuxA flexible, stable operating system to support hybrid cloud innovation.\\n                      Red Hat OpenShiftA container platform to build, modernize, and deploy applications at scale.\\n                      Red Hat Ansible Automation PlatformNew versionA foundation for implementing enterprise-wide automation.\\n                      Try & buyStart a trialAssess a product with a no-cost trial.\\n                      Buy onlineBuy select products and services in the Red Hat Store.\\n                      Integrate with major cloud providersBuy Red Hat solutions using committed spend from providers, including:\\n                          Featured\\n      Red Hat Enterprise Linux AI\\n    \\n      Red Hat OpenShift AI\\n    \\n      Red Hat OpenShift Virtualization\\n    \\n      Red Hat OpenShift Service on AWS\\n    \\n      Microsoft Azure Red Hat OpenShift\\n    See all products\\n              Application platformSimplify the way you build, deploy, manage, and secure apps across the hybrid cloud.\\n                      Artificial intelligenceBuild, deploy, and monitor AI models and apps with Red Hat\\'s open source platforms.\\n                      Edge computingDeploy workloads closer to the source with security-focused edge technology.\\n                      IT automationUnite disparate tech, teams, and environments with 1 comprehensive automation platform.\\n                      Linux standardizationGet consistency across operating environments with an open, flexible infrastructure.\\n                      SecurityDeliver software using trusted platforms and real-time security scanning and remediation.\\n                      VirtualizationModernize operations using a single platform for virtualized and containerized workloads.\\n                      By industry\\n      Automotive\\n    \\n      Financial services\\n    \\n      Healthcare\\n    \\n      Industrial sector\\n    \\n      Media and entertainment\\n    \\n      Public sector\\n    \\n      Telecommunications\\n    Explore solutions\\n              \\n              Services\\n          \\n      Consulting\\n    \\n      Open Innovation Labs\\n    \\n      Technical Account Management\\n    \\n              Training & certification\\n          \\n      All courses and exams\\n    \\n      All certifications\\n    \\n      Verify a certification\\n    \\n      Skills assessment\\n    \\n      Learning subscription\\n    \\n      Learning community\\n    \\n      Red Hat Academy\\n    \\n      FAQs\\n    \\n      Connect with learning experts\\n    Featured\\n      Ansible Basics: Automation Technical Overview (No cost)\\n    \\n      Containers, Kubernetes and Red Hat OpenShift Technical Overview (No cost)\\n    \\n      Red Hat Enterprise Linux Technical Overview (No cost)\\n    \\n      Red Hat Certified System Administrator exam\\n    \\n      Red Hat System Administration I\\n    Explore services\\n              Topics\\n      AI\\n    \\n      Application modernization\\n    \\n      Automation\\n    \\n      Cloud computing\\n    \\n      Cloud-native applications\\n    \\n      Containers\\n    \\n      DevOps\\n    \\n      Edge computing\\n    \\n      Linux\\n    \\n      Virtualization\\n    \\n      See all topics\\n    Articles\\n      What is InstructLab?\\n    \\n      What are cloud services?\\n    \\n      What is edge computing?\\n    \\n      What is hybrid cloud?\\n    \\n      Why build a Red Hat cloud?\\n    \\n      Cloud vs. edge\\n    \\n      Red Hat OpenShift vs. Kubernetes\\n    \\n      Learning Ansible basics\\n    More to explore\\n      Blog\\n    \\n      Customer success stories\\n    \\n      Events and webinars\\n    \\n      Newsroom\\n    \\n      Podcasts and video series\\n    \\n      Documentation\\n    \\n      Resource library\\n    \\n      Training and certification\\n    Explore resources\\n              For customers\\n      Our partners\\n    \\n      Red Hat Ecosystem Catalog\\n    \\n      Find a partner\\n    For partners\\n      Partner Connect\\n    \\n      Become a partner\\n    \\n      Training\\n    \\n      Support\\n    \\n      Access the partner portal\\n    About us\\n      Our company\\n    \\n      How we work\\n    \\n      Our social impact\\n    \\n      Development model\\n    \\n      Subscription model\\n    \\n      Product support\\n    Open source\\n      Open source commitments\\n    \\n      How we contribute\\n    \\n      Red Hat on GitHub\\n    Company details\\n      Analyst relations\\n    \\n      Blog\\n    \\n      Locations\\n    \\n      Newsroom\\n    Explore Red Hat\\n              Contact us\\n              For customersCustomer supportSubscription managementSupport casesRed Hat Ecosystem CatalogFind a partnerFor partnersPartner portalPartner supportBecome a partner Try, buy, & sellRed Hat MarketplaceRed Hat StoreContact salesStart a trialLearning resourcesDocumentationTraining and certification Hybrid cloud learning hubInteractive labsLearning communityRed Hat TVOpen source communitiesAnsibleGlobal advocacyHow we contribute\\n      For you\\n      NewRecommendationsAs you browse redhat.com, we\\'ll recommend resources you may like. For now, try these.All Red Hat productsTech topicsRed Hat resourcesSupportDocumentationConsoleDevelopersStart a trialContactSelect a language简体中文EnglishFrançaisDeutschItaliano日本語한국어PortuguêsEspañol\\n        Contact us\\n    \\n      English\\n    Select a language简体中文EnglishFrançaisDeutschItaliano日本語한국어PortuguêsEspañolRed HatProductsSolutionsTraining & servicesResourcesPartnersAboutMenu\\n        Search\\n      \\n        For you\\n      \\n        Contact us\\n      \\n        English\\n      \\n        Log in\\n      ProductsSolutionsTraining & servicesResourcesPartnersAboutContact usSelect a language简体中文EnglishFrançaisDeutschItaliano日本語한국어PortuguêsEspañol\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nJump to section\\n \\n\\n\\n\\n\\n\\n\\nJump to section\\n\\n\\n\\n\\n \\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n                                Topics                                                            Understanding AI                            \\n                    What is InstructLab?                     \\nWhat is InstructLab?\\n\\n\\nUpdated May 7, 2024 •%t-minute readCopy URL \\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nOverviewInstructLab is an open source project for enhancing\\xa0large language models (LLMs) used in\\xa0generative artificial intelligence (gen AI) applications. Created by IBM and Red Hat, the InstructLab community project provides a cost-effective solution for improving the alignment of LLMs and opens the doors for those with minimal machine learning experience to contribute.Join the InstructLab community \\n\\n\\n\\nWhat does InstructLab do?LLMs can power a range of useful applications like chatbots and coding assistants. These LLMs can be proprietary (such as OpenAI’s GPT models and Anthropic’s Claude models) or offer varying degrees of openness around pretraining data and usage restrictions (such as Meta’s Llama models, Mistral AI’s Mistral models, and IBM’s Granite models).AI practitioners often need to adapt a pretrained LLM to suit a particular business purpose. But there are limits to the ways you can modify an LLM:Fine-tuning an LLM to understand a specific area of knowledge or skills typically involves forking an existing open model, then running expensive, resource-intensive training.There’s no way to incorporate improvements back to the upstream project, and thus no way for models to continuously improve from community contributions.LLM refinements have typically required large amounts of human-generated data, which can be time-consuming and expensive to get.InstructLab follows an approach that punches through those limitations. It can enhance an LLM using far less human-generated information and far fewer computing resources than are typically used to retrain a model. And it makes it possible for upstream contributions to continuously make the model better.InstructLab is named after and based on IBM Research’s work on Large-scale Alignment for chatBots, abbreviated as LAB. The LAB method is described in a\\xa02024 research paper by members of the MIT-IBM Watson AI Lab and IBM Research.InstructLab is not model-specific. It can provide supplemental skills and knowledge fine-tuning to an LLM of your choice. This “tree of skills and knowledge” improves continuously from community contributions and can be applied to support regular builds of an enhanced LLM. InstructLab maintains an\\xa0enhanced version of IBM Granite. Two other lab-enhanced models released by IBM include\\xa0Labradorite, which is derived from Llama 2, and\\xa0Merlinite, which is derived from Mistral. The InstructLab project prioritizes fast iteration and intends to retrain models on a regular basis. Organizations can also use the InstructLab model alignment tools to train their own private LLMs with their own proprietary skills and knowledge. \\nHow does InstructLab work?The LAB method consists of 3 components:Taxonomy-driven data curation. Taxonomy is a set of diverse training data curated by humans as examples of new knowledge and skills for the model.Large-scale synthetic data generation. The model is then used to generate new examples based on the seed training data. Recognizing that synthetic data can vary in quality, the LAB method adds an automated step to refine the example answers, making sure they’re grounded and safe.Iterative, large-scale alignment tuning. Finally, the model is retrained based on the set of synthetic data. The LAB method includes 2 tuning phases: knowledge tuning, followed by skill tuning.The contributions of data from the community can lead to regular iterative builds of enhanced LLMs, each made better by the tree of skills generated from community contributions. \\nHow is InstructLab different from other methods of training an LLM?Let’s compare InstructLab to the other steps in creating and improving an LLM.PretrainingDuring pretraining, an LLM is trained to predict the next token using trillions of tokens of unlabeled data. This gets really expensive, sometimes requiring thousands of GPUs and months of time. Pretraining a highly capable LLM is only possible for organizations with significant resources.Alignment tuningAfter pretraining, LLMs undergo alignment tuning to make the model’s answers as accurate and useful as possible. The 1st step in alignment tuning is typically instruction tuning, in which a model is trained directly on specific tasks of interest. Next is preference tuning, which can include reinforcement learning from human feedback (RLHF). In this step, humans test the model and rate its output, noting if the model’s answers are preferred or unpreferred. An RLHF process may include multiple rounds of feedback and refinement to optimize a model.Researchers have found that the amount of feedback at this alignment tuning stage can be much smaller than the initial set of training data―tens of thousands of human annotations, compared to the trillions of tokens of data required for pretraining―and still unlock latent capabilities of the model.InstructLabThe LAB method emerged from the idea that it should be possible to realize the benefits of model alignment from an even smaller set of human-generated data. An AI model can use a handful of human examples to generate a large amount of synthetic data―then refine that list for quality―and use that high-quality synthetic data set for further tuning and training. In contrast to instruction tuning, which typically need thousands of examples of human feedback, LAB can make a model significantly better using relatively few examples provided by humans.How is InstructLab different from retrieval-augmented generation (RAG)?The short answer is InstructLab and retrieval-augmented generation (RAG) solve different problems.RAG is a cost-efficient method for supplementing an LLM with domain-specific knowledge that wasn’t part of its pretraining. RAG makes it possible for a chatbot to accurately answer questions related to a specific field or business without retraining the model. Knowledge documents are stored in a vector database, then retrieved in chunks and sent to the model as part of user queries. This is helpful for anyone who wants to add proprietary data to an LLM without giving up control of their information, or who needs an LLM to access timely information.\\xa0This is in contrast to the InstructLab method, which sources end-user contributions to support regular builds of an enhanced version of an LLM. InstructLab helps add knowledge and unlock new skills of an LLM.It’s possible to \"supercharge\" a RAG process by using the RAG technique on an InstructLab-tuned model.Learn more about RAG\\xa0 \\nWhat are the components of the InstructLab project?InstructLab is composed of several projects.TaxonomyInstructLab is driven by taxonomies, which are largely created manually and with care. InstructLab contains a taxonomy tree that lets users create models tuned with human-provided data, which is then enhanced with synthetic data generation.Command-line interface (CLI)The InstructLab CLI lets contributors test their contributions using their laptop or workstation. Community members can use the InstructLab technique to generate a low-fidelity approximation of synthetic data generation and model-instruction tuning without access to specialized hardware.Model training infrastructureFinally, there’s the process of creating the enhanced LLMs. It takes GPU-intensive infrastructure to regularly retrain models based on new contributions from the community. IBM donates and maintains the infrastructure necessary to frequently retrain the InstructLab project’s enhanced models.Dig deeper into AI infrastructureIcon-Red_Hat-Directional-A-Black-RGB \\nDiscover Red Hat Enterprise Linux AIWhen you’re ready to bring AI to the enterprise, Red Hat® Enterprise Linux® AI brings together the Granite family of open source-licensed LLMs, InstructLab model alignment tools, a bootable image of Red Hat Enterprise Linux, enterprise-grade technical support, and model intellectual property indemnification.Red Hat Enterprise Linux is the world’s leading enterprise Linux platform, certified on hundreds of clouds and with thousands of hardware and software vendors. With the technological foundation of Linux, containers, and automation, Red Hat’s open hybrid cloud strategy gives you the flexibility to run your AI applications anywhere you need them.Red Hat Enterprise Linux AI and the InstructLab project further deliver on this vision, breaking down the cost and resource barriers to experimenting with and building AI models while providing the tools, data, and concepts needed to fuel the next wave of intelligent workloads.Explore Red Hat Enterprise Linux AIIcon-Red_Hat-Directional-A-Black-RGB \\n\\n\\n\\n\\n\\n\\n\\n\\n \\n\\n\\n\\n\\n\\n\\nKeep reading\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nArticleWhat is generative AI?\\n\\n\\n\\nGenerative AI relies on deep learning models trained on large data sets to create new content.\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nLearn moreIcon-Red_Hat-Directional-A-Black-RGB\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nArticleWhat is machine learning?\\n\\n\\n\\nMachine learning is the technique of training a computer to find patterns, make predictions, and learn from experience without being explicitly programmed.\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nLearn moreIcon-Red_Hat-Directional-A-Black-RGB\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nArticleWhat are foundation models?\\n\\n\\n\\nA foundation model is a type of machine learning (ML) model that is pre-trained to perform a range of tasks.\\xa0\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nLearn moreIcon-Red_Hat-Directional-A-Black-RGB\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n \\n\\n\\n\\n\\n\\nMore about AI/ML\\n\\n\\n\\n\\n\\nProducts\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nA foundation model platform used to seamlessly develop, test, and run Granite family LLMs for enterprise applications.\\n\\n\\n\\n\\n\\nLearn moreIcon-Red_Hat-Directional-A-Black-RGB\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nAn AI-focused portfolio that provides tools to train, tune, serve, monitor, and manage AI/ML experiments and models on Red Hat OpenShift.\\n\\n\\n\\n\\n\\nLearn moreIcon-Red_Hat-Directional-A-Black-RGB\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nAn enterprise application platform with a unified set of tested services for bringing apps to market on your choice of infrastructure.\\xa0\\n\\n\\n\\n\\n\\nLearn moreIcon-Red_Hat-Directional-A-Black-RGB\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nRed Hat Ansible Lightspeed with IBM watsonx Code Assistant is a generative AI service designed by and for Ansible automators, operators, and developers.\\xa0\\n\\n\\n\\n\\n\\nLearn moreIcon-Red_Hat-Directional-A-Black-RGB\\n\\n\\n\\n\\n\\n\\nRelated articles\\n\\n\\n\\n\\n\\n\\n\\nWhat is generative AI?What is deep learning?What are large language models?What are foundation models?AI infrastructure explainedUnderstanding AIWhat is an AI platform?What is LLMOps?AI in bankingWhat are intelligent applications?What is agentic AI?\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nWhat are Granite models?What is MLOps?What is machine learning?Understanding AI/ML use casesWhat is AI in healthcare?AI/ML on Red Hat OpenShiftWhat is edge AI?AI infrastructure explainedWhat is AIOps?RAG vs. fine-tuningLLMs vs SLMs\\xa0\\n\\n\\n\\n\\n\\n\\nResources\\n\\n\\n\\n\\n\\n\\n\\n\\n                    e-book\\n                                                                             \\n\\n\\n\\nTop considerations for building a production-ready AI/ML environment\\n\\n\\n\\n\\n\\nRead itIcon-Red_Hat-Directional-A-Black-RGB\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n                    Analyst Material\\n                                                                             \\n\\n\\n\\nThe Total Economic Impact™ Of Red Hat Hybrid Cloud Platform For MLOps\\n\\n\\n\\n\\n\\nLearn moreIcon-Red_Hat-Directional-A-Black-RGB\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n                    Webinar\\n                                                                             \\n\\n\\n\\nGetting the most out of AI with open source and Kubernetes\\n\\n\\n\\n\\n\\nWatch it on demandIcon-Red_Hat-Directional-A-Black-RGB\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nKeep exploring \\n\\n\\n\\nPORTFOLIOAI from Red HatE-BOOKAdvance your business with AI and MLBLOGWhat is AI/ML and why does it matter to your business?PARTNERSExplore Red Hat’s AI partner ecosystemSUCCESS STORYBanco Galicia speeds new customer onboarding\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nLinkedInYouTubeFacebookXProductsRed Hat Enterprise LinuxRed Hat OpenShiftRed Hat Ansible Automation PlatformCloud servicesSee all productsToolsTraining and certificationMy accountCustomer supportDeveloper resourcesFind a partnerRed Hat Ecosystem CatalogRed Hat value calculatorDocumentationTry, buy, & sellProduct trial centerRed Hat MarketplaceRed Hat StoreBuy online (Japan)ConsoleCommunicateContact salesContact customer serviceContact trainingSocialAbout Red HatWe’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.Select a languageEnglish简体中文EnglishFrançaisDeutschItaliano日本語한국어PortuguêsEspañolRed Hat legal and privacy linksAbout Red HatJobsEventsLocationsContact Red HatRed Hat BlogDiversity, equity, and inclusionCool Stuff StoreRed Hat Summit© 2024 Red Hat, Inc.Red Hat legal and privacy linksPrivacy statementTerms of useAll policies and guidelinesDigital accessibility\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n')]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loader = WebBaseLoader([\"https://www.ibm.com/topics/langchain\", \"https://www.redhat.com/en/topics/ai/what-is-instructlab\"])\n",
    "data = loader.load()\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load from WORD files\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Docx2txtLoader` is utilized to convert Word documents into a document format suitable for further processing.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-12-03 14:11:12--  https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/94hiHUNLZdb0bLMkrCh79g/file-sample.docx\n",
      "Resolving cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)... 169.63.118.104, 169.63.118.104\n",
      "Connecting to cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)|169.63.118.104|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1311881 (1.3M) [application/vnd.openxmlformats-officedocument.wordprocessingml.document]\n",
      "Saving to: ‘file-sample.docx’\n",
      "\n",
      "file-sample.docx    100%[===================>]   1.25M  --.-KB/s    in 0.02s   \n",
      "\n",
      "2024-12-03 14:11:12 (61.7 MB/s) - ‘file-sample.docx’ saved [1311881/1311881]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/94hiHUNLZdb0bLMkrCh79g/file-sample.docx\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = Docx2txtLoader(\"file-sample.docx\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': 'file-sample.docx'}, page_content='Demonstration of DOCX support in calibre\\n\\nThis document demonstrates the ability of the calibre DOCX Input plugin to convert the various typographic features in a Microsoft Word (2007 and newer) document. Convert this document to a modern ebook format, such as AZW3 for Kindles or EPUB for other ebook readers, to see it in action.\\n\\nThere is support for images, tables, lists, footnotes, endnotes, links, dropcaps and various types of text and paragraph level formatting.\\n\\nTo see the DOCX conversion in action, simply add this file to calibre using the “Add Books” button and then click “Convert”.  Set the output format in the top right corner of the conversion dialog to EPUB or AZW3 and click “OK”.\\n\\n\\n\\nText Formatting\\n\\nInline formatting\\n\\nHere, we demonstrate various types of inline text formatting and the use of embedded fonts.\\n\\nHere is some bold, italic, bold-italic, underlined and struck out  text. Then, we have a superscript and a subscript. Now we see some red, green and blue text. Some text with a yellow highlight. Some text in a box. Some text in inverse video.\\n\\nA paragraph with styled text: subtle emphasis  followed by strong text and intense emphasis. This paragraph uses document wide styles for styling rather than inline text properties as demonstrated in the previous paragraph — calibre can handle both with equal ease.\\n\\nFun with fonts\\n\\nThis document has embedded the Ubuntu font family. The body text is in the Ubuntu typeface, here is some text in the Ubuntu Mono typeface, notice how every letter has the same width, even i and m. Every embedded font will automatically be embedded in the output ebook during conversion. \\n\\nParagraph level formatting\\n\\nYou can do crazy things with paragraphs, if the urge strikes you. For instance this paragraph is right aligned and has a right border. It has also been given a light gray background.\\n\\nFor the lovers of poetry amongst you, paragraphs with hanging indents, like this often come in handy. You can use hanging indents to ensure that a line of poetry retains its individual identity as a line even when the screen is  too narrow to display it as a single line. Not only does this paragraph have a hanging indent, it is also has an extra top margin, setting it apart from the preceding paragraph.\\n\\nTables\\n\\nITEM\\n\\nNEEDED\\n\\nBooks\\n\\n1\\n\\nPens\\n\\n3\\n\\nPencils\\n\\n2\\n\\nHighlighter\\n\\n2 colors\\n\\nScissors\\n\\n1 pair\\n\\nTables in Word can vary from the extremely simple to the extremely complex. calibre tries to do its best when converting tables. While you may run into trouble with the occasional table, the vast majority of common cases should be converted very well, as demonstrated in this section. Note that for optimum results, when creating tables in Word, you should set their widths using percentages, rather than absolute units.  To the left of this paragraph is a floating two column table with a nice green border and header row.\\n\\nNow let’s look at a fancier table—one with alternating row colors and partial borders. This table is stretched out to take 100% of the available width.\\n\\nCity or Town\\n\\nPoint A\\n\\nPoint B\\n\\nPoint C\\n\\nPoint D\\n\\nPoint E\\n\\nPoint A\\n\\n—\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nPoint B\\n\\n87\\n\\n—\\n\\n\\n\\n\\n\\n\\n\\nPoint C\\n\\n64\\n\\n56\\n\\n—\\n\\n\\n\\n\\n\\nPoint D\\n\\n37\\n\\n32\\n\\n91\\n\\n—\\n\\n\\n\\nPoint E\\n\\n93\\n\\n35\\n\\n54\\n\\n43\\n\\n—\\n\\n\\n\\nNext, we see a table with special formatting in various locations. Notice how the formatting for the header row and sub header rows is preserved.\\n\\nCollege\\n\\nNew students\\n\\nGraduating students\\n\\nChange\\n\\n\\n\\nUndergraduate\\n\\n\\n\\n\\n\\nCedar University\\n\\n110\\n\\n103\\n\\n+7\\n\\nOak Institute\\n\\n202\\n\\n210\\n\\n-8\\n\\n\\n\\nGraduate\\n\\n\\n\\n\\n\\nCedar University\\n\\n24\\n\\n20\\n\\n+4\\n\\nElm College\\n\\n43\\n\\n53\\n\\n-10\\n\\nTotal\\n\\n998\\n\\n908\\n\\n90\\n\\nSource: Fictitious data, for illustration purposes only\\n\\nNext, we have something a little more complex, a nested table, i.e. a table inside another table. Additionally, the inner table has some of its cells merged. The table is displayed horizontally centered.\\n\\nOne\\n\\nThree\\n\\nTwo\\n\\n\\n\\nFour\\n\\n\\n\\nTo the left is a table inside a table, with some cells merged.\\n\\n\\n\\nWe end with a fancy calendar, note how much of the original formatting is preserved. Note that this table will only display correctly on relatively wide screens. In general, very wide tables or tables whose cells have fixed width requirements don’t fare well in ebooks.\\n\\nDecember 2007\\n\\nSun\\n\\n\\n\\nMon\\n\\n\\n\\nTue\\n\\n\\n\\nWed\\n\\n\\n\\nThu\\n\\n\\n\\nFri\\n\\n\\n\\nSat\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n1\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n2\\n\\n\\n\\n3\\n\\n\\n\\n4\\n\\n\\n\\n5\\n\\n\\n\\n6\\n\\n\\n\\n7\\n\\n\\n\\n8\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n9\\n\\n\\n\\n10\\n\\n\\n\\n11\\n\\n\\n\\n12\\n\\n\\n\\n13\\n\\n\\n\\n14\\n\\n\\n\\n15\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n16\\n\\n\\n\\n17\\n\\n\\n\\n18\\n\\n\\n\\n19\\n\\n\\n\\n20\\n\\n\\n\\n21\\n\\n\\n\\n22\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n23\\n\\n\\n\\n24\\n\\n\\n\\n25\\n\\n\\n\\n26\\n\\n\\n\\n27\\n\\n\\n\\n28\\n\\n\\n\\n29\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n30\\n\\n\\n\\n31\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nStructural Elements\\n\\nMiscellaneous structural elements you can add to your document, like footnotes, endnotes, dropcaps and the like. \\n\\nFootnotes & Endnotes\\n\\nFootnotes and endnotes are automatically recognized and both are converted to endnotes, with backlinks for maximum ease of use in ebook devices.\\n\\nDropcaps\\n\\nD\\n\\nrop caps are used to emphasize the leading paragraph at the start of a section. In Word it is possible to specify how many lines of text a drop-cap should use. Because of limitations in ebook technology, this is not possible when converting.  Instead, the converted drop cap will use font size and line height to simulate the effect as well as possible. While not as good as the original, the result is usually tolerable. This paragraph has a “D” dropcap set to occupy three lines of text with a font size of 58.5 pts. Depending on the screen width and capabilities of the device you view the book on, this dropcap can look anything from perfect to ugly.\\n\\nLinks\\n\\nTwo kinds of links are possible, those that refer to an external website and those that refer to locations inside the document itself. Both are supported by calibre. For example, here is a link pointing to the calibre download page. Then we have a link that points back to the section on paragraph level formatting in this document.\\n\\nTable of Contents\\n\\nThere are two approaches that calibre takes when generating a Table of Contents. The first is if the Word document has a Table of Contents itself. Provided that the Table of Contents uses hyperlinks, calibre will automatically use it. The levels of the Table of Contents are identified by their left indent, so if you want the ebook to have a multi-level Table of Contents, make sure you create a properly indented Table of Contents in Word.\\n\\nIf no Table of Contents is found in the document, then a table of contents is automatically generated from the headings in the document. A heading is identified as something that has the Heading 1 or Heading 2, etc. style applied to it. These headings are turned into a Table of Contents with Heading 1 being the topmost level, Heading 2 the second level and so on.\\n\\n You can see the Table of Contents created by calibre by clicking the Table of Contents button in whatever viewer you are using to view the converted ebook. \\n\\n\\tDemonstration of DOCX support in calibre\\t1\\n\\n\\tText Formatting\\t2\\n\\n\\tInline formatting\\t2\\n\\n\\tFun with fonts\\t2\\n\\n\\tParagraph level formatting\\t2\\n\\n\\tTables\\t3\\n\\n\\tStructural Elements\\t5\\n\\n\\tFootnotes & Endnotes\\t5\\n\\n\\tDropcaps\\t5\\n\\n\\tLinks\\t5\\n\\n\\tTable of Contents\\t5\\n\\n\\tImages\\t7\\n\\n\\tLists\\t8\\n\\n\\tBulleted List\\t8\\n\\n\\tNumbered List\\t8\\n\\n\\tMulti-level Lists\\t8\\n\\n\\tContinued Lists\\t8\\n\\n\\n\\n\\n\\nImages\\n\\nImages can be of three main types. Inline images are images that are part of the normal text flow, like this image of a green dot . Inline images do not cause breaks in the text and are usually small in size. The next category of image is a floating image, one that “floats “ on the page and is surrounded by text. Word supports more types of floating images than are possible with current ebook technology, so the conversion maps floating images to simple left and right floats, as you can see with the left and right arrow images on the sides of this paragraph.\\n\\nThe final type of image is a “block” image, one that becomes a paragraph on its own and has no text on either side. Below is a centered green dot.\\n\\nCentered images like this are useful for large pictures that should be a focus of attention. \\n\\nGenerally, it is not possible to translate the exact positioning of images from a Word document to an ebook. That is because in Word, image positioning is specified in absolute units from the page boundaries.  There is no analogous technology in ebooks, so the conversion will usually end up placing the image either centered or floating close to the point in the text where it was inserted, not necessarily where it appears on the page in Word.\\n\\nLists\\n\\nAll types of lists are supported by the conversion, with the exception of lists that use fancy bullets, these get converted to regular bullets.\\n\\nBulleted List\\n\\nOne\\n\\nTwo\\n\\nNumbered List\\n\\nOne, with a very long line to demonstrate that the hanging indent for the list is working correctly\\n\\nTwo\\n\\nMulti-level Lists\\n\\nOne\\n\\nTwo\\n\\nThree\\n\\nFour with a very long line to demonstrate that the hanging indent for the list is working correctly.\\n\\nFive\\n\\nSix\\n\\nA Multi-level list with bullets:\\n\\nOne\\n\\nTwo\\n\\nThis bullet uses an image as the bullet item\\n\\nFour\\n\\nFive\\n\\nContinued Lists\\n\\nOne\\n\\nTwo\\n\\nAn interruption in our regularly scheduled listing, for this essential and very relevant public service announcement.\\n\\nWe now resume our normal programming\\n\\nFour')]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load from Unstructured Files\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, we need to load content from various text sources and formats without writing a separate loader for each one. Additionally, when a new file format emerges, we want to save time by not having to write a new loader for it. `UnstructuredFileLoader` addresses this need by supporting the loading of multiple file types. Currently, `UnstructuredFileLoader` can handle text files, PowerPoints, HTML, PDFs, images, and more.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For example, we can load `.txt` file.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': 'new-Policies.txt'}, page_content=\"1. Code of Conduct\\n\\nOur Code of Conduct establishes the core values and ethical standards that all members of our organization must adhere to. We are committed to fostering a workplace characterized by integrity, respect, and accountability.\\n\\nIntegrity: We commit to the highest ethical standards by being honest and transparent in all our dealings, whether with colleagues, clients, or the community. We protect sensitive information and avoid conflicts of interest.\\n\\nRespect: We value diversity and every individual's contribution. Discrimination, harassment, or any form of disrespect is not tolerated. We promote an inclusive environment where differences are respected, and everyone is treated with dignity.\\n\\nAccountability: We are responsible for our actions and decisions, complying with all relevant laws and regulations. We aim for continuous improvement and report any breaches of this code, supporting investigations into such matters.\\n\\nSafety: We prioritize the safety of our employees, clients, and the community. We encourage a culture of safety, including reporting any unsafe practices or conditions.\\n\\nEnvironmental Responsibility: We strive to reduce our environmental impact and promote sustainable practices.\\n\\nThis Code of Conduct is the cornerstone of our organizational culture. We expect every employee to uphold these principles and act as role models, ensuring our reputation for ethical conduct, integrity, and social responsibility.\\n\\n2. Recruitment Policy\\n\\nOur Recruitment Policy is dedicated to attracting, selecting, and integrating the most qualified and diverse candidates into our organization. The success of our company depends on the talent, skills, and commitment of our employees.\\n\\nEqual Opportunity: We are an equal opportunity employer and do not discriminate based on race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, or any other protected status. We actively support diversity and inclusion.\\n\\nTransparency: We maintain a transparent recruitment process. Job vacancies are advertised both internally and externally when appropriate. Job descriptions and requirements are clear and accurately reflect the role.\\n\\nSelection Criteria: We base our selection on qualifications, experience, and skills relevant to the role. Our interviews and assessments are objective, and decisions are made impartially.\\n\\nData Privacy: We are dedicated to protecting candidates' personal information and comply with all applicable data protection laws.\\n\\nFeedback: Candidates receive timely and constructive feedback on their applications and interview performance.\\n\\nOnboarding: New hires receive thorough onboarding to help them integrate effectively, including an overview of our culture, policies, and expectations.\\n\\nEmployee Referrals: We welcome employee referrals as they help build a strong and engaged team.\\n\\nThis policy lays the foundation for a diverse, inclusive, and talented workforce. It ensures that we hire candidates who align with our values and contribute to our success. We regularly review and update this policy to incorporate best practices in recruitment.\\n\\n3. Internet and Email Policy\\n\\nOur Internet and Email Policy ensures the responsible and secure use of these tools within our organization, recognizing their importance in daily operations and the need for compliance with security, productivity, and legal standards.\\n\\nAcceptable Use: Company-provided internet and email are primarily for job-related tasks. Limited personal use is permitted during non-work hours as long as it does not interfere with work duties.\\n\\nSecurity: Protect your login credentials and avoid sharing passwords. Be cautious with email attachments and links from unknown sources, and promptly report any unusual online activity or potential security threats.\\n\\nConfidentiality: Use email for confidential information, trade secrets, and sensitive customer data only with encryption. Be careful when discussing company matters on public platforms or social media.\\n\\nHarassment and Inappropriate Content: Internet and email must not be used for harassment, discrimination, or the distribution of offensive content. Always communicate respectfully and sensitively online.\\n\\nCompliance: Adhere to all relevant laws and regulations concerning internet and email use, including copyright and data protection laws.\\n\\nMonitoring: The company reserves the right to monitor internet and email usage for security and compliance purposes.\\n\\nConsequences: Violations of this policy may lead to disciplinary action, including potential termination.\\n\\nThis policy promotes the safe and responsible use of digital communication tools in line with our values and legal obligations. Employees must understand and comply with this policy. Regular reviews will ensure it remains relevant with changing technology and security standards.\\n\\n4. Mobile Phone Policy\\n\\nOur Mobile Phone Policy defines standards for responsible use of mobile devices within the organization to ensure alignment with company values and legal requirements.\\n\\nAcceptable Use: Mobile devices are primarily for work-related tasks. Limited personal use is allowed if it does not disrupt work responsibilities.\\n\\nSecurity: Secure your mobile device and credentials. Be cautious with app downloads and links from unknown sources, and report any security issues promptly.\\n\\nConfidentiality: Avoid sharing sensitive company information via unsecured messaging apps or emails. Exercise caution when discussing company matters in public.\\n\\nCost Management: Personal use of mobile phones should be separate from company accounts, and any personal charges on company-issued phones must be reimbursed.\\n\\nCompliance: Comply with all relevant laws and regulations concerning mobile phone usage, including data protection and privacy laws.\\n\\nLost or Stolen Devices: Immediately report any lost or stolen mobile devices to the IT department or your supervisor.\\n\\nConsequences: Non-compliance with this policy may result in disciplinary actions, including potential loss of mobile phone privileges.\\n\\nThis policy encourages the responsible use of mobile devices in line with legal and ethical standards. Employees are expected to understand and follow these guidelines. The policy is regularly reviewed to stay current with evolving technology and security best practices.\")]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loader = UnstructuredFileLoader(\"new-Policies.txt\")\n",
    "data = loader.load()\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also can load `.md` file.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': 'markdown-sample.md'}, page_content='An h1 header\\n\\nParagraphs are separated by a blank line.\\n\\n2nd paragraph. Italic, bold, and monospace. Itemized lists\\nlook like:\\n\\nthis one\\n\\nthat one\\n\\nthe other one\\n\\nNote that --- not considering the asterisk --- the actual text\\ncontent starts at 4-columns in.\\n\\nBlock quotes are\\nwritten like so.\\n\\nThey can span multiple paragraphs,\\nif you like.\\n\\nUse 3 dashes for an em-dash. Use 2 dashes for ranges (ex., \"it\\'s all\\nin chapters 12--14\"). Three dots ... will be converted to an ellipsis.\\nUnicode is supported. ☺\\n\\nAn h2 header\\n\\nHere\\'s a numbered list:\\n\\nfirst item\\n\\nsecond item\\n\\nthird item\\n\\nNote again how the actual text starts at 4 columns in (4 characters\\nfrom the left side). Here\\'s a code sample:\\n\\nAs you probably guessed, indented 4 spaces. By the way, instead of\\nindenting the block, you can use delimited blocks, if you like:\\n\\n~~~\\ndefine foobar() {\\n    print \"Welcome to flavor country!\";\\n}\\n~~~\\n\\n(which makes copying & pasting easier). You can optionally mark the\\ndelimited block for Pandoc to syntax highlight it:\\n\\n~~~python\\nimport time\\n\\nQuick, count to ten!\\n\\nfor i in range(10):\\n    # (but not too quick)\\n    time.sleep(0.5)\\n    print i\\n~~~\\n\\nAn h3 header\\n\\nNow a nested list:\\n\\nFirst, get these ingredients:\\n\\ncarrots\\ncelery\\nlentils\\n\\nBoil some water.\\n\\nDump everything in the pot and follow\\n    this algorithm:\\nfind wooden spoon\\nuncover pot\\nstir\\ncover pot\\nbalance wooden spoon precariously on pot handle\\nwait 10 minutes\\ngoto first step (or shut off burner when done)\\n\\nDo not bump wooden spoon or it will fall.\\n\\nNotice again how text always lines up on 4-space indents (including\\nthat last line which continues item 3 above).\\n\\nHere\\'s a link to a website, to a local\\ndoc, and to a section heading in the current\\ndoc. Here\\'s a footnote [^1].\\n\\n[^1]: Footnote text goes here.\\n\\nTables can look like this:\\n\\nsize  material      color\\n\\n9     leather       brown\\n10    hemp canvas   natural\\n11    glass         transparent\\n\\nTable: Shoes, their sizes, and what they\\'re made of\\n\\n(The above is the caption for the table.) Pandoc also supports\\nmulti-line tables:\\n\\nkeyword   text\\n\\nred       Sunsets, apples, and\\n          other red or reddish\\n          things.\\n\\ngreen     Leaves, grass, frogs\\n          and other things it\\'s\\n          not easy being.\\n\\nA horizontal rule follows.\\n\\nHere\\'s a definition list:\\n\\napples\\n  : Good for making applesauce.\\noranges\\n  : Citrus!\\ntomatoes\\n  : There\\'s no \"e\" in tomatoe.\\n\\nAgain, text is indented 4 spaces. (Put a blank line between each\\nterm/definition pair to spread things out more.)\\n\\nHere\\'s a \"line block\":\\n\\n| Line one\\n|   Line too\\n| Line tree\\n\\nand images can be specified like so:\\n\\nInline math equations go in like so: $\\\\omega = d\\\\phi / dt$. Display\\nmath should get its own line and be put in in double-dollarsigns:\\n\\n$$I = \\\\int \\\\rho R^{2} dV$$\\n\\nAnd note that you can backslash-escape any punctuation characters\\nwhich you wish to be displayed literally, ex.: `foo`, *bar*, etc.')]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loader = UnstructuredFileLoader(\"markdown-sample.md\")\n",
    "data = loader.load()\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Multiple files with different formats\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can even load a list of files with different formats.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "files = [\"markdown-sample.md\", \"new-Policies.txt\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = UnstructuredFileLoader(files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': ['markdown-sample.md', 'new-Policies.txt']}, page_content='An h1 header\\n\\nParagraphs are separated by a blank line.\\n\\n2nd paragraph. Italic, bold, and monospace. Itemized lists\\nlook like:\\n\\nthis one\\n\\nthat one\\n\\nthe other one\\n\\nNote that --- not considering the asterisk --- the actual text\\ncontent starts at 4-columns in.\\n\\nBlock quotes are\\nwritten like so.\\n\\nThey can span multiple paragraphs,\\nif you like.\\n\\nUse 3 dashes for an em-dash. Use 2 dashes for ranges (ex., \"it\\'s all\\nin chapters 12--14\"). Three dots ... will be converted to an ellipsis.\\nUnicode is supported. ☺\\n\\nAn h2 header\\n\\nHere\\'s a numbered list:\\n\\nfirst item\\n\\nsecond item\\n\\nthird item\\n\\nNote again how the actual text starts at 4 columns in (4 characters\\nfrom the left side). Here\\'s a code sample:\\n\\nAs you probably guessed, indented 4 spaces. By the way, instead of\\nindenting the block, you can use delimited blocks, if you like:\\n\\n~~~\\ndefine foobar() {\\n    print \"Welcome to flavor country!\";\\n}\\n~~~\\n\\n(which makes copying & pasting easier). You can optionally mark the\\ndelimited block for Pandoc to syntax highlight it:\\n\\n~~~python\\nimport time\\n\\nQuick, count to ten!\\n\\nfor i in range(10):\\n    # (but not too quick)\\n    time.sleep(0.5)\\n    print i\\n~~~\\n\\nAn h3 header\\n\\nNow a nested list:\\n\\nFirst, get these ingredients:\\n\\ncarrots\\ncelery\\nlentils\\n\\nBoil some water.\\n\\nDump everything in the pot and follow\\n    this algorithm:\\nfind wooden spoon\\nuncover pot\\nstir\\ncover pot\\nbalance wooden spoon precariously on pot handle\\nwait 10 minutes\\ngoto first step (or shut off burner when done)\\n\\nDo not bump wooden spoon or it will fall.\\n\\nNotice again how text always lines up on 4-space indents (including\\nthat last line which continues item 3 above).\\n\\nHere\\'s a link to a website, to a local\\ndoc, and to a section heading in the current\\ndoc. Here\\'s a footnote [^1].\\n\\n[^1]: Footnote text goes here.\\n\\nTables can look like this:\\n\\nsize  material      color\\n\\n9     leather       brown\\n10    hemp canvas   natural\\n11    glass         transparent\\n\\nTable: Shoes, their sizes, and what they\\'re made of\\n\\n(The above is the caption for the table.) Pandoc also supports\\nmulti-line tables:\\n\\nkeyword   text\\n\\nred       Sunsets, apples, and\\n          other red or reddish\\n          things.\\n\\ngreen     Leaves, grass, frogs\\n          and other things it\\'s\\n          not easy being.\\n\\nA horizontal rule follows.\\n\\nHere\\'s a definition list:\\n\\napples\\n  : Good for making applesauce.\\noranges\\n  : Citrus!\\ntomatoes\\n  : There\\'s no \"e\" in tomatoe.\\n\\nAgain, text is indented 4 spaces. (Put a blank line between each\\nterm/definition pair to spread things out more.)\\n\\nHere\\'s a \"line block\":\\n\\n| Line one\\n|   Line too\\n| Line tree\\n\\nand images can be specified like so:\\n\\nInline math equations go in like so: $\\\\omega = d\\\\phi / dt$. Display\\nmath should get its own line and be put in in double-dollarsigns:\\n\\n$$I = \\\\int \\\\rho R^{2} dV$$\\n\\nAnd note that you can backslash-escape any punctuation characters\\nwhich you wish to be displayed literally, ex.: `foo`, *bar*, etc.\\n\\n1. Code of Conduct\\n\\nOur Code of Conduct establishes the core values and ethical standards that all members of our organization must adhere to. We are committed to fostering a workplace characterized by integrity, respect, and accountability.\\n\\nIntegrity: We commit to the highest ethical standards by being honest and transparent in all our dealings, whether with colleagues, clients, or the community. We protect sensitive information and avoid conflicts of interest.\\n\\nRespect: We value diversity and every individual\\'s contribution. Discrimination, harassment, or any form of disrespect is not tolerated. We promote an inclusive environment where differences are respected, and everyone is treated with dignity.\\n\\nAccountability: We are responsible for our actions and decisions, complying with all relevant laws and regulations. We aim for continuous improvement and report any breaches of this code, supporting investigations into such matters.\\n\\nSafety: We prioritize the safety of our employees, clients, and the community. We encourage a culture of safety, including reporting any unsafe practices or conditions.\\n\\nEnvironmental Responsibility: We strive to reduce our environmental impact and promote sustainable practices.\\n\\nThis Code of Conduct is the cornerstone of our organizational culture. We expect every employee to uphold these principles and act as role models, ensuring our reputation for ethical conduct, integrity, and social responsibility.\\n\\n2. Recruitment Policy\\n\\nOur Recruitment Policy is dedicated to attracting, selecting, and integrating the most qualified and diverse candidates into our organization. The success of our company depends on the talent, skills, and commitment of our employees.\\n\\nEqual Opportunity: We are an equal opportunity employer and do not discriminate based on race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, or any other protected status. We actively support diversity and inclusion.\\n\\nTransparency: We maintain a transparent recruitment process. Job vacancies are advertised both internally and externally when appropriate. Job descriptions and requirements are clear and accurately reflect the role.\\n\\nSelection Criteria: We base our selection on qualifications, experience, and skills relevant to the role. Our interviews and assessments are objective, and decisions are made impartially.\\n\\nData Privacy: We are dedicated to protecting candidates\\' personal information and comply with all applicable data protection laws.\\n\\nFeedback: Candidates receive timely and constructive feedback on their applications and interview performance.\\n\\nOnboarding: New hires receive thorough onboarding to help them integrate effectively, including an overview of our culture, policies, and expectations.\\n\\nEmployee Referrals: We welcome employee referrals as they help build a strong and engaged team.\\n\\nThis policy lays the foundation for a diverse, inclusive, and talented workforce. It ensures that we hire candidates who align with our values and contribute to our success. We regularly review and update this policy to incorporate best practices in recruitment.\\n\\n3. Internet and Email Policy\\n\\nOur Internet and Email Policy ensures the responsible and secure use of these tools within our organization, recognizing their importance in daily operations and the need for compliance with security, productivity, and legal standards.\\n\\nAcceptable Use: Company-provided internet and email are primarily for job-related tasks. Limited personal use is permitted during non-work hours as long as it does not interfere with work duties.\\n\\nSecurity: Protect your login credentials and avoid sharing passwords. Be cautious with email attachments and links from unknown sources, and promptly report any unusual online activity or potential security threats.\\n\\nConfidentiality: Use email for confidential information, trade secrets, and sensitive customer data only with encryption. Be careful when discussing company matters on public platforms or social media.\\n\\nHarassment and Inappropriate Content: Internet and email must not be used for harassment, discrimination, or the distribution of offensive content. Always communicate respectfully and sensitively online.\\n\\nCompliance: Adhere to all relevant laws and regulations concerning internet and email use, including copyright and data protection laws.\\n\\nMonitoring: The company reserves the right to monitor internet and email usage for security and compliance purposes.\\n\\nConsequences: Violations of this policy may lead to disciplinary action, including potential termination.\\n\\nThis policy promotes the safe and responsible use of digital communication tools in line with our values and legal obligations. Employees must understand and comply with this policy. Regular reviews will ensure it remains relevant with changing technology and security standards.\\n\\n4. Mobile Phone Policy\\n\\nOur Mobile Phone Policy defines standards for responsible use of mobile devices within the organization to ensure alignment with company values and legal requirements.\\n\\nAcceptable Use: Mobile devices are primarily for work-related tasks. Limited personal use is allowed if it does not disrupt work responsibilities.\\n\\nSecurity: Secure your mobile device and credentials. Be cautious with app downloads and links from unknown sources, and report any security issues promptly.\\n\\nConfidentiality: Avoid sharing sensitive company information via unsecured messaging apps or emails. Exercise caution when discussing company matters in public.\\n\\nCost Management: Personal use of mobile phones should be separate from company accounts, and any personal charges on company-issued phones must be reimbursed.\\n\\nCompliance: Comply with all relevant laws and regulations concerning mobile phone usage, including data protection and privacy laws.\\n\\nLost or Stolen Devices: Immediately report any lost or stolen mobile devices to the IT department or your supervisor.\\n\\nConsequences: Non-compliance with this policy may result in disciplinary actions, including potential loss of mobile phone privileges.\\n\\nThis policy encourages the responsible use of mobile devices in line with legal and ethical standards. Employees are expected to understand and follow these guidelines. The policy is regularly reviewed to stay current with evolving technology and security best practices.')]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercises\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1 - Try to use other PDF loaders\n",
    "\n",
    "There are many other PDF loaders in LangChain, for example, `PyPDFium2Loader`. Can you use this PDF loader to load the PDF and see the difference?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pypdfium2\n",
      "  Downloading pypdfium2-4.30.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (48 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m48.5/48.5 kB\u001b[0m \u001b[31m6.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading pypdfium2-4.30.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.8 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.8/2.8 MB\u001b[0m \u001b[31m52.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: pypdfium2\n",
      "Successfully installed pypdfium2-4.30.0\n"
     ]
    }
   ],
   "source": [
    "# Your code here\n",
    "\n",
    "!pip install pypdfium2\n",
    "\n",
    "from langchain_community.document_loaders import PyPDFium2Loader\n",
    "\n",
    "loader = PyPDFium2Loader(pdf_url)\n",
    "\n",
    "data = loader.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "\n",
    "\n",
    "```python\n",
    "\n",
    "!pip install pypdfium2\n",
    "\n",
    "from langchain_community.document_loaders import PyPDFium2Loader\n",
    "\n",
    "loader = PyPDFium2Loader(pdf_url)\n",
    "\n",
    "data = loader.load()\n",
    "\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2 - Load from Arxiv\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes we have paper that we want to load from Arxiv, can you load this [paper](https://arxiv.org/abs/1605.08386) using `ArxivLoader`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting arxiv\n",
      "  Downloading arxiv-2.1.3-py3-none-any.whl.metadata (6.1 kB)\n",
      "Collecting feedparser~=6.0.10 (from arxiv)\n",
      "  Downloading feedparser-6.0.11-py3-none-any.whl.metadata (2.4 kB)\n",
      "Requirement already satisfied: requests~=2.32.0 in /home/jupyterlab/.local/lib/python3.11/site-packages (from arxiv) (2.32.3)\n",
      "Collecting sgmllib3k (from feedparser~=6.0.10->arxiv)\n",
      "  Downloading sgmllib3k-1.0.0.tar.gz (5.8 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.11/site-packages (from requests~=2.32.0->arxiv) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.11/site-packages (from requests~=2.32.0->arxiv) (3.7)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.11/site-packages (from requests~=2.32.0->arxiv) (2.2.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.11/site-packages (from requests~=2.32.0->arxiv) (2024.8.30)\n",
      "Downloading arxiv-2.1.3-py3-none-any.whl (11 kB)\n",
      "Downloading feedparser-6.0.11-py3-none-any.whl (81 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m81.3/81.3 kB\u001b[0m \u001b[31m10.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hBuilding wheels for collected packages: sgmllib3k\n",
      "  Building wheel for sgmllib3k (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for sgmllib3k: filename=sgmllib3k-1.0.0-py3-none-any.whl size=6049 sha256=044fa4ad8fae4a87d98652ca3d3edc7173f5391431ba118e82d803775b1296dc\n",
      "  Stored in directory: /home/jupyterlab/.cache/pip/wheels/3b/25/2a/105d6a15df6914f4d15047691c6c28f9052cc1173e40285d03\n",
      "Successfully built sgmllib3k\n",
      "Installing collected packages: sgmllib3k, feedparser, arxiv\n",
      "Successfully installed arxiv-2.1.3 feedparser-6.0.11 sgmllib3k-1.0.0\n",
      "arXiv:1605.08386v1  [math.CO]  26 May 2016\n",
      "HEAT-BATH RANDOM WALKS WITH MARKOV BASES\n",
      "CAPRICE STANLEY AND TOBIAS WINDISCH\n",
      "Abstract. Graphs on lattice points are studied whose edges come from a ﬁnite set of\n",
      "allowed moves of arbitrary length. We show that the diameter of these graphs on ﬁbers of a\n",
      "ﬁxed integer matrix can be bounded from above by a constant. We then study the mixing\n",
      "behaviour of heat-bath random walks on these graphs. We also state explicit conditions\n",
      "on the set of moves so that the heat-bath random walk, a generalization of the Glauber\n",
      "dynamics, is an expander in ﬁxed dimension.\n",
      "Contents\n",
      "1.\n",
      "Introduction\n",
      "1\n",
      "2.\n",
      "Graphs and statistics\n",
      "3\n",
      "3.\n",
      "Bounds on the diameter\n",
      "4\n",
      "4.\n",
      "Heat-bath random walks\n",
      "8\n",
      "5.\n",
      "Augmenting Markov bases\n",
      "14\n",
      "References\n",
      "19\n",
      "1. Introduction\n",
      "A ﬁber graph is a graph on the ﬁnitely many lattice points F ⊂Zd of a polytope where\n",
      "two lattice points are connected by an edge if their diﬀerence lies in a ﬁnite set of allowed\n",
      "moves M ⊂Zd. The implicit structure of these graphs \n"
     ]
    }
   ],
   "source": [
    "# Your code here\n",
    "\n",
    "!pip install arxiv\n",
    "\n",
    "from langchain_community.document_loaders import ArxivLoader\n",
    "\n",
    "docs = ArxivLoader(query=\"1605.08386\", load_max_docs=2).load()\n",
    "\n",
    "print(docs[0].page_content[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "    <summary>Click here for Solution</summary>\n",
    "    \n",
    "```python\n",
    "\n",
    "!pip install arxiv\n",
    "\n",
    "from langchain_community.document_loaders import ArxivLoader\n",
    "\n",
    "docs = ArxivLoader(query=\"1605.08386\", load_max_docs=2).load()\n",
    "\n",
    "print(docs[0].page_content[:1000])\n",
    "\n",
    "```\n",
    "\n",
    "</details>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Kang Wang](https://www.linkedin.com/in/kangwang95/)\n",
    "\n",
    "Kang Wang is a Data Scientist in IBM. He is also a PhD Candidate in the University of Waterloo.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other Contributors\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Joseph Santarcangelo](https://www.linkedin.com/in/joseph-s-50398b136/)\n",
    "\n",
    "Joseph has a Ph.D. in Electrical Engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD.\n",
    "\n",
    "[Hailey Quach](https://author.skills.network/instructors/hailey_quach)\n",
    "\n",
    "Hailey is a Data Scientist at IBM. She is also an undergraduate student at Concordia University, Montreal.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "© Copyright IBM Corporation. All rights reserved.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  },
  "prev_pub_hash": "8ccd87848ab9e79d16c766e68c2292b6bf1eff17098bb52f22c15a7b9da59990"
 },
 "nbformat": 4,
 "nbformat_minor": 4
}