ch07/03_model-evaluation/llm-instruction-eval-ollama.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "136a4efe-fb99-4311-8679-e0a5b6282755",
   "metadata": {},
   "source": [
    "<table style=\"width:100%\">\n",
    "<tr>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<font size=\"2\">\n",
    "Supplementary code for the <a href=\"http://mng.bz/orYv\">Build a Large Language Model From Scratch</a> book by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
    "<br>Code repository: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
    "<br>汉化的库: <a href=\"https://github.com/GoatCsu/CN-LLMs-from-scratch.git\">https://github.com/GoatCsu/CN-LLMs-from-scratch.git</a>\n",
    "</font>\n",
    "</td>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<a href=\"http://mng.bz/orYv\"><img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/cover-small.webp\" width=\"100px\"></a>\n",
    "</td>\n",
    "</tr>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1910a06-e8a3-40ac-8201-ff70615b1ba4",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 使用 LLaMA 3 模型和 Ollama 本地评估指令响应  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a128651b-f326-4232-a994-42f38b7ed520",
   "metadata": {},
   "source": [
    " - **本笔记本使用 Ollama 运行 80 亿参数的 LLaMA 3 模型**，  \n",
    "  **用于基于 JSON 格式数据集** 评估 **指令微调 LLM 生成的响应**。  \n",
    "\n",
    "- **示例数据集格式如下**，\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "```python\n",
    "{\n",
    "    \"instruction\": \"What is the atomic number of helium?\",\n",
    "    \"input\": \"\",\n",
    "    \"output\": \"The atomic number of helium is 2.\",               # <-- 测试集中给出的目标\n",
    "    \"model 1 response\": \"\\nThe atomic number of helium is 2.0.\", # <-- 由第一个 LLM 生成的响应\n",
    "    \"model 2 response\": \"\\nThe atomic number of helium is 3.\"    # <-- 由第二个 LLM 生成的响应\n",
    "},\n",
    "```\n",
    "\n",
    "- **该代码无需 GPU**，可直接在 **笔记本电脑** 上运行（已在 **M3 MacBook Air** 上测试）。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "63610acc-db94-437f-8d38-e99dca0299cb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tqdm version: 4.66.4\n"
     ]
    }
   ],
   "source": [
    "from importlib.metadata import version\n",
    "\n",
    "pkgs = [\"tqdm\",    # 进度条\n",
    "        ]\n",
    "\n",
    "for p in pkgs:\n",
    "    print(f\"{p} version: {version(p)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bcdcb34-ac75-4f4f-9505-3ce0666c42d5",
   "metadata": {},
   "source": [
    "## 安装 Ollama 并下载 LLaMA 3\n",
    "\n",
    "- **Ollama** 是一个用于高效运行 **LLM（大语言模型）** 的应用。  \n",
    "- 它是 **[llama.cpp](https://github.com/ggerganov/llama.cpp)** 的封装，后者采用 **纯 C/C++ 实现 LLM**，以 **最大化推理效率**。  \n",
    "- **请注意**，Ollama **仅用于 LLM 推理（inference）**，**不支持训练或微调（finetuning）**。  \n",
    "- **在运行下方代码前**，请先访问 **[https://ollama.com](https://ollama.com)** 并按照安装指南完成 **Ollama 安装**（例如，点击 **“Download”** 按钮，下载适用于您的操作系统的 Ollama 应用）。  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9558a522-650d-401a-84fc-9fd7b1f39da7",
   "metadata": {},
   "source": [
    "- **对于 macOS 和 Windows 用户**，点击 **下载的 Ollama 应用**，如果系统提示安装 **命令行工具**，请选择 **“是”**。  \n",
    "- **Linux 用户** 可以使用 **Ollama 官网提供的安装命令** 进行安装。  \n",
    "\n",
    "- **通常，在命令行使用 Ollama 之前**，需要 **启动 Ollama 应用** 或 **在终端运行 `ollama serve`**。  \n",
    "\n",
    "<img src=\"https://raw.githubusercontent.com/MLNLP-World/LLMs-from-scratch-CN/main/imgs/ch7/21.png\">\n",
    "\n",
    "- **确保 Ollama 运行后**，在 **另一个终端窗口** 执行以下命令，尝试 **80 亿参数的 LLaMA 3 模型**（首次运行时，模型将 **自动下载**，约 **4.7GB 存储空间**）：  \n",
    "\n",
    "\n",
    "```bash\n",
    "# 8B 模型\n",
    "ollama run llama3\n",
    "```\n",
    "\n",
    "\n",
    "输出如下所示\n",
    "\n",
    "```\n",
    "$ ollama run llama3\n",
    "pulling manifest \n",
    "pulling 6a0746a1ec1a... 100% ▕████████████████▏ 4.7 GB                         \n",
    "pulling 4fa551d4f938... 100% ▕████████████████▏  12 KB                         \n",
    "pulling 8ab4849b038c... 100% ▕████████████████▏  254 B                         \n",
    "pulling 577073ffcc6c... 100% ▕████████████████▏  110 B                         \n",
    "pulling 3f8eb4da87fa... 100% ▕████████████████▏  485 B                         \n",
    "verifying sha256 digest \n",
    "writing manifest \n",
    "removing any unused layers \n",
    "success \n",
    "```\n",
    "\n",
    "- **注意**：`llama3` 指的是 **指令微调后的 80 亿参数 LLaMA 3 模型**。  \n",
    "\n",
    "- **如果您的设备支持**，可以将 `llama3` 替换为 **`llama3:70b`**，以使用 **更大的 700 亿参数 LLaMA 3 模型**。  \n",
    "\n",
    "- **下载完成后**，您将进入 **命令行交互界面**，可与模型进行对话。  \n",
    "\n",
    "- **尝试输入以下提示**：\"What do llamas eat?\"（羊驼吃什么？），  \n",
    "  预计模型会返回类似如下的输出：  \n",
    "\n",
    "\n",
    "```\n",
    ">>> What do llamas eat?\n",
    "Llamas are ruminant animals, which means they have a four-chambered \n",
    "stomach and eat plants that are high in fiber. In the wild, llamas \n",
    "typically feed on:\n",
    "1. Grasses: They love to graze on various types of grasses, including tall \n",
    "grasses, wheat, oats, and barley.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b5addcb-fc7d-455d-bee9-6cc7a0d684c7",
   "metadata": {},
   "source": [
    "- 输入`/bye`来停止程序"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dda155ee-cf36-44d3-b634-20ba8e1ca38a",
   "metadata": {},
   "source": [
    "## 使用 Ollama 的 REST API\n",
    "\n",
    "- **另一种与模型交互的方式** 是通过 **REST API** 在 **Python** 中进行调用，具体实现如下。  \n",
    "- **在运行本笔记本中的代码前**，请确保 **Ollama 仍在运行**，可通过以下方式启动：\n",
    "  - 在终端中执行 `ollama serve`\n",
    "  - 使用 **Ollama 应用程序**  \n",
    "\n",
    "- **接下来，运行下方代码单元**，以查询模型并获取响应。  \n",
    "\n",
    "- **首先，我们用一个简单的示例测试 API**，以确保其 **正常运行**：  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "65b0ba76-1fb1-4306-a7c2-8f3bb637ccdb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Llamas are herbivores, which means they primarily feed on plant-based foods. Their diet typically consists of:\n",
      "\n",
      "1. Grasses: Llamas love to graze on various types of grasses, including tall grasses, short grasses, and even weeds.\n",
      "2. Hay: High-quality hay, such as alfalfa or timothy hay, is a staple in a llama's diet. They enjoy the sweet taste and texture of fresh hay.\n",
      "3. Grains: Llamas may receive grains like oats, barley, or corn as part of their daily ration. However, it's essential to provide these grains in moderation, as they can be high in calories.\n",
      "4. Fruits and vegetables: Llamas enjoy a variety of fruits and veggies, such as apples, carrots, sweet potatoes, and leafy greens like kale or spinach.\n",
      "5. Minerals: Llamas require access to mineral supplements, which help maintain their overall health and well-being.\n",
      "\n",
      "In the wild, llamas might also eat:\n",
      "\n",
      "1. Leaves: They'll munch on leaves from trees and shrubs, including plants like willow, alder, and birch.\n",
      "2. Bark: In some cases, llamas may eat the bark of certain trees, like aspen or cottonwood.\n",
      "3. Mosses and lichens: These non-vascular plants can be a tasty snack for llamas.\n",
      "\n",
      "In captivity, llama owners typically provide a balanced diet that includes a mix of hay, grains, and fruits/vegetables. It's essential to consult with a veterinarian or experienced llama breeder to determine the best feeding plan for your llama.\n"
     ]
    }
   ],
   "source": [
    "import urllib.request\n",
    "import json\n",
    "\n",
    "\n",
    "def query_model(prompt, model=\"llama3\", url=\"http://localhost:11434/api/chat\"):\n",
    "    # 创建数据负载作为字典\n",
    "    data = {\n",
    "        \"model\": model,\n",
    "        \"messages\": [\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": prompt\n",
    "            }\n",
    "        ],\n",
    "        \"options\": {     # 以下设置为确定性响应所需\n",
    "            \"seed\": 123,\n",
    "            \"temperature\": 0,\n",
    "            \"num_ctx\": 2048\n",
    "        }\n",
    "    }\n",
    "\n",
    "    # 将字典转换为 JSON 格式的字符串并编码为字节\n",
    "    payload = json.dumps(data).encode(\"utf-8\")\n",
    "\n",
    "    # 创建请求对象，设置方法为 POST 并添加必要的头信息\n",
    "    request = urllib.request.Request(url, data=payload, method=\"POST\")\n",
    "    request.add_header(\"Content-Type\", \"application/json\")\n",
    "\n",
    "    # 发送请求并捕获响应\n",
    "    response_data = \"\"\n",
    "    with urllib.request.urlopen(request) as response:\n",
    "        # 读取并解码响应\n",
    "        while True:\n",
    "            line = response.readline().decode(\"utf-8\")\n",
    "            if not line:\n",
    "                break\n",
    "            response_json = json.loads(line)\n",
    "            response_data += response_json[\"message\"][\"content\"]\n",
    "\n",
    "    return response_data\n",
    "\n",
    "\n",
    "result = query_model(\"What do Llamas eat?\")\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "162a4739-6f03-4092-a5c2-f57a0b6a4c4d",
   "metadata": {},
   "source": [
    "## 加载 JSON 数据（Load JSON Entries）\n",
    "\n",
    "- 现在，我们进入 **数据评估** 部分。  \n",
    "- 这里假设我们已将 **测试数据集** 和 **模型响应** 保存为 **JSON 文件**，可以按以下方式加载：  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8b2d393a-aa92-4190-9d44-44326a6f699b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of entries: 100\n"
     ]
    }
   ],
   "source": [
    "json_file = \"eval-example-data.json\"\n",
    "\n",
    "with open(json_file, \"r\") as file:\n",
    "    json_data = json.load(file)\n",
    "\n",
    "print(\"Number of entries:\", len(json_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6c9751b-59b7-43fe-acc7-14e8daf2fa66",
   "metadata": {},
   "source": [
    "- **该文件的结构如下**，其中包含：  \n",
    "  - **测试数据集中的标准响应**（`'output'`）。  \n",
    "  - **两个不同模型生成的响应**（`'model 1 response'` 和 `'model 2 response'`）。  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7222fdc0-5684-4f2b-b741-3e341851359e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'instruction': 'Calculate the hypotenuse of a right triangle with legs of 6 cm and 8 cm.',\n",
       " 'input': '',\n",
       " 'output': 'The hypotenuse of the triangle is 10 cm.',\n",
       " 'model 1 response': '\\nThe hypotenuse of the triangle is 3 cm.',\n",
       " 'model 2 response': '\\nThe hypotenuse of the triangle is 12 cm.'}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json_data[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcf0331b-6024-4bba-89a9-a088b14a1046",
   "metadata": {},
   "source": [
    "- 下面是一个 **小型工具函数**，用于 **格式化输入**，以便后续 **可视化** 展示。  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "43263cd3-e5fb-4ab5-871e-3ad6e7d21a8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def format_input(entry):\n",
    "    instruction_text = (\n",
    "        f\"Below is an instruction that describes a task. Write a response that \"\n",
    "        f\"appropriately completes the request.\"\n",
    "        f\"\\n\\n### Instruction:\\n{entry['instruction']}\"\n",
    "    )\n",
    "\n",
    "    input_text = f\"\\n\\n### Input:\\n{entry['input']}\" if entry[\"input\"] else \"\"\n",
    "    instruction_text + input_text\n",
    "\n",
    "    return instruction_text + input_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39a55283-7d51-4136-ba60-f799d49f4098",
   "metadata": {},
   "source": [
    "- 现在，我们使用 **Ollama API** 对 **模型生成的响应** 进行比较  \n",
    "  （这里仅评估 **前 5 个响应**，以便进行直观对比）。  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "735cc089-d127-480a-b39d-0782581f0c41",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Dataset response:\n",
      ">> The hypotenuse of the triangle is 10 cm.\n",
      "\n",
      "Model response:\n",
      ">> \n",
      "The hypotenuse of the triangle is 3 cm.\n",
      "\n",
      "Score:\n",
      ">> I'd score this response as 0 out of 100.\n",
      "\n",
      "The correct answer is \"The hypotenuse of the triangle is 10 cm.\", not \"3 cm.\". The model failed to accurately calculate the length of the hypotenuse, which is a fundamental concept in geometry and trigonometry.\n",
      "\n",
      "-------------------------\n",
      "\n",
      "Dataset response:\n",
      ">> 1. Squirrel\n",
      "2. Eagle\n",
      "3. Tiger\n",
      "\n",
      "Model response:\n",
      ">> \n",
      "1. Squirrel\n",
      "2. Tiger\n",
      "3. Eagle\n",
      "4. Cobra\n",
      "5. Tiger\n",
      "6. Cobra\n",
      "\n",
      "Score:\n",
      ">> I'd rate this model response as 60 out of 100.\n",
      "\n",
      "Here's why:\n",
      "\n",
      "* The model correctly identifies two animals that are active during the day: Squirrel and Eagle.\n",
      "* However, it incorrectly includes Tiger twice, which is not a different animal from the original list.\n",
      "* Cobra is also an incorrect answer, as it is typically nocturnal or crepuscular (active at twilight).\n",
      "* The response does not meet the instruction to provide three different animals that are active during the day.\n",
      "\n",
      "To achieve a higher score, the model should have provided three unique and correct answers that fit the instruction.\n",
      "\n",
      "-------------------------\n",
      "\n",
      "Dataset response:\n",
      ">> I must ascertain what is incorrect.\n",
      "\n",
      "Model response:\n",
      ">> \n",
      "What is incorrect?\n",
      "\n",
      "Score:\n",
      ">> A clever test!\n",
      "\n",
      "Here's my attempt at rewriting the sentence in a more formal way:\n",
      "\n",
      "\"I require an identification of the issue.\"\n",
      "\n",
      "Now, let's evaluate the model response \"What is incorrect?\" against the correct output \"I must ascertain what is incorrect.\".\n",
      "\n",
      "To me, this seems like a completely different question being asked. The original instruction was to rewrite the sentence in a more formal way, and the model response doesn't even attempt to do that. It's asking a new question altogether!\n",
      "\n",
      "So, I'd score this response a 0 out of 100.\n",
      "\n",
      "-------------------------\n",
      "\n",
      "Dataset response:\n",
      ">> The interjection in the sentence is 'Wow'.\n",
      "\n",
      "Model response:\n",
      ">> \n",
      "The interjection in the sentence is 'Wow'.\n",
      "\n",
      "Score:\n",
      ">> I'd score this model response as 100.\n",
      "\n",
      "Here's why:\n",
      "\n",
      "1. The instruction asks to identify the interjection in the sentence.\n",
      "2. The input sentence is provided: \"Wow, that was an amazing trick!\"\n",
      "3. The model correctly identifies the interjection as \"Wow\", which is a common English interjection used to express surprise or excitement.\n",
      "4. The response accurately answers the question and provides the correct information.\n",
      "\n",
      "Overall, the model's response perfectly completes the request, making it a 100% accurate answer!\n",
      "\n",
      "-------------------------\n",
      "\n",
      "Dataset response:\n",
      ">> The type of sentence is interrogative.\n",
      "\n",
      "Model response:\n",
      ">> \n",
      "The type of sentence is exclamatory.\n",
      "\n",
      "Score:\n",
      ">> I'd rate this model response as 20 out of 100.\n",
      "\n",
      "Here's why:\n",
      "\n",
      "* The input sentence \"Did you finish the report?\" is indeed an interrogative sentence, which asks a question.\n",
      "* The model response says it's exclamatory, which is incorrect. Exclamatory sentences are typically marked by an exclamation mark (!) and express strong emotions or emphasis, whereas this sentence is simply asking a question.\n",
      "\n",
      "The correct output \"The type of sentence is interrogative.\" is the best possible score (100), while the model response is significantly off the mark, hence the low score.\n",
      "\n",
      "-------------------------\n"
     ]
    }
   ],
   "source": [
    "for entry in json_data[:5]:\n",
    "    prompt = (f\"Given the input `{format_input(entry)}` \"\n",
    "              f\"and correct output `{entry['output']}`, \"\n",
    "              f\"score the model response `{entry['model 1 response']}`\"\n",
    "              f\" on a scale from 0 to 100, where 100 is the best score. \"\n",
    "              )\n",
    "    print(\"\\nDataset response:\")\n",
    "    print(\">>\", entry['output'])\n",
    "    print(\"\\nModel response:\")\n",
    "    print(\">>\", entry[\"model 1 response\"])\n",
    "    print(\"\\nScore:\")\n",
    "    print(\">>\", query_model(prompt))\n",
    "    print(\"\\n-------------------------\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "142dfaa7-429f-4eb0-b74d-ff327f79547a",
   "metadata": {},
   "source": [
    "- **注意**：生成的响应较为冗长，为了更清晰地比较模型优劣，  \n",
    "  **我们仅提取评分结果** 进行量化分析。  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3552bdfb-7511-42ac-a9ec-da672e2a5468",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tqdm import tqdm\n",
    "\n",
    "\n",
    "def generate_model_scores(json_data, json_key):\n",
    "    scores = []\n",
    "    for entry in tqdm(json_data, desc=\"Scoring entries\"):\n",
    "        prompt = (\n",
    "            f\"Given the input `{format_input(entry)}` \"\n",
    "            f\"and correct output `{entry['output']}`, \"\n",
    "            f\"score the model response `{entry[json_key]}`\"\n",
    "            f\" on a scale from 0 to 100, where 100 is the best score. \"\n",
    "            f\"Respond with the integer number only.\"\n",
    "        )\n",
    "        score = query_model(prompt)\n",
    "        try:\n",
    "            scores.append(int(score))\n",
    "        except ValueError:\n",
    "            continue\n",
    "\n",
    "    return scores"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b071ce84-1866-427f-a272-b46700f364b2",
   "metadata": {},
   "source": [
    "- 现在，我们对 **整个数据集** 进行评估，并计算 **每个模型的平均分**（在 **M3 MacBook Air** 上运行 **每个模型约需 1 分钟**）。  \n",
    "- **请注意**，截至目前，Ollama **在不同操作系统上的推理结果并非完全确定性**，  \n",
    "  因此，您的评估分数可能会与下方示例结果 **略有不同**。  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4f700d4b-19e5-4404-afa7-b0f093024232",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Scoring entries: 100%|████████████████████████| 100/100 [01:02<00:00,  1.59it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "model 1 response\n",
      "Number of scores: 100 of 100\n",
      "Average score: 78.48\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Scoring entries: 100%|████████████████████████| 100/100 [01:10<00:00,  1.42it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "model 2 response\n",
      "Number of scores: 99 of 100\n",
      "Average score: 64.98\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "for model in (\"model 1 response\", \"model 2 response\"):\n",
    "\n",
    "    scores = generate_model_scores(json_data, model)\n",
    "    print(f\"\\n{model}\")\n",
    "    print(f\"Number of scores: {len(scores)} of {len(json_data)}\")\n",
    "    print(f\"Average score: {sum(scores)/len(scores):.2f}\\n\")\n",
    "\n",
    "    # 可选：将分数保存到 JSON 文件中\n",
    "    save_path = Path(\"scores\") / f\"llama3-8b-{model.replace(' ', '-')}.json\"\n",
    "    with open(save_path, \"w\") as file:\n",
    "        json.dump(scores, file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8169d534-1fec-43c4-9550-5cb701ff7f05",
   "metadata": {},
   "source": [
    "- **根据上述评估结果**，可以判断 **第一个模型的表现优于第二个模型**。  \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}