updated

Author: afizs

agents/haystack/vllm_inference_engine.ipynb

@@ -18,11 +18,7 @@
       "source": [
         "# Use the ⚡ vLLM inference engine with Haystack\n",
         "\n",
-        "<img src=\"https://haystack.deepset.ai/images/haystack-ogimage.png\" width=\"430\" style=\"display:inline;\">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<img src=\"https://docs.vllm.ai/en/stable/assets/logos/vllm-logo-text-light.png\" width=\"500\" style=\"display:inline;\">\n",
-        "\n",
-        "[vLLM](https://docs.vllm.ai/en/stable/) is a high-throughput and memory-efficient inference and serving engine for LLMs.\n",
-        "\n",
-        "This notebook shows how to use it with Haystack."
+        "This notebook demonstrates how to integrate vLLM, a high-throughput and memory-efficient inference engine for Large Language Models (LLMs), with the Haystack framework."
       ]
     },
     {
@@ -33,8 +29,7 @@
       "source": [
         "## Install vLLM + Haystack integration\n",
         "\n",
-        "- we install vLLM using uv ([installation docs](https://docs.vllm.ai/en/stable/getting_started/installation.html)). For production use cases, there are also other options, including Docker ([docs](https://docs.vllm.ai/en/stable/deployment/docker)).\n",
-        "- we also install `vllm-haystack`, the vLLM/Haystack integration.\n"
+        "This section covers the installation of vLLM using `uv` and the `vllm-haystack` integration package. For production environments, alternative installation methods like Docker are available."
       ]
     },
     {
@@ -56,17 +51,7 @@
       "source": [
         "## Serving and using Generative Language Models\n",
         "\n",
-        "vLLM primarily supports most open-weights Generative Language Models.\n",
-        "\n",
-        "`vllm serve` launches an OpenAI-compatible server.\n",
-        "\n",
-        "In the following cell we spin up a server running on `localhost:8000`.\n",
-        "\n",
-        "We serve `Qwen/Qwen3-0.6B`, a very small but good Language Model, with reasoning and tool calling capabilities. Some parameters like `reasoning-parser` and `tool-call-parser` follow the values indicated in [Qwen documentation](https://qwen.readthedocs.io/en/latest/deployment/vllm.html).\n",
-        "\n",
-        "We also set some parameters specific for our Colab environment:\n",
-        "- `--enforce-eager`: disables the construction of CUDA graph. This negatively impacts performance but reduces memory requirements and server start time.\n",
-        "- `--gpu-memory-utilization 0.5`: limits GPU utilization. This is needed since later we'll run two other servers in this notebook for embedding and ranking models.\n"
+        "vLLM supports a wide range of open-weight Generative Language Models. This section details how to launch an OpenAI-compatible vLLM server to serve models like `Qwen/Qwen3-0.6B`. Specific parameters are used to optimize for a Colab environment, such as `--enforce-eager` and `--gpu-memory-utilization`."
       ]
     },
     {
@@ -108,11 +93,7 @@
       "source": [
         "### Chat with the model\n",
         "\n",
-        "Once we have launched the vLLM server,\n",
-        "we can simply initialize a `VLLMChatGenerator` ([docs](https://docs.haystack.deepset.ai/docs/vllmchatgenerator)) pointing to the vLLM server URL and start chatting!\n",
-        "\n",
-        "Here we disable reasoning ([Qwen docs](https://qwen.readthedocs.io/en/latest/deployment/vllm.html#thinking-non-thinking-modes)) and set a `streaming_callback` to allow streaming.\n",
-        "\n"
+        "After setting up the vLLM server, a `VLLMChatGenerator` can be initialized to interact with the served model. This example shows how to configure the generator, including disabling reasoning and enabling streaming output via a `streaming_callback`."
       ]
     },
     {
@@ -177,8 +158,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Not bad but, given \"hello\" as input, this would print \"o l l e h\".\n",
-        "Let's see if reasoning helps."
+        "The previous response was functional but could be improved. Let's see if enabling reasoning yields a better result."
       ],
       "metadata": {
         "id": "Fl-ElT77rK1L"
@@ -189,7 +169,7 @@
       "source": [
         "### With reasoning\n",
         "\n",
-        "We simply enable reasoning ([Qwen docs](https://qwen.readthedocs.io/en/latest/deployment/vllm.html#thinking-non-thinking-modes))."
+        "This section demonstrates the model's ability to engage in a 'thinking' mode before generating a response by enabling reasoning in the `VLLMChatGenerator` configuration."
       ],
       "metadata": {
         "id": "fV2CFCr8rR23"
@@ -287,9 +267,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Better!\n",
-        "\n",
-        "We can also easily extract reasoning as follows:"
+        "The response with reasoning provided a more accurate and robust solution. The model's reasoning process can also be extracted for analysis."
       ],
       "metadata": {
         "id": "z4OciCMar4V1"
@@ -326,9 +304,7 @@
       "source": [
         "### Structured outputs\n",
         "\n",
-        "This model also supports structured outputs.\n",
-        "\n",
-        "Let's try this, with reasoning and 0 temperature. These settings should help produce a more reliable output."
+        "This model also supports generating structured outputs, such as JSON. By defining a `response_format` and setting `temperature` to 0.0, the model can produce more reliable and structured data."
       ],
       "metadata": {
         "id": "MJ-YlCdmvy8s"
@@ -398,7 +374,7 @@
       "source": [
         "### Tool-calling Agent\n",
         "\n",
-        "Now let's build a simple agent with a weather tool. The example below requires handling parallel tool calls, which our model supports."
+        "This section illustrates how to build a simple agent using the `Agent` component from Haystack, integrated with a custom `get_weather` tool. The example demonstrates the model's capability to handle parallel tool calls."
       ],
       "metadata": {
         "id": "4T9AhOvcrxXe"
@@ -532,7 +508,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Well done!"
+        "The agent successfully used the tools to answer the user's query, demonstrating effective tool-calling capabilities."
       ],
       "metadata": {
         "id": "b7pCJc-3tP_T"
@@ -543,11 +519,7 @@
       "source": [
         "## Serving and using Embedding Models\n",
         "\n",
-        "vLLM also supports [embedding models](https://docs.vllm.ai/en/latest/models/pooling_models/embed/), used for computing semantic vectors from text.\n",
-        "\n",
-        "We serve the classic `sentence-transformers/all-MiniLM-L6-v2` embedding model, on port `8001`.\n",
-        "\n",
-        "`--enforce-eager` and `--gpu-memory-utilization` are only used to limit memory utilization in Colab."
+        "vLLM supports serving embedding models, which are used to compute semantic vectors from text. This section shows how to serve `sentence-transformers/all-MiniLM-L6-v2` on a specific port, with memory utilization adjusted for Colab."
       ],
       "metadata": {
         "id": "nrdR0j3fO3V_"
@@ -582,7 +554,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Let's try both [VLLMTextEmbedder (docs)](https://docs.haystack.deepset.ai/docs/vllmtextembedder) and [VLLMDocumentEmbedder (docs)](https://docs.haystack.deepset.ai/docs/vllmdocumentembedder) and create a simple retrieval pipeline."
+        "This section demonstrates the use of both `VLLMTextEmbedder` and `VLLMDocumentEmbedder` to create a basic retrieval pipeline for semantic search."
       ],
       "metadata": {
         "id": "4ugjx_wtvYkz"
@@ -699,11 +671,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "vLLM also supports [ranking models](https://docs.vllm.ai/en/latest/models/pooling_models/scoring/), generally used after keyword or embedding retrieval to reorder the retrieved documents by relevance to the query. vLLM supports different ranking models, including cross-encoders and late interaction models.\n",
-        "\n",
-        "We serve the `BAAI/bge-reranker-base` cross-encoder model, on port `8002`.\n",
-        "\n",
-        "`--enforce-eager` and `--gpu-memory-utilization` are only used to limit memory utilization in Colab."
+        "vLLM also supports ranking models, which are crucial for reordering retrieved documents based on relevance. This example sets up a `BAAI/bge-reranker-base` cross-encoder model, with Colab-specific memory settings."
       ],
       "metadata": {
         "id": "XVJaI07JyTJQ"
@@ -738,9 +706,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Let's build a two-stage retrieval pipeline:\n",
-        "- [`InMemoryBM25Retriever`](https://docs.haystack.deepset.ai/docs/inmemorybm25retriever) for fast keyword retrieval\n",
-        "- [`VLLMRanker`](https://docs.haystack.deepset.ai/docs/vllmranker) to select only the relevant documents from the initial set"
+        "This section constructs a two-stage retrieval pipeline, combining `InMemoryBM25Retriever` for initial keyword retrieval with `VLLMRanker` for reranking and selecting the most relevant documents."
       ],
       "metadata": {
         "id": "bxlDHYE_zN1G"
@@ -833,7 +799,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "Nice work! 🇫🇷"
+        "The ranking pipeline effectively reordered the documents, placing the most relevant results at the top."
       ],
       "metadata": {
         "id": "hli_bzo10Qth"
@@ -842,15 +808,7 @@
     {
       "cell_type": "markdown",
       "source": [
-        "In this notebook, we explored serving Generative Language Models, Embedding Models and Ranking Models via vLLM and performing inference through Haystack.\n",
-        "\n",
-        "For more information, check the documentation:\n",
-        "- [`VLLMChatGenerator`](https://docs.haystack.deepset.ai/docs/vllmchatgenerator)\n",
-        "- [`VLLMTextEmbedder`](https://docs.haystack.deepset.ai/docs/vllmtextembedder)\n",
-        "- [`VLLMDocumentEmbedder`](https://docs.haystack.deepset.ai/docs/vllmdocumentembedder)\n",
-        "- [`VLLMRanker`](https://docs.haystack.deepset.ai/docs/vllmranker)\n",
-        "\n",
-        "*Notebook by [Stefano Fiorucci](https://github.com/anakin87)*"
+        "In this notebook, we explored serving Generative Language Models, Embedding Models, and Ranking Models using vLLM and integrating them with Haystack for various inference tasks. For more detailed information, refer to the official documentation for `VLLMChatGenerator`, `VLLMTextEmbedder`, `VLLMDocumentEmbedder`, and `VLLMRanker`."
       ],
       "metadata": {
         "id": "U_y3uAyEbnZx"
@@ -862,7 +820,6 @@
     "colab": {
       "gpuType": "T4",
       "provenance": [],
-      "toc_visible": true,
       "include_colab_link": true
     },
     "kernelspec": {