Update dependencies and tools

.github/workflows/python.yaml

@@ -16,7 +16,7 @@ jobs:
         os: ["ubuntu-latest"]
         python_version: ["3.10", "3.11", "3.12", "3.13"]
     steps:
-        - uses: actions/checkout@v5
+        - uses: actions/checkout@v6
         - name: Install uv
           uses: astral-sh/setup-uv@v7
           with:

.vscode/mcp.json → .mcp.json

@@ -1,12 +1,12 @@
 {
-    "servers": {
+    "mcpServers": {
         "image-search-local": {
             "type": "http",
             "url": "http://localhost:8001/mcp"
         },
         "image-search-azure": {
             "type": "http",
-            "url": "https://<your-mcp-container-app>/mcp"
+            "url": "https://ca-mcp-dzwimtrf5gzou.orangeriver-094d6c5a.westus.azurecontainerapps.io/mcp"
         }
     }
 }

README.md

@@ -21,13 +21,25 @@ urlFragment: image-search-aisearch
 
 This project creates an image search web application and an [MCP (Model Context Protocol)](https://modelcontextprotocol.io) server, both backed by [Azure AI Search](https://learn.microsoft.com/azure/search/search-what-is-azure-search). They allow you to search for images using natural language queries, leveraging multimodal embeddings from the Azure AI Vision API and image descriptions generated by the Azure OpenAI Service.
 
-The web app frontend is built with TypeScript/React and the backend with Python/Quart. The MCP server is built with Python/FastMCP, and includes two tools that can be used by any MCP client, such as VS Code GitHub Copilot. The `image_search` tool returns a list of search results with thumbnail previews and descriptions, and the `display_image_files` tool renders images using [MCP apps](https://apps.extensions.modelcontextprotocol.io/api/).
+The web application provides a simple search box and a gallery of matching results. The web app frontend is built with TypeScript/React and the backend with Python/Quart.
+
+![Screenshot of the image search web application](screenshot_webapp.png)
+
+The MCP server is built with Python/FastMCP, and includes two tools that can be used by any MCP client, such as VS Code GitHub Copilot.
+
+The `image_search` tool attaches a thumbnail for each match to the tool result, so the agent can inspect the images directly in chat:
+
+![Screenshot of GitHub Copilot searching the MCP server](screenshot_copilot_search.png)
+
+The `display_image_files` tool renders selected images in an interactive carousel using MCP apps:
+
+![GitHub Copilot displaying selected images in an MCP app carousel](screenshot_copilot_app.png)
 
 This project uses the sample nature data set from [Vision Studio](https://learn.microsoft.com/azure/ai-services/computer-vision/overview-vision-studio).
 
 ## Azure account requirements
 
-**IMPORTANT:** In order to deploy and run this example, you'll need:
+In order to deploy and run this example, you'll need:
 
 * **Azure account**. If you're new to Azure, [get an Azure account for free](https://azure.microsoft.com/free/cognitive-search/).
 * **Azure account permissions**:
@@ -88,30 +100,44 @@ When `setup_search_service.py` runs (automatically during `azd up`, or manually)
 1. **Blob upload** – Images from the `pictures/` folder are uploaded to an Azure Blob Storage container. Only files placed directly under `pictures/` are uploaded.
 2. **Data source** – A Search data source is configured to point at that blob container.
 3. **Skillset** – Two skills are applied to each image:
-   - [`VisionVectorizeSkill`](https://learn.microsoft.com/azure/search/cognitive-search-skill-vision-vectorize) generates a 1024-dimensional multimodal embedding using Azure AI Vision.
-   - [`ChatCompletionSkill`](https://learn.microsoft.com/azure/search/chat-completion-skill-example-usage) (optional) produces a natural-language description of each image, stored as `verbalized_image`.
+   * [`VisionVectorizeSkill`](https://learn.microsoft.com/azure/search/cognitive-search-skill-vision-vectorize) generates a 1024-dimensional multimodal embedding using Azure AI Vision.
+   * [`ChatCompletionSkill`](https://learn.microsoft.com/azure/search/chat-completion-skill-example-usage) (optional) produces a natural-language description of each image, stored as `verbalized_image`.
 4. **Index** – Results are written to a Search index with an `embedding` vector field backed by an HNSW algorithm, plus a built-in AI Vision vectorizer so that query text is embedded at search time without extra code.
 5. **Indexer** – Runs the pipeline: blob → normalized images → skills → index projections. Each image within a blob becomes its own indexed document.
 
 At query time, the search service vectorizes the text query using the same AI Vision model and performs an approximate nearest-neighbor search over the stored embeddings.
 
-## Running locally
+## Testing deployed endpoints
 
-You can only run locally **after** having successfully run the `azd up` command. If you haven't yet, follow the steps in [Azure deployment](#azure-deployment) above.
+After running `azd up`, you can test both deployed Container Apps.
 
-### Web app
+### Deployed web app
 
-1. Run `azd auth login`
-2. Change dir to `app` and run `./start.ps1` or `./start.sh` depending on your OS.
-3. Open a browser and navigate to `http://localhost:50505`
+1. Run `azd env get-value SERVICE_ACA_URI` to get the deployed web app URL.
+2. Open the URL in a browser.
 
-### MCP server
+### Deployed MCP server
 
-1. Run `azd auth login`
-2. Run `python app/backend/mcp_server.py`
-3. The server starts on `http://localhost:8001`. Add it to your MCP client configuration (e.g. VS Code or Claude Desktop) pointing at that URL.
+1. Run `azd env get-value SERVICE_MCP_URI` to get the deployed MCP server URL.
+2. Append `/mcp` to the URL.
+3. In `.mcp.json`, replace the `url` for the `image-search-azure` entry under `mcpServers` with the resulting URL.
+4. Open your MCP client and use the `image-search-azure` server.
+
+## Testing local endpoints
+
+You can only run the endpoints locally **after** successfully running `azd up`. If you haven't yet, follow the steps in [Azure deployment](#azure-deployment) above.
+
+### Local web app
+
+1. Run `azd auth login`.
+2. Change to the `app` directory and run `./start.ps1` or `./start.sh`, depending on your OS.
+3. Open `http://localhost:50505` in a browser.
+
+### Local MCP server
 
-To connect to a deployed MCP server instead, update `.vscode/mcp.json` with your deployed Container App hostname and append `/mcp` (for example, using the `SERVICE_MCP_URI` output from `azd up`).
+1. Run `azd auth login`.
+2. Run `python app/backend/mcp_server.py`.
+3. Open your MCP client and use the `image-search-local` server from `.mcp.json`, which connects to `http://localhost:8001/mcp`.
 
 ## Adding new images
 

app/backend/image-viewer.html

@@ -13,16 +13,17 @@
             display: flex;
             align-items: center;
             justify-content: center;
-            height: 340px;
+            height: 430px;
             width: 700px;
+            font-family: ui-sans-serif, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
         }
         #carousel {
             position: relative;
             width: 680px;
-            height: 320px;
+            height: 410px;
             display: flex;
             align-items: center;
-            justify-content: center;
+            flex-direction: column;
         }
         #frame {
             width: 580px;
@@ -61,12 +62,36 @@
         #next { right: 0; }
         #counter {
             position: absolute;
-            bottom: 4px;
-            left: 50%;
-            transform: translateX(-50%);
+            top: 286px;
+            right: 58px;
+            font-size: 12px;
+            color: gray;
+        }
+        #details {
+            width: 580px;
+            padding-top: 10px;
+            box-sizing: border-box;
+        }
+        #title {
+            margin: 0;
+            font-size: 16px;
+            font-weight: 600;
+            line-height: 22px;
+        }
+        #description {
+            margin: 3px 0 0;
+            font-size: 13px;
+            line-height: 18px;
+            color: gray;
+            display: -webkit-box;
+            -webkit-line-clamp: 2;
+            -webkit-box-orient: vertical;
+            overflow: hidden;
+        }
+        #metadata {
+            margin-top: 5px;
             font-size: 12px;
             color: gray;
-            font-family: sans-serif;
         }
     </style>
 </head>
@@ -76,6 +101,11 @@
         <div id="frame"></div>
         <button class="nav" id="next" type="button" aria-label="Next image">&#8250;</button>
         <span id="counter" aria-live="polite"></span>
+        <div id="details">
+            <h2 id="title"></h2>
+            <p id="description"></p>
+            <div id="metadata"></div>
+        </div>
     </div>
     <script type="module">
         import { App } from "https://unpkg.com/@modelcontextprotocol/ext-apps@0.4.0/app-with-deps";
@@ -89,15 +119,33 @@
         const prevBtn = document.getElementById("prev");
         const nextBtn = document.getElementById("next");
         const counter = document.getElementById("counter");
+        const title = document.getElementById("title");
+        const description = document.getElementById("description");
+        const metadata = document.getElementById("metadata");
+
+        function formatBytes(bytes) {
+            if (!Number.isFinite(bytes)) return "";
+            if (bytes < 1024) return `${bytes} B`;
+            if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB`;
+            return `${(bytes / (1024 * 1024)).toFixed(1)} MB`;
+        }
 
         function show(i) {
             index = i;
             const img = images[index];
             frame.innerHTML = "";
             const el = document.createElement("img");
             el.src = `data:${img.mimeType || "image/jpeg"};base64,${img.data}`;
-            el.alt = "Blob image";
+            el.alt = img.filename || "Image";
             frame.appendChild(el);
+            title.textContent = img.filename || "Image";
+            description.textContent = img.description || "";
+            description.hidden = !img.description;
+            metadata.textContent = [
+                img.width && img.height ? `${img.width} × ${img.height}` : "",
+                img.format || "",
+                formatBytes(img.sizeBytes),
+            ].filter(Boolean).join(" · ");
             prevBtn.disabled = index === 0;
             nextBtn.disabled = index === images.length - 1;
             counter.textContent = images.length > 1 ? `${index + 1} / ${images.length}` : "";
@@ -106,8 +154,13 @@
         prevBtn.addEventListener("click", () => { if (index > 0) show(index - 1); });
         nextBtn.addEventListener("click", () => { if (index < images.length - 1) show(index + 1); });
 
-        app.ontoolresult = ({ content }) => {
-            images = (content || []).filter((block) => block.type === "image");
+        app.ontoolresult = ({ content, structuredContent }) => {
+            const imageContent = (content || []).filter((block) => block.type === "image");
+            const imageMetadata = structuredContent?.images || [];
+            images = imageContent.map((image, imageIndex) => ({
+                ...image,
+                ...(imageMetadata[imageIndex] || {}),
+            }));
             if (images.length > 0) show(0);
         };
 

app/backend/mcp_server.py

@@ -3,7 +3,7 @@
 import os
 import subprocess
 import base64
-from typing import Annotated
+from typing import Annotated, cast
 from pathlib import Path
 from urllib.parse import unquote, urlparse
 
@@ -18,7 +18,7 @@
 from azure.storage.blob.aio import BlobServiceClient
 from dotenv import load_dotenv
 from fastmcp import FastMCP
-from fastmcp.server.apps import AppConfig, ResourceCSP
+from fastmcp.apps import AppConfig, ResourceCSP
 from fastmcp.server.lifespan import lifespan
 from fastmcp.tools.tool import ToolResult
 from fastmcp.utilities.types import File
@@ -203,28 +203,39 @@ async def display_image_files(
     filenames: Annotated[
         list[str], "List of blob filenames to retrieve and display in a carousel."
     ],
+    descriptions: Annotated[
+        list[str] | None,
+        "Optional image descriptions from image_search, in the same order as filenames.",
+    ] = None,
 ) -> ToolResult:
-    """Fetch images from blob storage by filename and render them in a carousel MCP App."""
+    """Fetch images by filename and render them in a carousel with descriptions and file details."""
     if len(filenames) < 1:
         raise ValueError("Provide at least one filename.")
+    if descriptions is not None and len(descriptions) != len(filenames):
+        raise ValueError(
+            "Descriptions must have the same number of items as filenames."
+        )
 
     blob_service_client = get_blob_service_client()
 
     image_blocks: list[types.ImageContent] = []
-    image_results: list[dict[str, str]] = []
-    for filename in filenames:
+    image_results: list[dict[str, str | int]] = []
+    for image_index, filename in enumerate(filenames):
         blob_client = blob_service_client.get_blob_client(
             container=IMAGE_CONTAINER_NAME, blob=filename
         )
         try:
             stream = await blob_client.download_blob()
-            image_bytes = await stream.readall()
+            image_bytes = cast(bytes, await stream.readall())
         except ResourceNotFoundError as exc:
             raise ValueError(
                 f"Blob '{filename}' was not found in container '{IMAGE_CONTAINER_NAME}'."
             ) from exc
 
         mime_type = get_image_mime_type(filename)
+        with Image.open(io.BytesIO(image_bytes)) as image:
+            width, height = image.size
+            image_format = image.format or get_image_format(filename).upper()
         image_blocks.append(
             types.ImageContent(
                 type="image",
@@ -235,7 +246,12 @@ async def display_image_files(
         image_results.append(
             {
                 "filename": filename,
+                "description": descriptions[image_index] if descriptions else "",
                 "mimeType": mime_type,
+                "width": width,
+                "height": height,
+                "format": image_format,
+                "sizeBytes": len(image_bytes),
             }
         )
 
@@ -250,7 +266,7 @@ async def display_image_files(
 @mcp.tool(annotations={"readOnlyHint": True})
 async def image_search(
     query: Annotated[
-        str, "Text description of images to find (e.g., 'red dress', 'blue shirt')"
+        str, "Text description of images to find (e.g., 'sunlit mountain lake')"
     ],
     max_results: Annotated[int, "Maximum number of images to return (1-20)"] = 5,
 ) -> ToolResult:
@@ -273,7 +289,7 @@ async def image_search(
                 k_nearest_neighbors=max_results, fields="embedding", text=query
             )
         ],
-        select="metadata_storage_path,verbalized_image",
+        select=["metadata_storage_path", "verbalized_image"],
     )
 
     blob_service_client = get_blob_service_client()

app/backend/requirements.txt

@@ -1,12 +1,12 @@
-aiohttp==3.13.1
+aiohttp==3.13.3
 azure-identity==1.25.1
 azure-mgmt-storage==24.0.0
 azure-search-documents==11.7.0b1
-azure-storage-blob==12.27.0
+azure-storage-blob==12.27.1
 Quart==0.20.0
 typing_extensions>=4.12.2
 gunicorn==23.0.0
 uvicorn==0.40.0
 python-dotenv==1.2.1
-fastmcp==3.0.0
+fastmcp==3.4.4
 Pillow==12.1.0