Add Lab D: observe, evaluate, and secure your agents

Closes the observability gap in the consolidated catalogue: there was no
tracing, evaluation or safety-testing content anywhere in the repo.

Instructions/Consolidated/
  D  landing page (generated task table from frontmatter)
  D0 getting started - project, Application Insights connection, agent to measure
  D1 trace your agent (core, L300) - OpenTelemetry + Azure Monitor, custom spans
  D2 evaluate answer quality (core, L300) - groundedness, relevance, similarity
  D3 red team your agent (optional, L400) - attack strategies + custom seed prompts

Labfiles/D-observe-evaluate-and-secure-agents/
  Python/ starter files with fill-in-the-blank placeholders, Solution/Python/
  reference code, a ground-truth JSONL dataset, custom attack objectives, and
  setup/check_env.py + setup/bootstrap_agent.py so the lab stands alone without
  Lab B. azd template, Bicep and write_env scripts are generated from
  Labfiles/_shared/ via manifest.yml.

Verified: Tier 0 checks (frontmatter, code blocks, links, line endings),
Labfiles/_shared/sync.py --check, and the Tier 1 SDK contract check against
azure-ai-projects 2.3.0 and azure-ai-evaluation 1.18.3. Not run end to end
against a live Foundry project.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Author: ivorb

Instructions/Consolidated/D-observe-evaluate-and-secure-agents.md

@@ -0,0 +1,180 @@
+---
+title: 'Observe, evaluate, and secure your agents'
+lab:
+    title: 'Observe, evaluate, and secure your agents'
+    description: 'Find out what your Tailwind Traders agent is actually doing: trace it with OpenTelemetry, score its answers against ground truth with built-in evaluators, and attack it with the AI Red Teaming Agent. A modular lab you can complete end to end or one task at a time.'
+    type: 'lab'
+    id: 'D'
+    order: 4
+    difficulty: 3
+    duration: 60
+    access: 'open'
+    level: 300
+    concepts: 'tracing, OpenTelemetry, evaluation, groundedness, AI red teaming'
+    islab: true
+    status: 'draft'
+---
+
+<!--
+PILOT NOTE (remove before publishing):
+"Lab D" is new content: there was no observability, evaluation or safety-testing
+material anywhere in this repo. It follows the same template as Labs A-C.
+Starter code lives in a single folder — Labfiles/D-observe-evaluate-and-secure-agents/Python/ —
+shared by every task (one virtual environment, one .env). The completed reference code is
+in Labfiles/D-observe-evaluate-and-secure-agents/Solution/Python/.
+
+This landing page is the lab overview. Setup lives in D0-getting-started.md and each task is
+its own page (D1-D3) so it can be completed on its own. The azd template and Bicep are
+generated from Labfiles/_shared/ — edit them there, not in the lab folder.
+-->
+
+# Observe, evaluate, and secure your agents
+
+**Level** ▰▰▰▱▱ **L300**  (**L100** beginner → **L500** expert)
+
+You can build an agent in an afternoon. Knowing whether it's any good — and whether it
+behaves when someone tries to make it misbehave — is a different job. This lab is about
+that job: seeing inside a running agent, measuring the quality of its answers, and
+attacking it before someone else does.
+
+![Anton](../Media/anton-avatar.png)<br /><strong>Meet Anton, your AI guide.</strong><br />You'll spot **Ask Anton** tips throughout this lab. Want more interactive, hands-on help? Chat with Anton in the *[Ask Anton](https://aka.ms/choose-anton)* app.
+
+<details>
+<summary><strong><i>About the Ask Anton app</i></strong></summary>
+
+<strong><i><a href="https://aka.ms/choose-anton" target="_blank">Ask Anton</a></i></strong> is a generative AI agent that can answer questions about AI concepts and Microsoft Foundry technologies. It's available in two versions at <code>https://aka.ms/choose-anton</code>:
+<ul>
+<li><strong>Azure-based</strong>: Best experience <i>(requires an Azure subscription and deployment of a model in a Foundry project)</i>.</li>
+<li><strong>Browser-based</strong>: Use a small language model in your browser <i>(reduced functionality - may be slow or work only in "basic" mode in older/lower-spec devices)</i>.</li>
+</ul>
+<blockquote><i>Ask Anton is <u>not</u> a supported Microsoft product or a component of Microsoft Learn or AI Skills Navigator.</i></blockquote>
+</details>
+
+<style>
+/* "Ask Anton" just-in-time concept blocks */
+details.concept { margin:.6rem 0 1rem; }
+details.concept > summary { display:inline-block; cursor:pointer; list-style:none;
+  font-size:.85em; font-weight:600; color:#6b4ba1; background:#6b4ba112;
+  border:1px solid #6b4ba133; border-radius:999px; padding:.2em .7em; }
+details.concept > summary::-webkit-details-marker { display:none; }
+details.concept > summary::before { content:"Ask Anton: "; font-weight:700;
+  padding-left:1.5em;
+  background:url("../Media/anton-avatar.png") left center / 1.25em 1.25em no-repeat; }
+details.concept > summary:hover { background:#6b4ba1; color:#fff; border-color:#6b4ba1; }
+details.concept[open] > summary { border-bottom-left-radius:0; border-bottom-right-radius:0; }
+details.concept .concept-body { border:1px solid #6b4ba133; border-top:none;
+  border-radius:0 8px 8px 8px; padding:.6rem .9rem; background:#6b4ba108; font-size:.95em; }
+</style>
+
+<details markdown="1" class="concept">
+<summary>Why can't I just read the output?</summary>
+<div class="concept-body" markdown="1">
+
+Because the output is the one part of an agent that looks fine when everything else isn't.
+An answer can be fluent and confidently wrong, or correct but produced by three retries and
+a tool call that timed out. **Tracing** shows you what happened on the way to the answer.
+**Evaluation** scores the answer against something you already know to be true.
+**Red teaming** tells you what the agent does when the question is hostile.
+
+</div>
+</details>
+
+**Your scenario:** you work at **Tailwind Traders**, an outdoor-gear retailer that also runs
+guided trips. The staff assistant you built in earlier labs is now answering real questions
+on the shop floor — and store managers are asking harder questions about it. *Why was that
+answer slow? Is it making things up about the returns policy? What happens if a customer
+tries to talk it into something it shouldn't say?* In this lab you answer all three with
+evidence rather than opinion.
+
+You'll start with the **Core** tasks, which get you from "it runs" to "I can prove how well
+it runs". The **Optional** task then goes after safety.
+
+> **Note**: Some of the technologies used in this exercise are in preview or in active
+> development. You may experience some unexpected behavior, warnings, or errors.
+
+## What you'll learn
+
+By completing the **Core** tasks of this exercise, you'll be able to:
+
+- **Trace an agent** with OpenTelemetry, export the traces to Azure Monitor, and read them
+  in the Foundry portal — including custom spans you add around your own code.
+- **Evaluate answer quality** against ground truth using built-in evaluators
+  (groundedness, relevance, similarity) and a JSONL dataset.
+
+The **Optional** task lets you additionally:
+
+- **Red team your agent** with the AI Red Teaming Agent: run adversarial attack strategies
+  and your own seed prompts against a deployed agent, and read the attack success rate.
+
+## How this lab is organized
+
+This lab is **modular**. Each task is written to be completed **on its own, starting fresh** —
+so you can pick a single task and do just that one. Every task also shares one starter folder,
+one virtual environment, and one `.env`, so if you'd rather work straight through, you can.
+
+1. **Start with [Getting started](D0-getting-started.md)** — create your Microsoft Foundry
+   project, connect Application Insights, get the starter code, and set up your `.env`. Every
+   task begins from here; if you're doing the whole lab in one sitting, you only need to do
+   this once.
+2. **Do any task.** Each task lists the setup it needs so you can start it independently. If
+   you're moving straight from the previous task, a short *"Continuing from a previous task?"*
+   note at the top lets you skip the repeated setup and keep going.
+
+## Lab at a glance
+
+Complete the **Core** tasks first (about **1 hour**) — they end with an agent you can see
+inside and a scorecard for its answers. Then add the **Optional** task if you want to test
+how it stands up to attack.
+
+{% include lab-tasks-table.html lab='D' %}
+
+**Choosing your path** — pick the tasks that fit the time you have:
+
+- **Core only (~1h):** do Tasks 1–2.
+- **Everything (~1h 35m):** add **Task 3**, the red team scan.
+
+> **One agent, three questions**: Task 1 traces an agent you create in code. Tasks 2 and 3
+> both point at the **grounded knowledge agent** from
+> [Lab B](B-integrate-agents-with-enterprise-knowledge-and-m365.md). If you haven't done Lab B,
+> one command creates an equivalent agent so this lab stands alone — see
+> [Getting started](D0-getting-started.md).
+
+## Measure, don't guess
+
+The three techniques in this lab answer different questions, and it's worth being clear about
+which is which:
+
+- **Tracing** answers *"what happened?"* It's a record of one run: which spans took how long,
+  which tools were called, what the model was sent. Use it when something is slow or broke.
+- **Evaluation** answers *"how good is it, on average?"* It's a score over a dataset, so it's
+  the only one of the three that tells you whether a change made things better or worse.
+- **Red teaming** answers *"what can I make it do?"* It's an adversarial probe, and a clean
+  result is a floor, not a guarantee.
+
+None of them replaces the others, and all three are cheap compared to finding out from a
+customer.
+
+## Summary
+
+Across this lab you:
+
+- **Instrumented an agent** with OpenTelemetry, exported traces to Application Insights, and
+  read them — including your own custom spans — in the Foundry portal.
+- **Evaluated a grounded agent** against a ground-truth dataset with built-in groundedness,
+  relevance and similarity evaluators, and got a score you can compare across changes.
+- (Optionally) **Red teamed the agent** with adversarial attack strategies and your own seed
+  prompts, and read the resulting attack success rate.
+
+Together these turn "the demo worked" into evidence you can show someone.
+
+## Clean up
+
+If you're finished, delete the resources you created to avoid unnecessary Azure costs.
+
+1. In the [Azure portal](https://portal.azure.com), navigate to the resource group that contains your Foundry resource.
+1. On the toolbar, select **Delete resource group**, enter the resource group name, and confirm.
+
+> The code you run in Task 1 deletes the agent version it creates. The agent Tasks 2 and 3
+> measure is removed when you delete the resource group. If you provisioned with `azd`, run
+> `azd down` instead — but note that Application Insights, if you created it from the Foundry
+> portal, is a separate resource and is deleted with the resource group rather than by `azd`.

Instructions/Consolidated/D0-getting-started.md

@@ -0,0 +1,189 @@
+---
+title: 'Getting started: set up your environment'
+lab:
+    title: 'Getting started: set up your environment'
+    description: 'Shared setup for the Observe, evaluate, and secure your agents lab: create a Microsoft Foundry project, connect Application Insights, get the starter code, and configure your environment. Complete this once before any task.'
+    type: 'task'
+    parent: 'D'
+    order: 0
+    section: 'setup'
+    access: 'open'
+    level: 300
+    concepts: 'environment setup, Microsoft Foundry project, Application Insights'
+    status: 'draft'
+---
+
+# Getting started
+
+This page sets up everything the **Observe, evaluate, and secure your agents** lab needs.
+**Every task begins here** — complete this page first. Each task is written so you can then
+do it on its own; if you're working through the whole lab in one sitting, you only need to
+do this setup once.
+
+**Your scenario:** you work at **Tailwind Traders**, an outdoor-gear retailer that also runs
+guided trips. The staff assistant is live, and this lab is how you find out what it's really
+doing: tracing it, scoring its answers, and attacking it.
+
+> **Note**: Some of the technologies used in this lab are in preview or in active
+> development. You may experience some unexpected behavior, warnings, or errors.
+
+## Prerequisites
+
+Before starting, ensure you have:
+
+- An [Azure subscription](https://azure.microsoft.com/free/) with sufficient permissions and quota to provision Azure AI resources
+- [Visual Studio Code](https://code.visualstudio.com/) installed on your local machine
+- [Python 3.13](https://www.python.org/downloads/) installed
+- [Git](https://git-scm.com/downloads) installed on your local machine
+- Basic familiarity with Python
+
+> \* Use Python **3.10–3.13**. Task 3 installs PyRIT, which doesn't support Python 3.14 yet.
+> The lab has been tested with Python 3.13.
+
+## Create a Microsoft Foundry project
+
+You need a Foundry project and a deployed model for every task. You can create these in the
+portal (the default), or provision them with one command using the Azure Developer CLI (`azd`).
+
+### Option A — Create the project in the portal (default)
+
+1. In a web browser, open the [Foundry portal](https://ai.azure.com) at `https://ai.azure.com` and sign in using your Azure credentials. Close any tips or quick start panes, and if necessary use the **Foundry** logo at the top left to navigate to the home page.
+
+    > **Important**: For this lab, you're using the **New** Foundry experience.
+
+1. In the top banner, select **Start building**.
+
+1. When prompted, create a **new** project and enter a valid name (for example, `observability-lab-project`).
+
+1. Expand **Advanced options** and specify:
+    - **Microsoft Foundry resource**: *A valid name for your Foundry resource*
+    - **Region**: *Select one available near you*\*
+    - **Subscription**: *Your Azure subscription*
+    - **Resource group**: *Select or create a resource group*
+
+    > \* If you plan to do **Task 3** (red teaming), the AI Red Teaming Agent is only available
+    > in **East US 2**, **France Central**, **Sweden Central**, **Switzerland West**, and
+    > **North Central US**. Choosing one of those now saves you creating a second project later.
+
+1. Select **Create** and wait for your project to be created.
+
+1. On the project **Overview** page, note the **project endpoint** and the name of the model
+    deployment that was created for you — you'll put both in your `.env`.
+
+### Option B — Provision with azd (optional, one command)
+
+If you'd rather not click through the portal, the lab ships an optional `azd` template that
+creates the Foundry resource, a project, and a model deployment for you.
+
+1. Install the [Azure Developer CLI](https://learn.microsoft.com/azure/developer/azure-developer-cli/install-azd).
+
+1. From the `Labfiles/D-observe-evaluate-and-secure-agents` folder, run:
+
+    ```
+    azd auth login
+    azd up
+    ```
+
+1. Answer the prompts (environment name, region). When it finishes, `azd` writes
+    `PROJECT_ENDPOINT` and `MODEL_DEPLOYMENT_NAME` into `Python/.env` for you.
+
+    > **Note**: `azd up` does **not** create the Application Insights resource Task 1 needs —
+    > connect that in the portal using the steps below. When you're done with the lab, run
+    > `azd down` to delete everything it created.
+
+## Connect Application Insights (needed for Task 1)
+
+Foundry stores traces in an **Application Insights** resource connected to your project. Connect
+one now — it takes a minute, and once it's connected Foundry starts recording server-side traces
+for your agents without any code at all.
+
+1. In the [Foundry portal](https://ai.azure.com), open your project.
+
+1. In the left navigation, select **Agents**, then select **Traces** at the top.
+
+1. Select **Connect**, then either pick an existing Application Insights resource or select
+    **Create new** and complete the wizard.
+
+    > If you don't see the **Connect** button, select **Manage** in the upper right, then
+    > **Project details** > **Connected resources** > **Add connection** > **Application Insights**.
+
+1. To *read* the traces you'll need the **Log Analytics Reader** role on that Application
+    Insights resource. If you created it yourself, you already have it.
+
+> **Why this matters**: your Foundry project can only hand your code a connection string if
+> something is connected. Task 1 asks the project for that string, so this step has to happen
+> first.
+
+## Get the starter code
+
+1. In VS Code, open the Command Palette (**Ctrl+Shift+P**), run **Git: Clone**, and enter:
+
+    ```
+    https://github.com/MicrosoftLearning/mslearn-ai-agents.git
+    ```
+
+1. Open the cloned repo, then **File > Open Folder** and select `mslearn-ai-agents/Labfiles/D-observe-evaluate-and-secure-agents/Python`. This single folder holds the starter code for **every** task in this lab — you use one virtual environment and one `.env` throughout.
+
+1. Right-click **requirements.txt** and choose **Open in Integrated Terminal**. Then create a virtual environment and install packages:
+
+    ```
+    python -m venv labenv
+    .\labenv\Scripts\Activate.ps1
+    pip install -r requirements.txt
+    ```
+
+    > This install is larger than the other labs — it includes the evaluation SDK and, for
+    > Task 3, PyRIT. Give it a few minutes.
+
+1. Open the **.env** file and set `PROJECT_ENDPOINT` to your project endpoint and `MODEL_DEPLOYMENT_NAME` to your model deployment name. Save the file. (If you used `azd up`, these are already filled in.)
+
+    > **Tip**: In the Foundry Toolkit VS Code extension, right-click your project deployment and select **Copy Project Endpoint** to get the endpoint URL.
+
+## Get an agent to measure (needed for Tasks 2 and 3)
+
+Tasks 2 and 3 measure a **grounded** agent — one that answers from the Tailwind Traders
+knowledge base rather than from the model's own memory. You have two ways to get one:
+
+- **You did [Lab B](B-integrate-agents-with-enterprise-knowledge-and-m365.md)**: set `AGENT_NAME`
+  in `.env` to that agent's name (`tailwind-knowledge-agent` if you kept the default) and you're
+  done.
+- **You didn't**: create an equivalent agent here. Sign in and run, from the `Python` folder with
+  the virtual environment active:
+
+    ```
+    az login
+    ```
+
+    ```
+    python ../setup/bootstrap_agent.py
+    ```
+
+    This uploads the documents in `Python/knowledge/`, grounds an agent named
+    `tailwind-knowledge-agent` on them with File Search, and writes `AGENT_NAME` into your `.env`.
+
+> Task 1 doesn't need this agent — it creates and deletes its own.
+
+## Check you're ready for a task
+
+Each task needs specific values in your `.env`. Before starting a task, run the preflight
+check from the `Python` folder you opened in VS Code — it reads your `.env` and tells you
+what (if anything) is missing:
+
+```
+python ../setup/check_env.py --task 1
+```
+
+Swap `1` for the task number you're about to start.
+
+> **Tip**: The preflight check uses only the Python standard library, so it's safe to run
+> before `pip install` and without the virtual environment active. It can't see whether
+> Application Insights is connected — that's a project setting, not a `.env` value — so
+> do the connection step above if you're starting at Task 1.
+
+That's it — head to any task:
+
+| Task | Page |
+| --- | --- |
+| Task 1 – Trace your agent | [D1](D1-trace-your-agent.md) |
+| Task 2 – Evaluate answer quality | [D2](D2-evaluate-answer-quality.md) |
+| Task 3 – Red team your agent | [D3](D3-red-team-your-agent.md) |