fix: wire real retrieval into optimized prompts + make Part 3 a genuine tool-calling agent

Final review found the optimizer never saw the (fictional) docs, so the
exact-match RAG lesson could not work, and Part 3's agent was dead code.

Part 1-2: attach retrieved context (rag_app.retrieve) to each eval row and
inject it via the ChatPrompt user template (retrieve-then-generate); retrieval
is fixed, the prompt is optimized. Print baseline vs optimized .system so the
refinement is visible. Verified offline: 18/18 exact answers appear in the
retrieved top-3 context.

Part 3: replace the never-optimized agentic_answer gate with a genuine
tool-calling agent (search_docs -> rag_app.retrieve, allow_tool_use) whose
system prompt is actually optimized; note optimize_tools=True as the one-line
tool-description add; FewShot tunes the same agent. Tool binding verified
against the optimizer's own resolve_toolcalling_tools. README Part 3 + the
Part 4 selection table updated to match.

Author: LeoRoccoBreedt

guides/prompt_agent_optimization/README.md

@@ -19,8 +19,9 @@ step is a comparable run.
 - **Part 1** ⭐ — your first optimization: exact-match metric + `MetaPromptOptimizer`.
 - **Part 2** — LLM-judge metrics, *how to trust a judge*, and multi-objective
   optimization with `MultiMetricObjective`.
-- **Part 3** — from prompt to agent: a retrieval gate, `FewShotBayesianOptimizer`,
-  and `ParameterOptimizer`.
+- **Part 3** — from prompt to agent: a tool-calling `search_docs` agent optimized
+  end-to-end, then `FewShotBayesianOptimizer` on the same agent (with a pointer to
+  `ParameterOptimizer`).
 - **Part 4** — choosing an optimizer (selection table + how to choose + chaining).
 - **Part 5** — promote the winner to the Prompt Library; pointers to Optimization
   Studio and the docs.

guides/prompt_agent_optimization/prompt_agent_optimization.ipynb

@@ -85,8 +85,17 @@
    "outputs": [],
    "source": [
     "exact_cases = data.load_exact_cases()\n",
+    "\n",
+    "# RAG step — retrieve. For each question, pull the most relevant docs with our\n",
+    "# real ChromaDB retriever and attach them as `context`. A production RAG system\n",
+    "# retrieves per query at answer time; we do it once here so every optimizer trial\n",
+    "# answers the SAME question from the SAME context. What we optimize is the\n",
+    "# *prompt*, not the retriever.\n",
+    "for case in exact_cases:\n",
+    "    case[\"context\"] = \"\\n\\n\".join(rag_app.retrieve(case[\"query\"]))\n",
+    "\n",
     "exact_dataset = data.build_dataset(client, \"ledgerline-exact\", exact_cases)\n",
-    "print(f\"Dataset 'ledgerline-exact' has {len(exact_cases)} cases\")"
+    "print(f\"Dataset 'ledgerline-exact' has {len(exact_cases)} cases (each with retrieved context)\")"
    ]
   },
   {
@@ -128,7 +137,9 @@
    "source": [
     "#### The starting prompt\n",
     "\n",
-    "Here is our baseline system prompt — deliberately mediocre, so there's room to improve. This is the `ChatPrompt` the optimizer will rewrite. `{query}` is filled from each dataset row."
+    "Here is our baseline system prompt — deliberately mediocre, so there's room to improve. This is the `ChatPrompt` the optimizer will rewrite.\n",
+    "\n",
+    "Look at the **user template**: `{context}` is filled with the docs we just retrieved and `{query}` with the question — both come from each dataset row. That's the **retrieve-then-generate** shape of a real RAG system. Retrieval is held fixed; what we optimize is how the **system prompt** tells the model to turn that context into a correct, concise answer."
    ]
   },
   {
@@ -145,7 +156,7 @@
     "prompt = ChatPrompt(\n",
     "    name=\"ledgerline-answer\",\n",
     "    system=BASELINE_SYSTEM,\n",
-    "    user=\"{query}\",\n",
+    "    user=\"Context:\\n{context}\\n\\nQuestion: {query}\",\n",
     "    model=config.GEN_MODEL,\n",
     ")"
    ]
@@ -188,7 +199,13 @@
     "\n",
     "print(\"Baseline score:\", result.initial_score)\n",
     "print(\"Best score:    \", result.score)\n",
-    "print(\"\\nOptimized system prompt:\\n\", result.prompt)"
+    "\n",
+    "# See HOW the prompt was refined: the optimizer rewrote the *system* instructions.\n",
+    "# result.prompt is a ChatPrompt; .system is the optimized system text.\n",
+    "print(\"\\n--- Baseline system prompt ---\")\n",
+    "print(BASELINE_SYSTEM)\n",
+    "print(\"\\n--- Optimized system prompt ---\")\n",
+    "print(result.prompt.system)"
    ]
   },
   {
@@ -198,7 +215,7 @@
    "source": [
     "#### See it in Opik\n",
     "\n",
-    "Open **Evaluation → Optimization runs** in your Opik workspace. You'll see this run with every candidate prompt, its score, and the trace for each trial. Compare the baseline row to the best row — that delta is your improvement.\n",
+    "The cell above printed the **baseline vs optimized system prompt** side by side — that rewrite is the concrete refinement the optimizer found. Now open **Evaluation → Optimization runs** in your Opik workspace: you'll see this run with every candidate prompt, its score, and the trace for each trial. Compare the baseline row to the best row — that delta is your improvement.\n",
     "\n",
     "🎓 **This is where the live workshop ends.** You've run a real optimization and improved a prompt, measured against a dataset, stored in Opik. Everything below builds on exactly this loop."
    ]
@@ -289,12 +306,14 @@
    "outputs": [],
    "source": [
     "judge_cases = data.load_judge_cases()\n",
+    "for case in judge_cases:\n",
+    "    case[\"context\"] = \"\\n\\n\".join(rag_app.retrieve(case[\"query\"]))\n",
     "judge_dataset = data.build_dataset(client, \"ledgerline-judge\", judge_cases)\n",
     "\n",
     "judge_prompt = ChatPrompt(\n",
     "    name=\"ledgerline-answer-judge\",\n",
     "    system=BASELINE_SYSTEM,\n",
-    "    user=\"{query}\",\n",
+    "    user=\"Context:\\n{context}\\n\\nQuestion: {query}\",\n",
     "    model=config.GEN_MODEL,\n",
     ")\n",
     "\n",
@@ -352,7 +371,7 @@
     "    n_samples=8,\n",
     ")\n",
     "print(\"Multi-objective best score:\", multi_result.score)\n",
-    "print(\"\\nOptimized prompt:\\n\", multi_result.prompt)"
+    "print(\"\\nOptimized system prompt:\\n\", multi_result.prompt.system)"
    ]
   },
   {
@@ -372,7 +391,11 @@
    "source": [
     "## Part 3 — From prompt to agent\n",
     "\n",
-    "So far we optimized a single answer prompt. Real systems are agents: they *decide* what to do. Our RAG app can grow a **retrieval gate** — decide whether a question even needs a docs lookup (cheap questions skip retrieval). That decision is itself a prompt, and the same optimizer loop tunes it."
+    "So far retrieval was **fixed**: we retrieved once, put the docs in the prompt, and optimized the wording. Real systems are agents — they *decide* what to do. Here we hand the model a **`search_docs` tool** wired to our retriever and let it choose when to call it. That turns the prompt into an **agent**, and the same optimizer loop tunes it.\n",
+    "\n",
+    "**What \"optimizing an agent\" means:** not rewriting the tool's code — the retriever is fixed. It means optimizing the natural-language surface the agent reasons over: its **system prompt** (when to search, how to answer from results) and, optionally, its **tool descriptions** (`optimize_tools=True`) so it calls the tool at the right moments.\n",
+    "\n",
+    "*(A lighter alternative to a tool is a yes/no retrieval gate — see `rag_app.should_retrieve` — optimized with the same loop. We use a real tool here because it better reflects a production agent.)*"
    ]
   },
   {
@@ -382,18 +405,54 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "@opik.track\n",
-    "def agentic_answer(query: str, system_prompt: str) -> str:\n",
-    "    # Agent step: decide whether to retrieve, then answer accordingly.\n",
-    "    if rag_app.should_retrieve(query):\n",
-    "        return rag_app.answer(query, system_prompt=system_prompt)\n",
-    "    messages = [\n",
-    "        {\"role\": \"system\", \"content\": system_prompt},\n",
-    "        {\"role\": \"user\", \"content\": query},\n",
-    "    ]\n",
-    "    import litellm\n",
-    "    resp = litellm.completion(model=config.GEN_MODEL, messages=messages)\n",
-    "    return resp.choices[0].message.content"
+    "# The tool the agent may call. It wraps our real ChromaDB retriever; the *agent*\n",
+    "# decides when to call it. Returning one string keeps the tool result clean.\n",
+    "def search_docs(query: str) -> str:\n",
+    "    \"\"\"Search the Ledgerline documentation and return the most relevant snippets.\"\"\"\n",
+    "    return \"\\n\\n\".join(rag_app.retrieve(query))\n",
+    "\n",
+    "\n",
+    "SEARCH_DOCS_TOOL = {\n",
+    "    \"type\": \"function\",\n",
+    "    \"function\": {\n",
+    "        \"name\": \"search_docs\",\n",
+    "        \"description\": \"Search the Ledgerline product documentation for relevant snippets.\",\n",
+    "        \"parameters\": {\n",
+    "            \"type\": \"object\",\n",
+    "            \"properties\": {\n",
+    "                \"query\": {\"type\": \"string\", \"description\": \"What to look up in the docs.\"},\n",
+    "            },\n",
+    "            \"required\": [\"query\"],\n",
+    "        },\n",
+    "    },\n",
+    "}\n",
+    "\n",
+    "AGENT_SYSTEM = \"You are a Ledgerline support agent. Use tools when they help.\"\n",
+    "\n",
+    "# tools + function_map make this ChatPrompt an agent: on a tool call, the\n",
+    "# optimizer executes search_docs and feeds the result back to the model.\n",
+    "agent_prompt = ChatPrompt(\n",
+    "    name=\"ledgerline-agent\",\n",
+    "    system=AGENT_SYSTEM,\n",
+    "    user=\"{query}\",\n",
+    "    tools=[SEARCH_DOCS_TOOL],\n",
+    "    function_map={\"search_docs\": search_docs},\n",
+    "    model=config.GEN_MODEL,\n",
+    ")\n",
+    "\n",
+    "# optimize_prompts defaults to \"system\": we tune the agent's instructions.\n",
+    "# (Flip optimize_tools=True to ALSO let the optimizer refine the tool description.)\n",
+    "agent_result = optimizer.optimize_prompt(\n",
+    "    prompt=agent_prompt,\n",
+    "    dataset=judge_dataset,\n",
+    "    metric=answer_relevance,\n",
+    "    max_trials=8,\n",
+    "    n_samples=8,\n",
+    "    allow_tool_use=True,\n",
+    ")\n",
+    "print(\"Agent baseline:\", agent_result.initial_score, \"-> best:\", agent_result.score)\n",
+    "print(\"\\n--- Optimized agent system prompt ---\")\n",
+    "print(agent_result.prompt.system)"
    ]
   },
   {
@@ -403,7 +462,7 @@
    "source": [
     "#### When demonstrations matter: Few-Shot Bayesian\n",
     "\n",
-    "If the win comes from *showing examples* rather than rewording instructions, reach for `FewShotBayesianOptimizer` — it uses Bayesian search (Optuna) to pick the best set and order of few-shot demonstrations to attach to your prompt. Same API as before."
+    "If the win comes from *showing examples* rather than rewording instructions, reach for `FewShotBayesianOptimizer` — it uses Bayesian search (Optuna) to pick the best set and order of few-shot demonstrations to attach. We point it at the **same agent**, so it tunes the agent's examples rather than a fresh prompt."
    ]
   },
   {
@@ -418,7 +477,7 @@
     "fewshot_optimizer = FewShotBayesianOptimizer(model=config.OPTIMIZER_MODEL, n_threads=4)\n",
     "\n",
     "fewshot_result = fewshot_optimizer.optimize_prompt(\n",
-    "    prompt=judge_prompt,\n",
+    "    prompt=agent_prompt,\n",
     "    dataset=judge_dataset,\n",
     "    metric=answer_relevance,\n",
     "    n_samples=8,\n",
@@ -452,7 +511,7 @@
     "| **Few-Shot Bayesian** | Picking the best demonstrations | Part 3 |\n",
     "| **Evolutionary** | Exploring diverse structures; multi-objective | (see multi-objective, Part 2) |\n",
     "| **GEPA** | Single-turn, reflection-heavy tasks (`pip install gepa`) | (try on your own) |\n",
-    "| **Parameter** | Temperature / top_p, prompt unchanged | Part 3 |\n",
+    "| **Parameter** | Temperature / top_p, prompt unchanged | Part 3 (described) |\n",
     "\n",
     "**How to choose, in four questions:**\n",
     "1. **What's the constraint** — wording, examples, tool use, or sampling params?\n",