|
| 1 | +--- |
| 2 | +id: monitor-ai-agents |
| 3 | +title: Monitor your AI agents |
| 4 | +sidebar_label: Monitor your AI agents |
| 5 | +description: "Monitor and troubleshoot your AI agent process instances in Camunda 8 using Operate and Optimize." |
| 6 | +keywords: ["agentic ai", "AI agents"] |
| 7 | +--- |
| 8 | + |
| 9 | +import Tabs from "@theme/Tabs"; |
| 10 | +import TabItem from "@theme/TabItem"; |
| 11 | + |
| 12 | +Monitor and troubleshoot your AI agent process instances in Camunda 8 using Operate and Optimize. |
| 13 | + |
| 14 | +## About |
| 15 | + |
| 16 | +In this guide, you will: |
| 17 | + |
| 18 | +- Inspect an AI agent process instance in Operate. |
| 19 | +- Understand agent’s tool usage and metadata such as tool call inputs and results. |
| 20 | +- Analyze the agent context and how it is stored. |
| 21 | +- Examine process data in Optimize. |
| 22 | + |
| 23 | +:::note |
| 24 | +Operate enables inspection of execution paths, tool usage, and agent metadata. However, certain runtime artifacts, such as document storage contents, may require additional configuration. |
| 25 | +::: |
| 26 | + |
| 27 | +After completing this guide, you will be able to inspect, debug, and monitor AI agent executions in Camunda 8. |
| 28 | + |
| 29 | +## Prerequisites |
| 30 | + |
| 31 | +- You have the [AI Agent Chat Quick Start](https://marketplace.camunda.com/en-US/apps/587865) model blueprint deployed in [Modeler](/components/modeler/about-modeler.md). |
| 32 | +- You are familiar with [Operate](/components/operate/operate-introduction.md). |
| 33 | + |
| 34 | +:::important |
| 35 | +This guide is a follow-up to [Build your first AI agent](../../guides/getting-started-agentic-orchestration.md), where you will use the same example AI agent process. Therefore, the same [prerequisites apply](../..//guides/getting-started-agentic-orchestration.md#prerequisites). However, it can be applied to other AI agent process implementations. |
| 36 | +::: |
| 37 | + |
| 38 | +## Step 1: Run your AI agent process |
| 39 | + |
| 40 | +Run your process instance using a prompt to trigger the AI Agent connector. |
| 41 | +For example: |
| 42 | + |
| 43 | +1. Enter "Visit _docs.camunda.io_ and tell me about it" in the **How can I help you today?** field. |
| 44 | +1. Click **Start instance**. |
| 45 | + |
| 46 | +## Step 2: Open the process instance in Operate |
| 47 | + |
| 48 | +1. Open [Operate](/components/operate/operate-introduction.md). |
| 49 | +2. Locate the process instance created by your prompt. See [View a deployed process](/components/operate/userguide/basic-operate-navigation.md#view-a-deployed-process) for more details. |
| 50 | +3. Open your process instance view by clicking on its process instance key. |
| 51 | + |
| 52 | +At this point, you should see the process progressing through your model. |
| 53 | + |
| 54 | +## Step 3: Understand what Operate shows |
| 55 | + |
| 56 | +With Operate, you can track the agent activity and see which tool tasks are called. |
| 57 | + |
| 58 | +1. To show how many times each BPMN element is triggered, select **Execution count**. For this particular prompt example, you can see: |
| 59 | + - The AI Agent connector was triggered once. |
| 60 | + - Within it, the agent executed the **Fetch URL** tool. This aligns with your prompt example of extracting information from a website. |
| 61 | + |
| 62 | +2. Select the **Fetch URL** tool element: |
| 63 | + - In the bottom-left pane, you can see where the element belongs in the execution tree. |
| 64 | + - In the bottom-right pane, the element details are displayed, including the [**Variables**](components/concepts/variables.md) and [**Input/Output Mappings**](/components/concepts/variables.md#inputoutput-variable-mappings) columns, among others. |
| 65 | + However, the actual tool inputs and results are stored in a **parent scope** and are accessible via the element's inner instance in the execution tree. |
| 66 | + |
| 67 | +## Step 4: Inspect tool calls |
| 68 | + |
| 69 | +Each tool execution produces an inner instance where you can find: |
| 70 | + |
| 71 | +- The inputs passed into the tool. |
| 72 | +- The results. |
| 73 | + |
| 74 | +To see the **Fetch URL** tool input and results: |
| 75 | + |
| 76 | +1. In the execution tree, select the **AI_Agent#innerInstance** parent element of the **Fetch URL** tool. You will see: |
| 77 | + - The `toolCall` variable (the _input_). In its value field, you can see the URL you asked the agent to fetch information from. |
| 78 | + - The `toolCallResult` variable (the _results_). See [Tool call responses](/components/connectors/out-of-the-box-connectors/agentic-ai-aiagent-tool-definitions.md#tool-call-responses) for more details. |
| 79 | + |
| 80 | +2. To better inspect the results, click the pencil icon to enter edit mode for `toolCallResult`. |
| 81 | +3. Click the two-squares icon to open the JSON editor modal. With this, you can inspect the full payload of the variable value. The result is a reference to a document where the actual output is stored (you will learn more about this in [Step 6](#step-6-understand-how-agent-memory-is-stored)). |
| 82 | + |
| 83 | +Here's the `toolCallResult` value: |
| 84 | + |
| 85 | +```json |
| 86 | +{ |
| 87 | + "storeId": "gcp", |
| 88 | + "documentId": "1aa0e4f6-7cee-4c19-ac2e-cebee3c7b151", |
| 89 | + "contentHash": "649a8575bf8ced157fbbba7acc02ba33058bb2e3f40fb37b0f303730fb64fba0", |
| 90 | + "metadata": { |
| 91 | + "contentType": "text/html; charset=utf-8", |
| 92 | + "size": 33722, |
| 93 | + "fileName": "1aa0e4f6-7cee-4c19-ac2e-cebee3c7b151" |
| 94 | + }, |
| 95 | + "camunda.document.type": "camunda" |
| 96 | +} |
| 97 | +``` |
| 98 | + |
| 99 | +:::note |
| 100 | +If a tool is executed more than once, select the desired tool invocation in **Instance History**, then open the corresponding inner instance to view the actual inputs and results. |
| 101 | +::: |
| 102 | + |
| 103 | +## Step 5: Analyze the agent context |
| 104 | + |
| 105 | +Within the AI Agent connector, you can examine the agent context. |
| 106 | +To view it: |
| 107 | + |
| 108 | +1. Select the **AI Agent** element in the execution tree. |
| 109 | +2. To better inspect the value, click the pencil icon to enter edit mode for the `agentContext` variable. |
| 110 | +3. Click the two-squares icon to open the JSON editor modal. With this, you can inspect the full payload of the variable value. |
| 111 | + |
| 112 | +In the JSON payload, you can find information about: |
| 113 | + |
| 114 | +- Defined tools. |
| 115 | +- The conversation, including your prompts and agent's replies. |
| 116 | +- Tool calls invoked by the agent. |
| 117 | +- Tool call inputs and results. |
| 118 | +- Additional metadata, such as reasoning traces and token usage. |
| 119 | + |
| 120 | +Here's the example conversation stored in the agent's context: |
| 121 | + |
| 122 | +```json |
| 123 | +"type": "in-process", |
| 124 | +"conversationId": "130bbad7-b316-4cac-9653-a77e37e11f8b", |
| 125 | +"messages": [ |
| 126 | + { |
| 127 | + "role": "system", |
| 128 | + "content": [ |
| 129 | + { |
| 130 | + "type": "text", |
| 131 | + "text": "You are a helpful, generic chat agent which can answer a wide amount of questions based on your knowledge and an optional set of available tools.\n\nIf tools are provided, you should prefer them instead of guessing an answer. You can call the same tool multiple times by providing different input values. Don't guess any tools which were not explicitely configured. If no tool matches the request, try to generate an answer. If you're not able to find a good answer, return with a message stating why you're not able to.\n\nIf you are prompted to interact with a person, never guess contact details, but use available user/person lookup tools instead and return with an error if you're not able to look up appropriate data.\n\nThinking, step by step, before you execute your tools, you think using the template `<thinking><context></context><reflection></reflection></thinking>`" |
| 132 | + } |
| 133 | + ] |
| 134 | + }, |
| 135 | + { |
| 136 | + "role": "user", |
| 137 | + "content": [ |
| 138 | + { |
| 139 | + "type": "text", |
| 140 | + "text": "Visit https://docs.camunda.io/ and tell me about it" |
| 141 | + } |
| 142 | + ], |
| 143 | + "metadata": { |
| 144 | + "timestamp": "2026-02-23T11:48:03.90923378Z" |
| 145 | + } |
| 146 | + }, |
| 147 | + { |
| 148 | + "role": "assistant", |
| 149 | + "content": [ |
| 150 | + { |
| 151 | + "type": "text", |
| 152 | + "text": "<thinking>\n<context>\nThe user wants me to visit a specific URL (https://docs.camunda.io/) and tell them about it. I have a tool called \"Fetch_URL\" that can fetch the contents of a given URL. This seems like a perfect match for the request.\n</context>\n<reflection>\nI should use the Fetch_URL tool to get the contents of https://docs.camunda.io/ and then analyze what I find to provide information about it to the user.\n</reflection>\n</thinking>" |
| 153 | + } |
| 154 | + ], |
| 155 | + "toolCalls": [ |
| 156 | + { |
| 157 | + "id": "tooluse_kBcf61pS50OS31KYQnRFzy", |
| 158 | + "name": "Fetch_URL", |
| 159 | + "arguments": { |
| 160 | + "url": "https://docs.camunda.io/" |
| 161 | + } |
| 162 | + } |
| 163 | + ], |
| 164 | +``` |
| 165 | + |
| 166 | +## Step 6: Understand how agent memory is stored |
| 167 | + |
| 168 | +In Modeler, within the AI Agent sub-process, you can define how the conversation memory is stored using the **Memory storage type** field. |
| 169 | + |
| 170 | +By default, agent memory uses the **In Process** type, which stores it as part of the agent context. |
| 171 | +With this option, you can view it in Operate within the agent context, as you did in the previous step, [Analyze the agent context](#step-5-analyze-the-agent-context). |
| 172 | + |
| 173 | +With the **Camunda Document Storage** option instead: |
| 174 | + |
| 175 | +- You can't view the full conversation and chain-of-thought traces in Operate. Operate only show a **document reference** and metadata. |
| 176 | +- Use this option for long conversations, where Operate variable limits might be exceeded. |
| 177 | + |
| 178 | +See [Memory](/components/connectors/out-of-the-box-connectors/agentic-ai-aiagent-subprocess.md#memory) for more details. |
| 179 | + |
| 180 | +:::note Agent memory storage |
| 181 | + |
| 182 | +- Use **In Process** for testing and debugging scenarios: Better visibility in Operate. |
| 183 | +- Use **Camunda Document Storage** for production scenarios: Better scalability and runtime behavior for long contexts. |
| 184 | + |
| 185 | +::: |
| 186 | + |
| 187 | +## Step 7: Examine process data in Optimize |
| 188 | + |
| 189 | +By leveraging data collected during process execution, you can use [Optimize](/components/optimize/what-is-optimize.md) to explore reports and process dashboards, and identify areas for improvement in your AI agent processes. |
| 190 | + |
| 191 | +1. Open Optimize. |
| 192 | +2. Go to the **Dashboards** tab. |
| 193 | +3. Select your AI agent process, **AI Agent Chat With Tools**. |
| 194 | +4. Verify that Optimize shows data for the recently executed process instance in the **Business Operations** section, including the running process model diagram and other statistics below. |
| 195 | +5. Explore other metrics provided in the **Business Reporting** and **Process Improvement** sections. |
| 196 | + |
| 197 | +See [Getting started](/components/optimize/improve-processes-with-optimize.md) for more details on what you can do with Optimize for business intelligence. |
| 198 | + |
| 199 | +## Step 8: Review the results |
| 200 | + |
| 201 | +Go back to Operate. In the **User Feedback** element, you will see the execution count in green. This means the process instance execution is stopped there and waiting for action. |
| 202 | + |
| 203 | +In this case, the required action is to provide feedback on the agent results. To do so: |
| 204 | + |
| 205 | +1. Select the **User Feedback** element. |
| 206 | +2. Open [Tasklist](/components/tasklist/introduction-to-tasklist.md). |
| 207 | +3. Select the user feedback task and assign to yourself by clicking **Assign to me**. |
| 208 | +4. Analyze the result. You will see an overview of the website URL requested in the prompt. |
| 209 | +5. You can follow up with more prompts to continue testing your AI agent. |
| 210 | +6. Select the **Are you satisfied with the result?** checkbox when you want to finish the process, then click **Complete task**. |
| 211 | +7. Go back to Operate. You will see the process instance is now completed, and the end event has been triggered. |
| 212 | + |
| 213 | +## Next steps |
| 214 | + |
| 215 | +Now that you know how to monitor your AI agents, you can: |
| 216 | + |
| 217 | +- Learn more about [Camunda agentic orchestration](/components/agentic-orchestration/agentic-orchestration-overview.md) and the [AI Agent connector](/components/connectors/out-of-the-box-connectors/agentic-ai-aiagent.md). |
| 218 | +- Explore other [AI agent model blueprints](https://marketplace.camunda.com/en-US/listing?q=ai&cat=107793&locale=en-US) from Camunda marketplace. |
0 commit comments