AzureAISearch: Problems with Setup and Citations

[justuswolff-audit]

Hi,
we moved from a file search tool, to an AzureAISearchTool and now face some issues. While the search is good, we wonder how the search result is returned to the agent. Let's look at the metadata of a FileSearch tool call in AIFoundry:

{
  "name": "file_search",
  "context": {
    "trace_id": "thread_vPnz5fB1vQiiSjs0PzDXU0T6",
    "span_id": "call_bzgwiiU0Vsdk2rzjTGMhCAqW",
    "thread_id": "thread_vPnz5fB1vQiiSjs0PzDXU0T6"
  },
  "kind": "Tool",
  "parent_id": "step_TXcQvWzIKIvTBJilRdFG0wZb",
  "start_time": "2025-08-01T08:17:14.000Z",
  "end_time": "2025-08-01T08:17:15.000Z",
  "status": {
    "status_code": "Ok",
    "description": "completed"
  },
  "attributes": {
    "span_type": "Tool",
    "id": "call_bzgwiiU0Vsdk2rzjTGMhCAqW",
    "type": "file_search",
    "file_search": {
      "ranking_options": {
        "ranker": "default_2024_08_21",
        "score_threshold": 0
      },
      "results": [
        {
          "file_id": "assistant-id",
          "file_name": "fileName - 1.pdf",
          "score": 0.01666666753590107,
          "attributes": {},
          "content": [
            {
              "type": "text",
              "text": "REDACTED"
            }
          ]
        },
        {
          "file_id": "assistant-id",
          "file_name": "fileName - 1.pdf",
          "score": 0.016393441706895828,
          "attributes": {},
          "content": [
            {
              "type": "text",
              "text": "REDACTED"
            }
          ]
        }
      ]
    },
    "output": [
      {
        "file_id": "assistant-id",
        "file_name": "fileName - 1.pdf",
        "score": 0.01666666753590107,
        "attributes": {},
        "content": [
          {
            "type": "text",
            "text": "REDACTED"
          }
        ]
      }
    ]
  }
}

This has a nice, structured response. However, if I compare that to the metadata of an AzureAISearch call:

{
  "name": "azure_ai_search",
  "context": {
    "trace_id": "thread_q6w2pvVFUpQxqnXL06RUWmfx",
    "span_id": "call_hZaZpiLVOo1jeIZ70g4p1rr4",
    "thread_id": "thread_q6w2pvVFUpQxqnXL06RUWmfx"
  },
  "kind": "Tool",
  "parent_id": "step_kkusDWUD469XL2K0dwAsyfrV",
  "start_time": "2025-08-01T08:19:00.000Z",
  "end_time": "2025-08-01T08:19:01.000Z",
  "status": {
    "status_code": "Ok",
    "description": "completed"
  },
  "attributes": {
    "span_type": "Tool",
    "id": "call_hZaZpiLVOo1jeIZ70g4p1rr4",
    "type": "azure_ai_search",
    "azure_ai_search": {
      "input": "search term",
      "output": {
        "summary": "Retrieved 2 documents.",
        "metadata": {
          "urls": ["doc_0", "doc_1"],
          "titles": ["doc_0", "doc_1"],
          "get_urls": [
            "REDACTED?api-version=2024-07-01&$select=ChunkId,Content,Source",
            "REDACTED?api-version=2024-07-01&$select=ChunkId,Content,Source"
          ],
          "ids": ["REDACTED", "REDACTED"],
          "command": "search",
          "query_type": "simple",
          "top_k": 5,
          "filter": true
        }
      }
    },
    "output": {
      "summary": "Retrieved 2 documents.",
      "metadata": {
        "urls": ["doc_0", "doc_1"],
        "titles": ["doc_0", "doc_1"],
        "get_urls": [
          "REDACTED?api-version=2024-07-01&$select=ChunkId,Content,Source",
          "REDACTED?api-version=2024-07-01&$select=ChunkId,Content,Source"
        ],
        "ids": ["REDACTED", "REDACTED"],
        "command": "search",
        "query_type": "simple",
        "top_k": 5,
        "filter": true
      }
    }
  }
}

it seems like there is no content returned, just urls.

Now I've got a couple of questions:

1. It seems like the search result content is not sent with the tool call. Does the agent service queries it from the search index itself (after the tool call)?
2. Where is the 'select=ChunkId,Content,Source' set? I couldn't find a parameter where I could set the fields to be selected.
3. How do I cite content returned from the AISearch? With the file tools I have the urls and file names in the response object, however with the AISearch I am left with:

"annotations": [
    {
      "type": "url_citation",
      "text": "【3:0†source】",
      "start_index": 635,
      "end_index": 647,
      "url_citation": { "url": "doc_0", "title": "doc_0" }
    },
    {
      "type": "url_citation",
      "text": "【3:1†source】",
      "start_index": 647,
      "end_index": 659,
      "url_citation": { "url": "doc_1", "title": "doc_1" }
    }
  ]

[hfritz34]

1. Content retrieval pattern: Yes, the azure_ai_search tool uses a two-phase approach. The initial tool call only returns metadata (URLs, IDs, titles) and the get_urls with API endpoints. The agent service then makes separate requests to retrieve the actual content using those get_urls endpoints behind the scenes. This is why you only see summary metadata in the tool output rather than the full content like with file_search.

2. Select parameter configuration: The $select=ChunkId,Content,Source appears to be hardcoded in the azure_ai_search tool implementation. Currently, there's no exposed parameter to customize which fields are selected from your search index. This is a limitation of the current tool design; you can't override the default field selection through the tool configuration.

3. Citation handling: The generic doc_0, doc_1 citations you're seeing suggest the azure_ai_search tool isn't properly mapping your search index metadata to meaningful citation information. For proper citations, you need:

• Ensure your search index has fields that map to meaningful document titles/names
• The tool should be using your index's title/source fields instead of generating generic doc_X labels
• You may need to configure your search index schema to include proper metadata fields that the tool can use for citations

Recommendation: This looks like the azure_ai_search tool is still missing some functionality compared to file_search. You might want to file feedback with Microsoft about exposing field selection parameters and improving citation metadata handling for better parity with the file_search experience.