How can I orchestrate multiple semantic skills with chain-of-thought planning(Azure AI Search as DataSource)

[KishoreKunalPraval]

I’m working on a project where I need to build a conversational sales assistant using Semantic Kernel. My goal is to integrate multiple semantic skills—each wrapping an Azure Cognitive Search index (for example, one for network service providers, another for capacity-related queries, and a third for property details).

The high-level workflow I’m aiming for is as follows:

Composite Prompt Construction:
I want to merge a predefined set of instructions, tool descriptions, and conversation history (e.g., from Cosmos DB), along with the current user query, into a single composite prompt.

Chain-of-Thought Planner:
I’d like to use the Azure OpenAI chat completion service to generate a chain-of-thought (CoT) reasoning on the composite prompt. The CoT should decide which semantic skills to invoke based on the query’s requirements.

Semantic Skills Execution:
Each semantic skill is implemented as a function (or plugin) for a specific search index (e.g., NSP, Capacity, Property) and registered into the Semantic Kernel instance so that they’re callable via SK function invocation.

Final Synthesis:
After executing the selected skills, I plan to aggregate their outputs and send the combined result back to the Azure OpenAI service for the final answer synthesis.

Question

What is the recommended way to structure and load these semantic skills into the kernel so that they are easily invoked based on the CoT reasoning?

How can I best design the chain-of-thought planner within Semantic Kernel to decide dynamically which skills to invoke?

Are there any best practices for aggregating outputs from multiple skills and synthesizing a coherent final answer via the Azure OpenAI chat completion service?

[qqq694637644]

This is my implementation of RAG using Semantic Kernel and samples. I let llm first retrieve the question, then plan, view the samples, examine the code implementation of Semantic Kernel, and finally answer the question. Is it very effective?

Based on the analysis of the Semantic Kernel architecture and functionality, here are the answers to your questions:

1. Recommended Method for Structuring Semantic Skills and Loading Them into the Kernel

In the context of Chain-of-Thought (CoT) reasoning, the following method is recommended for structuring semantic skills:

1. Domain-Oriented Plugin Design:

   • Create dedicated plugin classes for each search index (e.g., NSPPlugin, CapacityPlugin, PropertyPlugin).
   • Each plugin contains functions related to its domain, marked with the @kernel_function decorator.

2. Integrate Azure Cognitive Search:

   • Encapsulate access to specific Azure Cognitive Search indexes within each plugin.
   • Use AzureCognitiveSearchMemoryStore to interact with the Azure Cognitive Search service.
   • Ensure each plugin function provides clear descriptions and parameter annotations, which are crucial for the CoT planner.

3. Plugin Loading Example:

# Create and register your domain plugins
nsp_plugin = kernel.add_plugin(NSPPlugin(search_endpoint, admin_key, embedding_model), "NSP")
capacity_plugin = kernel.add_plugin(CapacityPlugin(search_endpoint, admin_key, embedding_model), "Capacity")
property_plugin = kernel.add_plugin(PropertyPlugin(search_endpoint, admin_key, embedding_model), "Property")

2. Best Approach to Designing the Chain-of-Thought Planner

The best approach to designing a Chain-of-Thought planner in Semantic Kernel to dynamically decide which skills to invoke:

1. Use FunctionCallingStepwisePlanner:

   • The built-in FunctionCallingStepwisePlanner in Semantic Kernel is well-suited for this scenario, as it can dynamically select functions to invoke based on user queries.
   • This planner uses Chain-of-Thought reasoning to determine the next steps.

2. Custom Planning Prompts:

   • Create specialized initial plan YAML configurations and step prompts to guide the planner in handling domain-specific sales consultation tasks.
   • Customize the planner's behavior using FunctionCallingStepwisePlannerOptions.

3. Implementation Example:

from semantic_kernel.planners import FunctionCallingStepwisePlanner, FunctionCallingStepwisePlannerOptions

# 1. Create custom planner configuration
planner_options = FunctionCallingStepwisePlannerOptions(
    get_initial_plan=lambda: custom_plan_yaml,
    get_step_prompt=lambda: custom_step_prompt,
    max_iterations=10,
    max_completion_tokens=2000
)

# 2. Instantiate the planner
stepwise_planner = FunctionCallingStepwisePlanner(
    service_id="chat-completion-service",
    options=planner_options
)

# 3. Use the planner to handle user queries
result = await stepwise_planner.invoke(
    kernel=kernel,
    question=composite_prompt,  # Composite prompt containing history and current query
    arguments=arguments
)

3. Best Practices for Aggregating Outputs from Multiple Skills

Best practices for aggregating outputs from multiple skills and synthesizing a coherent final answer using Azure OpenAI Chat Completion service:

1. Use Plan Object to Collect Results:

   • Use the Plan object generated by the planner to track the results of each skill execution.
   • Collect the outputs of each step and maintain the execution context.

2. Result Synthesis Strategy:

   • Create a specialized synthesis prompt template that describes how to integrate the outputs of multiple skills.
   • Use a separate kernel function call to pass all results to the large language model for generating the final answer.

3. Implementation Example:

# 1. Execute the planner to get the Chain-of-Thought result
planner_result = await stepwise_planner.invoke(kernel, user_query)

# 2. Extract information from the result (already includes all skill results)
chat_history = planner_result.chat_history

# 3. Build the final synthesis prompt
final_prompt = f"""
Based on the following skill execution results, generate a comprehensive and helpful response:

Conversation History:
{chat_history}

The final answer should:
1. Integrate all relevant information
2. Respond in the tone of a sales assistant
3. Provide specific, actionable advice
"""

# 4. Call the model for final synthesis
final_response = await chat_completion.complete(final_prompt)

Overall Architecture Recommendation

The following high-level architecture is recommended for implementing your conversational sales assistant:

1. Domain Plugin Layer: Create independent plugin classes, each handling a specific domain (NSP, Capacity, Property).
2. Chain-of-Thought Planning Layer: Use FunctionCallingStepwisePlanner to decide which plugin functions to invoke.
3. Synthesis Layer: Collect the results from the planner execution and generate a coherent response through a final synthesis step.
4. Prompt Management: Treat composite prompt construction, Chain-of-Thought prompts, and result synthesis prompts as configurable components.

This architecture provides a modular, scalable approach to handling complex sales advisor interactions while fully leveraging the planning and reasoning capabilities of Semantic Kernel.

[KishoreKunalPraval]

https://learn.microsoft.com/en-us/semantic-kernel/concepts/ai-services/chat-completion/function-calling/function-choice-behaviors?utm_source=chatgpt.com&pivots=programming-language-python

according to this we don;t need semantic planner anymore

[qqq694637644]

I'll help you set up a conversational sales assistant using Semantic Kernel with multiple semantic skills backed by Azure Cognitive Search. Let me explore the best approach for your architecture.

Search files...

Search files...

Search files...

Search files...
Based on my research, I'll provide you with a structured approach to implement your conversational sales assistant using Semantic Kernel, focusing on integrating multiple semantic skills backed by Azure Cognitive Search indices.

1. Setting Up Semantic Skills with Azure Cognitive Search

Each semantic skill will be implemented as a plugin that wraps an Azure Cognitive Search index. Here's a recommended structure:

import asyncio
from semantic_kernel import Kernel
from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion, AzureTextEmbedding
from semantic_kernel.connectors.memory.azure_cognitive_search import AzureCognitiveSearchMemoryStore
from semantic_kernel.memory import SemanticTextMemory
from semantic_kernel.core_plugins import TextMemoryPlugin

# Initialize Kernel with Azure OpenAI services
kernel = Kernel()
kernel.add_service(
    AzureChatCompletion(
        service_id="chat",
        deployment_name="your-gpt-deployment",
        endpoint="your-azure-endpoint",
        api_key="your-azure-api-key"
    )
)
kernel.add_service(
    AzureTextEmbedding(
        service_id="embedding",
        deployment_name="your-embedding-deployment",
        endpoint="your-azure-endpoint",
        api_key="your-azure-api-key"
    )
)

# Create memory stores for each domain (NSP, Capacity, Property)
async def setup_memory_stores():
    # Network Service Provider memory
    nsp_memory_store = await AzureCognitiveSearchMemoryStore(
        vector_size=1536,  # Size depends on your embedding model
        search_endpoint="your-search-endpoint",
        admin_key="your-search-api-key"
    ).acreate()
    nsp_memory = SemanticTextMemory(storage=nsp_memory_store)
    
    # Capacity memory
    capacity_memory_store = await AzureCognitiveSearchMemoryStore(
        vector_size=1536,
        search_endpoint="your-search-endpoint",
        admin_key="your-search-api-key"
    ).acreate()
    capacity_memory = SemanticTextMemory(storage=capacity_memory_store)
    
    # Property memory
    property_memory_store = await AzureCognitiveSearchMemoryStore(
        vector_size=1536,
        search_endpoint="your-search-endpoint",
        admin_key="your-search-api-key"
    ).acreate()
    property_memory = SemanticTextMemory(storage=property_memory_store)
    
    return nsp_memory, capacity_memory, property_memory

2. Implementing Domain-Specific Skills

Create each semantic skill as a plugin that wraps a specific search index:

class NSPSkill:
    def __init__(self, memory: SemanticTextMemory, collection_name: str = "nsp"):
        self.memory = memory
        self.collection_name = collection_name
    
    @kernel_function(
        description="Searches for information about network service providers",
        name="search_nsp"
    )
    async def search_nsp(self, query: str, relevance: float = 0.7, limit: int = 5) -> str:
        results = await self.memory.search(
            collection=self.collection_name,
            query=query,
            limit=limit,
            min_relevance_score=relevance
        )
        
        # Format results into a coherent response
        if not results:
            return "No information found about network service providers for this query."
        
        response = "Here's information about network service providers:\n\n"
        for match, score in results:
            response += f"• {match.text} (relevance: {score:.2f})\n"
        
        return response

# Similarly create CapacitySkill and PropertySkill classes

3. Composite Prompt Construction

Create a function to build the composite prompt from instructions, tool descriptions, and conversation history:

async def build_composite_prompt(kernel: Kernel, user_query: str, conversation_history: list):
    # System instructions that describe the assistant's role
    system_instructions = """
    You are a sales assistant specialized in telecommunications services. 
    You have access to several tools that can search for information:
    - NSP tool: Searches for network service provider information
    - Capacity tool: Searches for capacity and bandwidth information
    - Property tool: Searches for property details and locations
    
    Analyze each user query carefully. Consider which tools are needed to provide a complete response.
    Always use a step-by-step approach to determine the most relevant tools to use based on the query content.
    """
    
    # Format conversation history
    formatted_history = ""
    for message in conversation_history:
        role = "User" if message["role"] == "user" else "Assistant"
        formatted_history += f"{role}: {message['content']}\n"
    
    # Create the composite prompt
    composite_prompt = f"""
    {system_instructions}
    
    ### Conversation History:
    {formatted_history}
    
    ### Current User Query:
    {user_query}
    
    ### Think step by step:
    1. What information is the user looking for?
    2. Which tools would be most relevant to answer this query?
    3. What specific aspects of the query should I search for with each tool?
    """
    
    return composite_prompt

4. Chain-of-Thought Planner

Implement the CoT planner that determines which skills to invoke:

from semantic_kernel.planners.function_calling_stepwise_planner import FunctionCallingStepwisePlanner, FunctionCallingStepwisePlannerOptions

async def create_cot_planner(kernel: Kernel):
    # Configure options for the planner
    options = FunctionCallingStepwisePlannerOptions()
    options.max_iterations = 15
    options.max_completion_tokens = 2000
    
    # Create the planner
    planner = FunctionCallingStepwisePlanner(service_id="chat", options=options)
    
    return planner

async def execute_cot_plan(kernel: Kernel, planner, user_query: str, conversation_history: list):
    # Register all semantic skills to the kernel
    nsp_memory, capacity_memory, property_memory = await setup_memory_stores()
    
    # Register the semantic skills as plugins
    kernel.add_plugin(NSPSkill(nsp_memory), "NSPSkill")
    kernel.add_plugin(CapacitySkill(capacity_memory), "CapacitySkill")
    kernel.add_plugin(PropertySkill(property_memory), "PropertySkill")
    
    # Create arguments with conversation history
    arguments = {"history": conversation_history, "input": user_query}
    
    # Execute the plan
    plan_result = await planner.invoke(kernel=kernel, question=user_query, arguments=arguments)
    
    return plan_result

5. Final Synthesis

Implement the final answer synthesis:

async def synthesize_final_answer(kernel: Kernel, plan_result, user_query: str):
    # Extract relevant information from plan execution
    skill_outputs = plan_result.skill_outputs
    
    # Prepare a prompt for final synthesis
    synthesis_prompt = f"""
    The user asked: "{user_query}"
    
    Based on the information retrieved from various sources:
    
    {skill_outputs}
    
    Please synthesize a comprehensive, coherent answer that addresses the user's query.
    The answer should be conversational, helpful, and concise.
    """
    
    # Use the chat service to generate the final response
    chat_service = kernel.get_service("chat")
    
    chat_history = ChatHistory()
    chat_history.add_system_message("You are a telecommunications sales assistant. Synthesize a comprehensive answer based on the information provided.")
    chat_history.add_user_message(synthesis_prompt)
    
    response = await chat_service.complete_chat(chat_history)
    
    return response.content

6. Complete Workflow Integration

Now, put everything together:

async def handle_user_query(kernel: Kernel, user_query: str, conversation_history: list):
    # 1. Create the CoT planner
    planner = await create_cot_planner(kernel)
    
    # 2. Execute the plan to determine and call relevant skills
    plan_result = await execute_cot_plan(kernel, planner, user_query, conversation_history)
    
    # 3. Synthesize the final answer from all skill outputs
    final_answer = await synthesize_final_answer(kernel, plan_result, user_query)
    
    # 4. Update conversation history
    conversation_history.append({"role": "user", "content": user_query})
    conversation_history.append({"role": "assistant", "content": final_answer})
    
    return final_answer, conversation_history

Best Practices

1. Structuring Skills: Each semantic skill should have a clear domain focus and well-defined function descriptions that help the planner understand when to use them.

2. Chain-of-Thought Planning: Use the FunctionCallingStepwisePlanner which provides a structured approach for planning. Include detailed reasoning in your prompts to guide the model in selecting the right skills.

3. Dynamic Function Selection: The function calling capabilities in Azure OpenAI services allow for more dynamic selection of which functions to call, which is perfect for your use case.

4. Result Aggregation: Structure the output from each skill in a consistent format to make it easier to aggregate results.

5. Memory Management: Use Azure Cosmos DB to store conversation history, which can be integrated into the composite prompt for context.

6. Error Handling: Implement robust error handling to account for cases where search indices might not return results.

7. Prompt Engineering: Craft detailed prompts that explicitly instruct the model to reason through which skills to invoke based on the query content.

This architecture leverages Semantic Kernel's ability to seamlessly integrate Azure Cognitive Search indices as searchable knowledge bases and provides a structured approach to dynamically select and combine information from multiple sources based on the user's query.

[sophialagerkranspandey]

Tagging @alliscode