Plan for AutoGen Integration moving forward

[pgarz]

I know there this PR to add AutoGen capabilities to Semantic Kernel, but it already seems stale: https://github.com/microsoft/semantic-kernel/pull/3448/files#diff-78fb74f8da2858e1d262d86343b2c59e6f5991c778e002e44cd951b2c0942920

There's also this supporting article, but said code isn't part of the current package so it's not directly usable: https://devblogs.microsoft.com/semantic-kernel/autogen-agents-meet-semantic-kernel/

Using the function calling paradigm of Semantic Kernel with the the Agent orchestration of AutoGen sounds like something that I really want to do, but there's quite a lot of clunky work-around to try to get them to coexist.

What's the plan for AutoGen with respect to Semantic Kernel moving forward?

[joslat]

It has been mentioned a few times on the office-hours, there is one tomorrow and the previous ones are recorded in YouTube ;)
As far as I know,

• The Agents have to be decoupled from OpenAI, so they can run on any LLM/model.
• Once those abstractions are in there will come support for Autogen architecture
• And Autogen is redeveloped in .NET, I suppose will be adjusted to use Semantic Kernel once the first two points are in place

Makes sense, right? :)

[pgarz]

Seems like within AutoGen, Azure endpoints are usable https://github.com/microsoft/autogen/blob/main/notebook/oai_completion.ipynb

And so are local LLMs: https://microsoft.github.io/autogen/blog/2023/07/14/Local-LLMs

[guru98]

Hi, i'm a little bit confused about this topic. Right now there is an autogen implementation in development. This version have the possibility to use agents developed with semantic kernel. Also have a connector to extend type of messages handled by the semantic kernel agent. The semantic kernel agents support streaming. What i see in the first implementations here is something that not seems fully alligned like you want implement a autogen architecture inside semantic kernel potentially not fully compatible with autogen.net. Could you please explain better the strategy?

[madsbolaris]

This is a great question; we're currently working with the team that is building the .NET version of AutoGen. The goal is to eventually have that implementation use the abstractions within Semantic Kernel. This has not yet happened though because they are needing an abstraction around Function Calls first. We've captured that work here: #5153

@guru98, regarding your comment that the implementations aren't "fully aligned". We've been designing the current agent framework to include a superset of functionality in AutoGen. We're now actively working on rewriting all of the multi-agent samples in AutoGen using the Semantic Kernel abstractions to validate that we haven't lost any functionality.

It should be noted that the plans to reuse abstractions across AutoGen and Semantic Kernel is only in the .NET space. They will remain separate for Python.

[ColtonBehannon]

Hi @matthewbolanos , could you elaborate a bit on what you mean by "they will remain separate for Python"? My understanding from some of the office hours is that that Semantic Kernel will handle enterprise ready implementations of LLM orchestration, but that AutoGen is just for research. If that's the case, does this mean that Semantic Kernel (Python) will not be pursuing any multi-agent features? Or is the idea that the concepts accomplished by AutoGen be done through other means in Semantic Kernel i.e. instead of supporting a "group chat" of agents with tools like AutoGen; SK just use Plugins (and separate LLMs) as needed to accomplish the same feat?

[ymuichiro]

That expression also caught my attention. If possible, I would like to work on developing more advanced agents with the Python version of Semantic Kernel and AutoGen.

If we need to consider that they will be implemented separately, would it be best to create my own wrapper, as shown in the following diagram, like a microservice?

sequenceDiagram
    participant User 
    participant Autogen
    participant WAOAI as Wrapped OpenAI API Server
    participant BAPI as Bing API, etc. SK Plugin

    User->>Autogen: request
    Autogen->>WAOAI: as AssistantAgent
    note right of WAOAI: Semantic Kernel Agent with an API<br> server compatible with OpenAI API
    WAOAI->>BAPI: Function Calling
    BAPI->>WAOAI: context
    WAOAI->>Autogen: Considering the search results<br> obtained from the Bing API
    Autogen->>User: response

Loading

[ymuichiro]

Of course, if it's just about implementing web integration functions, it would be fine to use the features on the Autogen side. This flow is for when you want to use agents developed with the Semantic Kernel.

[guru98]

Thanks for the answer. I'm still studing the 2 implementation and do great test with your implementation (including group chat). I note that they manage more types of messages, also they have the concept of middleware that, if I understand well, is mapped to the channel in semantic framework agents, correct?

…

________________________________ Da: Matthew Bolaños ***@***.***> Inviato: martedì 16 aprile 2024 17:44 A: microsoft/semantic-kernel ***@***.***> Cc: Ugo Castellani ***@***.***>; Mention ***@***.***> Oggetto: Re: [microsoft/semantic-kernel] Plan for AutoGen Integration moving forward (Discussion #4795) This is a great question; we're currently working with the team that is building the .NET version of AutoGen. The goal is to eventually have that implementation use the abstractions within Semantic Kernel. This has not yet happened though because they are needing an abstraction around Function Calls first. We've captured that work here: #5153<#5153> @guru98<https://github.com/guru98>, regarding your comment that the implementations aren't "fully aligned". We've been designing the current agent framework to include a superset of functionality in AutoGen. We're now actively working on rewriting all of the multi-agent samples in AutoGen using the Semantic Kernel abstractions to validate that we haven't lost any functionality. — Reply to this email directly, view it on GitHub<#4795 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AA77TTUHJRPPZZ5IBRPRL4LY5VBO3AVCNFSM6AAAAABCSGEM56VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCMZRHEYTC>. You are receiving this because you were mentioned.Message ID: ***@***.***>