ag2 with local openai provider, such as Ollama or vLLM?

[ntsarb]

Hello, is it possible to use ag2 with a local LLM provider that implements the OpenAI API, such as Ollama or vLLM?

[pdaxt]

Yes, AG2 works great with local OpenAI-compatible providers. Here's how to set it up:

vLLM Setup

from autogen import ConversableAgent

config_list = [{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "base_url": "http://localhost:8000/v1",
    "api_key": "not-needed"  # vLLM doesn't require a key
}]

agent = ConversableAgent(
    name="local_agent",
    llm_config={"config_list": config_list}
)

Ollama Setup

config_list = [{
    "model": "llama3.1",
    "base_url": "http://localhost:11434/v1",
    "api_key": "ollama"
}]

Production Tips

1. vLLM is faster for multi-agent workloads due to continuous batching
2. Set max_tokens explicitly - some local models default to very short responses
3. For tool use, ensure your model supports function calling (Llama 3.1+, Mistral v0.3+)
4. Memory management - Run vLLM with --max-model-len set appropriately for your GPU

vllm serve meta-llama/Llama-3.1-8B-Instruct \\
    --max-model-len 8192 \\
    --gpu-memory-utilization 0.9

I've been running AG2 + vLLM for agent workflows and it works well. The main gotcha is ensuring your model handles system prompts correctly - some quantized models struggle with AG2's structured prompts.