Using smolagents with Local LLMs via LM Studio

This repository contains examples of how to use smolagents with local LLMs running on LM Studio.

Summary

This repository demonstrates how to use the smolagents library with local LLMs running on LM Studio. It includes:

1. A basic example of connecting to LM Studio and creating a simple agent
2. A coding assistant that can help with Python programming tasks
3. A research assistant that can search for information and summarize text
4. A test script to verify the connection to LM Studio

The examples show how to:

• Connect to LM Studio's OpenAI-compatible API
• Create tools using the @tool decorator
• Use the ToolCallingAgent to create agents that can use tools
• Handle conversation state in a conversational interface

Project Structure

smolagents-lmstudio-examples/
├── README.md                     # Project documentation
├── requirements.txt              # Python dependencies
├── setup.py                      # Package setup script
├── install.sh                    # Unix/Mac installation script
├── install.bat                   # Windows installation script
├── .gitignore                    # Git ignore file
├── smolagents_rules.mdc          # Cursor rules for smolagents
├── test_lm_studio_connection.py  # Script to test LM Studio connection
├── local_llm_example.py          # Basic example of using a local LLM
├── coding_assistant_agent.py     # Example of a coding assistant agent
└── research_assistant_agent.py   # Example of a research assistant agent

Prerequisites

• Python 3.8+
• smolagents with OpenAI support (pip install 'smolagents[openai]')
• requests (pip install requests)
• LM Studio installed on your computer

Installation

Using Installation Scripts

On macOS/Linux:

# Make the script executable
chmod +x install.sh

# Run the installation script
./install.sh

On Windows:

# Run the installation script
install.bat

Manual Installation

1. Create a virtual environment (optional but recommended):

   python -m venv venv
   
   # On macOS/Linux
   source venv/bin/activate
   
   # On Windows
   venv\Scripts\activate

2. Install dependencies:

   pip install -r requirements.txt

Using setup.py

pip install -e .
# Or with OpenAI support
pip install -e ".[openai]"

Setup Instructions

1. Install LM Studio:

   • Download and install LM Studio from lmstudio.ai
   • Launch LM Studio and download a model of your choice

2. Start the Local Server in LM Studio:

   • Load a model in LM Studio
   • Click on the "Local Server" tab
   • Click "Start Server"
   • The server should be running at http://localhost:1234

3. Test the Connection:

   • Run the test script to verify that you can connect to LM Studio:

     python test_lm_studio_connection.py
     

   • If successful, you'll see a confirmation message
   • If not, the script will provide troubleshooting instructions

4. Run the Example Scripts:

   • Make sure the LM Studio server is running
   • Run one of the example scripts:

     python local_llm_example.py
     

     or

     python coding_assistant_agent.py
     

     or

     python research_assistant_agent.py
     

Example Files

• test_lm_studio_connection.py: A script to test if the connection to LM Studio is working properly
• local_llm_example.py: A simple example showing how to connect to LM Studio and create a basic agent
• coding_assistant_agent.py: A more advanced example demonstrating how to create a coding assistant with tools
• research_assistant_agent.py: An example of a research assistant that can search for information and summarize text

Configuration

The key configuration for connecting to LM Studio is:

from smolagents import OpenAIServerModel

model = OpenAIServerModel(
    model_id="local-model",  # This can be any name, LM Studio will use whatever model you have loaded
    api_base="http://localhost:1234/v1",  # Default LM Studio API endpoint
    api_key="not-needed",  # LM Studio doesn't require an API key by default
)

You can adjust the api_base URL if your LM Studio server is running on a different port or host.

Using Tools with Local LLMs

The examples demonstrate how to create and use tools with your local LLMs:

1. Import the necessary classes:

   from smolagents import OpenAIServerModel, ToolCallingAgent, tool

2. Define a tool function with the @tool decorator:

   @tool
   def search_web(query: str) -> str:
       """
       Search the web for information.
       
       Args:
           query: The search query
           
       Returns:
           str: Search results
       """
       # Tool implementation
       return result

3. Add the tool to your agent:

   agent = ToolCallingAgent(
       name="AssistantName",
       model=model,
       tools=[search_web],
   )

4. Run the agent:

   response = agent.run("Your query here")
   print(response)

Handling Conversation State

ToolCallingAgent doesn't maintain conversation state between runs. To create a conversational interface, you need to:

1. Store conversation history manually:

   conversation_history = []

2. Create a new agent for each interaction:

   agent = ToolCallingAgent(
       name="AssistantName",
       model=model,
       tools=[tools_list],
   )

3. Include conversation history in the prompt:

   # Add user input to history
   conversation_history.append(f"User: {user_input}")
   
   # Create prompt with history
   full_prompt = "\n".join(conversation_history) + "\n\nPlease respond to the latest message."
   
   # Run the agent
   response = agent.run(full_prompt)
   
   # Add response to history
   conversation_history.append(f"Assistant: {response}")

Available Agent Types

smolagents provides several types of agents:

• ToolCallingAgent: A general-purpose agent that can use tools via function calling
• CodeAgent: An agent that writes and executes Python code to solve tasks
• MultiStepAgent: A base class for agents that solve tasks in multiple steps

For most simple use cases with local LLMs, ToolCallingAgent is recommended.

Notes on Model Performance

• The quality of responses depends on the model you have loaded in LM Studio
• Larger models (7B+ parameters) will generally provide better results
• Some models may not support all features (like function calling for tools)
• If you encounter issues with tools, try using a model that supports OpenAI function calling format
• For best results with tools, use models that have been fine-tuned for function calling

Recommended Models for LM Studio

For the best experience with these examples, consider using these models in LM Studio:

• Gemma 3 12B Instruct (gemma-3-12b-it)
• Gemma 3 27B Instruct (gemma-3-27b-it)

The Gemma 3 models are highly recommended as they provide excellent function calling capabilities and overall performance. The 27B model is comparable to Gemini 1.5 Pro in quality, while the 12B model offers a good balance between performance and resource requirements.

Troubleshooting

• Connection Error: Make sure LM Studio server is running and the port matches your configuration
• Poor Responses: Try loading a different/larger model in LM Studio
• Tool Execution Issues: Some local models may not support function calling properly. Try a different model or simplify your prompts.
• API Errors: If you get API errors, check the LM Studio logs for more information
• Import Errors: Make sure you're using the correct import paths:
  • from smolagents import OpenAIServerModel, ToolCallingAgent, tool
  • There is no Agent class in smolagents, use ToolCallingAgent or CodeAgent instead
• Method Errors: ToolCallingAgent uses run() method, not chat():
  • Use agent.run("Your query") instead of agent.chat("Your query")
• Parameter Errors: ToolCallingAgent doesn't accept a system_prompt parameter. The required parameters are:
  • name: A name for the agent
  • model: The model to use
  • tools: A list of tools (can be empty [])
• ModuleNotFoundError: If you get an error about missing 'openai', install the OpenAI dependency:
  • pip install 'smolagents[openai]'
• Tool Creation Errors: Use the @tool decorator pattern to create tools:

  @tool
  def my_tool(param: str) -> str:
      """Tool description with Args: section"""
      return result