Using Locally deployed NVIDIA NIMs as the model for CrewAI agents

[ghn010590]

What is the key name that needs to be used for specifying the URL of the locally hosted NIM both LLM (for the agent) and embedding model(while using WebsiteSearchTool)? The Litellm used under the hood by default connects to https://integrate.api.nvidia.com/v1. How do i override it to use the local endpoint?

[Ribin-Baby]

1. Deploy the NIM Container

Use the following command to deploy the NIM model using Docker:

docker run -d --rm \
  --gpus all \
  --shm-size=16GB \
  -e NGC_API_KEY="$NGC_API_KEY" \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -u $(id -u) \
  -p 8000:8000 \
  nvcr.io/nim/meta/llama-3.1-8b-instruct:latest

---

2. Test the API Endpoint

Once the container is running, you can test the model with a simple curl request:

curl -X 'POST' 'http://0.0.0.0:8000/v1/chat/completions' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "meta/llama-3.1-8b-instruct",
    "messages": [{"role": "user", "content": "hi greet the user."}],
    "max_tokens": 64
}'

---

3. Run Inference from Python

You can call the model programmatically using Python like this:

from crewai import LLM

llm = LLM(
    model="openai/meta/llama-3.1-8b-instruct",
    base_url="http://localhost:8000/v1",
    api_key="localnimnoapikey"  # Any non-empty value works
)

response = llm.call(messages=[
    {"role": "user", "content": "What's a good name for a dog?"}
])

print('\n')
print(response)

Note: There is no fixed api_key required. Just make sure it’s not empty.

---

4. Set Environment Variables

Before running your application, make sure the following environment variables are set:

import os

os.environ["OPENAI_API_KEY"]   = "localnimnoapikey"
os.environ["OPENAI_API_BASE"]  = "http://localhost:8000/v1"  
os.environ["OPENAI_MODEL_NAME"] = "openai/meta/llama-3.1-8b-instruct"

The model name must start with openai/ in order to use the local NIM.

---

5. Use the LLM Instance in Your CrewAI Agents

Make sure to reuse the llm instance initialized above in your agent definitions:

from crewai import Agent, Task, Crew

code_explorer = Agent(
  config=agents_config['code_explorer'],
  llm=llm,
  system_template=system_template,
  prompt_template=prompt_template,
  response_template=response_template,
  tools=[DirectoryReadTool(), FileReadTool()]
)

documentation_planner = Agent(
  config=agents_config['documentation_planner'],
  llm=llm,
  system_template=system_template,
  prompt_template=prompt_template,
  response_template=response_template,
  tools=[DirectoryReadTool(), FileReadTool()]
)

analyze_codebase = Task(
  config=tasks_config['analyze_codebase'],
  agent=code_explorer
)

create_documentation_plan = Task(
  config=tasks_config['create_documentation_plan'],
  agent=documentation_planner,
  output_pydantic=DocPlan
)

planning_crew = Crew(
  agents=[code_explorer, documentation_planner],
  tasks=[analyze_codebase, create_documentation_plan],
  verbose=False
)

---

By following the steps above, you’ll be able to fully integrate a local NIM model with your CrewAI agents.