Skip to content

ollama.md

Latest commit

 

History

History
248 lines (179 loc) · 7.04 KB

ollama.md

File metadata and controls

248 lines (179 loc) · 7.04 KB

OpenChat Playground with Ollama

This page describes how to run OpenChat Playground (OCP) with Ollama integration.

Get the repository root

  1. Get the repository root.

    # bash/zsh
REPOSITORY_ROOT=$(git rev-parse --show-toplevel)
    # PowerShell
$REPOSITORY_ROOT = git rev-parse --show-toplevel

Run on local machine

  1. Make sure Ollama is installed and running on your local machine. If not, install Ollama from ollama.com and start the service.

    ollama serve

  2. Pull the model you want to use. The default model OCP uses is llama3.2.

    ollama pull llama3.2

    Alternatively, if you want to run with a different model, say qwen3 other than the default one, download it first by running the following command.

    ollama pull qwen3

  3. Make sure you are at the repository root.

    cd $REPOSITORY_ROOT

  4. Run the app.

    # bash/zsh
dotnet run --project $REPOSITORY_ROOT/src/OpenChat.PlaygroundApp -- \
    --connector-type Ollama
    # PowerShell
dotnet run --project $REPOSITORY_ROOT\src\OpenChat.PlaygroundApp -- `
    --connector-type Ollama

    Alternatively, if you want to run with a different model, say qwen3, make sure you've already downloaded the model by running the ollama pull qwen3 command.

    # bash/zsh
dotnet run --project $REPOSITORY_ROOT/src/OpenChat.PlaygroundApp -- \
    --connector-type Ollama \
    --model qwen3
    # PowerShell
dotnet run --project $REPOSITORY_ROOT\src\OpenChat.PlaygroundApp -- `
    --connector-type Ollama `
    --model qwen3

  5. Open your web browser, navigate to http://localhost:5280, and enter prompts.

Run on local container

  1. Make sure the Ollama server is up and running.

    ollama serve

  2. Download the Ollama model. The default model OCP uses is llama3.2.

    ollama pull llama3.2

    Alternatively, if you want to run with a different model, say qwen3, other than the default one, download it first by running the following command.

    ollama pull qwen3

  3. Make sure you are at the repository root.

    cd $REPOSITORY_ROOT

  4. Build a container.

    docker build -f Dockerfile -t openchat-playground:latest .

  5. Run the app.

    # bash/zsh - from locally built container
docker run -i --rm -p 8080:8080 openchat-playground:latest \
  --connector-type Ollama \
  --base-url http://host.docker.internal:11434 \
    # PowerShell - from locally built container
docker run -i --rm -p 8080:8080 openchat-playground:latest `
    --connector-type Ollama `
    --base-url http://host.docker.internal:11434
    # bash/zsh - from GitHub Container Registry
docker run -i --rm -p 8080:8080 ghcr.io/aliencube/open-chat-playground/openchat-playground:latest \
    --connector-type Ollama \
    --base-url http://host.docker.internal:11434
    # PowerShell - from GitHub Container Registry
docker run -i --rm -p 8080:8080 ghcr.io/aliencube/open-chat-playground/openchat-playground:latest `
    --connector-type Ollama `
    --base-url http://host.docker.internal:11434

    Alternatively, if you want to run with a different model, say qwen3, make sure you've already downloaded the model by running the ollama pull qwen3 command.

    ollama pull qwen3
    # bash/zsh - from locally built container
docker run -i --rm -p 8080:8080 openchat-playground:latest \
    --connector-type Ollama \
    --base-url http://host.docker.internal:11434 \
    --model qwen3
    # PowerShell - from locally built container
docker run -i --rm -p 8080:8080 openchat-playground:latest `
    --connector-type Ollama `
    --base-url http://host.docker.internal:11434 `
    --model qwen3
    # bash/zsh - from GitHub Container Registry
docker run -i --rm -p 8080:8080 ghcr.io/aliencube/open-chat-playground/openchat-playground:latest \
    --connector-type Ollama \
    --base-url http://host.docker.internal:11434 \
    --model qwen3
    # PowerShell - from GitHub Container Registry
docker run -i --rm -p 8080:8080 ghcr.io/aliencube/open-chat-playground/openchat-playground:latest `
    --connector-type Ollama `
    --base-url http://host.docker.internal:11434 `
    --model qwen3

    NOTE: Use host.docker.internal:11434 to connect to Ollama running on the host machine from inside the container.

  6. Open your web browser, navigate to http://localhost:8080, and enter prompts.

Run on Azure

  1. Make sure you are at the repository root.

    cd $REPOSITORY_ROOT

  2. Login to Azure.

    # Login to Azure Dev CLI
azd auth login

  3. Check login status.

    # Azure Dev CLI
azd auth login --check-status

  4. Initialize azd template.

    azd init

    NOTE: You will be asked to provide environment name for provisioning.

  5. Set the connector type to Ollama.

    azd env set CONNECTOR_TYPE "Ollama"

    The default model OCP uses is llama3.2. If you want to run with a different model, say qwen3 other than the default one, add it to azd environment variables.

    azd env set OLLAMA_MODEL "qwen3"

  6. As a default, the app uses a Serverless GPU with NVIDIA T4 (NC8as-T4). If you want to use NVIDIA A100, set the GPU profile.

    azd env set GPU_PROFILE_NAME "NC24-A100"

    If you want to know more about Serverless GPU, visit Using serverless GPUs in Azure Container Apps.

  7. Run the following commands in order to provision and deploy the app.

    azd up

    NOTE: You will be asked to provide Azure subscription and location for deployment. IMPORTANT: Due to the limitation for GPU support, the available regions are limited to Australia East, Sweden Central and West US 3. For more details, visit Using serverless GPUs in Azure Container Apps.

    Once deployed, you will be able to see the deployed OCP app URL.

  8. Open your web browser, navigate to the OCP app URL, and enter prompts.

  9. Clean up all the resources.

    azd down --force --purge