Do I need to deploy an LLM as an accessory (to build a a simple chatbot)?

[charnould]

Hi,
I'm currently playing with LLM and I try to selfhost a small chatbot using Kamal!
To keep things super simple, I'd like:

• A minimal JS server (let's say Express) = my chatbot via a simple HTML form...
• ...which interacts with another "server" hosting/using ollama.ai Docker image for my LLM which makes available a REST API like:

curl http://localhost:11434/api/generate -d '{
  "model": "llama2",
  "prompt":"Why is the sky blue?"
}'

I'm currently struggling to understand how to set this up with Kamal.

I know how to deploy the JS server, but what about the ML stuff?

• Is it an accessory? But how do I update it if - for example - I want to change my LLM model?
• Or can I/do I need to build a "big" Docker image containing both the JS and ML parts?

Could you please give me some guidance? Thanks.

[charnould]

Just create an accessory (and boot it):

# in deploy.yml
accessories:
  ollama:
    service: ollama
    image: ollama/ollama
    host: your-ip

Then in app code:

export const my_function = async () => {
  await fetch('http://ollama:11434/api/pull', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ model: '...' })
  })
}