This repo has branches with different docker compose set-ups that I use to locally run AI and try things.
- NVIDIA Graphics card: I have set this up to leverage NVIDIA CUDA.
- Debian based OS: The supporting scripts are written for Debian.
- Docker repo added to APT
- Clone this repo
- Check out the branch you want to use
- Run
./set-up.sh - Run
docker compose up -d
This branch runs only Ollama. Run docker exec -t ollama ollama run gemma3:1b to start a very simple model and chat with it.
Run nvtop to monitor your GPU and evaluate whether the model is properly running on the GPU.
Run docker exec -t ollama ollama ps to see how which models are running, how much memory they use, and how this is distributed across CPU and GPU.