LLM deployment issues (will be updated frequently)

[seokho-son]

Ollama 관련

• https://github.com/ollama/ollama
• 단일 서버에 다중 모델 병렬 실행 요청시, 상당한 지연 발생 (Ollama가 아직 다중 모델 병렬 구동을 잘 지원하지 않는 것으로 보임)
• CUDA 설치 필요 (GPU 병렬 활용 등)

curl -fsSL https://ollama.com/install.sh | sh
sudo sed -i '/\[Service\]/a Environment=\"OLLAMA_HOST=0.0.0.0:3000\"' /etc/systemd/system/ollama.service
sudo sed -i 's/User=ollama/User=root/' /etc/systemd/system/ollama.service

sudo sed -i 's/Group=ollama/Group=root/' /etc/systemd/system/ollama.service
sudo systemctl daemon-reload; sudo systemctl restart ollama
sudo systemctl status ollama --no-pager

Open WebUI 관련

• https://github.com/open-webui/open-webui

sudo docker run -d -p 80:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

• 환경변수 OLLAMA_BASE_URLS 활용할 필요. 리스트는 ; 세퍼레이트임.

  • https://docs.openwebui.com/getting-started/env-configuration/#ollama_base_url

• 다른 서버별 모델 다운로드시, 서버 지정에 오류 있음 (Ollama로 직접 지정하는 것이 좋을 듯)

• https://ollama.com/library

OLLAMA_HOST=0.0.0.0:3000 ollama list
OLLAMA_HOST=0.0.0.0:3000 ollama pull llama3

OLLAMA_HOST=0.0.0.0:3000 ollama pull llama3 &
OLLAMA_HOST=0.0.0.0:3000 ollama pull phi3 &
OLLAMA_HOST=0.0.0.0:3000 ollama pull mistral &
OLLAMA_HOST=0.0.0.0:3000 ollama pull gemma &
OLLAMA_HOST=0.0.0.0:3000 ollama pull mixtral &
OLLAMA_HOST=0.0.0.0:3000 ollama pull llava &
OLLAMA_HOST=0.0.0.0:3000 ollama pull yi &
OLLAMA_HOST=0.0.0.0:3000 ollama pull falcon2 &
OLLAMA_HOST=0.0.0.0:3000 ollama pull solar &
OLLAMA_HOST=0.0.0.0:3000 ollama pull llama2 &

• https://github.com/ollama/ollama/blob/main/docs/gpu.md

GPU 드라이버 설치 관련

• https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=22.04&target_type=deb_local
• CUDA 설치 및 Nvidia 드라이버 설치, 다양한 시도 필요
• CSP 이미지 활용하면 간편하지만, 이미지 및 스펙별 검토가 추가적으로 필요

sudo lspci | grep -i nvidia

sudo wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin
sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600
sudo wget https://developer.download.nvidia.com/compute/cuda/12.5.0/local_installers/cuda-repo-ubuntu2204-12-5-local_12.5.0-555.42.02-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2204-12-5-local_12.5.0-555.42.02-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2204-12-5-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update

sudo DEBIAN_FRONTEND=noninteractive apt-get -y install gcc
gcc --version
sudo DEBIAN_FRONTEND=noninteractive apt-get -y install cuda-12-5

echo 'export PATH=/usr/local/cuda-12.5/bin${PATH:+:${PATH}}' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-12.5/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}' >> ~/.bashrc
source ~/.bashrc

sudo reboot

nvcc --version
nvidia-smi