LLMs

[cboettig]

• Host LLMs with sleep and wake endpoint. Support Multiple Models vllm-project/vllm#299 (comment)
• Move LLM hosting to k8s-based service deploy
• Move shiny apps to k8s-based deploy.

[cboettig]

• add olmo with rope for longer context. (e.g. see vllm instructions on qwen https://huggingface.co/Qwen/Qwen3-235B-A22B-FP8#switching-between-thinking-and-non-thinking-mode
• add a qwen3 model

[cboettig]

For sleeping:

curl --data "level=1" https://llm.cirrus.carlboettiger.info/sleep -H "Authorization: Bearer $TOKEN"
curl -X GET "https://llm.cirrus.carlboettiger.info/is_sleeping"

curl --data "level=1" https://llm.cirrus.carlboettiger.info/wake_up -H "Authorization: Bearer $TOKEN"

Requires running with:

--env "VLLM_SERVER_DEV_MODE=1" as env var and `--enable-sleep-mode` as vllm parameter

[cboettig]

potential hosted models target list:

• olmo w/ rope
• qwen3 w/ reasoning
• https://huggingface.co/PleIAs/Pleias-RAG-1B (small model for RAG)
• https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B for tool use