Skip to content

Latest commit

 

History

History
30 lines (20 loc) · 1.15 KB

serving_dinfer_openai.md

File metadata and controls

30 lines (20 loc) · 1.15 KB

Demo of dinfer openai serving

Serving

export SPECIAL_MODEL_DIR=/models/LLaDA2.0-mini--572899f-C8 # Download from https://www.modelscope.cn/models/inclusionAI/LLaDA2.0-mini
export TASK_DLLM_BATCH_SIZE=2
python3 serving_dinfer_openai.py

Client

date && curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer YOUR_API_KEY" -N -d '{"messages": [{"role": "user", "content": "你好， 我是小明"}], "stream": false}' http://0.0.0.0:48000/v1/chat/completions && date

date && curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer YOUR_API_KEY" -N -d '{"messages": [{"role": "user", "content": "你好， 我是小明"}], "stream": true}' http://0.0.0.0:48000/v1/chat/completions && date

Web demo

date && docker pull ghcr.io/open-webui/open-webui:main && date

mkdir data-open-webui
cd data-open-webui

date && docker run -d -p 60111:8080 -v $PWD:/app/backend/data --name open-webui ghcr.io/open-webui/open-webui:main && date

Config your open-webui with http://0.0.0.0:48000/v1 like this