GLM4

Glm-4

1. Supported Models

• glm-4-9B

2. Setup

2.1 Start Docker Container

Refer to the README.md

2.2 Clone Model

# 1. create directory
mkdir -p /data/mtt/models /data/mtt/models_convert

# 2. clone model
cd /data/mtt/models
git lfs install
git clone xxxxx    # Please download glm-4-9b-chat model by yourself!

If you encounter any issues while downloading the model, please refer to the README.

2.3 Weight Conversion

python -m mttransformer.convert_weight \
	--in_file /data/mtt/models/glm-4-9b-chat/ \
	--saved_dir /data/mtt/models_convert/glm-4-9b-chat-fp16-tp1-convert/ \
	--tensor-para-size 1
    --model_type chatglm2

3. Inference

• start server

python -m vllm.entrypoints.openai.api_server \
    --model /data/mtt/models_convert/glm-4-9b-chat-fp16-tp1-convert/ \
    --trust-remote-code \
    --tensor-parallel-size 1 \
    -pp 1 \
    --block-size 64 \
    --max-model-len 4096 \
    --disable-log-stats \
    --disable-log-requests \
    --gpu-memory-utilization 0.95 \
    --device "musa"

• send message

curl http://0.0.0.0:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
        "model": "/data/mtt/models_convert/glm-4-9b-chat-fp16-tp1-convert/",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Who won the world series in 2020?"}
        ]
}'