Using 32*A800 but with low performance for deepseekr1

[HoyTiger]

I deployed the bf16 DeepSeek-R1 model using 32 A800 GPUs with the following command:

However, the token usage seems consistently low (around 0.02) with a throughput of approximately 18 tokens/s. The GPU utilization remains at 100%, which suggests not just partial parameter activation per token.

Could you advise if there are specific hyperparameters I should configure to address this issue?

[HoyTiger]

note that The throughput per individual request is 6-7 tokens/s.

[zhyncs]

I remember that @ispobock got around 30 tokens/s in A100.

[HoyTiger]

Is this the result of a single request or a concurrent request? If possible, can you provide the result of a single request？