[Bug]: ucm0.5.0rc1同时5并发时报错

[ppzze]

Your current environment

The output of `python collect_env.py`

ucm0.5.0rc1 多并发时报错

🐛 Describe the bug

[qyh111]

Can you provide more infomation like platform,vllm version, ucm_config?

[qyh111]

And which storage backend did you use, local disk? remote mount point like nfs over rdma?

[ppzze]

Can you provide more infomation like platform,vllm version, ucm_config?

plantform:
Ascend 910B4 2node

version:
ucm0.5.0rc1 \
vllm-ascend 0.17.0rc1
GLM4.7

ucm_config:
Ucm_connector_name:"UcmPipelineStore"
ucm_connector_config:
store_pipeline: "Cache|Posix"storage_backends:"/mnt/test"
io_direct: false

storage backend: nfs

[qyh111]

This situation is most likely caused by insufficient bandwidth. Have you tried measuring the bandwidth of the mount point?

[ppzze]

This situation is most likely caused by insufficient bandwidth. Have you tried measuring the bandwidth of the mount point?

yes， i try and it seems like the error of Prometheus.
and now i have another question：
that is the output is “request_id: chatcmpl-84ac254c7dbad905-a9b7c011, total_blocks_num: 47, hit hbm: 0, hit external: 0”
I’ve been trying all night. The HBM hit has data, but the external hit is always 0. I found that the mounted external storage isn’t being written to at all. Under what circumstances will /mnt/test write to the external storage?

[qyh111]

This situation is most likely caused by insufficient bandwidth. Have you tried measuring the bandwidth of the mount point?

yes， i try and it seems like the error of Prometheus. and now i have another question： that is the output is “request_id: chatcmpl-84ac254c7dbad905-a9b7c011, total_blocks_num: 47, hit hbm: 0, hit external: 0” I’ve been trying all night. The HBM hit has data, but the external hit is always 0. I found that the mounted external storage isn’t being written to at all. Under what circumstances will /mnt/test write to the external storage?

Do you mean there's no dirs/files under /mnt/test ? Does /mnt/test is your mount point?

[ppzze]

This situation is most likely caused by insufficient bandwidth. Have you tried measuring the bandwidth of the mount point?

yes， i try and it seems like the error of Prometheus. and now i have another question： that is the output is “request_id: chatcmpl-84ac254c7dbad905-a9b7c011, total_blocks_num: 47, hit hbm: 0, hit external: 0” I’ve been trying all night. The HBM hit has data, but the external hit is always 0. I found that the mounted external storage isn’t being written to at all. Under what circumstances will /mnt/test write to the external storage?

Do you mean there's no dirs/files under /mnt/test ? Does /mnt/test is your mount point?

This path /mnt/test is defined in the config file examples/ucm_config_example.yaml with the following settings:ucm_connector_config:store_pipeline: "Cache|Posix"storage_backends: "/mnt/test"io_direct: false
It should correspond to the mounted external storage. Folders have been created in /mnt/test, yet they are all empty. I wonder if this is the expected behavior.

[UESTC-AHao]

1.When HBM PC is enabled (--enable-prefix-caching), data retrieval prioritizes HBM hits if the data is available in HBM; external storage hits only occur when the data volume exceeds HBM capacity.
2.A miss in external storage does not mean the data was not written to external storage. Based on the information you provided, there are only 47 blocks, while we pre-created 4096 directories. Due to the hash-based distribution of KV Cache, files will only be visible in a small subset of these directories. Did you randomly open some folders and find them empty? Or did you use a specific interface to traverse all 4096 directories and find them empty?
3.To verify whether data has been written to external storage or to force external storage hits, you can disable HBM PC (--no-enable-prefix-caching).

[ppzze]

1.When HBM PC is enabled (--enable-prefix-caching), data retrieval prioritizes HBM hits if the data is available in HBM; external storage hits only occur when the data volume exceeds HBM capacity. 2.A miss in external storage does not mean the data was not written to external storage. Based on the information you provided, there are only 47 blocks, while we pre-created 4096 directories. Due to the hash-based distribution of KV Cache, files will only be visible in a small subset of these directories. Did you randomly open some folders and find them empty? Or did you use a specific interface to traverse all 4096 directories and find them empty? 3.To verify whether data has been written to external storage or to force external storage hits, you can disable HBM PC (--no-enable-prefix-caching).

1.I was previously using vllm-ascend 0.17.0, but the tools_call function was unavailable due to a bug in vllm itself. I switched to vllm-ascend 0.14.0, and now data can be written to both memory and external storage normally. However, an error occurs at 10 concurrency, as shown in the figure below for node0:

2.In addition, node1 keeps reporting the message: No available shared memory broadcast block found in 60 seconds. Is this caused by insufficient shared memory or other issues?

[sumingZero]

Try reducing the cache_buffer_capacity_gb value in ucm_config_example.yaml to a smaller number(default 256), for example, set it to 32. Additionally, also lower the gpu-memory-utilization setting when launching vLLM—ideally keep it below 0.92.

[ppzze]

Try reducing the cache_buffer_capacity_gb value in ucm_config_example.yaml to a smaller number(default 256), for example, set it to 32. Additionally, also lower the gpu-memory-utilization setting when launching vLLM—ideally keep it below 0.92.

Please, after I set cache_buffer_capacity_gb to 32, it still reports an error under 10 concurrent connections: