Questions about UCX_MEMTYPE_CAHE

[EmmaLee-c]

At present, even if UCX_MEMTYPE_CAHE=yes is enabled, ucx still needs to perform memory type detection when processing send/recv data and finding addresses in memtype cache:

tag_recv.c:70 UCX REQ req 0x18b2d40: recv_nbx buffer 0x7ff399200000 dt 0x8 count 4194304 tag 640000000000/ffffffffffffffff
memtype_cache.c:356 UCX DATA 0x7ff399200000..0x7ff399600000 found in 0x189d330 [0x7ff399200000..0x7ff399600000] unknown dev
ucp_context.h:683 UCX REQ address 0x7ff399200000 length 4194304: memtype cache returned 'unknown'
ucp_context.c:2516 UCX REQ address 0x7ff399200000 length 4194304: md rocm_cpy detected as type 'rocm' GPU3

Moreover, if the address cannot be found in memtype cache, it may also be mistakenly classified as a host type.
So, may I ask, what is the function of UCX_MEMTYPE_CAHE?

Thank you very much for your answer!

[yosefe]

The memory detection is done only once: first a memory allocation hook adds entries to the cache as "unknown" memory type, and the first time this entry is used in send/recv operation the actual type is detected and stored back in the cache.

[EmmaLee-c]

As shown in the UCX log above, the same send was sent twice in a row, and the memory type was detected twice

[EmmaLee-c]

I looked at the ucx source code and found that CUDA memory will call ucs_memtype_cache_update to update after the first detection, but ROCM memory will not

[yosefe]

adding @edgargabriel to comment on ROCm

[edgargabriel]

@EmmaLee-c I will look into this, thank you for pointing it out.