how to tunne margo

[lzw06061139]

I am using margo in my distribution cache program which uses linux fuse to mount shared filesystem directory on local directory，accelerates file access with local nvme ssd ，and the program composes clients(margo clients) and servers(margo servers) on multi nodes, but the overall performace of this program doesn't well. To analize the performance issue, I enabled margo monitoring, and got some *stats.json files.I tried to analize the *stats.json files and diagnose the bottlenecks following the instructions list in mochi-hpc/mochi-performance-anaysis, but i still don't know how to tunne margo to fix the perfmance issue. Could you help to analize the issue and give me some advises? The following *stats.json snippet generated from a dd read test: dd of=/dev/null if=/mountpoint/4gfile bs=128K count=32768, /mountpoint is the fuse mountpoint of the cache program， the 4gfile is already on local cache（nvme ssd）， so the client（margo client）reads file data from server（margo）on nvme ssd acctually.

The client（origin）side shows the iforward_wait occupies the most latency time:

The server（target）side shows the irespond:relative_timestamp_from_ult_start occupies the most:

Thank you in advance！

[lzw06061139]

the default margo config info is below:

{
  "version": "0.18.2",
  "argobots": {
    "pools": [
      {
        "kind": "fifo_wait",
        "name": "__primary__",
        "access": "mpmc"
      },
      {
        "kind": "fifo_wait",
        "name": "__pool_1__",
        "access": "mpmc"
      },
      {
        "kind": "fifo_wait",
        "name": "__pool_2__",
        "access": "mpmc"
      }
    ],
    "xstreams": [
      {
        "scheduler": {
          "type": "basic_wait",
          "pools": [0]
        },
        "name": "__primary__"
      },
      {
        "scheduler": {
          "type": "basic_wait",
          "pools": [1]
        },
        "name": "__xstream_1__"
      },
      {
        "scheduler": {
          "type": "basic_wait",
          "pools": [2]
        },
        "name": "__xstream_2__"
      },
      {
        "scheduler": {
          "type": "basic_wait",
          "pools": [2]
        },
        "name": "__xstream_3__"
      },
      {
        "scheduler": {
          "type": "basic_wait",
          "pools": [2]
        },
        "name": "__xstream_4__"
      },
      {
        "scheduler": {
          "type": "basic_wait",
          "pools": [2]
        },
        "name": "__xstream_5__"
      }
    ],
    "abt_mem_max_num_stacks": 8,
    "abt_thread_stacksize": 2097152,
    "profiling_dir": ".",
    "lazy_stack_alloc": false
  },
  "mercury": {
    "version": "2.4.0",
    "address": ..............................",
    "listening": true,
    "input_eager_size": 4064,
    "output_eager_size": 4080,
    "request_post_init": 256,
    "request_post_incr": 256,
    "auto_sm": false,
    "no_bulk_eager": false,
    "no_loopback": false,
    "stats": false,
    "na_no_block": false,
    "na_no_retry": false,
    "max_contexts": 0,
    "na_max_unexpected_size": 0,
    "na_max_expected_size": 0,
    "na_request_mem_device": false,
    "checksum_level": "none",
    "na_addr_format": "unspec"
  },
  "progress_timeout_ub_msec": 100,
  "progress_spindown_msec": 10,
  "handle_cache_size": 32,
  "enable_abt_profiling": false,
  "progress_pool": 1,
  "rpc_pool": 2
}

[mdorier]

Do I understand correctly that your Margo client is the FUSE daemon? i.e. you have the following:

[process 1: dd] <=(1)=> [process 2: FUSE + margo client] <=(2)=> [process 3: margo server] <= (3) => [SSD]

Could you attach the full statistics rather than screenshots? I'd like to see some other sections. Also could you give me some details about the machines you're working on (number of cores, in particular), the protocol you use, and whether you are using 2 nodes or this is all local. Thanks!

[lzw06061139]

Do I understand correctly that your Margo client is the FUSE daemon? i.e. you have the following:

[process 1: dd] <=(1)=> [process 2: FUSE + margo client] <=(2)=> [process 3: margo server] <= (3) => [SSD]

Could you attach the full statistics rather than screenshots? I'd like to see some other sections. Also could you give me some details about the machines you're working on (number of cores, in particular), the protocol you use, and whether you are using 2 nodes or this is all local. Thanks!

Yes, the process flow is as you listed above, I ran the program on two docker containers, each equiped with 4vcpu + 2GiB mem + 25Gbps tcp network. The full stats file see the attachments.

margo.gpu-lg-cmc-h-h200-0472.host.h.pjlab.org.cn.195.stats.json
margo.gpu-lg-cmc-h-h200-0472.host.h.pjlab.org.cn.295.stats.json

[mdorier]

From the name of the files, I assume both containers run on the same host?
There is a 3.5 sec difference between what the client sees (14.5 sec) and what the server sees (11 sec), but that's across 32768 RPCs, so about 100 usec per RPC. This sounds reasonable. Most of the time in your RPC is spent doing the bulk transfer (3.7 sec) and non-margo-related stuff (I assume reading from the SSD).

I assume the configuration you posted is the one for the server? If you have 4 VCPUs (and assuming they map to actual individual CPUs), you don't want to have 6 execution streams as you are oversubscribing your cores. I assume you initialized margo with something like margo_init(... , 1, 4). If your server program's main thread is just blocking on margo_wait_for_finalize, there is no reason to have use_progress_thread = 1, it can be set to 0. Then you can try 0, 1, 2, or 3 for the rpc_thread_count argument.

Note that from our experience, when you run in production, FUSE will end up being the bottleneck as it imposes data copies and context-switches through the kernel. Just something to be careful of.

[carns]

A couple of more quick general tips:

• If you need to use TCP for communication, you will likely get better performance from the libfabric TCP provider than the libfabric sockets provider. You should be able to activate this by using "tcp://" for initialization, but you can force it if needed with "ofi+tcp://".
• Is your client sometimes communicating with a daemon that is on the same guest system? If so, you can set "auto_sm":true in the mercury section of the json. This will cause Mercury to bypass the libfabric transport and use its own local shared memory transport when it recognizes that the process is local. There is some more information on this at https://mochi.readthedocs.io/en/latest/margo/09_config.html. To be honest we should make that the default behavior at this point; in the past we did not enable it automatically because it did not interact well with certain transports that are no longer in common use.

[lzw06061139]

@mdorier @carns

1. The fuse client distributes requests based on file path hash. On my previous test, the request was forwared to the same host as the client exactly.
2. I have a question: Whether the bulk times is part of the irespond:relative_timestamp_from_utl_start, if not, which part time it contrubute to?
3. Yes, the server initializes margo with margo_init, but waits on a hearbeat loop, so I think use_progress_thread = 1 is need.

I re-run the test these updates, I seems better than before, please have a look:

1. scale-up each container from 4vCPU to 8vCPU, others remain un-changed.

2. start margo(client and server) with tcp instread of sockets

3. I will test auto_sm=true later, and give you a feedback. For other options, I still don't have enough skill to tunne.

The dd commands:

1. write
   dd if=/dev/zero of=/mountpoint/mnt/44gfile1 bs=1M count=4096 oflag=direct
   4096+0 records in
   4096+0 records out
   4294967296 bytes (4.3 GB, 4.0 GiB) copied, 6.55726 s, 655 MB/s

2. read
   dd of=/dev/null if=/mountpoint/mnt/44gfile1 bs=128K count=32768
   32768+0 records in
   32768+0 records out
   4294967296 bytes (4.3 GB, 4.0 GiB) copied, 9.39006 s, 457 MB/s

These are the new stats files (the client pid is 271, the server pid is195):

margo.gpu-lg-cmc-h-h200-0472.host.h.pjlab.org.cn.195.stats.json
margo.gpu-lg-cmc-h-h200-0472.host.h.pjlab.org.cn.211.stats.json
margo.gpu-lg-cmc-h-h200-0472.host.h.pjlab.org.cn.216.stats.json
margo.gpu-lg-cmc-h-h200-0472.host.h.pjlab.org.cn.221.stats.json
margo.gpu-lg-cmc-h-h200-0472.host.h.pjlab.org.cn.226.stats.json
margo.gpu-lg-cmc-h-h200-0472.host.h.pjlab.org.cn.231.stats.json
margo.gpu-lg-cmc-h-h200-0472.host.h.pjlab.org.cn.247.stats.json
margo.gpu-lg-cmc-h-h200-0472.host.h.pjlab.org.cn.253.stats.json
margo.gpu-lg-cmc-h-h200-0472.host.h.pjlab.org.cn.257.stats.json
margo.gpu-lg-cmc-h-h200-0472.host.h.pjlab.org.cn.259.stats.json
margo.gpu-lg-cmc-h-h200-0472.host.h.pjlab.org.cn.262.stats.json
margo.gpu-lg-cmc-h-h200-0472.host.h.pjlab.org.cn.265.stats.json
margo.gpu-lg-cmc-h-h200-0472.host.h.pjlab.org.cn.271.stats.json
margo.gpu-lg-cmc-h-h200-0472.host.h.pjlab.org.cn.276.stats.json

[lzw06061139]

I call margo_init_ext with the margo.json(details see the attached file margo.json) to initialize margo as below, but it failed with:

# [2406314.720876] mercury->fatal [error] /root/mercury-2.4.0/src/na/na.c:859 NA_Initialize_opt2() No suitable plugin found that matches uid
[error] Could not initialize hg_class with protocol uid

The code snippet:

	json_object *jobj = json_object_from_file("./margo.json");
	if (jobj == NULL) {
		log_fatal("read margo.json failed, abort %s", json_util_get_last_err());
	}
	const char *margo_json_config = json_object_to_json_string(jobj);
	struct margo_init_info args = {
		.json_config = margo_json_config,
	};
	log_info("margo config: %s", margo_json_config);

	mid = margo_init_ext(info_string, MARGO_SERVER_MODE, &args);
	if (mid == MARGO_INSTANCE_NULL)
		log_fatal("margo_init failed, abort");
	json_object_put(jobj);

	char *margo_config = margo_get_config(mid);
	log_info("margo config: %s", margo_config);
	free(margo_config);

I dump out the final json config , find the mercury::address has been set to : "uid://313262592#na+sm://227134-0#ofi+tcp://172.18.0.2:38329", what's wrong, where is mis-configured?

{"version":"0.18.2","argobots":{"pools":[{"kind":"fifo_wait","name":"__primary__","access":"mpmc"},{"kind":"fifo_wait","name":"__pool_1__","access":"mpmc"},{"kind":"fifo_wait","name":"__pool_2__","access":"mpmc"}],"xstreams":[{"scheduler":{"type":"basic_wait","pools":[0]},"name":"__primary__"},{"scheduler":{"type":"basic_wait","pools":[1]},"name":"__xstream_1__"},{"scheduler":{"type":"basic_wait","pools":[2]},"name":"__xstream_2__"},{"scheduler":{"type":"basic_wait","pools":[2]},"name":"__xstream_3__"},{"scheduler":{"type":"basic_wait","pools":[2]},"name":"__xstream_4__"},{"scheduler":{"type":"basic_wait","pools":[2]},"name":"__xstream_5__"}],"abt_mem_max_num_stacks":8,"abt_thread_stacksize":2097152,"profiling_dir":".","lazy_stack_alloc":false},"mercury":{"version":"2.4.0",  **"address":"uid://313262592#na+sm://227134-0#ofi+tcp://172.18.0.2:38329"**,"listening":true,"input_eager_size":4064,"output_eager_size":4080,"request_post_init":256,"request_post_incr":256,"auto_sm":true,"no_bulk_eager":false,"no_loopback":false,"stats":false,"na_no_block":false,"na_no_retry":false,"max_contexts":0,"na_max_unexpected_size":0,"na_max_expected_size":0,"na_request_mem_device":false,"checksum_level":"none","na_addr_format":"unspec"},"progress_timeout_ub_msec":100,"progress_spindown_msec":10,"handle_cache_size":32,"enable_abt_profiling":false,"progress_pool":1,"rpc_pool":2}

The attachment is the origin json config file:

margo.json

[mdorier]

• I have a question: Whether the bulk times is part of the irespond:relative_timestamp_from_utl_start, if not, which part time it contrubute to?

irespond:relative_timestamp_from_utl_start = "when did you call margo_respond, relative to the start of the ULT". I am guessing your RPC handler does something like this:

• Get the RPC arguments (margo_get_input);
• Read the file from SSD into a buffer;
• Expose this buffer for RDMA;
• Do a bulk transfer (margo_bulk_tranfer);
• Call margo_respond;
  So irespond:relative_timestamp_from_utl_start being relative to the start, it includes the bulk transfer time (as well as all the other operations in the ULT).

• Yes, the server initializes margo with margo_init, but waits on a hearbeat loop, so I think use_progress_thread = 1 is need.

It depends on how you setup this heartbeat. Your main ES could do something like this:

• Call margo_init;
• Register your RPCs;
• Call margo_timer_create to create a timer;
• Call margo_timer_start with a specific duration;
• Call margo_wait_for_finalize;
• In the timer's callback function, do the work needed when the timer fires, then call margo_timer_start to restart the timer.

With this model you don't need use_progress_thread = 1.

I call margo_init_ext with the margo.json(details see the attached file margo.json) to initialize margo as below, but it failed with:

Your margo.json looks fine, I suspect the problem is the info_string argument passed to margo_init_ext. Margo takes the protocol from there, not from the JSON. What do you pass as info_string? It should just be `"ofi+tcp".

The resulting address in the output JSON is normal, it's Mercury's way of formatting a combination of two addresses: one for shared memory (if processes live on the same machine and container), one for TCP.

Looking at the logs you sent (only at inode_read_rdma RPCs), the client spends 9 seconds cumulatively on these RPCs, the server spends 8.61 seconds on the corresponding ULT, of which 7.55 seconds are between the start of the ULT and the call to margo_respond. This means there is about 1.5 seconds for the latency of sending the RPC and receiving the response (this 1.5 seconds include serialization of the input arguments and serialization of the output value), so about 45 usec per RPC, which is as expected. The bulk transfer inside the RPC takes 2 seconds (transfer_cb:relative_timestamp_from_itransfer_start), so about 5.55 seconds are used for non-Margo stuff (i.e., reading the data from SSD). This is about the same as when you used sockets, if I recall. So you successfully reduced the overhead of Margo by switching to TCP, now I'd suggest you look into optimizing the access to the SSD as this is where time is spent.

[lzw06061139]

@mdorier Thank you for your comprehensive reply, I got a lot.

To the margo_init_ext issue, the info_string is ofi+tcp://172.18.0.2.

2025-06-13 09:15:15.509193505 +0000: <info> information string ofi+tcp://172.18.0.2

But I find if remove the mercury part from the margo.json file, margo_init_ext works, I still don't know why.

see the following json string, if remove the mercury part, margo can initialize. otherwise raise the same error as I presented above.

static char margo_client_info[] ="{ \
  \"argobots\": {					\
    \"abt_mem_max_num_stacks\": 8,	\
    \"abt_thread_stacksize\": 2097152,\
    \"profiling_dir\": \".\",		\
    \"lazy_stack_alloc\": false		\
  },								\
    \"mercury\": {					\
    \"listening\": false,			\
    \"input_eager_size\": 4064,		\
    \"output_eager_size\": 4080,	\
    \"request_post_init\": 256,		\
    \"request_post_incr\": 256,		\
    \"auto_sm\": true,				\
    \"no_bulk_eager\": false,		\
    \"no_loopback\": false,			\
    \"stats\": false,				\
    \"na_no_block\": false,			\
    \"na_no_retry\": false,			\
    \"max_contexts\": 0,			\
    \"na_max_unexpected_size\": 0,	\
    \"na_max_expected_size\": 0,	\
    \"na_request_mem_device\": false,\
    \"checksum_level\": \"none\",	\
    \"na_addr_format\": \"unspec\"	\
  },								\
  \"use_progress_thread\": true,	\
  \"rpc_thread_count\": 0			\
}";

[mdorier]

Thanks, that's odd, we'll have to look into it.

[lzw06061139]

Thanks, that's odd, we'll have to look into it.

In normal case, after initializing, the address is: ofi+tcp://172.18.0.2:40871, if margo_init_info includes mercury section, the address would be: uid://313262592#na+sm://239232-0#ofi+tcp://172.18.0.2:40871, I think the mercury takes uid as protocol, which is a unknown protocol, so raise the error:

# [2406314.720876] mercury->fatal [error] /root/mercury-2.4.0/src/na/na.c:859 NA_Initialize_opt2() No suitable plugin found that matches uid
[error] Could not initialize hg_class with protocol uid

[mdorier]

Do you have margo installed with the +plumber variant (or --with-mochi-plumber if you built manually)?

@carns I think if plumber is enabled and auto_sm is enabled too, this call is going to call this function and resolve the protocol as uid.

[carns]

Do you have margo installed with the +plumber variant (or --with-mochi-plumber if you built manually)?

@carns I think if plumber is enabled and auto_sm is enabled too, this call is going to call this function and resolve the protocol as uid.

I don't see that in my reading of the code. That canonicalize function just makes sure the resulting address has a "://" in it to simplify parsing later (it is legal to initialize Mercury with just a plain "tcp" or similar).

[carns]

Something is wrong with the initialization but I can't spot it yet.

Mercury will honor "uid://" addresses just fine, but only as remote addresses. They cannot be used to specify the process's own address at initialization time (so it should not show up in the NA_Initialize_opt2() path; that code should only see "tcp://" or "ofi+tcp://").

@lzw06061139 , what is your info_string variable set to in the code snippet that you provided?

[lzw06061139]

Something is wrong with the initialization but I can't spot it yet.

Mercury will honor "uid://" addresses just fine, but only as remote addresses. They cannot be used to specify the process's own address at initialization time (so it should not show up in the NA_Initialize_opt2() path; that code should only see "tcp://" or "ofi+tcp://").

@lzw06061139 , what is your info_string variable set to in the code snippet that you provided?

@carns
the info_string is ofi+tcp://172.18.0.2