Why does the performance of Tree creation and removal decrease after adding parameter -u ?

[pyworker]

The test backend is the same distributed storage system with 3 clients. Two test results show that in test2, both Directory removal and File removal performed better than test1. However, why did Tree removal performance degrade?

Operation	Test 1: No parameter -u	Test 2: Add parameter -u
Directory creation	933.02	1,726.39
Directory stat	583,623.01	631,672.29
Directory rename	562.85	1,523.19
Directory removal	617.00	1,749.92
File creation	498.29	1,495.06
File stat	555,782.33	603,496.98
File read	2,982.81	2,961.35
File removal	643.97	1,817.04
Tree creation	545.85	182.58
Tree removal	877.10	235.40

Detailed test command

root@test-hostname:~# cat hosts 
10.2.0.199
10.2.0.197
10.2.0.196
root@test-hostname:~# mpirun --allow-run-as-root -np 3 --hostfile hosts mdtest -n 100 -d /3fs/stage/test/ 
-- started at 03/12/2025 20:29:45 --

mdtest-4.0.0 was launched with 3 total task(s) on 1 node(s)
Command line used: mdtest '-n' '100' '-d' '/3fs/stage/test/'
WARNING: Read bytes is 0, thus, a read test will actually just open/close
WARNING: Read bytes is 0, thus, a read test will actually just open/close
WARNING: Read bytes is 0, thus, a read test will actually just open/close
Path                : /3fs/stage/test/
FS                  : 335.2 TiB   Used FS: 79.4%   Inodes: 0.0 Mi   Used Inodes: -nan%
Nodemap: 111
3 tasks, 300 files/directories

SUMMARY rate: (of 1 iterations)
   Operation                     Max            Min           Mean        Std Dev
   ---------                     ---            ---           ----        -------
   Directory creation            933.016        933.016        933.016          0.000
   Directory stat             583623.006     583623.006     583623.006          0.000
   Directory rename              562.850        562.850        562.850          0.000
   Directory removal             617.002        617.002        617.002          0.000
   File creation                 498.294        498.294        498.294          0.000
   File stat                  555782.332     555782.332     555782.332          0.000
   File read                    2982.814       2982.814       2982.814          0.000
   File removal                  643.966        643.966        643.966          0.000
   Tree creation                 545.849        545.849        545.849          0.000
   Tree removal                  877.102        877.102        877.102          0.000
-- finished at 03/12/2025 20:29:49 --

root@test-hostname:~# mpirun --allow-run-as-root -np 3 --hostfile hosts mdtest -n 100 -d /3fs/stage/test/  -u
-- started at 03/12/2025 20:29:55 --

mdtest-4.0.0 was launched with 3 total task(s) on 1 node(s)
Command line used: mdtest '-n' '100' '-d' '/3fs/stage/test/' '-u'
WARNING: Read bytes is 0, thus, a read test will actually just open/close
WARNING: Read bytes is 0, thus, a read test will actually just open/close
WARNING: Read bytes is 0, thus, a read test will actually just open/close
Path                : /3fs/stage/test/
FS                  : 335.2 TiB   Used FS: 79.4%   Inodes: 0.0 Mi   Used Inodes: -nan%
Nodemap: 111
3 tasks, 300 files/directories

SUMMARY rate: (of 1 iterations)
   Operation                     Max            Min           Mean        Std Dev
   ---------                     ---            ---           ----        -------
   Directory creation           1726.392       1726.392       1726.392          0.000
   Directory stat             631672.289     631672.289     631672.289          0.000
   Directory rename             1523.191       1523.191       1523.191          0.000
   Directory removal            1749.916       1749.916       1749.916          0.000
   File creation                1495.059       1495.059       1495.059          0.000
   File stat                  603496.978     603496.978     603496.978          0.000
   File read                    2961.354       2961.354       2961.354          0.000
   File removal                 1817.037       1817.037       1817.037          0.000
   Tree creation                 182.583        182.583        182.583          0.000
   Tree removal                  235.397        235.397        235.397          0.000
-- finished at 03/12/2025 20:29:57 --

[JulianKunkel]

It is difficult to analyze and judge already.
The small number of files makes it even harder.
I would recommend to run the benchmark for a longer period of time.
Also just run with "-D" for dirs or files only to differentiate.

The difference is that one has all folders in the same directory while the other has per rank a folder. Caching can be part of the reason. The code that measures it is also slightly different causing some minor offsets.

[pyworker]

It has nothing to do with the amount of data. The performance is always like this in my repeated tests.

I have a set of data that was tested using the following command.
mpirun --allow-run-as-root -np 3 --hostfile hosts mdtest -n 60000 -d /3fs/stage/test/ [-u]

Operation	Test 1: No parameter -u	Test 2: Add parameter -u
Directory creatio	953	2,124
Directory stat	12,821	14,680
Directory rename	491	1,622
Directory removal	590	1,795
File creation	492	1,926
File stat	13,721	14,508
File read	2,808	2,836
File removal	603	1,872
Tree creation	369	116
Tree removal	6	6

Theoretically, after using the parameter -u, it is "has per rank a folder", and the performance should be better. This can also be seen from the results of Directory and File. However, the performance of Tree has deteriorated. Are there any special circumstances in the statistical mechanism of mdtest? I also got similar results when testing the local file system.

[JulianKunkel]

The following happens: When not using -u, then only Rank0 does all the tree creates, but there are less. When using -u, then all processes are involved but the tree is more complex. The same is true for removal. Thus, one process is faster to create fewer dirs than multiple to create more... So that makes sense. To sum up: the tree creation isn't really a helpful benchmark stage. To understand what you measure, run with -n 0 and -C flag. With 2 procs I get the following subtrees: Without -u /test-dir.0-0 /test-dir.0-0/mdtest_tree.0

…

-u option /test-dir.0-0 /test-dir.0-0/mdtest_tree.0.0 /test-dir.0-0/mdtest_tree.1.0 Why isn't that parallelized? Well as the tree is needed to be created hierarchically, only multiple subdirs (rarely used feature) can be parallelized... Please close this issue, if this clarified the behavior. Of course, the community would surely appreciate a PR with a nicer description...

Message ID: ***@***.***>