Parallel writing in BP5

[dmitry-ganyushin]

I made some profiling of BP5 writing, and it looks like on one node only one process is writing data, and the others are making a data copy operation to the writing process. Is it the best way to write data from N process to one shared file?

[pnorbert]

Excellent question and an excellent chart. The default AggregationType for BP5 is TwoLevelShm (see doc), and the chart is showing what is going on. By default we have 1 file per compute node and your example is on a single node, hence a single file. Only one process on the node writes to the file, the others are sharing their data with that process through a shared memory segment. They do this one by one of course, filling one half of the shared memory segment, while the main process is writing the other half. All of this is designed for scalable write performance on many nodes, working beautifully up the 9000 nodes of Frontier. It is very inefficient for a single node write performance. On a single node, you could change AggregatorRatio to 1 to have each process create their own data file and write independently, or any number less than the number of processes to seek for better performance but there is a better solution.

The other aggregation types are EveryoneWrites and EveryoneWritesSerial. In both cases, every process is writing their own data to a file. Still the default is one file per compute node and every process on a node writes to the same file. The first strategy lets everyone write immediately, which is probably always the best for a small application, as long as the file system can handle all the write requests from all processes at once. As it can be expected, it scales horribly, performance will drop off sharply when the file system is stressed to much. The second version serializes the writers to a single file, so that if there are N files being written then, at any time, there are N active writers, one per file, which scales better.

Actually, EveryoneWritesSerial is the best performing on Lustre as far as we can see on current big machines (Frontier, Perlmutter), however, it was slow on GPFS on Summit and hence the default aggregation type is still TwoLevelShm in the current release.

[pnorbert]

To illustrate this, here is a performance chart from early access to Frontier, with early version of BP5 aggregation but on an under-utilized system. Please don't take this as a publication quality result, this is just from one set of runs from the past.

SHM=TwoLevelSHM
EWS=EveryoneWritesSerial
EW=EveryoneWrites.
EW-AR1 is EveryoneWrites with AggregatorRatio=1, i.e. generating one data file per process, eight times more files than the rest (with their default settings of one file per node), wanted to see when Lustre was breaking. Tests on 2k/4k nodes (16384 and 32768 files, resp.) did not finish...

We were surprised to see how the Orion filesystem of Frontier could deal with lots of writers (and lots of files) at once, but still only to a certain point. Our overall experience for the scalability of EW is not that great as on this machine, and would never suggest it as a strategy for HPC codes. SHM keeps up well with EWS so it is not really a problem to have this set as default.

[dmitry-ganyushin]

Thanks for your answers. If we consider the TwoLevelShm aggregation scheme, could it be possible that we could open N times the same file with every MPI process on one node in the beginning, and append this file serially one MPI process after another? In this approach, we could avoid coping the large amount of data using shared memory segments. Would it be more efficient?

[pnorbert]

Isn't this what EveryoneWritesSerial does with the default settings?