Reduce frequency of flushes of the output buffer in Bunch::print and SyncParticle::print
#75
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Per suggestions from @shishlo, I investigated the frequency that
Bunch::printwould flush the output buffer. This PR replaces all instances ofstd::endlwith'\n', as the former invokes a flush operation for each line written to the output file. For very large bunches, this substantially inflates write times. I've also removed manual flushes of the output buffer, as it is flushed automatically when the buffer is full anyway.In testing with a bunch size of
N=100_000_000, with MPI enabled and utilizing 8 cores, I observe these changes to reduce bunch write time by 60-70% across several trials, with no significant increase in RAM consumption relative to the existing implementation.As an aside, I experimented with pre-allocating the output buffer with various sizes ranging from
64kBto16384 kB, while also varying thechunkSizecontrolling the amount of data communicated to the rank 0 process at once. Neither seemed to yield any further decrease in write times.