Skip to content

Comments

Use buffered writing to write records#196

Merged
nkaz001 merged 3 commits intonkaz001:masterfrom
volemont:patch-1
Mar 30, 2025
Merged

Use buffered writing to write records#196
nkaz001 merged 3 commits intonkaz001:masterfrom
volemont:patch-1

Conversation

@volemont
Copy link
Contributor

I noticed that writing backtest results to csv takes a long time for large assets such as BTCUSDT. This PR uses buffered writing to speed-up the writing significantly.

@nkaz001
Copy link
Owner

nkaz001 commented Mar 28, 2025

Isn't it enough to replacelet mut file = File::create(file_path)?; with let mut file = BufWriter::new(File::create(file_path)?); to get the improvement?

@volemont
Copy link
Contributor Author

volemont commented Mar 28, 2025

The code is courtesy of ChatGPT. Chain of thought:

Key Improvements

  1. Buffered I/O: Using BufWriter reduces the number of system calls by buffering writes into memory before flushing them to the disk. This significantly improves performance when writing large files.
  2. Avoid Redundant String Formatting: Instead of writing the header and data line-by-line with multiple calls to writeln!, the code now writes lines using write_all and pre-formats strings more efficiently.
  3. Efficient File Path Creation: The file path creation is consolidated into a single format! call for better readability and performance.
  4. Minimize Temporary Allocations: The line string is reused within the loop instead of allocating a new one for every field.”

Do you prefer to remove the other changes and just keep let mut file = BufWriter::new(File::create(file_path)?);?

@nkaz001
Copy link
Owner

nkaz001 commented Mar 29, 2025

I think using BufWriter alone provides a similar improvement, without additional changes. Could you test it to confirm and update accordingly?

@volemont
Copy link
Contributor Author

I reverted my previous commit and implemented your requested changes in 45db00d.

Master (commit ac9cbc9) with test script (master...volemont:hftbacktest-1:master):

❯ python -u 5_backtest.py
PEOPLEUSDT: 0

Elapsed time: 0:00:17.057809

This PR branch with commit 45db00d:

❯ python -u 5_backtest.py
PEOPLEUSDT: 0

Elapsed time: 0:00:10.035907

@nkaz001 nkaz001 merged commit 0454f6b into nkaz001:master Mar 30, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants