fix: buffered writes when using io.Copy
#318
Conversation
When copying files to a fat filesystem and if using `io.Copy` on the `filesystem.OpenFile` it was slow which the entries were flushed for each write which is in-efficient. Only flush once on `Close()`. Signed-off-by: Noel Georgi <[email protected]>
|
@deitch this is more of a draft stuff and I'll leave it upto the maintainers to do any fixes, most of this was generated with Claude so I don't feel fully confident in the implementation, I've tried to add tests. The benchmark test shows how bad is when using Can be run as |
| return totalRead, retErr | ||
| } | ||
|
|
||
| // ReadFrom reads data from r until EOF and writes it to the file. |
There was a problem hiding this comment.
from go doc io.Copy
go doc io.Copy
package io // import "io"
func Copy(dst Writer, src Reader) (written int64, err error)
Copy copies from src to dst until either EOF is reached on src or an
error occurs. It returns the number of bytes copied and the first error
encountered while copying, if any.
A successful Copy returns err == nil, not err == EOF. Because Copy is
defined to read from src until EOF, it does not treat an EOF from Read as an
error to be reported.
If src implements WriterTo, the copy is implemented by calling
src.WriteTo(dst). Otherwise, if dst implements ReaderFrom, the copy is
implemented by calling dst.ReadFrom(src).
otherwise it was calling Write with a default buffer which is extremely slow
| // Try to determine source size for optimal allocation using Seeker interface | ||
| var knownSize int64 = -1 | ||
|
|
||
| if seeker, ok := r.(io.Seeker); ok { |
There was a problem hiding this comment.
this helps pre-allocate by checking if we source file implements io.Seeker if reading from a sourcefile with os.Open this works great
| ) | ||
|
|
||
| const ( | ||
| benchmarkFileSize = (4 * 1024 * 1024 * 1024) - 1 // 4GB |
There was a problem hiding this comment.
tried to use the max fat32 size, since that would be the most extreme case
1 participant
When copying files to a fat filesystem and if using
io.Copyon thefilesystem.OpenFileit was slow which the entries were flushed for each write which is in-efficient. Only flush once onClose().