Slow flashing performance on macOS

[babldev]

Flashing images on macOS is too slow to be usable. Some major performance improvements may not be too difficult (and benefit all platforms).

Some notes:

• Flashing code is here https://github.com/beagleboard/bb-imager-rs/blob/main/bb-flasher-sd/src/flashing.rs
• Balena etcher code is here (much faster)
  • https://github.com/balena-io/etcher/blob/391164bf15810aeab53a1f0d3c959c9944c944fa/lib/util/child-writer.ts#L166
  • https://github.com/balena-io-modules/etcher-sdk/blob/abb420b721fa094320e9b0cc2d28900c0790e56c/lib/source-destination/block-device.ts#L72

I'll take a look in case there are simple improvements. The rust approach has a lot of blocking code and we may be able to pipe streams together in a more efficient way.

[Ayush1325]

Well, the blocking itself is not a problem. At least on Linux, blocking (running on a different os thread from UI) is faster than async code.

I will look at balena code more to check things. And compare the performance on Linux.

[babldev]

@Ayush1325 I'm seeing about a 7.5x speedup on my Mac by writing 1 MB at a time instead of 8 kB. Not sure how this impacts linux performance.

https://github.com/beagleboard/bb-imager-rs/compare/main...goat-hill:bb-imager-rs:fast-flash?expand=1

I switched from using 512 bytes + BufWriter (8kB) to direct writes of 1MB.

[Ayush1325]

@Ayush1325 I'm seeing about a 7.5x speedup on my Mac by writing 1 MB at a time instead of 8 kB. Not sure how this impacts linux performance.

https://github.com/beagleboard/bb-imager-rs/compare/main...goat-hill:bb-imager-rs:fast-flash?expand=1

I switched from using 512 bytes + BufWriter (8kB) to direct writes of 1MB.

It will destory performance on Linux. Not sure about Windows.

With that said, we can just have different sized buffers for different platforms.

[Ayush1325]

@Ayush1325 I'm seeing about a 7.5x speedup on my Mac by writing 1 MB at a time instead of 8 kB. Not sure how this impacts linux performance.

https://github.com/beagleboard/bb-imager-rs/compare/main...goat-hill:bb-imager-rs:fast-flash?expand=1

I switched from using 512 bytes + BufWriter (8kB) to direct writes of 1MB.

Also, can you try 512 bytes + BufWriter (1MB)?

And, can you also post some of your tests. I want to understand exactly how far behind balena is our performance.

[babldev]

I'll try to gather some more concrete measurements this week. I did a quick test on Ubuntu running in a VM on my mac and it seemed around the same speed as Mac with this change.

Most dd examples for flashing SD cards quickly tend to use bs=1M or around that value. I assume that generally reduces the number of system calls. Although maybe that's more of a gain on Mac since people are reporting linux is already pretty fast.

[Ayush1325]

One other thing to keep in mind is xz decompression. Most of our images are currently using xz compression.
In my past tests, I found that trying to decompress big chunks can end up making the whole process slower. It has the potential to have high variation from machine to machine though, so not sure if that should be taken into account now.

[babldev]

Recapping discussion from Discord:

Balana Etcher is opening the file in O_DIRECT mode to bypass the cache, additionally F_NOCACHE flag on darwin. Some of these tweaks are worth experimenting with.
https://github.com/balena-io-modules/etcher-sdk/blob/master/lib/source-destination/block-device.ts
https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/fcntl.2.html