Increase BUFFER_SIZE

[gsabdiv]

When reading from disk, lr_checksum_fd does so in chunks of 2kB, while modern disks usually have 4kB, ZFS has 128k and NFS mounts have 1 MB and page sizes are 4 kB and more. I think 256 kB would speed it up quite a lot.

However, I'm not sure if
librepo/checksum.c:40:#define BUFFER_SIZE 2048
is the only line that needs to be changed, there are other lines where a buffer of 2048 bytes appears out of nowhere:
librepo/util.c:314: const int bufsize = 2048;
tests/test_repoconf.c:41: char array[2048];

[ppisar]

Different file systems and block devices have different block size. That means that one size does no fit them all. However, various layers between the device and the file read ahead and cache, so tuning the size of the chunk becomes not so important. Changing the size rather affects an overhead of the system calls. The less system calls, the better. (Contrary when the file size reaches a size of the larges cache (physical memory), it becomes hindrance for the system itself. So there must be a reasonable upper limit, again specific for each system.)

Therefore I'd like first to see numbers to be sure this change is worth if it. Can you provide some benchmarks?

At the end, in case of librepo, overhead of the syscalls can be negligible comparing to time needed for parsing the downloaded files.

[gsabdiv]

I'm using reposync to mirror some repositories and it takes hours if nothing needs to be copied. Here's a dd showing the difference of 2k and 256k chunks while the filesystem (a local one, block size 128 KB) is very busy:

$ for s in 2k 256k 2k 256k 2k 256k; do echo ${s}:; time dd if=some_huge.rpm bs=$s of=/dev/zero; echo ---;done
2k:
1550790+1 records in
1550790+1 records out
3176018976 bytes (3.2 GB, 3.0 GiB) copied, 7.59685 s, 418 MB/s

real    0m7.600s
user    0m0.267s
sys     0m7.291s
---
256k:
12115+1 records in
12115+1 records out
3176018976 bytes (3.2 GB, 3.0 GiB) copied, 0.667176 s, 4.8 GB/s

real    0m0.671s
user    0m0.005s
sys     0m0.661s
---
2k:
1550790+1 records in
1550790+1 records out
3176018976 bytes (3.2 GB, 3.0 GiB) copied, 7.48997 s, 424 MB/s

real    0m7.493s
user    0m0.261s
sys     0m7.193s
---
256k:
12115+1 records in
12115+1 records out
3176018976 bytes (3.2 GB, 3.0 GiB) copied, 0.535222 s, 5.9 GB/s

real    0m0.539s
user    0m0.003s
sys     0m0.529s
---
2k:
1550790+1 records in
1550790+1 records out
3176018976 bytes (3.2 GB, 3.0 GiB) copied, 7.15667 s, 444 MB/s

real    0m7.160s
user    0m0.243s
sys     0m6.884s
---
256k:
12115+1 records in
12115+1 records out
3176018976 bytes (3.2 GB, 3.0 GiB) copied, 0.39045 s, 8.1 GB/s

real    0m0.394s
user    0m0.005s
sys     0m0.387s
---

I'm using reposync to mirror a huge repository and if nothing need to be copied, the reposync process consumes about 2% of one core on the CPU, but it takes hours to not copy anything.

# timeout 1m strace -fcp `pgrep reposync`
strace: Process 36055 attached
strace: Process 36055 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 93.78    0.340964           1    313648           read
  2.30    0.008371           6      1212           access
  1.26    0.004581           7       606           openat
  0.90    0.003261           2      1212           stat
  0.70    0.002550           2      1212      1212 fsetxattr
  0.29    0.001038           1       606           close
  0.22    0.000810           1       606       606 fgetxattr
  0.21    0.000759           1       606           fsync
  0.18    0.000655           1       606           fstat
  0.16    0.000586           0       606           lseek
------ ----------- ----------- --------- --------- ----------------
100.00    0.363575           1    320920      1818 total

And I traced it down to coming from librepo.

[ppisar]

The dd example only demonstrates read()/write() syscall overhead because the input is cached and output goes to /dev/null.

The strace reposync example naturally will show most time spend in read() if nothing changes in the repository.

I believe the reported spent time won't depend on read size much because strace takes the data from ptrace and ptrace takes them from kernel-space hooks that happens after switching from user space to kernel space. The most expensive thing is a context switch which happens before the ptrace hook is called.

What would be interesting is running time reposync with the current 2-KB blocks and and then with a larger block. Comparing the the sys+user time after finishing the process could should what is the effect of the read size.

[gsabdiv]

time reposync on RHEL9 for one repo with the 2k buffer:
real 3m8.801s
user 0m36.076s
sys 0m31.630s

with a 1M buffer:
real 1m7.337s
user 0m28.761s
sys 0m7.727s

[ppisar]

Those are good results. Provided modern storages have 4-KB blocks, increasing the buffer makes sense. 1 MB is not unreasonably wasteful.

However, before increasing the buffers, the functions doing read(2)/write(2) needs to be rewritten to properly deal with partial reads/writes. E.g. code in lr_copy_content(), your librepo/util.c:314: const int bufsize = 2048; example, is not safe.

[gsabdiv]

For benchmarking, I changed this:

diff -r librepo-1.20.0_org/librepo/checksum.c librepo-1.20.0/librepo/checksum.c
40c40
< #define BUFFER_SIZE             2048
---
> #define BUFFER_SIZE             1024*1024
diff -r librepo-1.20.0_org/librepo/util.c librepo-1.20.0/librepo/util.c
314c314
<     const int bufsize = 2048;
---
>     const int bufsize = 1024*1024;
diff -r librepo-1.20.0_org/tests/test_repoconf.c librepo-1.20.0/tests/test_repoconf.c
41c41
<     char array[2048];
---
>     char array[1024*1024];

I did not check whether this makes sense or is safe.